Design Document for Driftless

Goals

Configuration management
- Applies declarative system configs idempotently
- Requires list of configuration tasks
- Writes audit/diff logs to a local directory (i.e. NFS), HTTP endpoint, or S3 bucket
- Alternative to Ansible/Chef/Puppet
- Crates: git2 nix reqwest rust-s3 serde
Metrics gathering
- Gather host metrics (CPU, mem, disk, etc)
- Requires list of metrics to collect, poll interval, and thresholds
- Export metrics via /metrics endpoint or push to S3 bucket
- Alternative to Prometheus Node Exporter
- Crates: prometheus rust-s3 sysinfo
Log collection
- Tails and forwards logs
- Requires list of paths to tail and filters/parsers to use
- Writes logs to local directory (i.e. NFS), S3 bucket, syslog, or HTTP (i.e. ELK stack)
- Alternative to FileBeat
- Crates: flate2 reqwest rust-s3
Secrets management
- Remove secrets from configuration using variable substitution
- Reads secrets from environment variables and env files outside input directory
- Crates: secret_vault

Runs as a CLI
Uses the given directory of configuration files as input (i.e. cloned Git repo)
Configuration files use an JSON, TOML, or YAML syntax (auto-detect file extension)
Configuration file schemas include:
- apply: Idempotent system configuration tasks
- facts: Facts, metrics, and other information gathering tasks
- logs: Log file tailing and forwarding tasks
Default configuration directory: /etc/driftless/config (system-wide) or ~/.config/driftless/config (user)
Secrets passed via environment variables, /etc/driftless/secrets.yml, /etc/driftless/secrets.env, ~/.config/driftless/secrets.yml, or ~/.config/driftless/secrets.env
Sub-command names mirror file schemas (i.e. apply, facts, logs) for running tasks
- The apply sub-command should include a --dry-run flag or similar to only output diffs
Additional agent sub-command activates agent mode
- Gathers built-in facts
- If configured, starts Prometheus metrics endpoint (i.e. 0.0.0.0:8000/metrics)
- Starts an event loop
  - Reads configuration files from directory
  - Gathers configured additional facts and metrics at requested interval
  - Starts collecting and forwarding configured log files
  - Runs apply tasks at requested interval

The nix crate provides Rust bindings to *nix APIs. We can leverage this for:

Create task prompts in the TODO list that adds support for macOS and Windows operating systems in all applicable areas of the codebase
Review usages of dead_code, unsafe, and unused_imports to silence warnings and determine if code should be used or cleaned up according to Rust best practices. Use this opportunity to cleanup unused code and dependencies to reduce release binary size and improve maintainability.
Review the codebase for consistent error-handling patterns and improve as needed
Ensure all dependencies in Cargo.toml are up-to-date with the latest stable versions
Review the codebase for usage of Rust best-practices and guidelines
Review the codebase for safety and security vulnerabilities and apply mitigations as needed
Ensure comprehensive test coverage and cleanup any clippy warnings. Tests should be written for the intent of the code not the implementation details.
Review the auto-generated and manually-managed documentation in the docs/ directory and validate information is accurate against the current codebase. Look for cleanup, clarification, expansion, and reorganization opportunities. Ensure all auto-generated documentation contains a banner indicating it is auto-generated and should not be manually edited.
Perform a final review of the entire codebase, documentation, and project structure to ensure consistency, quality, and readiness for production use.