attune-system/attune
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

re-uploading work

Browse Source
This commit is contained in:
David Culbreth
2026-02-04 17:46:30 -06:00
commit 3b14c65998
1388 changed files with 381262 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

337
docs/configuration/CONFIG_README.md Normal file
View File

@@ -0,0 +1,337 @@
# Attune Configuration Guide
This document explains how to configure Attune using YAML configuration files.
## Quick Start
1. **Copy the example configuration:**
```bash
cp config.example.yaml config.yaml
```
2. **Edit your configuration:**
```bash
nano config.yaml
```
3. **Generate secure secrets:**
```bash
# Generate JWT secret (64 bytes, base64 encoded)
openssl rand -base64 64
# Generate encryption key (32 bytes minimum, base64 encoded)
openssl rand -base64 32
```
4. **Run the application:**
```bash
cargo run --bin attune-api
```
## Configuration Files
### Available Files
- **`config.example.yaml`** - Template for new installations (safe to commit)
- **`config.development.yaml`** - Development environment settings (can be committed)
- **`config.test.yaml`** - Test environment settings (can be committed)
- **`config.production.yaml`** - Production template with placeholders (safe to commit)
- **`config.yaml`** - Your actual configuration (DO NOT COMMIT - add to .gitignore)
### Loading Priority
Configuration is loaded in this order (later overrides earlier):
1. **Base config file** - `config.yaml` or path from `ATTUNE_CONFIG` environment variable
2. **Environment-specific file** - `config.{environment}.yaml` (e.g., `config.development.yaml`)
3. **Environment variables** - Variables with `ATTUNE__` prefix
**Example:**
```bash
# Use production config with secret from environment
export ATTUNE_CONFIG=config.production.yaml
export ATTUNE__SECURITY__JWT_SECRET=$(openssl rand -base64 64)
cargo run --bin attune-api
```
## Configuration Sections
### Required Settings
These settings must be configured before running:
```yaml
database:
url: postgresql://user:password@localhost:5432/attune
security:
jwt_secret: your-secret-key-here # Generate with: openssl rand -base64 64
encryption_key: your-encryption-key-here # Generate with: openssl rand -base64 32
```
### Optional Settings
```yaml
server:
host: 0.0.0.0
port: 8080
cors_origins:
- http://localhost:3000
log:
level: info # trace, debug, info, warn, error
format: json # json, pretty
redis: # Optional: for caching
url: redis://localhost:6379
message_queue: # Optional: for async processing
url: amqp://guest:guest@localhost:5672/%2f
```
## Environment Variables
Override any setting using environment variables with the `ATTUNE__` prefix:
```bash
# Database
export ATTUNE__DATABASE__URL=postgresql://localhost/attune
export ATTUNE__DATABASE__MAX_CONNECTIONS=100
# Server
export ATTUNE__SERVER__PORT=3000
export ATTUNE__SERVER__CORS_ORIGINS="https://app.example.com,https://www.example.com"
# Security (recommended for production)
export ATTUNE__SECURITY__JWT_SECRET=$(openssl rand -base64 64)
export ATTUNE__SECURITY__ENCRYPTION_KEY=$(openssl rand -base64 32)
# Logging
export ATTUNE__LOG__LEVEL=debug
export ATTUNE__LOG__FORMAT=pretty
```
**Syntax:** `ATTUNE__<section>__<key>=value`
- Use double underscores (`__`) to separate nested keys
- For arrays, use comma-separated values
- Boolean values: `true`, `false`, `1`, `0`, `yes`, `no`
## Environment-Specific Configuration
### Development
Create or use `config.development.yaml`:
```yaml
environment: development
database:
url: postgresql://postgres:postgres@localhost:5432/attune
log_statements: true
log:
level: debug
format: pretty
security:
jwt_access_expiration: 86400 # 24 hours for dev convenience
```
Run with:
```bash
export ATTUNE__ENVIRONMENT=development
cargo run --bin attune-api
```
### Production
Create `config.production.yaml` with placeholders:
```yaml
environment: production
database:
url: postgresql://user@db.example.com/attune
max_connections: 100
log:
level: info
format: json
file: /var/log/attune/attune.log
server:
cors_origins:
- https://app.example.com
```
**IMPORTANT:** Override secrets with environment variables:
```bash
export ATTUNE_CONFIG=config.production.yaml
export ATTUNE__SECURITY__JWT_SECRET=$(cat /run/secrets/jwt_secret)
export ATTUNE__SECURITY__ENCRYPTION_KEY=$(cat /run/secrets/encryption_key)
export ATTUNE__DATABASE__URL=$(cat /run/secrets/db_url)
```
### Testing
The `config.test.yaml` is automatically used during tests:
```bash
cargo test # Uses config.test.yaml
```
## Docker Deployment
### Option 1: Mount Configuration File
```bash
docker run -v $(pwd)/config.production.yaml:/app/config.yaml \
-e ATTUNE__SECURITY__JWT_SECRET=$JWT_SECRET \
attune-api
```
### Option 2: Environment Variables Only
```bash
docker run \
-e ATTUNE__DATABASE__URL=postgresql://db/attune \
-e ATTUNE__SECURITY__JWT_SECRET=$JWT_SECRET \
-e ATTUNE__SECURITY__ENCRYPTION_KEY=$ENCRYPTION_KEY \
-e ATTUNE__SERVER__PORT=8080 \
attune-api
```
### Docker Compose
```yaml
version: '3.8'
services:
api:
image: attune-api
volumes:
- ./config.production.yaml:/app/config.yaml
environment:
ATTUNE__SECURITY__JWT_SECRET: ${JWT_SECRET}
ATTUNE__SECURITY__ENCRYPTION_KEY: ${ENCRYPTION_KEY}
ATTUNE__DATABASE__URL: postgresql://postgres:postgres@db:5432/attune
```
## Security Best Practices
### 1. Never Commit Secrets
Your `.gitignore` is already configured to exclude `config.yaml`. Always use:
- `config.example.yaml` for templates (safe to commit)
- `config.yaml` for local development (never commit)
- Environment variables for production secrets
### 2. Generate Strong Secrets
```bash
# JWT secret (64 bytes recommended)
openssl rand -base64 64
# Encryption key (32 bytes minimum)
openssl rand -base64 32
# Or use a password manager to generate and store secrets
```
### 3. Use Environment Variables in Production
```bash
# Good: Load from secure secret store
export ATTUNE__SECURITY__JWT_SECRET=$(vault read -field=jwt_secret secret/attune)
# Bad: Hardcode in config file (especially if committed)
# config.yaml:
# security:
# jwt_secret: hardcoded-secret-in-git # ❌ NEVER DO THIS
```
### 4. Restrict File Permissions
```bash
chmod 600 config.yaml
chown appuser:appuser config.yaml
```
## Troubleshooting
### Configuration Not Loading
**Problem:** Application can't find `config.yaml`
**Solution:**
```bash
# Check file exists
ls -la config.yaml
# Or specify path explicitly
export ATTUNE_CONFIG=/path/to/config.yaml
cargo run --bin attune-api
```
### YAML Syntax Error
**Problem:** Configuration fails to parse
**Solution:**
```bash
# Validate YAML syntax
python3 -c "import yaml; yaml.safe_load(open('config.yaml'))"
# Or use yq
yq eval config.yaml
```
**Common issues:**
- Use spaces, not tabs for indentation
- Ensure consistent indentation (2 spaces recommended)
- Quote strings with special characters: `:`, `#`, `@`, `%`
### Environment Variables Not Working
**Problem:** Environment variables not overriding config
**Solution:**
```bash
# Check variable is set
env | grep ATTUNE
# Ensure correct format
export ATTUNE__DATABASE__URL=... # ✅ Correct (double underscore)
export ATTUNE_DATABASE_URL=... # ❌ Wrong (single underscore)
```
### Validation Errors
**Problem:** "JWT secret is required when authentication is enabled"
**Solution:**
```yaml
# Either disable auth for development
security:
enable_auth: false
# Or provide a secret
security:
enable_auth: true
jwt_secret: your-secret-here
```
## Complete Configuration Reference
See [docs/configuration.md](docs/configuration.md) for complete documentation of all configuration options.
## Migration from .env
If you're migrating from `.env` files, see [docs/env-to-yaml-migration.md](docs/env-to-yaml-migration.md) for a complete migration guide.
## Getting Help
- **Configuration Guide:** [docs/configuration.md](docs/configuration.md)
- **Quick Start:** [docs/quick-start.md](docs/quick-start.md)
- **Migration Guide:** [docs/env-to-yaml-migration.md](docs/env-to-yaml-migration.md)
- **README:** [README.md](README.md)

554
docs/configuration/config-troubleshooting.md Normal file
View File

@@ -0,0 +1,554 @@
# Configuration Troubleshooting Guide
This guide helps you troubleshoot common configuration issues with Attune.
## Quick Diagnostics
### Check Configuration is Valid
```bash
# Test configuration loading
cargo run --bin attune-api -- --help
# Enable debug logging to see config details
ATTUNE__LOG__LEVEL=debug cargo run --bin attune-api
```
### Verify Configuration Files Exist
```bash
# Check for config files
ls -la config*.yaml
# View current configuration
cat config.yaml
```
### Test Environment Variables
```bash
# List all Attune environment variables
env | grep ATTUNE
# Test a specific variable
echo $ATTUNE__DATABASE__URL
```
## Common Issues
### 1. Encryption Key Must Be At Least 32 Characters
**Error:**
```
Error: Validation error: Encryption key must be at least 32 characters
```
**Cause:** The `encryption_key` in your configuration is less than 32 characters.
**Solution:**
Generate a proper encryption key:
```bash
# Generate a 32-byte key (base64 encoded = ~44 characters)
openssl rand -base64 32
# Example output:
# R3mHJ8qP5xK2nB9vW7tY4uZ1aE6cF0dG8hI2jK4lM5n=
```
Update your configuration:
```yaml
security:
encryption_key: R3mHJ8qP5xK2nB9vW7tY4uZ1aE6cF0dG8hI2jK4lM5n=
```
Or use an environment variable:
```bash
export ATTUNE__SECURITY__ENCRYPTION_KEY=$(openssl rand -base64 32)
```
**Note:** The key must be **at least 32 characters** after base64 decoding. A 32-byte random value base64-encoded is typically 44 characters.
### 2. JWT Secret Required
**Error:**
```
Error: Validation error: JWT secret is required when authentication is enabled
```
**Cause:** Authentication is enabled but no JWT secret is configured.
**Solution Option 1 - Provide a JWT secret:**
```yaml
security:
jwt_secret: your-secret-here
enable_auth: true
```
Generate a secure secret:
```bash
openssl rand -base64 64
```
**Solution Option 2 - Disable authentication (dev only):**
```yaml
security:
enable_auth: false
```
### 3. Configuration File Not Found
**Error:**
```
Error: Configuration error: ...
```
**Cause:** The application can't find `config.yaml`.
**Solution:**
Check if the file exists:
```bash
ls -la config.yaml
```
If missing, create it from the example:
```bash
cp config.example.yaml config.yaml
nano config.yaml
```
Or specify a custom path:
```bash
export ATTUNE_CONFIG=/path/to/config.yaml
cargo run --bin attune-api
```
### 4. Database Connection Failed
**Error:**
```
Error: Database error: connection refused
Error: Database error: password authentication failed
```
**Cause:** Cannot connect to PostgreSQL database.
**Solution:**
1. **Verify PostgreSQL is running:**
```bash
pg_isready -h localhost -p 5432
```
2. **Check connection string:**
```yaml
database:
url: postgresql://username:password@host:port/database
```
3. **Test connection manually:**
```bash
psql postgresql://postgres:postgres@localhost:5432/attune
```
4. **Verify database exists:**
```bash
psql -U postgres -c "SELECT datname FROM pg_database WHERE datname='attune';"
```
5. **Create database if needed:**
```bash
createdb -U postgres attune
```
### 5. Invalid Log Level
**Error:**
```
Error: Invalid log level: TRACE. Must be one of: ["trace", "debug", "info", "warn", "error"]
```
**Cause:** Log level must be lowercase.
**Solution:**
```yaml
log:
level: debug # Use lowercase
```
Not:
```yaml
log:
level: DEBUG # Wrong - uppercase not allowed
```
### 6. YAML Syntax Error
**Error:**
```
Error: Configuration error: invalid type: string "true", expected a boolean
Error: while parsing a block mapping
```
**Cause:** Invalid YAML syntax.
**Common mistakes:**
**Wrong - Quotes around boolean:**
```yaml
enable_auth: "true" # This is a string, not a boolean
```
**Correct:**
```yaml
enable_auth: true
```
**Wrong - Mixed spaces and tabs:**
```yaml
database:
url: postgresql://... # Tab used for indentation
```
**Correct - Use spaces only:**
```yaml
database:
url: postgresql://... # 2 spaces
```
**Wrong - Missing space after colon:**
```yaml
level:debug # No space after colon
```
**Correct:**
```yaml
level: debug # Space after colon
```
**Validate YAML syntax:**
```bash
# Using Python
python3 -c "import yaml; yaml.safe_load(open('config.yaml'))"
# Using yq
yq eval config.yaml
# Using Ruby
ruby -ryaml -e "YAML.load_file('config.yaml')"
```
### 7. Environment Variable Not Working
**Error:** Configuration value not being overridden by environment variable.
**Cause:** Incorrect environment variable format.
**Correct format:**
```bash
ATTUNE__SECTION__KEY=value
```
**Common mistakes:**
**Wrong - Single underscore:**
```bash
ATTUNE_DATABASE_URL=... # Wrong
```
**Correct - Double underscore:**
```bash
ATTUNE__DATABASE__URL=... # Correct
```
**Wrong - Missing ATTUNE prefix:**
```bash
DATABASE__URL=... # Wrong
```
**Correct - With ATTUNE prefix:**
```bash
ATTUNE__DATABASE__URL=... # Correct
```
**Wrong - Incorrect case:**
```bash
attune__database__url=... # Wrong (lowercase prefix)
```
**Correct - ATTUNE in uppercase:**
```bash
ATTUNE__DATABASE__URL=... # Correct
```
**For arrays (comma-separated):**
```bash
ATTUNE__SERVER__CORS_ORIGINS="http://localhost:3000,http://localhost:5173"
```
### 8. Port Already in Use
**Error:**
```
Error: Address already in use (os error 98)
```
**Cause:** Another process is using port 8080.
**Solution:**
1. **Find the process:**
```bash
lsof -i :8080
# Or
netstat -tulpn | grep 8080
```
2. **Kill the process:**
```bash
kill <PID>
```
3. **Or change the port:**
```yaml
server:
port: 8081
```
Or with environment variable:
```bash
ATTUNE__SERVER__PORT=8081 cargo run --bin attune-api
```
### 9. CORS Issues
**Error in browser console:**
```
Access to fetch at 'http://localhost:8080/api/v1/...' from origin 'http://localhost:3000'
has been blocked by CORS policy
```
**Cause:** Frontend origin not in allowed CORS origins.
**Solution:**
Add your frontend URL to CORS origins:
```yaml
server:
cors_origins:
- http://localhost:3000
- http://localhost:5173 # Vite default
```
Or use environment variable:
```bash
export ATTUNE__SERVER__CORS_ORIGINS="http://localhost:3000,http://localhost:5173"
```
### 10. Configuration Changes Not Taking Effect
**Cause:** Cached binary or incorrect environment.
**Solution:**
1. **Rebuild the application:**
```bash
cargo clean
cargo build --bin attune-api
```
2. **Check which config file is being loaded:**
```bash
ATTUNE__LOG__LEVEL=debug cargo run --bin attune-api 2>&1 | grep -i config
```
3. **Verify environment:**
```bash
env | grep ATTUNE
```
4. **Use explicit config file:**
```bash
ATTUNE_CONFIG=./config.development.yaml cargo run --bin attune-api
```
## Advanced Diagnostics
### Debug Configuration Loading
Enable debug logging to see configuration details:
```bash
ATTUNE__LOG__LEVEL=debug cargo run --bin attune-api 2>&1 | tee debug.log
```
Look for lines containing:
- "Configuration loaded"
- "Environment:"
- "Loaded configuration from"
### Verify Configuration Merging
Test the configuration priority:
1. **Base config only:**
```bash
cargo run --bin attune-api
```
2. **With environment-specific config:**
```bash
ATTUNE__ENVIRONMENT=development cargo run --bin attune-api
```
3. **With environment variable override:**
```bash
ATTUNE__SERVER__PORT=9999 cargo run --bin attune-api
```
### Check Database Connection
Test database connectivity separately:
```bash
# Using psql
psql -U postgres -h localhost -d attune -c "SELECT version();"
# Using sqlx-cli
sqlx database create --database-url postgresql://postgres:postgres@localhost/attune
sqlx migrate run --database-url postgresql://postgres:postgres@localhost/attune
```
### Validate All Configuration Values
```bash
# Dump configuration (be careful - may contain secrets!)
ATTUNE__LOG__LEVEL=debug cargo run --bin attune-api 2>&1 | grep -A 50 "Configuration loaded"
```
## Security Checklist
When troubleshooting, ensure:
- [ ] Secrets are not logged or displayed
- [ ] Configuration files with secrets are in `.gitignore`
- [ ] Production secrets use environment variables
- [ ] File permissions are restrictive (`chmod 600 config.yaml`)
- [ ] JWT secrets are cryptographically strong (64+ bytes)
- [ ] Encryption keys are at least 32 characters
## Getting More Help
### Enable Verbose Logging
```bash
RUST_LOG=debug,attune_common=trace cargo run --bin attune-api
```
### Check Application Logs
```bash
# If logging to file
tail -f /var/log/attune/attune.log
# With pretty formatting
ATTUNE__LOG__FORMAT=pretty cargo run --bin attune-api
```
### Validate Configuration Programmatically
Create a test file `check_config.rs`:
```rust
use attune_common::config::Config;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let config = Config::load()?;
config.validate()?;
println!("✓ Configuration is valid");
println!("Environment: {}", config.environment);
println!("Database URL: {}", config.database.url);
println!("Server: {}:{}", config.server.host, config.server.port);
Ok(())
}
```
Run it:
```bash
cargo run --bin check_config
```
## Reference
- [Configuration Guide](configuration.md) - Complete reference
- [Migration Guide](env-to-yaml-migration.md) - Migrating from .env
- [Quick Start](quick-start.md) - Getting started
- [CONFIG_README.md](../CONFIG_README.md) - Quick reference
## Common Configuration Patterns
### Development Setup
```yaml
environment: development
database:
url: postgresql://postgres:postgres@localhost:5432/attune
log_statements: true
server:
host: 127.0.0.1
port: 8080
log:
level: debug
format: pretty
security:
jwt_secret: dev-secret-change-in-production
encryption_key: dev-key-at-least-32-characters-long
enable_auth: true
```
### Production Setup
```yaml
environment: production
database:
url: postgresql://user:pass@db.example.com/attune
max_connections: 100
server:
host: 0.0.0.0
cors_origins:
- https://app.example.com
log:
level: info
format: json
file: /var/log/attune/attune.log
```
With environment variables:
```bash
export ATTUNE__SECURITY__JWT_SECRET=$(cat /run/secrets/jwt_secret)
export ATTUNE__SECURITY__ENCRYPTION_KEY=$(cat /run/secrets/encryption_key)
```
### Testing Setup
```yaml
environment: test
database:
url: postgresql://postgres:postgres@localhost:5432/attune_test
server:
port: 0 # Random port
log:
level: warn
security:
jwt_secret: test-secret-not-secure
encryption_key: test-key-at-least-32-characters-long
```
---
If you continue to experience issues, please check the [GitHub Issues](https://github.com/yourusername/attune/issues) or open a new issue with:
1. Error message
2. Your configuration (with secrets redacted)
3. Output of `cargo --version` and `rustc --version`
4. Operating system and PostgreSQL version

624
docs/configuration/configuration.md Normal file
View File

@@ -0,0 +1,624 @@
# Configuration Guide
This guide explains how to configure the Attune automation platform using YAML configuration files.
## Overview
Attune uses a layered configuration system with YAML files as the primary configuration source. Configuration can be overridden using environment variables, making it flexible for different deployment scenarios.
## Configuration Loading Priority
Configuration is loaded in the following order (later sources override earlier ones):
1. **Base configuration file** (`config.yaml` or path from `ATTUNE_CONFIG` environment variable)
2. **Environment-specific configuration** (e.g., `config.development.yaml`, `config.production.yaml`)
3. **Environment variables** (prefix: `ATTUNE__`, separator: `__`)
This layered approach allows you to:
- Keep common settings in `config.yaml`
- Override settings per environment (dev, test, production)
- Override sensitive values with environment variables (recommended for production)
## Quick Start
1. **Copy the example configuration:**
```bash
cp config.example.yaml config.yaml
```
2. **Edit the configuration:**
```bash
nano config.yaml
```
3. **Set required values:**
- Database URL
- JWT secret key
- Encryption key
- CORS origins (if applicable)
4. **Run the application:**
```bash
cargo run --bin attune-api
```
## Configuration File Structure
### Basic Example
```yaml
service_name: attune
environment: development
database:
url: postgresql://postgres:postgres@localhost:5432/attune
max_connections: 50
server:
host: 0.0.0.0
port: 8080
cors_origins:
- http://localhost:3000
security:
jwt_secret: your-secret-key-here
jwt_access_expiration: 3600
log:
level: info
format: json
```
## Configuration Sections
### Service Metadata
```yaml
service_name: attune # Service identifier
environment: development # Environment: development, test, staging, production
```
### Database Configuration
```yaml
database:
# PostgreSQL connection URL
url: postgresql://username:password@host:port/database
# Connection pool settings
max_connections: 50 # Maximum pool size
min_connections: 5 # Minimum pool size
connect_timeout: 30 # Connection timeout (seconds)
idle_timeout: 600 # Idle connection timeout (seconds)
# Debugging
log_statements: false # Log SQL statements (set true for debugging)
```
**Environment variable override:**
```bash
export ATTUNE__DATABASE__URL=postgresql://user:pass@localhost/attune
export ATTUNE__DATABASE__MAX_CONNECTIONS=100
```
### Server Configuration
```yaml
server:
host: 0.0.0.0 # Bind address (0.0.0.0 for all interfaces)
port: 8080 # HTTP port
request_timeout: 30 # Request timeout (seconds)
enable_cors: true # Enable CORS middleware
# Allowed CORS origins
cors_origins:
- http://localhost:3000
- https://app.example.com
max_body_size: 10485760 # Max request body size (10MB in bytes)
```
**Environment variable override:**
```bash
export ATTUNE__SERVER__PORT=3000
export ATTUNE__SERVER__CORS_ORIGINS="https://app.com,https://www.app.com"
```
### Security Configuration
```yaml
security:
# JWT secret for signing tokens (REQUIRED if enable_auth is true)
# Generate with: openssl rand -base64 64
jwt_secret: your-secret-key-here
# Token expiration times (seconds)
jwt_access_expiration: 3600 # Access token: 1 hour
jwt_refresh_expiration: 604800 # Refresh token: 7 days
# Encryption key for storing secrets (must be at least 32 characters)
# Generate with: openssl rand -base64 32
encryption_key: your-32-char-encryption-key-here
# Enable/disable authentication
enable_auth: true
```
**Environment variable override (recommended for production):**
```bash
export ATTUNE__SECURITY__JWT_SECRET=$(openssl rand -base64 64)
export ATTUNE__SECURITY__ENCRYPTION_KEY=$(openssl rand -base64 32)
```
### Logging Configuration
```yaml
log:
# Log level: trace, debug, info, warn, error
level: info
# Log format: json (structured), pretty (human-readable)
format: json
# Enable console output
console: true
# Optional: write logs to file
file: /var/log/attune/attune.log
```
**Environment variable override:**
```bash
export ATTUNE__LOG__LEVEL=debug
export ATTUNE__LOG__FORMAT=pretty
```
### Redis Configuration (Optional)
```yaml
redis:
url: redis://localhost:6379
pool_size: 10
```
### Message Queue Configuration (Optional)
```yaml
message_queue:
url: amqp://guest:guest@localhost:5672/%2f
exchange: attune
enable_dlq: true # Enable dead letter queue
message_ttl: 3600 # Message TTL (seconds)
```
### Worker Configuration (Optional)
```yaml
worker:
name: attune-worker-1
worker_type: local # local, remote, container
max_concurrent_tasks: 10
heartbeat_interval: 30 # seconds
task_timeout: 300 # seconds
```
## Environment-Specific Configuration
### Development Environment
Create `config.development.yaml`:
```yaml
environment: development
database:
url: postgresql://postgres:postgres@localhost:5432/attune
log_statements: true
server:
host: 127.0.0.1
cors_origins:
- http://localhost:3000
- http://localhost:5173
log:
level: debug
format: pretty
security:
jwt_secret: dev-secret-not-for-production
jwt_access_expiration: 86400 # 24 hours for dev convenience
```
### Production Environment
Create `config.production.yaml`:
```yaml
environment: production
database:
url: postgresql://attune_user:CHANGE_ME@db.example.com/attune
max_connections: 100
log_statements: false
server:
cors_origins:
- https://app.example.com
- https://www.example.com
log:
level: info
format: json
file: /var/log/attune/attune.log
security:
# Override with environment variables!
jwt_secret: CHANGE_ME_USE_ENV_VAR
encryption_key: CHANGE_ME_USE_ENV_VAR
```
**Important:** Always override sensitive values in production using environment variables:
```bash
export ATTUNE__SECURITY__JWT_SECRET=$(openssl rand -base64 64)
export ATTUNE__SECURITY__ENCRYPTION_KEY=$(openssl rand -base64 32)
export ATTUNE__DATABASE__URL=postgresql://user:pass@db.example.com/attune
```
### Test Environment
Create `config.test.yaml`:
```yaml
environment: test
database:
url: postgresql://postgres:postgres@localhost:5432/attune_test
max_connections: 10
server:
port: 0 # Random port for tests
log:
level: warn
format: pretty
security:
jwt_secret: test-secret-not-secure
jwt_access_expiration: 300 # 5 minutes
```
## Environment Variables
You can override any configuration value using environment variables with the `ATTUNE__` prefix and `__` separator for nested keys.
### Syntax
```
ATTUNE__<section>__<key>=value
```
### Examples
```bash
# Database
export ATTUNE__DATABASE__URL=postgresql://localhost/attune
export ATTUNE__DATABASE__MAX_CONNECTIONS=100
# Server
export ATTUNE__SERVER__PORT=8080
export ATTUNE__SERVER__HOST=0.0.0.0
# Security
export ATTUNE__SECURITY__JWT_SECRET=my-secret-key
export ATTUNE__SECURITY__ENABLE_AUTH=true
# Logging
export ATTUNE__LOG__LEVEL=debug
export ATTUNE__LOG__FORMAT=pretty
# Arrays (comma-separated)
export ATTUNE__SERVER__CORS_ORIGINS="https://app.com,https://www.app.com"
```
### Boolean Values
Boolean values can be set using:
- `true`, `1`, `yes`, `on` → true
- `false`, `0`, `no`, `off` → false
```bash
export ATTUNE__DATABASE__LOG_STATEMENTS=true
export ATTUNE__SECURITY__ENABLE_AUTH=false
```
## Custom Configuration File Path
By default, Attune looks for `config.yaml` in the current directory. You can specify a custom path:
```bash
export ATTUNE_CONFIG=/path/to/custom-config.yaml
cargo run --bin attune-api
```
Or:
```bash
ATTUNE_CONFIG=/etc/attune/config.yaml attune-api
```
## Configuration Validation
The application validates configuration on startup and will fail with clear error messages if:
- Required fields are missing (e.g., JWT secret when auth is enabled)
- Values are invalid (e.g., invalid log level)
- Security requirements are not met (e.g., encryption key too short)
Example validation errors:
```
Error: Configuration validation failed: JWT secret is required when authentication is enabled
Error: Invalid log level: trace. Must be one of: ["trace", "debug", "info", "warn", "error"]
Error: Encryption key must be at least 32 characters
```
## Security Best Practices
### 1. Never Commit Secrets
Add sensitive config files to `.gitignore`:
```gitignore
config.yaml
config.*.yaml
!config.example.yaml
!config.development.yaml
!config.test.yaml
```
### 2. Use Strong Secrets
Generate cryptographically secure secrets:
```bash
# JWT secret (64 bytes, base64 encoded)
openssl rand -base64 64
# Encryption key (32 bytes minimum, base64 encoded)
openssl rand -base64 32
```
### 3. Environment Variables in Production
Always use environment variables for sensitive values in production:
```bash
# In production deployment
export ATTUNE__SECURITY__JWT_SECRET=$(openssl rand -base64 64)
export ATTUNE__SECURITY__ENCRYPTION_KEY=$(openssl rand -base64 32)
export ATTUNE__DATABASE__URL=postgresql://user:${DB_PASSWORD}@db.example.com/attune
```
### 4. File Permissions
Restrict access to configuration files:
```bash
chmod 600 config.yaml
chown appuser:appuser config.yaml
```
### 5. Separate Configs Per Environment
Use environment-specific files and never share production secrets with development:
```
config.yaml # Base (committed)
config.example.yaml # Example (committed)
config.development.yaml # Development (committed, no real secrets)
config.production.yaml # Production template (committed, placeholders only)
config.yaml # Actual config (NOT committed, local overrides)
```
## Docker/Container Deployments
### Option 1: Mount Configuration File
```bash
docker run -v /path/to/config.yaml:/app/config.yaml attune-api
```
### Option 2: Environment Variables (Recommended)
```bash
docker run \
-e ATTUNE__DATABASE__URL=postgresql://db/attune \
-e ATTUNE__SECURITY__JWT_SECRET=$(cat /run/secrets/jwt_secret) \
-e ATTUNE__SERVER__PORT=8080 \
attune-api
```
### Docker Compose Example
```yaml
version: '3.8'
services:
api:
image: attune-api
environment:
ATTUNE__DATABASE__URL: postgresql://postgres:postgres@db:5432/attune
ATTUNE__SERVER__PORT: 8080
ATTUNE__LOG__LEVEL: info
secrets:
- jwt_secret
- encryption_key
command: sh -c "
export ATTUNE__SECURITY__JWT_SECRET=$$(cat /run/secrets/jwt_secret) &&
export ATTUNE__SECURITY__ENCRYPTION_KEY=$$(cat /run/secrets/encryption_key) &&
attune-api
"
secrets:
jwt_secret:
file: ./secrets/jwt_secret.txt
encryption_key:
file: ./secrets/encryption_key.txt
```
## Kubernetes Deployments
### ConfigMap for Base Configuration
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: attune-config
data:
config.yaml: |
service_name: attune
environment: production
server:
host: 0.0.0.0
port: 8080
log:
level: info
format: json
```
### Secrets for Sensitive Values
```yaml
apiVersion: v1
kind: Secret
metadata:
name: attune-secrets
type: Opaque
stringData:
jwt-secret: "your-base64-jwt-secret"
encryption-key: "your-base64-encryption-key"
database-url: "postgresql://user:pass@db/attune"
```
### Deployment
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: attune-api
spec:
template:
spec:
containers:
- name: api
image: attune-api:latest
env:
- name: ATTUNE__SECURITY__JWT_SECRET
valueFrom:
secretKeyRef:
name: attune-secrets
key: jwt-secret
- name: ATTUNE__SECURITY__ENCRYPTION_KEY
valueFrom:
secretKeyRef:
name: attune-secrets
key: encryption-key
- name: ATTUNE__DATABASE__URL
valueFrom:
secretKeyRef:
name: attune-secrets
key: database-url
volumeMounts:
- name: config
mountPath: /app/config.yaml
subPath: config.yaml
volumes:
- name: config
configMap:
name: attune-config
```
## Troubleshooting
### Configuration Not Loading
1. **Check file exists:**
```bash
ls -la config.yaml
```
2. **Check YAML syntax:**
```bash
# Install yq if needed: brew install yq
yq eval config.yaml
```
3. **Check environment variable:**
```bash
echo $ATTUNE_CONFIG
```
### Environment Variables Not Working
1. **Check variable name format:**
- Must start with `ATTUNE__`
- Use double underscores for nesting
- Case-sensitive
2. **Verify variable is set:**
```bash
env | grep ATTUNE
```
3. **Check for typos:**
```bash
export ATTUNE__DATABASE__URL=... # Correct
export ATTUNE_DATABASE_URL=... # Wrong! (single underscore)
```
### Validation Errors
Enable debug logging to see detailed configuration:
```bash
ATTUNE__LOG__LEVEL=debug cargo run --bin attune-api
```
## Migration from .env Files
If you're migrating from `.env` files, here's a conversion guide:
### Before (.env):
```bash
ATTUNE__DATABASE__URL=postgresql://localhost/attune
ATTUNE__SERVER__PORT=8080
ATTUNE__LOG__LEVEL=info
ATTUNE__SECURITY__JWT_SECRET=my-secret
```
### After (config.yaml):
```yaml
database:
url: postgresql://localhost/attune
server:
port: 8080
log:
level: info
security:
jwt_secret: my-secret
```
**Note:** You can still use environment variables for overrides, but the base configuration is now in YAML.
## Additional Resources
- [Quick Start Guide](quick-start.md)
- [Authentication Documentation](authentication.md)
- [API Reference](api-reference.md)
- [Deployment Guide](deployment.md)

553
docs/configuration/env-to-yaml-migration.md Normal file
View File

@@ -0,0 +1,553 @@
# Migration Guide: .env to YAML Configuration
This guide helps you migrate from `.env` file configuration to the new YAML-based configuration system.
## Why Migrate to YAML?
The YAML configuration format offers several advantages:
- **Better readability**: Clear hierarchical structure
- **Comments**: Document your configuration inline
- **Complex structures**: Native support for arrays, nested objects
- **Type safety**: Automatic type conversion
- **Environment-specific configs**: Easy to maintain separate configs per environment
- **No prefix noise**: No need for `ATTUNE__` prefixes everywhere
## Quick Migration Steps
1. **Copy the example config**
```bash
cp config.example.yaml config.yaml
```
2. **Convert your .env values** (see conversion examples below)
3. **Test the new configuration**
```bash
cargo run --bin attune-api
```
4. **Remove old .env file** (optional, after verification)
## Conversion Reference
### Database Configuration
**Before (.env):**
```bash
ATTUNE__DATABASE__URL=postgresql://postgres:postgres@localhost:5432/attune
ATTUNE__DATABASE__MAX_CONNECTIONS=50
ATTUNE__DATABASE__MIN_CONNECTIONS=5
ATTUNE__DATABASE__CONNECT_TIMEOUT=30
ATTUNE__DATABASE__IDLE_TIMEOUT=600
ATTUNE__DATABASE__LOG_STATEMENTS=true
```
**After (config.yaml):**
```yaml
database:
url: postgresql://postgres:postgres@localhost:5432/attune
max_connections: 50
min_connections: 5
connect_timeout: 30
idle_timeout: 600
log_statements: true
```
### Server Configuration
**Before (.env):**
```bash
ATTUNE__SERVER__HOST=0.0.0.0
ATTUNE__SERVER__PORT=8080
ATTUNE__SERVER__REQUEST_TIMEOUT=30
ATTUNE__SERVER__ENABLE_CORS=true
ATTUNE__SERVER__CORS_ORIGINS=http://localhost:3000,http://localhost:5173
ATTUNE__SERVER__MAX_BODY_SIZE=10485760
```
**After (config.yaml):**
```yaml
server:
host: 0.0.0.0
port: 8080
request_timeout: 30
enable_cors: true
cors_origins:
- http://localhost:3000
- http://localhost:5173
max_body_size: 10485760
```
### Security Configuration
**Before (.env):**
```bash
ATTUNE__SECURITY__JWT_SECRET=your-secret-key-here
ATTUNE__SECURITY__JWT_ACCESS_EXPIRATION=3600
ATTUNE__SECURITY__JWT_REFRESH_EXPIRATION=604800
ATTUNE__SECURITY__ENCRYPTION_KEY=your-32-char-encryption-key
ATTUNE__SECURITY__ENABLE_AUTH=true
```
**After (config.yaml):**
```yaml
security:
jwt_secret: your-secret-key-here
jwt_access_expiration: 3600
jwt_refresh_expiration: 604800
encryption_key: your-32-char-encryption-key
enable_auth: true
```
### Logging Configuration
**Before (.env):**
```bash
ATTUNE__LOG__LEVEL=info
ATTUNE__LOG__FORMAT=json
ATTUNE__LOG__CONSOLE=true
ATTUNE__LOG__FILE=/var/log/attune/attune.log
```
**After (config.yaml):**
```yaml
log:
level: info
format: json
console: true
file: /var/log/attune/attune.log
```
### Redis Configuration
**Before (.env):**
```bash
ATTUNE__REDIS__URL=redis://localhost:6379
ATTUNE__REDIS__POOL_SIZE=10
```
**After (config.yaml):**
```yaml
redis:
url: redis://localhost:6379
pool_size: 10
```
### Message Queue Configuration
**Before (.env):**
```bash
ATTUNE__MESSAGE_QUEUE__URL=amqp://guest:guest@localhost:5672/%2f
ATTUNE__MESSAGE_QUEUE__EXCHANGE=attune
ATTUNE__MESSAGE_QUEUE__ENABLE_DLQ=true
ATTUNE__MESSAGE_QUEUE__MESSAGE_TTL=3600
```
**After (config.yaml):**
```yaml
message_queue:
url: amqp://guest:guest@localhost:5672/%2f
exchange: attune
enable_dlq: true
message_ttl: 3600
```
## Automated Conversion Script
If you have a complex `.env` file, you can use this Python script to help convert it:
```python
#!/usr/bin/env python3
"""
Convert .env file to YAML configuration
Usage: python env_to_yaml.py .env > config.yaml
"""
import sys
import re
def parse_env_line(line):
"""Parse a single .env line"""
line = line.strip()
if not line or line.startswith('#'):
return None, None
match = re.match(r'ATTUNE__(.+?)=(.+)', line)
if not match:
return None, None
key = match.group(1)
value = match.group(2)
# Remove quotes if present
value = value.strip('"').strip("'")
# Convert boolean strings
if value.lower() in ('true', 'false'):
value = value.lower()
# Try to convert to number
elif value.isdigit():
value = int(value)
return key, value
def build_yaml_structure(env_vars):
"""Build nested YAML structure from flat env vars"""
structure = {}
for key, value in env_vars.items():
parts = key.lower().split('__')
# Navigate/create nested structure
current = structure
for part in parts[:-1]:
if part not in current:
current[part] = {}
current = current[part]
# Handle special cases (arrays)
final_key = parts[-1]
if isinstance(value, str) and ',' in value:
# Assume comma-separated list
current[final_key] = [v.strip() for v in value.split(',')]
else:
current[final_key] = value
return structure
def print_yaml(data, indent=0):
"""Print YAML structure"""
for key, value in data.items():
if isinstance(value, dict):
print(f"{' ' * indent}{key}:")
print_yaml(value, indent + 1)
elif isinstance(value, list):
print(f"{' ' * indent}{key}:")
for item in value:
print(f"{' ' * (indent + 1)}- {item}")
elif isinstance(value, bool):
print(f"{' ' * indent}{key}: {str(value).lower()}")
elif isinstance(value, (int, float)):
print(f"{' ' * indent}{key}: {value}")
else:
# Quote strings that might be ambiguous
if any(c in str(value) for c in [':', '#', '@', '%']):
print(f"{' ' * indent}{key}: \"{value}\"")
else:
print(f"{' ' * indent}{key}: {value}")
def main():
if len(sys.argv) < 2:
print("Usage: python env_to_yaml.py <.env file>", file=sys.stderr)
sys.exit(1)
env_file = sys.argv[1]
env_vars = {}
# Parse .env file
with open(env_file, 'r') as f:
for line in f:
key, value = parse_env_line(line)
if key and value is not None:
env_vars[key] = value
# Build and print YAML structure
structure = build_yaml_structure(env_vars)
print("# Attune Configuration")
print("# Generated from .env file")
print("# Please review and adjust as needed")
print()
print_yaml(structure)
if __name__ == '__main__':
main()
```
**Usage:**
```bash
python env_to_yaml.py .env > config.yaml
```
**Note:** Always review the generated YAML file and adjust as needed.
## Handling Environment-Specific Configurations
One of the benefits of YAML is easy environment-specific configurations.
### Create Base Configuration
`config.yaml` (common settings):
```yaml
service_name: attune
database:
max_connections: 50
connect_timeout: 30
server:
enable_cors: true
max_body_size: 10485760
log:
console: true
```
### Development Override
`config.development.yaml`:
```yaml
environment: development
database:
url: postgresql://postgres:postgres@localhost:5432/attune
log_statements: true
server:
host: 127.0.0.1
port: 8080
log:
level: debug
format: pretty
security:
jwt_secret: dev-secret-not-for-production
jwt_access_expiration: 86400 # 24 hours
```
### Production Configuration
`config.production.yaml`:
```yaml
environment: production
database:
url: postgresql://user:password@db.prod.example.com/attune
max_connections: 100
server:
host: 0.0.0.0
port: 8080
cors_origins:
- https://app.example.com
- https://www.example.com
log:
level: info
format: json
file: /var/log/attune/attune.log
# Use environment variables for secrets!
# ATTUNE__SECURITY__JWT_SECRET
# ATTUNE__SECURITY__ENCRYPTION_KEY
```
## Still Using Environment Variables
Environment variables still work for overrides! This is especially useful for:
- **Container deployments** (Docker, Kubernetes)
- **Production secrets** (don't commit secrets to YAML files)
- **CI/CD pipelines**
- **Quick overrides** without editing files
**Example:**
```bash
# Base config in config.yaml, override database URL
export ATTUNE__DATABASE__URL=postgresql://prod-db/attune
cargo run --bin attune-api
```
## Migration Checklist
- [ ] Back up your existing `.env` file
- [ ] Copy `config.example.yaml` to `config.yaml`
- [ ] Convert `.env` values to YAML format
- [ ] Add comments to document your configuration
- [ ] Test the application starts successfully
- [ ] Verify all features work (database, authentication, etc.)
- [ ] Create environment-specific configs if needed
- [ ] Update deployment scripts/documentation
- [ ] Set up environment variables for production secrets
- [ ] Add `config.yaml` to `.gitignore` (if it contains secrets)
- [ ] Commit example configs to version control
## Rollback Plan
If you need to roll back to `.env` files:
1. The old `.env` file still works (keep your backup)
2. Rename/remove `config.yaml` temporarily
3. Environment variables with `ATTUNE__` prefix still work
**Note:** The application will still load config from environment variables if no YAML file is present.
## Common Issues
### Issue: Configuration not loading
**Problem:** Application can't find `config.yaml`
**Solution:**
- Ensure `config.yaml` is in the current working directory
- Or set `ATTUNE_CONFIG` environment variable:
```bash
export ATTUNE_CONFIG=/path/to/config.yaml
```
### Issue: YAML syntax error
**Problem:** YAML parsing fails
**Solution:**
- Check indentation (use spaces, not tabs)
- Validate YAML syntax:
```bash
# Using Python
python -c "import yaml; yaml.safe_load(open('config.yaml'))"
# Using yq
yq eval config.yaml
```
### Issue: Boolean values not working
**Problem:** Boolean value treated as string
**Solution:** Use lowercase without quotes:
```yaml
# Correct
enable_auth: true
log_statements: false
# Incorrect
enable_auth: "true" # String, not boolean
LOG_STATEMENTS: FALSE # Wrong case
```
### Issue: Array values not parsing
**Problem:** CORS origins not loading correctly
**Solution:** Use YAML array syntax:
```yaml
# Correct
cors_origins:
- http://localhost:3000
- http://localhost:5173
# Also correct (inline)
cors_origins: [http://localhost:3000, http://localhost:5173]
# For env vars, use comma-separated
export ATTUNE__SERVER__CORS_ORIGINS="http://localhost:3000,http://localhost:5173"
```
### Issue: Special characters in strings
**Problem:** URL or path not parsing correctly
**Solution:** Quote strings with special characters:
```yaml
# With special characters
database:
url: "postgresql://user:p@ssw0rd!@localhost/attune"
# Or use single quotes
redis:
url: 'redis://localhost:6379'
```
## Tips and Best Practices
### 1. Use Comments
Document your configuration:
```yaml
database:
# Production database endpoint
url: postgresql://user@db.example.com/attune
# Adjust based on expected load
max_connections: 100 # Increased for production traffic
```
### 2. Group Related Settings
Keep related configuration together:
```yaml
# All server-related settings
server:
host: 0.0.0.0
port: 8080
cors_origins:
- https://app.example.com
```
### 3. Use Environment-Specific Files
Don't duplicate settings—use base + overrides:
```
config.yaml # Common settings
config.development.yaml # Dev overrides
config.production.yaml # Prod overrides
```
### 4. Never Commit Secrets
Use environment variables for sensitive data:
```yaml
# config.production.yaml (committed to git)
security:
# Override these with environment variables!
jwt_secret: CHANGE_ME_USE_ENV_VAR
encryption_key: CHANGE_ME_USE_ENV_VAR
```
```bash
# In production (not in git)
export ATTUNE__SECURITY__JWT_SECRET=$(openssl rand -base64 64)
export ATTUNE__SECURITY__ENCRYPTION_KEY=$(openssl rand -base64 32)
```
### 5. Validate Configuration
Always validate after migration:
```bash
# The application will validate on startup
cargo run --bin attune-api
# Check for validation errors in logs
```
## Getting Help
If you encounter issues during migration:
1. **Check the logs**: Enable debug logging
```bash
ATTUNE__LOG__LEVEL=debug cargo run --bin attune-api
```
2. **Review documentation**:
- [Configuration Guide](configuration.md)
- [Quick Start Guide](quick-start.md)
3. **Validate YAML syntax**: Use online validators or CLI tools
4. **Compare with examples**: Check `config.example.yaml`
## Summary
The migration from `.env` to YAML provides:
✅ Better readability and maintainability
✅ Native support for complex data types
✅ Easy environment-specific configurations
✅ Inline documentation with comments
✅ Backward compatibility with environment variables
The YAML configuration system gives you more flexibility while maintaining the security and override capabilities of environment variables.