attune/work-summary/sessions/2025-01-13-yaml-configuration.md

Work Summary: YAML Configuration Migration

Date: 2025-01-13
Task: Migrate from .env to YAML configuration format

Overview

Successfully migrated the Attune project from .env file-based configuration to a modern YAML configuration system with environment variable overrides. This provides better readability, type safety, and support for complex nested structures.

Changes Made

1. Configuration System Refactoring

File: crates/common/src/config.rs

• Replaced .env loading logic with YAML-based configuration
• Implemented layered configuration loading:
  1. Base config file (config.yaml or ATTUNE_CONFIG path)
  2. Environment-specific config (e.g., config.development.yaml)
  3. Environment variables (with ATTUNE__ prefix for overrides)
• Added Config::load() method with automatic environment detection
• Added Config::load_from_file() for explicit file loading
• Enhanced documentation with examples and usage patterns
• Added support for comma-separated lists in environment variables

2. Example Configuration Files

Created comprehensive YAML configuration templates:

config.yaml - Base configuration template

• All configuration sections documented
• Sensible defaults for development
• Comments explaining each setting

config.example.yaml - Safe-to-commit example

• Template for new installations
• Placeholder values for secrets
• Instructions for generating secure keys

config.development.yaml - Development environment

• Debug logging enabled
• Verbose SQL statements
• Extended token expiration for convenience
• Local CORS origins

config.production.yaml - Production template

• Production-ready settings
• Placeholder secrets (must override with env vars)
• Stricter security settings
• JSON logging for aggregation

config.test.yaml - Test environment

• Separate test database
• Minimal logging for clean test output
• Fixed secrets for reproducible tests
• Random port assignment

3. Dependency Cleanup

Removed dotenvy dependency from:

• Cargo.toml (workspace)
• crates/common/Cargo.toml (dev-dependencies)
• crates/api/Cargo.toml

Updated code:

• crates/api/src/main.rs - Removed .env loading calls
• crates/common/tests/helpers.rs - Use environment variables instead

4. Documentation

Created comprehensive guides:

docs/configuration.md (624 lines)

• Complete configuration reference
• All sections documented with examples
• Environment variable override syntax
• Security best practices
• Docker/Kubernetes deployment examples
• Troubleshooting guide
• Migration from .env instructions

docs/env-to-yaml-migration.md (553 lines)

• Step-by-step migration guide
• Before/after conversion examples
• Python script for automated conversion
• Environment-specific configuration patterns
• Common issues and solutions
• Tips and best practices

Updated existing documentation:

• README.md - Replaced .env examples with YAML
• docs/quick-start.md - Updated all configuration examples
• Added YAML configuration instructions
• Updated troubleshooting sections

5. Git Configuration

.gitignore updates:

# Configuration files (keep *.example.yaml)
config.yaml
config.*.yaml
!config.example.yaml
!config.development.yaml
!config.test.yaml

This ensures:

• Actual config files are not committed (may contain secrets)
• Example and safe configs are version-controlled
• Development and test configs can be shared

Configuration Format Comparison

Before (.env):

ATTUNE__DATABASE__URL=postgresql://localhost/attune
ATTUNE__DATABASE__MAX_CONNECTIONS=50
ATTUNE__SERVER__PORT=8080
ATTUNE__SERVER__CORS_ORIGINS=http://localhost:3000,http://localhost:5173
ATTUNE__SECURITY__JWT_SECRET=secret
ATTUNE__LOG__LEVEL=info

After (config.yaml):

database:
  url: postgresql://localhost/attune
  max_connections: 50

server:
  port: 8080
  cors_origins:
    - http://localhost:3000
    - http://localhost:5173

security:
  jwt_secret: secret

log:
  level: info

Benefits

Readability

• Clear hierarchical structure
• Native support for comments
• No prefix noise (ATTUNE__)
• Better IDE support with syntax highlighting

Maintainability

• Environment-specific configs (dev, test, prod)
• Easy to see all configuration in one place
• Type-safe parsing (booleans, numbers, arrays)

Flexibility

• Complex nested structures
• Native array support (no comma-separated strings)
• Still supports environment variable overrides
• Multiple config file support

Security

• Secrets can be overridden with env vars
• Safe examples can be version-controlled
• Clear separation of template vs actual config

Configuration Loading Priority

The system loads configuration in this order (later overrides earlier):

1. Base YAML file - config.yaml or $ATTUNE_CONFIG
2. Environment-specific YAML - config.{environment}.yaml
3. Environment variables - ATTUNE__* for overrides

Example:

# Use production config with secret override
export ATTUNE_CONFIG=config.production.yaml
export ATTUNE__SECURITY__JWT_SECRET=$(openssl rand -base64 64)
cargo run --bin attune-api

Usage Examples

Development

# Uses config.development.yaml automatically
export ATTUNE__ENVIRONMENT=development
cargo run --bin attune-api

Production

# Use production config with env var secrets
export ATTUNE_CONFIG=config.production.yaml
export ATTUNE__SECURITY__JWT_SECRET=$SECRET_FROM_VAULT
export ATTUNE__DATABASE__URL=$DB_CONNECTION_STRING
attune-api

Testing

# Uses config.test.yaml
export ATTUNE__ENVIRONMENT=test
cargo test

Docker

docker run \
  -v /path/to/config.yaml:/app/config.yaml \
  -e ATTUNE__SECURITY__JWT_SECRET=$SECRET \
  attune-api

Migration Steps for Users

1. Copy example configuration:

   cp config.example.yaml config.yaml
   

2. Edit with your settings:

   nano config.yaml
   

3. Generate secure secrets:

   openssl rand -base64 64  # JWT secret
   openssl rand -base64 32  # Encryption key
   

4. Test the application:

   cargo run --bin attune-api
   

Testing

• Configuration loads from YAML files
• Environment variables override YAML values
• Environment-specific configs load correctly
• All services start successfully
• Database connection works
• Authentication still functions
• CORS configuration applies correctly
• Test suite passes with new config

Files Changed

Created

• config.yaml (base configuration)
• config.example.yaml (safe example)
• config.development.yaml (dev overrides)
• config.production.yaml (prod template)
• config.test.yaml (test configuration)
• docs/configuration.md (comprehensive guide)
• docs/env-to-yaml-migration.md (migration guide)

Modified

• crates/common/src/config.rs (YAML loading logic)
• crates/api/src/main.rs (removed dotenvy)
• crates/common/tests/helpers.rs (removed dotenvy)
• Cargo.toml (removed dotenvy dependency)
• crates/common/Cargo.toml (removed dev-dependency)
• crates/api/Cargo.toml (removed dependency)
• .gitignore (added config file rules)
• README.md (updated configuration section)
• docs/quick-start.md (updated all examples)

Deprecated

• .env files (no longer used, but env vars still work)
• dotenvy crate (removed from dependencies)

Notes

Backward Compatibility

Environment variables with the ATTUNE__ prefix still work for overrides. This ensures:

• Existing container deployments continue working
• CI/CD pipelines don't need changes
• Secrets can be injected via env vars (recommended)

Security Considerations

• Never commit config.yaml if it contains real secrets
• Use environment variables for production secrets
• The .gitignore is configured to prevent accidental commits
• Example files use placeholder values only

Future Enhancements

Potential improvements for future consideration:

• Config validation on startup with detailed error messages ✅ (already implemented)
• Hot-reload configuration without restart
• Config schema validation with JSON Schema
• Encrypted config file support
• Remote config loading (e.g., from Consul, etcd)

Documentation Links

• Configuration Guide - Complete reference
• Migration Guide - .env to YAML migration
• Quick Start - Updated with YAML examples
• README - High-level overview

Issues Encountered and Resolved

Encryption Key Length Validation

Problem: After initial implementation, the application failed to start with error:

Error: Validation error: Encryption key must be at least 32 characters

Root Cause: The placeholder encryption keys in config.development.yaml and config.test.yaml were exactly 31 characters, one character short of the required 32-character minimum.

Solution: Updated all configuration files to use encryption key placeholders that are at least 32 characters:

• config.yaml: dev-encryption-key-at-least-32-characters-long-change-this (58 chars)
• config.example.yaml: dev-encryption-key-at-least-32-characters-long-change-this (58 chars)
• config.development.yaml: test-encryption-key-32-chars-okay (33 chars)
• config.test.yaml: test-encryption-key-32-chars-okay (33 chars)
• config.production.yaml: CHANGE_ME_USE_ENV_VAR_PLACEHOLDER_32_CHARS_MINIMUM (50 chars)

Verification: Application now starts successfully with the message:

INFO Starting Attune API Service
INFO Configuration loaded successfully
INFO Database connection established
INFO Starting server on 127.0.0.1:8080
INFO Server listening on 127.0.0.1:8080

Conclusion

The migration to YAML configuration provides a more maintainable, readable, and flexible configuration system while maintaining full backward compatibility with environment variable overrides. The comprehensive documentation ensures smooth adoption for all users.

All existing functionality has been preserved, and the new system is production-ready.