attune-system/attune
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

re-uploading work

Browse Source
This commit is contained in:
David Culbreth
2026-02-04 17:46:30 -06:00
commit 3b14c65998
1388 changed files with 381262 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

339
work-summary/sessions/2025-01-13-yaml-configuration.md Normal file
View File

@@ -0,0 +1,339 @@
# Work Summary: YAML Configuration Migration
**Date:** 2025-01-13
**Task:** Migrate from .env to YAML configuration format
## Overview
Successfully migrated the Attune project from `.env` file-based configuration to a modern YAML configuration system with environment variable overrides. This provides better readability, type safety, and support for complex nested structures.
## Changes Made
### 1. Configuration System Refactoring
**File: `crates/common/src/config.rs`**
- Replaced `.env` loading logic with YAML-based configuration
- Implemented layered configuration loading:
1. Base config file (`config.yaml` or `ATTUNE_CONFIG` path)
2. Environment-specific config (e.g., `config.development.yaml`)
3. Environment variables (with `ATTUNE__` prefix for overrides)
- Added `Config::load()` method with automatic environment detection
- Added `Config::load_from_file()` for explicit file loading
- Enhanced documentation with examples and usage patterns
- Added support for comma-separated lists in environment variables
### 2. Example Configuration Files
Created comprehensive YAML configuration templates:
**`config.yaml`** - Base configuration template
- All configuration sections documented
- Sensible defaults for development
- Comments explaining each setting
**`config.example.yaml`** - Safe-to-commit example
- Template for new installations
- Placeholder values for secrets
- Instructions for generating secure keys
**`config.development.yaml`** - Development environment
- Debug logging enabled
- Verbose SQL statements
- Extended token expiration for convenience
- Local CORS origins
**`config.production.yaml`** - Production template
- Production-ready settings
- Placeholder secrets (must override with env vars)
- Stricter security settings
- JSON logging for aggregation
**`config.test.yaml`** - Test environment
- Separate test database
- Minimal logging for clean test output
- Fixed secrets for reproducible tests
- Random port assignment
### 3. Dependency Cleanup
**Removed `dotenvy` dependency from:**
- `Cargo.toml` (workspace)
- `crates/common/Cargo.toml` (dev-dependencies)
- `crates/api/Cargo.toml`
**Updated code:**
- `crates/api/src/main.rs` - Removed `.env` loading calls
- `crates/common/tests/helpers.rs` - Use environment variables instead
### 4. Documentation
**Created comprehensive guides:**
**`docs/configuration.md`** (624 lines)
- Complete configuration reference
- All sections documented with examples
- Environment variable override syntax
- Security best practices
- Docker/Kubernetes deployment examples
- Troubleshooting guide
- Migration from .env instructions
**`docs/env-to-yaml-migration.md`** (553 lines)
- Step-by-step migration guide
- Before/after conversion examples
- Python script for automated conversion
- Environment-specific configuration patterns
- Common issues and solutions
- Tips and best practices
**Updated existing documentation:**
- `README.md` - Replaced .env examples with YAML
- `docs/quick-start.md` - Updated all configuration examples
- Added YAML configuration instructions
- Updated troubleshooting sections
### 5. Git Configuration
**`.gitignore` updates:**
```gitignore
# Configuration files (keep *.example.yaml)
config.yaml
config.*.yaml
!config.example.yaml
!config.development.yaml
!config.test.yaml
```
This ensures:
- Actual config files are not committed (may contain secrets)
- Example and safe configs are version-controlled
- Development and test configs can be shared
## Configuration Format Comparison
### Before (.env):
```bash
ATTUNE__DATABASE__URL=postgresql://localhost/attune
ATTUNE__DATABASE__MAX_CONNECTIONS=50
ATTUNE__SERVER__PORT=8080
ATTUNE__SERVER__CORS_ORIGINS=http://localhost:3000,http://localhost:5173
ATTUNE__SECURITY__JWT_SECRET=secret
ATTUNE__LOG__LEVEL=info
```
### After (config.yaml):
```yaml
database:
url: postgresql://localhost/attune
max_connections: 50
server:
port: 8080
cors_origins:
- http://localhost:3000
- http://localhost:5173
security:
jwt_secret: secret
log:
level: info
```
## Benefits
### Readability
- Clear hierarchical structure
- Native support for comments
- No prefix noise (`ATTUNE__`)
- Better IDE support with syntax highlighting
### Maintainability
- Environment-specific configs (dev, test, prod)
- Easy to see all configuration in one place
- Type-safe parsing (booleans, numbers, arrays)
### Flexibility
- Complex nested structures
- Native array support (no comma-separated strings)
- Still supports environment variable overrides
- Multiple config file support
### Security
- Secrets can be overridden with env vars
- Safe examples can be version-controlled
- Clear separation of template vs actual config
## Configuration Loading Priority
The system loads configuration in this order (later overrides earlier):
1. **Base YAML file** - `config.yaml` or `$ATTUNE_CONFIG`
2. **Environment-specific YAML** - `config.{environment}.yaml`
3. **Environment variables** - `ATTUNE__*` for overrides
Example:
```bash
# Use production config with secret override
export ATTUNE_CONFIG=config.production.yaml
export ATTUNE__SECURITY__JWT_SECRET=$(openssl rand -base64 64)
cargo run --bin attune-api
```
## Usage Examples
### Development
```bash
# Uses config.development.yaml automatically
export ATTUNE__ENVIRONMENT=development
cargo run --bin attune-api
```
### Production
```bash
# Use production config with env var secrets
export ATTUNE_CONFIG=config.production.yaml
export ATTUNE__SECURITY__JWT_SECRET=$SECRET_FROM_VAULT
export ATTUNE__DATABASE__URL=$DB_CONNECTION_STRING
attune-api
```
### Testing
```bash
# Uses config.test.yaml
export ATTUNE__ENVIRONMENT=test
cargo test
```
### Docker
```bash
docker run \
-v /path/to/config.yaml:/app/config.yaml \
-e ATTUNE__SECURITY__JWT_SECRET=$SECRET \
attune-api
```
## Migration Steps for Users
1. Copy example configuration:
```bash
cp config.example.yaml config.yaml
```
2. Edit with your settings:
```bash
nano config.yaml
```
3. Generate secure secrets:
```bash
openssl rand -base64 64 # JWT secret
openssl rand -base64 32 # Encryption key
```
4. Test the application:
```bash
cargo run --bin attune-api
```
## Testing
- [x] Configuration loads from YAML files
- [x] Environment variables override YAML values
- [x] Environment-specific configs load correctly
- [x] All services start successfully
- [x] Database connection works
- [x] Authentication still functions
- [x] CORS configuration applies correctly
- [x] Test suite passes with new config
## Files Changed
### Created
- `config.yaml` (base configuration)
- `config.example.yaml` (safe example)
- `config.development.yaml` (dev overrides)
- `config.production.yaml` (prod template)
- `config.test.yaml` (test configuration)
- `docs/configuration.md` (comprehensive guide)
- `docs/env-to-yaml-migration.md` (migration guide)
### Modified
- `crates/common/src/config.rs` (YAML loading logic)
- `crates/api/src/main.rs` (removed dotenvy)
- `crates/common/tests/helpers.rs` (removed dotenvy)
- `Cargo.toml` (removed dotenvy dependency)
- `crates/common/Cargo.toml` (removed dev-dependency)
- `crates/api/Cargo.toml` (removed dependency)
- `.gitignore` (added config file rules)
- `README.md` (updated configuration section)
- `docs/quick-start.md` (updated all examples)
### Deprecated
- `.env` files (no longer used, but env vars still work)
- `dotenvy` crate (removed from dependencies)
## Notes
### Backward Compatibility
Environment variables with the `ATTUNE__` prefix **still work** for overrides. This ensures:
- Existing container deployments continue working
- CI/CD pipelines don't need changes
- Secrets can be injected via env vars (recommended)
### Security Considerations
- Never commit `config.yaml` if it contains real secrets
- Use environment variables for production secrets
- The `.gitignore` is configured to prevent accidental commits
- Example files use placeholder values only
### Future Enhancements
Potential improvements for future consideration:
- Config validation on startup with detailed error messages ✅ (already implemented)
- Hot-reload configuration without restart
- Config schema validation with JSON Schema
- Encrypted config file support
- Remote config loading (e.g., from Consul, etcd)
## Documentation Links
- [Configuration Guide](../docs/configuration.md) - Complete reference
- [Migration Guide](../docs/env-to-yaml-migration.md) - .env to YAML migration
- [Quick Start](../docs/quick-start.md) - Updated with YAML examples
- [README](../README.md) - High-level overview
## Issues Encountered and Resolved
### Encryption Key Length Validation
**Problem:** After initial implementation, the application failed to start with error:
```
Error: Validation error: Encryption key must be at least 32 characters
```
**Root Cause:** The placeholder encryption keys in `config.development.yaml` and `config.test.yaml` were exactly 31 characters, one character short of the required 32-character minimum.
**Solution:** Updated all configuration files to use encryption key placeholders that are at least 32 characters:
- `config.yaml`: `dev-encryption-key-at-least-32-characters-long-change-this` (58 chars)
- `config.example.yaml`: `dev-encryption-key-at-least-32-characters-long-change-this` (58 chars)
- `config.development.yaml`: `test-encryption-key-32-chars-okay` (33 chars)
- `config.test.yaml`: `test-encryption-key-32-chars-okay` (33 chars)
- `config.production.yaml`: `CHANGE_ME_USE_ENV_VAR_PLACEHOLDER_32_CHARS_MINIMUM` (50 chars)
**Verification:** Application now starts successfully with the message:
```
INFO Starting Attune API Service
INFO Configuration loaded successfully
INFO Database connection established
INFO Starting server on 127.0.0.1:8080
INFO Server listening on 127.0.0.1:8080
```
## Conclusion
The migration to YAML configuration provides a more maintainable, readable, and flexible configuration system while maintaining full backward compatibility with environment variable overrides. The comprehensive documentation ensures smooth adoption for all users.
All existing functionality has been preserved, and the new system is production-ready.