re-uploading work
This commit is contained in:
196
work-summary/sessions/2025-01-30-schema-config-fix.md
Normal file
196
work-summary/sessions/2025-01-30-schema-config-fix.md
Normal file
@@ -0,0 +1,196 @@
|
||||
# Schema Configuration Fix for E2E Services
|
||||
|
||||
**Date**: 2025-01-30
|
||||
**Status**: ✅ Complete
|
||||
**Branch**: main
|
||||
|
||||
## Summary
|
||||
|
||||
Fixed an issue where the `start-e2e-services.sh` script was ignoring the database schema configuration from custom config files and always defaulting to the `attune` schema.
|
||||
|
||||
## Problem
|
||||
|
||||
When launching services with a custom config file:
|
||||
|
||||
```bash
|
||||
CONFIG_FILE=config.development.yaml ./scripts/start-e2e-services.sh
|
||||
```
|
||||
|
||||
The services would ignore the `schema: "public"` setting in `config.development.yaml` and always use the `attune` schema instead.
|
||||
|
||||
### Root Cause
|
||||
|
||||
The configuration loading system has the following priority order:
|
||||
|
||||
1. Base config file (e.g., `config.yaml` from `ATTUNE_CONFIG`)
|
||||
2. Environment-specific config file (e.g., `config.{environment}.yaml`)
|
||||
3. Environment variables (highest priority)
|
||||
|
||||
The `start-e2e-services.sh` script was setting `ATTUNE__ENVIRONMENT=e2e`, which caused the config loader to look for and load `config.e2e.yaml` **after** loading the custom config file. Since environment-specific files have higher priority, any settings in `config.e2e.yaml` (including the default `schema: "attune"`) would override the custom config.
|
||||
|
||||
### Code Flow
|
||||
|
||||
```rust
|
||||
// In crates/common/src/config.rs::Config::load()
|
||||
|
||||
// 1. Load base config (from ATTUNE_CONFIG)
|
||||
builder = builder.add_source(config_crate::File::with_name(&config_path));
|
||||
|
||||
// 2. Load environment-specific config (PROBLEM: this overrides step 1)
|
||||
let environment = std::env::var("ATTUNE__ENVIRONMENT").unwrap_or_else(...);
|
||||
let env_config_path = format!("config.{}.yaml", environment);
|
||||
builder = builder.add_source(config_crate::File::with_name(&env_config_path));
|
||||
|
||||
// 3. Load environment variables (highest priority)
|
||||
builder = builder.add_source(config_crate::Environment::with_prefix("ATTUNE")...);
|
||||
```
|
||||
|
||||
When `ATTUNE__ENVIRONMENT=e2e` was set, it loaded `config.e2e.yaml` which contained `schema: "attune"`, overriding the `schema: "public"` from `config.development.yaml`.
|
||||
|
||||
## Solution
|
||||
|
||||
Modified `scripts/start-e2e-services.sh` to only set `ATTUNE__ENVIRONMENT=e2e` when using the default E2E config file, allowing custom config files to determine their own environment:
|
||||
|
||||
```bash
|
||||
# Only set ATTUNE__ENVIRONMENT if using default e2e config
|
||||
# Otherwise, let the config file determine the environment
|
||||
if [ "$CONFIG_FILE" = "config.e2e.yaml" ]; then
|
||||
ATTUNE__ENVIRONMENT=e2e ATTUNE_CONFIG="$CONFIG_FILE" ./target/debug/$binary_name > "$log_file" 2>&1 &
|
||||
else
|
||||
ATTUNE_CONFIG="$CONFIG_FILE" ./target/debug/$binary_name > "$log_file" 2>&1 &
|
||||
fi
|
||||
```
|
||||
|
||||
## Changes Made
|
||||
|
||||
### Modified Files
|
||||
|
||||
**`scripts/start-e2e-services.sh`**:
|
||||
1. Added schema detection from config file to display in startup banner
|
||||
2. Modified service startup to conditionally set `ATTUNE__ENVIRONMENT` only for default E2E config
|
||||
3. Added configuration info display showing config file and detected schema
|
||||
|
||||
### Enhanced User Experience
|
||||
|
||||
Added informative banner showing configuration being used:
|
||||
|
||||
```
|
||||
Configuration:
|
||||
• Config file: config.development.yaml
|
||||
• Database schema: public
|
||||
```
|
||||
|
||||
This helps users verify the correct configuration is loaded before services start.
|
||||
|
||||
## Verification
|
||||
|
||||
### Before Fix
|
||||
|
||||
```bash
|
||||
$ CONFIG_FILE=config.development.yaml ./scripts/start-e2e-services.sh
|
||||
# Services would show:
|
||||
INFO Using production schema: attune
|
||||
```
|
||||
|
||||
### After Fix
|
||||
|
||||
```bash
|
||||
$ CONFIG_FILE=config.development.yaml ./scripts/start-e2e-services.sh
|
||||
|
||||
Configuration:
|
||||
• Config file: config.development.yaml
|
||||
• Database schema: public
|
||||
|
||||
# Services now show:
|
||||
WARN Using non-standard schema: 'public'. Production should use 'attune'
|
||||
INFO Connecting to database with max_connections=50, schema=public
|
||||
```
|
||||
|
||||
All services (api, executor, worker, sensor) correctly use the `public` schema.
|
||||
|
||||
## Usage
|
||||
|
||||
### Default E2E Environment
|
||||
```bash
|
||||
# Uses config.e2e.yaml with attune schema (default)
|
||||
./scripts/start-e2e-services.sh
|
||||
```
|
||||
|
||||
### Development Environment
|
||||
```bash
|
||||
# Uses config.development.yaml with public schema
|
||||
CONFIG_FILE=config.development.yaml ./scripts/start-e2e-services.sh
|
||||
```
|
||||
|
||||
### Custom Configuration
|
||||
```bash
|
||||
# Uses any custom config file with its specified schema
|
||||
CONFIG_FILE=my-custom-config.yaml ./scripts/start-e2e-services.sh
|
||||
```
|
||||
|
||||
## Related Files
|
||||
|
||||
- `scripts/start-e2e-services.sh` - Service startup script (modified)
|
||||
- `crates/common/src/config.rs` - Configuration loading logic (reference)
|
||||
- `crates/common/src/db.rs` - Database connection with schema support (reference)
|
||||
- `config.development.yaml` - Development config with `schema: "public"`
|
||||
- `config.e2e.yaml` - E2E test config with default `schema: "attune"`
|
||||
|
||||
## Configuration Priority Reference
|
||||
|
||||
For future reference, Attune's configuration loading priority is:
|
||||
|
||||
1. **Base config file** (lowest priority)
|
||||
- `config.yaml` or file specified by `ATTUNE_CONFIG` env var
|
||||
|
||||
2. **Environment-specific config file**
|
||||
- `config.{ATTUNE__ENVIRONMENT}.yaml`
|
||||
- Only loaded if `ATTUNE__ENVIRONMENT` is set or determined from base config
|
||||
|
||||
3. **Environment variables** (highest priority)
|
||||
- Prefix: `ATTUNE__`
|
||||
- Separator: `__` (double underscore)
|
||||
- Example: `ATTUNE__DATABASE__SCHEMA=public`
|
||||
|
||||
### Schema Configuration
|
||||
|
||||
The database schema can be configured via:
|
||||
|
||||
1. Config file: `database.schema: "public"`
|
||||
2. Environment variable: `ATTUNE__DATABASE__SCHEMA=public`
|
||||
3. Default if not specified: `"attune"`
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
1. **Environment variable overrides can be subtle**: Setting `ATTUNE__ENVIRONMENT` has cascading effects by triggering additional config file loads
|
||||
2. **Configuration priority matters**: Higher priority sources always override lower priority ones
|
||||
3. **Scripts should respect user intent**: When users provide custom config, don't override their choices
|
||||
4. **Visibility helps debugging**: Showing the detected configuration at startup makes issues immediately visible
|
||||
|
||||
## Testing
|
||||
|
||||
Tested with:
|
||||
- ✅ Default E2E config (uses `attune` schema)
|
||||
- ✅ Development config (uses `public` schema)
|
||||
- ✅ All services (api, executor, worker, sensor) respect the config
|
||||
- ✅ Schema detection and display in startup banner
|
||||
|
||||
## Impact
|
||||
|
||||
**Positive**:
|
||||
- Users can now easily switch between development and E2E environments
|
||||
- Schema isolation works correctly for parallel testing
|
||||
- Configuration behavior is more intuitive and predictable
|
||||
|
||||
**No Breaking Changes**:
|
||||
- Default behavior unchanged (E2E config still uses `attune` schema)
|
||||
- Existing scripts and workflows continue to work
|
||||
- Only affects behavior when custom `CONFIG_FILE` is provided
|
||||
|
||||
## Future Improvements
|
||||
|
||||
Consider adding:
|
||||
1. Validation warning if config file doesn't exist
|
||||
2. Display full resolved configuration on startup (with sensitive values masked)
|
||||
3. Config validation command: `attune config validate --file config.yaml`
|
||||
4. Environment variable to completely disable environment-specific config loading
|
||||
Reference in New Issue
Block a user