attune/work-summary/sessions/2025-01-30-schema-config-fix.md

Schema Configuration Fix for E2E Services

Date: 2025-01-30
Status: ✅ Complete
Branch: main

Summary

Fixed an issue where the start-e2e-services.sh script was ignoring the database schema configuration from custom config files and always defaulting to the attune schema.

Problem

When launching services with a custom config file:

CONFIG_FILE=config.development.yaml ./scripts/start-e2e-services.sh

The services would ignore the schema: "public" setting in config.development.yaml and always use the attune schema instead.

Root Cause

The configuration loading system has the following priority order:

1. Base config file (e.g., config.yaml from ATTUNE_CONFIG)
2. Environment-specific config file (e.g., config.{environment}.yaml)
3. Environment variables (highest priority)

The start-e2e-services.sh script was setting ATTUNE__ENVIRONMENT=e2e, which caused the config loader to look for and load config.e2e.yaml after loading the custom config file. Since environment-specific files have higher priority, any settings in config.e2e.yaml (including the default schema: "attune") would override the custom config.

Code Flow

// In crates/common/src/config.rs::Config::load()

// 1. Load base config (from ATTUNE_CONFIG)
builder = builder.add_source(config_crate::File::with_name(&config_path));

// 2. Load environment-specific config (PROBLEM: this overrides step 1)
let environment = std::env::var("ATTUNE__ENVIRONMENT").unwrap_or_else(...);
let env_config_path = format!("config.{}.yaml", environment);
builder = builder.add_source(config_crate::File::with_name(&env_config_path));

// 3. Load environment variables (highest priority)
builder = builder.add_source(config_crate::Environment::with_prefix("ATTUNE")...);

When ATTUNE__ENVIRONMENT=e2e was set, it loaded config.e2e.yaml which contained schema: "attune", overriding the schema: "public" from config.development.yaml.

Solution

Modified scripts/start-e2e-services.sh to only set ATTUNE__ENVIRONMENT=e2e when using the default E2E config file, allowing custom config files to determine their own environment:

# Only set ATTUNE__ENVIRONMENT if using default e2e config
# Otherwise, let the config file determine the environment
if [ "$CONFIG_FILE" = "config.e2e.yaml" ]; then
    ATTUNE__ENVIRONMENT=e2e ATTUNE_CONFIG="$CONFIG_FILE" ./target/debug/$binary_name > "$log_file" 2>&1 &
else
    ATTUNE_CONFIG="$CONFIG_FILE" ./target/debug/$binary_name > "$log_file" 2>&1 &
fi

Changes Made

Modified Files

scripts/start-e2e-services.sh:

1. Added schema detection from config file to display in startup banner
2. Modified service startup to conditionally set ATTUNE__ENVIRONMENT only for default E2E config
3. Added configuration info display showing config file and detected schema

Enhanced User Experience

Added informative banner showing configuration being used:

Configuration:
  • Config file: config.development.yaml
  • Database schema: public

This helps users verify the correct configuration is loaded before services start.

Verification

Before Fix

$ CONFIG_FILE=config.development.yaml ./scripts/start-e2e-services.sh
# Services would show:
INFO Using production schema: attune

After Fix

$ CONFIG_FILE=config.development.yaml ./scripts/start-e2e-services.sh

Configuration:
  • Config file: config.development.yaml
  • Database schema: public

# Services now show:
INFO Connecting to database with max_connections=50, schema=public

All services (api, executor, worker, sensor) correctly use the public schema.

Usage

Default E2E Environment

# Uses config.e2e.yaml with attune schema (default)
./scripts/start-e2e-services.sh

Development Environment

# Uses config.development.yaml with public schema
CONFIG_FILE=config.development.yaml ./scripts/start-e2e-services.sh

Custom Configuration

# Uses any custom config file with its specified schema
CONFIG_FILE=my-custom-config.yaml ./scripts/start-e2e-services.sh

Related Files

• scripts/start-e2e-services.sh - Service startup script (modified)
• crates/common/src/config.rs - Configuration loading logic (reference)
• crates/common/src/db.rs - Database connection with schema support (reference)
• config.development.yaml - Development config with schema: "public"
• config.e2e.yaml - E2E test config with default schema: "attune"

Configuration Priority Reference

For future reference, Attune's configuration loading priority is:

1. Base config file (lowest priority)

   • config.yaml or file specified by ATTUNE_CONFIG env var

2. Environment-specific config file

   • config.{ATTUNE__ENVIRONMENT}.yaml
   • Only loaded if ATTUNE__ENVIRONMENT is set or determined from base config

3. Environment variables (highest priority)

   • Prefix: ATTUNE__
   • Separator: __ (double underscore)
   • Example: ATTUNE__DATABASE__SCHEMA=public

Schema Configuration

The database schema can be configured via:

1. Config file: database.schema: "public"
2. Environment variable: ATTUNE__DATABASE__SCHEMA=public
3. Default if not specified: "attune"

Lessons Learned

1. Environment variable overrides can be subtle: Setting ATTUNE__ENVIRONMENT has cascading effects by triggering additional config file loads
2. Configuration priority matters: Higher priority sources always override lower priority ones
3. Scripts should respect user intent: When users provide custom config, don't override their choices
4. Visibility helps debugging: Showing the detected configuration at startup makes issues immediately visible

Testing

Tested with:

• ✅ Default E2E config (uses attune schema)
• ✅ Development config (uses public schema)
• ✅ All services (api, executor, worker, sensor) respect the config
• ✅ Schema detection and display in startup banner

Impact

Positive:

• Users can now easily switch between development and E2E environments
• Schema isolation works correctly for parallel testing
• Configuration behavior is more intuitive and predictable

No Breaking Changes:

• Default behavior unchanged (E2E config still uses attune schema)
• Existing scripts and workflows continue to work
• Only affects behavior when custom CONFIG_FILE is provided

Future Improvements

Consider adding:

1. Validation warning if config file doesn't exist
2. Display full resolved configuration on startup (with sensitive values masked)
3. Config validation command: attune config validate --file config.yaml
4. Environment variable to completely disable environment-specific config loading