142 lines
5.6 KiB
Markdown
142 lines
5.6 KiB
Markdown
# Migration Consolidation - Executive Summary
|
|
|
|
**Date**: 2026-02-04
|
|
**Status**: Pre-production - Safe to consolidate
|
|
**Impact**: No production deployments exist
|
|
|
|
## Overview
|
|
|
|
The Attune project has accumulated 22 migrations during active development. Since there are no production deployments, we can safely consolidate these into a clean initial state, removing items that were created and then dropped or modified.
|
|
|
|
## Key Findings
|
|
|
|
### Items Created Then Dropped (Remove Entirely)
|
|
|
|
1. **`runtime_type_enum`** - Created in 00001, dropped in 20260203000001
|
|
- Associated column: `runtime.runtime_type`
|
|
- Associated indexes: 4 indexes referencing this column
|
|
- **Action**: Don't create at all
|
|
|
|
2. **`workflow_task_execution` table** - Created in 00004, dropped in 20260127212500
|
|
- Consolidated into `execution.workflow_task JSONB` column
|
|
- **Action**: Don't create table, add JSONB column to execution from start
|
|
|
|
3. **Individual webhook columns (10 columns)** - Added in 20260120000001/000002, dropped in 20260127000001
|
|
- `webhook_secret`, `webhook_hmac_enabled`, `webhook_hmac_secret`, etc.
|
|
- Consolidated into single `webhook_config JSONB`
|
|
- **Action**: Only create `webhook_enabled`, `webhook_key`, `webhook_config` from start
|
|
|
|
4. **Runtime INSERT statements** - Added in 20260202000001, truncated in 20260203000001
|
|
- Now loaded from YAML files in `packs/core/runtimes/`
|
|
- **Action**: Remove all runtime data from migrations
|
|
|
|
### Items Added Later (Include From Start)
|
|
|
|
1. **Execution table workflow columns**:
|
|
- `is_workflow BOOLEAN` (added later)
|
|
- `workflow_def BIGINT` (added later)
|
|
- `workflow_task JSONB` (added in consolidation migration)
|
|
|
|
2. **Is adhoc flags** (added in 20260129140130):
|
|
- `action.is_adhoc`
|
|
- `sensor.is_adhoc`
|
|
- `rule.is_adhoc`
|
|
|
|
3. **Event table rule tracking** (added in 20260130000001):
|
|
- `event.rule BIGINT`
|
|
- `event.rule_ref TEXT`
|
|
|
|
4. **Worker role** (added in 20260131000001):
|
|
- `worker_role_enum` type
|
|
- `worker.worker_role` column
|
|
|
|
5. **Pack environments** (added in 20260203000002):
|
|
- `pack_environment_status_enum` type
|
|
- `pack.installers JSONB` column
|
|
- `pack_environment` table
|
|
|
|
6. **LISTEN/NOTIFY triggers** (added across 4 migrations):
|
|
- Execution notify
|
|
- Event notify
|
|
- Enforcement notify
|
|
- Consolidated into single migration
|
|
|
|
### Constraints Modified
|
|
|
|
1. **`runtime_ref_format`** - Original: `^[^.]+\.(action|sensor)\.[^.]+$`
|
|
- Expected format was `pack.type.name` (e.g., `core.action.python`)
|
|
- Changed to allow `pack.name` format (e.g., `core.python`)
|
|
- **Action**: Drop constraint entirely or create with final format
|
|
|
|
2. **`runtime_ref_lowercase`** - Dropped and not recreated
|
|
- **Action**: Determine if needed in final schema
|
|
|
|
## Recommended Consolidation Structure
|
|
|
|
```
|
|
migrations/
|
|
├── 20250101000001_initial_setup.sql # Enums, extensions (FINAL VERSIONS)
|
|
├── 20250101000002_identity_and_auth.sql # Identity, keys
|
|
├── 20250101000003_pack_system.sql # Pack, runtime (no runtime_type)
|
|
├── 20250101000004_action_sensor.sql # Action, sensor (with is_adhoc)
|
|
├── 20250101000005_trigger_event_rule.sql # Trigger (with webhook_config), event (with rule), rule (with is_adhoc)
|
|
├── 20250101000006_execution_system.sql # Execution (with workflow cols), enforcement, inquiry, policy
|
|
├── 20250101000007_workflow_system.sql # Workflow_definition only (no workflow_task_execution)
|
|
├── 20250101000008_worker_notification.sql # Worker (with role), notification
|
|
├── 20250101000009_artifacts.sql # Artifact table
|
|
├── 20250101000010_webhook_system.sql # Webhook tables, FINAL functions
|
|
├── 20250101000011_pack_environments.sql # Pack_environment table and enum
|
|
├── 20250101000012_pack_testing.sql # Pack_test_results table
|
|
├── 20250101000013_notify_triggers.sql # ALL LISTEN/NOTIFY triggers
|
|
└── README.md # Migration documentation
|
|
```
|
|
|
|
## Benefits
|
|
|
|
1. **Cleaner git history** - Future developers see logical progression
|
|
2. **Faster test setup** - Fewer migrations to run (13 vs 22)
|
|
3. **No dead code** - Nothing created just to be dropped
|
|
4. **Accurate from start** - Tables created with final schema
|
|
5. **Better documentation** - Clear purpose for each migration
|
|
|
|
## Risks
|
|
|
|
**NONE** - No production deployments exist. This is the ideal time to consolidate.
|
|
|
|
## Data Considerations
|
|
|
|
**Runtime metadata** is now managed in YAML files:
|
|
- Location: `packs/core/runtimes/*.yaml`
|
|
- Loaded by: Pack installation system
|
|
- Files: `python.yaml`, `nodejs.yaml`, `shell.yaml`, `native.yaml`, `sensor_builtin.yaml`
|
|
|
|
**Core pack data** should be checked for any other SQL insertions that should move to YAML.
|
|
|
|
## Validation Plan
|
|
|
|
1. Create consolidated migrations in new directory
|
|
2. Test on fresh database: `createdb attune_test && sqlx migrate run`
|
|
3. Compare schema output: `pg_dump --schema-only` before/after
|
|
4. Verify table counts, column counts, constraint counts match
|
|
5. Load core pack and verify all data loads correctly
|
|
6. Run full test suite
|
|
7. If successful, replace old migrations
|
|
|
|
## Timeline Estimate
|
|
|
|
- **Analysis complete**: ✅ Done
|
|
- **Create consolidated migrations**: 2-3 hours
|
|
- **Testing and validation**: 1-2 hours
|
|
- **Documentation updates**: 30 minutes
|
|
- **Total**: ~4-6 hours
|
|
|
|
## Recommendation
|
|
|
|
**PROCEED** with consolidation. This is a textbook case for migration consolidation:
|
|
- Pre-production system ✅
|
|
- No user data ✅
|
|
- Clear improvement to codebase ✅
|
|
- Low risk ✅
|
|
- High maintainability gain ✅
|
|
|
|
The longer we wait, the harder this becomes. Do it now while it's straightforward. |