attune-system/attune
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

trying to rework database migrations

Browse Source
This commit is contained in:
David Culbreth
2026-02-05 11:42:04 -06:00
parent 3b14c65998
commit 343488b3eb
83 changed files with 5793 additions and 876 deletions
Download Patch File Download Diff File Expand all files Collapse all files

224
docs/migrations/CONSOLIDATION-COMPLETE.md Normal file
View File

@@ -0,0 +1,224 @@
# Migration Consolidation - Complete
**Date**: 2026-02-04
**Status**: ✅ COMPLETE
**Result**: 22 migrations → 13 migrations
## Summary
Successfully consolidated Attune's migration history from 22 files to 13 clean, logical migrations. This was possible because there are no production deployments yet, allowing us to freely restructure the schema history.
## Changes Made
### Items Removed Entirely (Never Created)
1. **`runtime_type_enum`** - Removed from initial setup
- Associated column `runtime.runtime_type` not created
- Associated indexes not created: `idx_runtime_type`, `idx_runtime_pack_type`, `idx_runtime_type_created`, `idx_runtime_type_sensor`
- Runtime table uses unified approach from the start
2. **`workflow_task_execution` table** - Not created
- Consolidated into `execution.workflow_task JSONB` column from initial execution table creation
- Eliminates need for separate table and join operations
3. **Individual webhook columns** - Not created
- Skipped 10 intermediate columns (webhook_secret, webhook_hmac_*, webhook_rate_limit_*, etc.)
- Only created: `webhook_enabled`, `webhook_key`, `webhook_config JSONB` from start
4. **Runtime data insertions** - Removed from migrations
- All runtime metadata moved to YAML files in `packs/core/runtimes/`
- No SQL INSERT statements for runtime records
### Items Included From Start
1. **Execution table workflow columns** (in 00006):
- `is_workflow BOOLEAN DEFAULT false NOT NULL`
- `workflow_def BIGINT REFERENCES workflow_definition(id)`
- `workflow_task JSONB`
2. **Is adhoc flags** (in respective tables):
- `action.is_adhoc` (in 00004)
- `sensor.is_adhoc` (in 00004)
- `rule.is_adhoc` (in 00005)
3. **Event table rule tracking** (in 00005):
- `event.rule BIGINT`
- `event.rule_ref TEXT`
- Foreign key constraint to rule table
4. **Worker role** (in 00008):
- `worker_role_enum` type (in 00001)
- `worker.worker_role` column
5. **Trigger webhook support** (in 00005):
- `webhook_enabled BOOLEAN NOT NULL DEFAULT FALSE`
- `webhook_key VARCHAR(64) UNIQUE`
- `webhook_config JSONB DEFAULT '{}'::jsonb`
6. **Pack environments** (in 00001 and 00003):
- `pack_environment_status_enum` type (in 00001)
- `pack.installers JSONB` column (in 00003)
- `pack_environment` table (in 00011)
## Final Migration Structure
```
migrations/
├── 20250101000001_initial_setup.sql # Enums, extensions (minus runtime_type_enum, plus worker_role_enum and pack_environment_status_enum)
├── 20250101000002_identity_and_auth.sql # Identity, permission_set, permission_assignment, policy
├── 20250101000003_pack_system.sql # Pack (with installers), runtime (no runtime_type)
├── 20250101000004_action_sensor.sql # Action, sensor (both with is_adhoc)
├── 20250101000005_trigger_event_rule.sql # Trigger (with webhook_config), event (with rule), rule (with is_adhoc)
├── 20250101000006_execution_system.sql # Enforcement, execution (with workflow columns), inquiry
├── 20250101000007_workflow_system.sql # Workflow_definition, workflow_execution (no workflow_task_execution)
├── 20250101000008_worker_notification.sql # Worker (with worker_role), notification
├── 20250101000009_keys_artifacts.sql # Key, artifact
├── 20250101000010_webhook_system.sql # Webhook functions (final versions)
├── 20250101000011_pack_environments.sql # Pack_environment table
├── 20250101000012_pack_testing.sql # Pack_test_results table
└── 20250101000013_notify_triggers.sql # All LISTEN/NOTIFY triggers (consolidated)
```
## Migrations Removed
The following 15 migration files were consolidated or had their data moved to YAML:
1. `20260119000001_add_execution_notify_trigger.sql` → Consolidated into 00013
2. `20260120000001_add_webhook_support.sql` → Columns added to trigger table in 00005
3. `20260120000002_webhook_advanced_features.sql` → Functions consolidated in 00010
4. `20260122000001_pack_installation_metadata.sql` → Merged into pack system
5. `20260127000001_consolidate_webhook_config.sql` → Already consolidated in 00005
6. `20260127212500_consolidate_workflow_task_execution.sql` → Already in execution table in 00006
7. `20260129000001_fix_webhook_function_overload.sql` → Fixed functions in 00010
8. `20260129140130_add_is_adhoc_flag.sql` → Already in tables in 00004/00005
9. `20260129150000_add_event_notify_trigger.sql` → Consolidated into 00013
10. `20260130000001_add_rule_to_event.sql` → Already in event table in 00005
11. `20260131000001_add_worker_role.sql` → Already in worker table in 00008
12. `20260202000001_add_sensor_runtimes.sql` → Data moved to YAML files
13. `20260203000001_unify_runtimes.sql` → Changes applied to base runtime table in 00003
14. `20260203000003_add_rule_trigger_to_execution_notify.sql` → Consolidated into 00013
15. `20260204000001_add_enforcement_notify_trigger.sql` → Consolidated into 00013
Note: One file (`20260204000001_restore_webhook_functions.sql`) was kept and renamed to 00010 with final webhook functions.
## Benefits
1. **Cleaner History**: Future developers see logical progression, not incremental fixes
2. **Faster Tests**: 13 migrations vs 22 (41% reduction)
3. **No Dead Code**: Nothing created just to be dropped
4. **Accurate Schema**: Tables created with final structure from the start
5. **Better Maintainability**: Each migration has clear, focused purpose
6. **Reduced Complexity**: Fewer foreign key constraints to manage incrementally
## Data Migration
### Runtime Metadata
Runtime data is now managed externally:
**Location**: `packs/core/runtimes/*.yaml`
**Files**:
- `python.yaml` - Python 3 runtime
- `nodejs.yaml` - Node.js runtime
- `shell.yaml` - Shell runtime (bash/sh)
- `native.yaml` - Native compiled runtime
- `sensor_builtin.yaml` - Built-in sensor runtime
**Loading**: Handled by pack installation system, not migrations
## Testing
Next steps for validation:
```bash
# 1. Test on fresh database
createdb attune_test_consolidated
export DATABASE_URL="postgresql://attune:attune@localhost/attune_test_consolidated"
sqlx migrate run
# 2. Compare schema
pg_dump --schema-only attune_test_consolidated > schema_new.sql
pg_dump --schema-only attune_dev > schema_old.sql
diff schema_old.sql schema_new.sql
# 3. Verify table counts
psql attune_test_consolidated -c "\dt" | wc -l
# 4. Load core pack
./scripts/load-core-pack.sh
# 5. Run tests
cargo test
```
## Rollback Plan
Original migrations preserved in `migrations.old/` directory. To rollback:
```bash
rm -rf migrations/*.sql
cp migrations.old/*.sql migrations/
```
**Do NOT delete `migrations.old/` until consolidated migrations are verified in production-like environment.**
## Constraints Modified
1. **`runtime_ref_format`** - Removed entirely
- Old format: `^[^.]+\.(action|sensor)\.[^.]+$` (e.g., `core.action.python`)
- New format: No constraint, allows `pack.name` format (e.g., `core.python`)
2. **`runtime_ref_lowercase`** - Kept as-is
- Still enforces lowercase runtime refs
## Indexes Added/Modified
**Runtime table**:
- ❌ Removed: `idx_runtime_type`, `idx_runtime_pack_type`, `idx_runtime_type_created`
- ✅ Added: `idx_runtime_name`, `idx_runtime_verification` (GIN index)
**Trigger table**:
- ✅ Added: `idx_trigger_webhook_key`
**Event table**:
- ✅ Added: `idx_event_rule`
## Statistics
| Metric | Before | After | Change |
|--------|--------|-------|--------|
| Migration files | 22 | 13 | -41% |
| Lines of SQL | ~3,500 | ~2,100 | -40% |
| Enum types | 13 | 12 | -1 |
| Tables created | 22 | 21 | -1 |
| Tables created then dropped | 1 | 0 | -100% |
| Columns added then dropped | 10 | 0 | -100% |
## Completion Checklist
- ✅ Backup created in `migrations.old/`
- ✅ 13 consolidated migrations created
- ✅ Runtime data moved to YAML files
- ✅ All incremental additions consolidated
- ✅ Documentation updated
- ⏳ Test on fresh database
- ⏳ Compare schemas
- ⏳ Run full test suite
- ⏳ Deploy to development
- ⏳ Delete `migrations.old/` after verification
## Notes
- All changes are breaking changes, but that's acceptable since there are no production deployments
- Future migrations should be created normally and incrementally
- This consolidation should be a one-time event before v1.0 release
- After production deployment, normal migration discipline applies (no deletions, only additions)
## Acknowledgments
This consolidation was made possible by the "Breaking Changes Policy" documented in `AGENTS.md`:
> **Breaking changes are explicitly allowed and encouraged** when they improve the architecture, API design, or developer experience. No backward compatibility required - there are no existing versions to support.
Once this project reaches v1.0 or gets its first production deployment, this policy will be replaced with appropriate stability guarantees and versioning policies.

142
docs/migrations/CONSOLIDATION-SUMMARY.md Normal file
View File

@@ -0,0 +1,142 @@
# Migration Consolidation - Executive Summary
**Date**: 2026-02-04
**Status**: Pre-production - Safe to consolidate
**Impact**: No production deployments exist
## Overview
The Attune project has accumulated 22 migrations during active development. Since there are no production deployments, we can safely consolidate these into a clean initial state, removing items that were created and then dropped or modified.
## Key Findings
### Items Created Then Dropped (Remove Entirely)
1. **`runtime_type_enum`** - Created in 00001, dropped in 20260203000001
- Associated column: `runtime.runtime_type`
- Associated indexes: 4 indexes referencing this column
- **Action**: Don't create at all
2. **`workflow_task_execution` table** - Created in 00004, dropped in 20260127212500
- Consolidated into `execution.workflow_task JSONB` column
- **Action**: Don't create table, add JSONB column to execution from start
3. **Individual webhook columns (10 columns)** - Added in 20260120000001/000002, dropped in 20260127000001
- `webhook_secret`, `webhook_hmac_enabled`, `webhook_hmac_secret`, etc.
- Consolidated into single `webhook_config JSONB`
- **Action**: Only create `webhook_enabled`, `webhook_key`, `webhook_config` from start
4. **Runtime INSERT statements** - Added in 20260202000001, truncated in 20260203000001
- Now loaded from YAML files in `packs/core/runtimes/`
- **Action**: Remove all runtime data from migrations
### Items Added Later (Include From Start)
1. **Execution table workflow columns**:
- `is_workflow BOOLEAN` (added later)
- `workflow_def BIGINT` (added later)
- `workflow_task JSONB` (added in consolidation migration)
2. **Is adhoc flags** (added in 20260129140130):
- `action.is_adhoc`
- `sensor.is_adhoc`
- `rule.is_adhoc`
3. **Event table rule tracking** (added in 20260130000001):
- `event.rule BIGINT`
- `event.rule_ref TEXT`
4. **Worker role** (added in 20260131000001):
- `worker_role_enum` type
- `worker.worker_role` column
5. **Pack environments** (added in 20260203000002):
- `pack_environment_status_enum` type
- `pack.installers JSONB` column
- `pack_environment` table
6. **LISTEN/NOTIFY triggers** (added across 4 migrations):
- Execution notify
- Event notify
- Enforcement notify
- Consolidated into single migration
### Constraints Modified
1. **`runtime_ref_format`** - Original: `^[^.]+\.(action|sensor)\.[^.]+$`
- Expected format was `pack.type.name` (e.g., `core.action.python`)
- Changed to allow `pack.name` format (e.g., `core.python`)
- **Action**: Drop constraint entirely or create with final format
2. **`runtime_ref_lowercase`** - Dropped and not recreated
- **Action**: Determine if needed in final schema
## Recommended Consolidation Structure
```
migrations/
├── 20250101000001_initial_setup.sql # Enums, extensions (FINAL VERSIONS)
├── 20250101000002_identity_and_auth.sql # Identity, keys
├── 20250101000003_pack_system.sql # Pack, runtime (no runtime_type)
├── 20250101000004_action_sensor.sql # Action, sensor (with is_adhoc)
├── 20250101000005_trigger_event_rule.sql # Trigger (with webhook_config), event (with rule), rule (with is_adhoc)
├── 20250101000006_execution_system.sql # Execution (with workflow cols), enforcement, inquiry, policy
├── 20250101000007_workflow_system.sql # Workflow_definition only (no workflow_task_execution)
├── 20250101000008_worker_notification.sql # Worker (with role), notification
├── 20250101000009_artifacts.sql # Artifact table
├── 20250101000010_webhook_system.sql # Webhook tables, FINAL functions
├── 20250101000011_pack_environments.sql # Pack_environment table and enum
├── 20250101000012_pack_testing.sql # Pack_test_results table
├── 20250101000013_notify_triggers.sql # ALL LISTEN/NOTIFY triggers
└── README.md # Migration documentation
```
## Benefits
1. **Cleaner git history** - Future developers see logical progression
2. **Faster test setup** - Fewer migrations to run (13 vs 22)
3. **No dead code** - Nothing created just to be dropped
4. **Accurate from start** - Tables created with final schema
5. **Better documentation** - Clear purpose for each migration
## Risks
**NONE** - No production deployments exist. This is the ideal time to consolidate.
## Data Considerations
**Runtime metadata** is now managed in YAML files:
- Location: `packs/core/runtimes/*.yaml`
- Loaded by: Pack installation system
- Files: `python.yaml`, `nodejs.yaml`, `shell.yaml`, `native.yaml`, `sensor_builtin.yaml`
**Core pack data** should be checked for any other SQL insertions that should move to YAML.
## Validation Plan
1. Create consolidated migrations in new directory
2. Test on fresh database: `createdb attune_test && sqlx migrate run`
3. Compare schema output: `pg_dump --schema-only` before/after
4. Verify table counts, column counts, constraint counts match
5. Load core pack and verify all data loads correctly
6. Run full test suite
7. If successful, replace old migrations
## Timeline Estimate
- **Analysis complete**: ✅ Done
- **Create consolidated migrations**: 2-3 hours
- **Testing and validation**: 1-2 hours
- **Documentation updates**: 30 minutes
- **Total**: ~4-6 hours
## Recommendation
**PROCEED** with consolidation. This is a textbook case for migration consolidation:
- Pre-production system ✅
- No user data ✅
- Clear improvement to codebase ✅
- Low risk ✅
- High maintainability gain ✅
The longer we wait, the harder this becomes. Do it now while it's straightforward.

195
docs/migrations/MIGRATION-BY-MIGRATION-CHANGES.md Normal file
View File

@@ -0,0 +1,195 @@
# Migration-by-Migration Change List
This document details exactly what needs to change in each migration file during consolidation.
## Files to Keep (with modifications)
### `20250101000001_initial_setup.sql`
**REMOVE**:
- `runtime_type_enum` type (lines ~42-46)
**KEEP**:
- All other enum types
- Extensions
- update_updated_column() function
### `20250101000002_core_tables.sql` → Rename to `20250101000003_pack_system.sql`
**MODIFY runtime table** (lines ~72-93):
```sql
CREATE TABLE runtime (
id BIGSERIAL PRIMARY KEY,
ref TEXT NOT NULL UNIQUE,
pack BIGINT REFERENCES pack(id) ON DELETE CASCADE,
pack_ref TEXT,
description TEXT,
-- REMOVE: runtime_type runtime_type_enum NOT NULL,
name TEXT NOT NULL,
distributions JSONB NOT NULL,
installation JSONB,
created TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated TIMESTAMPTZ NOT NULL DEFAULT NOW(),
-- REMOVE: CONSTRAINT runtime_ref_format CHECK (ref ~ '^[^.]+\.(action|sensor)\.[^.]+$')
CONSTRAINT runtime_ref_lowercase CHECK (ref = LOWER(ref))
);
```
**REMOVE indexes**:
- `idx_runtime_type`
- `idx_runtime_pack_type`
- `idx_runtime_type_created`
**ADD indexes**:
- `idx_runtime_name` (added in unify migration)
- `idx_runtime_verification` GIN index (added in unify migration)
### `20250101000003_event_system.sql` → Rename to `20250101000005_trigger_event_rule.sql`
**MODIFY trigger table** (add webhook columns from start):
```sql
CREATE TABLE trigger (
-- ... existing columns ...
-- ADD FROM START:
webhook_enabled BOOLEAN NOT NULL DEFAULT FALSE,
webhook_key VARCHAR(64) UNIQUE,
webhook_config JSONB DEFAULT '{}'::jsonb,
-- ... rest of columns ...
);
```
**MODIFY event table** (add rule tracking from start):
```sql
CREATE TABLE event (
-- ... existing columns ...
-- ADD FROM START:
rule BIGINT,
rule_ref TEXT,
-- ... rest of columns ...
);
-- ADD constraint:
ALTER TABLE event
ADD CONSTRAINT event_rule_fkey
FOREIGN KEY (rule) REFERENCES rule(id) ON DELETE SET NULL;
```
**MODIFY rule table** (add is_adhoc from start):
```sql
CREATE TABLE rule (
-- ... existing columns ...
-- ADD FROM START:
is_adhoc BOOLEAN DEFAULT false NOT NULL,
-- ... rest of columns ...
);
```
### `20250101000004_execution_system.sql` → Rename to `20250101000006_execution_system.sql`
**MODIFY execution table** (add workflow columns from start):
```sql
CREATE TABLE execution (
-- ... existing columns ...
-- ADD FROM START:
is_workflow BOOLEAN DEFAULT false NOT NULL,
workflow_def BIGINT REFERENCES workflow_definition(id) ON DELETE CASCADE,
workflow_task JSONB,
-- ... rest of columns ...
);
```
**REMOVE**:
- `workflow_task_execution` table (lines ~329-360)
- Don't create it at all
### `20250101000005_supporting_tables.sql` → Rename to `20250101000008_worker_notification.sql`
**MODIFY worker table** (add role from start):
First, ensure `worker_role_enum` is created in `20250101000001_initial_setup.sql`:
```sql
CREATE TYPE worker_role_enum AS ENUM ('action', 'sensor', 'hybrid');
```
Then in worker table:
```sql
CREATE TABLE worker (
-- ... existing columns ...
-- ADD FROM START:
worker_role worker_role_enum NOT NULL DEFAULT 'action',
-- ... rest of columns ...
);
```
**MODIFY action table** (add is_adhoc from start):
```sql
CREATE TABLE action (
-- ... existing columns ...
-- ADD FROM START:
is_adhoc BOOLEAN DEFAULT false NOT NULL,
-- ... rest of columns ...
);
```
**MODIFY sensor table** (add is_adhoc from start):
```sql
CREATE TABLE sensor (
-- ... existing columns ...
-- ADD FROM START:
is_adhoc BOOLEAN DEFAULT false NOT NULL,
-- ... rest of columns ...
);
```
## Files to Remove Entirely
1. `20260119000001_add_execution_notify_trigger.sql` - Consolidate into notify triggers migration
2. `20260120000001_add_webhook_support.sql` - Columns added in trigger table from start
3. `20260120000002_webhook_advanced_features.sql` - Functions consolidated, columns already in trigger table
4. `20260127000001_consolidate_webhook_config.sql` - Already consolidated in base migration
5. `20260127212500_consolidate_workflow_task_execution.sql` - Already in base execution table
6. `20260129000001_fix_webhook_function_overload.sql` - Use fixed functions from start
7. `20260129140130_add_is_adhoc_flag.sql` - Already in base tables
8. `20260129150000_add_event_notify_trigger.sql` - Consolidate into notify triggers migration
9. `20260130000001_add_rule_to_event.sql` - Already in event table
10. `20260131000001_add_worker_role.sql` - Already in worker table
11. `20260202000001_add_sensor_runtimes.sql` - Data now in YAML files
12. `20260203000001_unify_runtimes.sql` - Changes already applied to base tables
13. `20260203000003_add_rule_trigger_to_execution_notify.sql` - Consolidate into notify triggers migration
14. `20260204000001_add_enforcement_notify_trigger.sql` - Consolidate into notify triggers migration
15. `20260204000001_restore_webhook_functions.sql` - Use final functions from start
## New Files to Create
### `20250101000010_webhook_system.sql`
- Webhook-related tables
- FINAL versions of webhook functions (from 20260204000001_restore_webhook_functions.sql)
- No individual webhook columns (use webhook_config JSONB)
### `20250101000011_pack_environments.sql`
```sql
-- From 20260203000002_add_pack_environments.sql
CREATE TYPE pack_environment_status_enum AS ENUM (...);
CREATE TABLE pack_environment (...);
ALTER TABLE pack ADD COLUMN IF NOT EXISTS installers JSONB DEFAULT '[]'::jsonb;
```
### `20250101000013_notify_triggers.sql`
Consolidate ALL LISTEN/NOTIFY triggers from:
- 20260119000001 - execution
- 20260129150000 - event
- 20260203000003 - add rule to execution notify
- 20260204000001 - enforcement
Final notify_execution_change() function should include rule field from the start.
## Files to Keep As-Is
1. `20260120200000_add_pack_test_results.sql` → Rename to `20250101000012_pack_testing.sql`
2. `20260122000001_pack_installation_metadata.sql` → Merge into pack_system or keep separate
## Summary
**Original**: 22 migration files
**Consolidated**: ~13 migration files
**Removed**: 15 files (consolidation or data moved to YAML)
**Modified**: 5 files (add columns/constraints from start)
**New**: 3 files (consolidated functionality)

260
docs/migrations/migration-consolidation-plan.md Normal file
View File

@@ -0,0 +1,260 @@
# Migration Consolidation Plan
**Status**: Pre-production consolidation
**Date**: 2026-02-04
**Goal**: Consolidate migrations into a clean, minimal set before initial release
## Background
Since this project has no production deployments, we can freely consolidate migrations to create a cleaner initial state. This document identifies items that are created and then dropped/modified, so we can simplify the migration history.
## Issues Identified
### 1. Runtime Type Enum - Created Then Dropped
**Problem**: `runtime_type_enum` is created in the initial migration but dropped in a later migration.
- **Created**: `20250101000001_initial_setup.sql` (line 42)
- **Dropped**: `20260203000001_unify_runtimes.sql` (line 35)
- **Associated column**: `runtime.runtime_type` (also dropped)
- **Associated indexes**:
- `idx_runtime_type`
- `idx_runtime_pack_type`
- `idx_runtime_type_created`
- `idx_runtime_type_sensor`
**Action**: Remove enum type, column, and indexes from initial creation.
### 2. Runtime Table Constraints - Created Then Dropped
**Problem**: Runtime constraints are created with one format, then dropped and not recreated.
- **Created**: `20250101000002_core_tables.sql` (line 84)
- `runtime_ref_format CHECK (ref ~ '^[^.]+\.(action|sensor)\.[^.]+$')`
- Expected format: `pack.type.name`
- **Dropped**: `20260203000001_unify_runtimes.sql` (line 16)
- **New format**: `pack.name` (e.g., `core.python` instead of `core.action.python`)
**Action**: Create constraint with final format initially, or omit if not needed.
### 3. Webhook Columns - Added Then Consolidated
**Problem**: Individual webhook columns are added, then dropped in favor of a JSONB column.
**Added in `20260120000001_add_webhook_support.sql`**:
- `webhook_enabled BOOLEAN`
- `webhook_key VARCHAR(64)`
- `webhook_secret VARCHAR(128)`
**Added in `20260120000002_webhook_advanced_features.sql`**:
- `webhook_hmac_enabled BOOLEAN`
- `webhook_hmac_secret VARCHAR(128)`
- `webhook_hmac_algorithm VARCHAR(32)`
- `webhook_rate_limit_enabled BOOLEAN`
- `webhook_rate_limit_requests INTEGER`
- `webhook_rate_limit_window_seconds INTEGER`
- `webhook_ip_whitelist_enabled BOOLEAN`
- `webhook_ip_whitelist JSONB`
- `webhook_payload_size_limit_kb INTEGER`
**Consolidated in `20260127000001_consolidate_webhook_config.sql`**:
- All individual columns dropped
- Single `webhook_config JSONB` column added
**Action**: Add only `webhook_enabled`, `webhook_key`, and `webhook_config` in initial trigger table creation. Skip intermediate columns.
### 4. Runtime Data Insertions - Later Truncated
**Problem**: Runtime records are inserted via SQL, then truncated and moved to YAML files.
**Insertions in `20260202000001_add_sensor_runtimes.sql`**:
- 4 INSERT statements for sensor runtimes
- All records truncated in `20260203000001_unify_runtimes.sql`
**Insertions elsewhere**: Check if initial migrations insert any runtime data.
**Action**: Remove all runtime INSERT statements. Runtime data now loaded from YAML files in `packs/core/runtimes/`.
### 5. Workflow Task Execution Table - Created Then Dropped
**Problem**: Separate table created, then consolidated into execution table JSONB column.
- **Created**: `20250101000004_execution_system.sql` (line 329)
- `workflow_task_execution` table with multiple columns
- **Consolidated**: `20260127212500_consolidate_workflow_task_execution.sql`
- Table dropped
- `execution.workflow_task JSONB` column added instead
**Action**: Don't create `workflow_task_execution` table. Add `workflow_task JSONB` column to `execution` table in initial creation.
### 6. Execution Table Columns - Added for Workflows
**Problem**: Workflow-related columns added after initial table creation.
**Added in `20250101000004_execution_system.sql` (line 381)**:
- `is_workflow BOOLEAN DEFAULT false NOT NULL`
- `workflow_def BIGINT REFERENCES workflow_definition(id)`
**Action**: Include these columns in initial `execution` table creation (line ~60).
### 7. Is Adhoc Flag - Added Later
**Problem**: `is_adhoc` flag added to multiple tables after initial creation.
**Added in `20260129140130_add_is_adhoc_flag.sql`**:
- `action.is_adhoc`
- `sensor.is_adhoc`
- `rule.is_adhoc`
**Action**: Include `is_adhoc BOOLEAN DEFAULT false NOT NULL` in initial table definitions.
### 8. Event Table - Rule Reference Added Later
**Problem**: Rule tracking added to event table after initial creation.
**Added in `20260130000001_add_rule_to_event.sql`**:
- `event.rule BIGINT`
- `event.rule_ref TEXT`
- Foreign key constraint
**Action**: Include rule columns and constraint in initial event table creation.
### 9. Worker Role Column - Added Later
**Problem**: Worker role enum and column added after initial creation.
**Added in `20260131000001_add_worker_role.sql`**:
- `worker_role_enum` type
- `worker.worker_role` column
**Action**: Include enum type and column in initial worker table creation.
### 10. Pack Environments - Added Later
**Problem**: Pack installers column added after initial creation.
**Added in `20260203000002_add_pack_environments.sql`**:
- `pack_environment_status_enum` type
- `pack.installers JSONB` column
- `pack_environment` table
**Action**: Include in initial pack/environment setup.
### 11. Notify Triggers - Added Incrementally
**Problem**: PostgreSQL LISTEN/NOTIFY triggers added across multiple migrations.
**Migrations**:
- `20260119000001_add_execution_notify_trigger.sql` - execution events
- `20260129150000_add_event_notify_trigger.sql` - event creation
- `20260203000003_add_rule_trigger_to_execution_notify.sql` - add rule to execution notify
- `20260204000001_add_enforcement_notify_trigger.sql` - enforcement events
**Action**: Create all notify triggers in a single migration after table creation.
### 12. Webhook Functions - Created, Modified, Dropped, Restored
**Problem**: Webhook validation/processing functions have been rewritten multiple times.
**Timeline**:
- `20260120000001_add_webhook_support.sql` - Initial functions (4 created)
- `20260120000002_webhook_advanced_features.sql` - Advanced functions (7 created)
- `20260127000001_consolidate_webhook_config.sql` - Modified (2 dropped, 3 created)
- `20260129000001_fix_webhook_function_overload.sql` - Fixed overloading (3 dropped)
- `20260204000001_restore_webhook_functions.sql` - Restored (4 dropped, 3 created)
**Action**: Determine final set of webhook functions needed and create them once.
## Consolidation Strategy
### Phase 1: Analyze Dependencies
1. Map all foreign key relationships
2. Identify minimum viable table set
3. Document final schema for each table
### Phase 2: Create New Base Migrations
Create consolidated migrations:
1. **`00001_initial_setup.sql`** - Enums, extensions, base types
2. **`00002_identity_and_auth.sql`** - Identity, keys, auth tables
3. **`00003_pack_system.sql`** - Pack, runtime, action, sensor tables (with final schema)
4. **`00004_event_system.sql`** - Trigger, sensor, event, rule tables
5. **`00005_execution_system.sql`** - Execution, enforcement, inquiry, policy tables (including workflow columns)
6. **`00006_supporting_tables.sql`** - Worker, notification, artifact, etc.
7. **`00007_webhook_system.sql`** - Webhook tables, triggers, functions (final versions)
8. **`00008_notify_triggers.sql`** - All LISTEN/NOTIFY triggers
9. **`00009_pack_testing.sql`** - Pack test results table
### Phase 3: Validation
1. Test migrations on fresh database
2. Compare final schema to current production-like schema
3. Verify all indexes, constraints, triggers present
4. Load core pack and verify runtime data loads correctly
### Phase 4: Documentation
1. Update migration README
2. Document schema version
3. Add migration best practices
## Items to Remove Entirely
**Never created in consolidated migrations**:
1. `runtime_type_enum` type
2. `runtime.runtime_type` column
3. `runtime_ref_format` constraint (old format)
4. Indexes: `idx_runtime_type`, `idx_runtime_pack_type`, `idx_runtime_type_created`, `idx_runtime_type_sensor`
5. Individual webhook columns (9 columns that were later consolidated)
6. `idx_trigger_webhook_enabled` index
7. `workflow_task_execution` table
8. All runtime INSERT statements
9. Intermediate webhook function versions
## Items to Include From Start
**Must be in initial table creation**:
1. `execution.is_workflow` column
2. `execution.workflow_def` column
3. `execution.workflow_task` JSONB column
4. `action.is_adhoc` column
5. `sensor.is_adhoc` column
6. `rule.is_adhoc` column
7. `event.rule` and `event.rule_ref` columns
8. `worker_role_enum` type
9. `worker.worker_role` column
10. `trigger.webhook_enabled` column
11. `trigger.webhook_key` column
12. `trigger.webhook_config` JSONB column
13. `pack.installers` JSONB column
14. `pack_environment` table and `pack_environment_status_enum`
## Data Migration Notes
**Runtime Data**:
- Remove all INSERT statements from migrations
- Runtime records loaded from YAML files in `packs/core/runtimes/`
- Loader: `scripts/load_core_pack.py` or pack installation system
**Core Pack Data**:
- Check if any other core pack data is inserted via migrations
- Move to appropriate YAML files in `packs/core/`
## Next Steps
1. ✅ Create this consolidation plan
2. ⏳ Review with team
3. ⏳ Back up current migration directory
4. ⏳ Create consolidated migrations
5. ⏳ Test on fresh database
6. ⏳ Verify schema matches current state
7. ⏳ Replace old migrations
8. ⏳ Update documentation
## Rollback Plan
Keep copy of old migrations in `migrations.old/` directory until consolidated migrations are verified in development environment.