217 lines
7.2 KiB
Markdown
217 lines
7.2 KiB
Markdown
# Migration Consolidation - January 16, 2025
|
|
|
|
## Summary
|
|
|
|
Consolidated 18 separate database migration files into 5 logically organized migrations for better maintainability and clarity in this early-stage project.
|
|
|
|
## Changes Made
|
|
|
|
### Migration Structure
|
|
|
|
**Before:** 18 migration files
|
|
- 12 initial table creation files (20240101 series)
|
|
- 6 patch/fix files (20240102-20240103 series)
|
|
|
|
**After:** 5 consolidated migration files
|
|
1. `20250101000001_initial_setup.sql` - Schema, enums, shared functions
|
|
2. `20250101000002_core_tables.sql` - Pack, runtime, worker, identity, permissions, policy, key (7 tables)
|
|
3. `20250101000003_event_system.sql` - Trigger, sensor, event, enforcement (4 tables)
|
|
4. `20250101000004_execution_system.sql` - Action, rule, execution, inquiry (4 tables)
|
|
5. `20250101000005_supporting_tables.sql` - Notification, artifact (2 tables)
|
|
|
|
### What Was Consolidated
|
|
|
|
Each new migration incorporates all patches and fixes:
|
|
|
|
1. **Initial Setup** - Combined:
|
|
- Schema creation
|
|
- All 12 enum type definitions
|
|
- Shared `update_updated_column()` function
|
|
|
|
2. **Core Tables** - Combined:
|
|
- Pack table with all indexes (including performance indexes)
|
|
- Runtime and worker tables
|
|
- Identity table with `password_hash` column (from patch 20240102000001)
|
|
- Permission_set, permission_assignment, policy tables
|
|
- Key table with ownership validation
|
|
|
|
3. **Event System** - Combined:
|
|
- Trigger table
|
|
- Sensor table with `config` column (from patch 20240103000001) and CASCADE foreign keys (from patch 20240102000002)
|
|
- Event table
|
|
- Enforcement table (with forward reference to rule)
|
|
|
|
4. **Execution System** - Combined:
|
|
- Action table
|
|
- Rule table with `action_params` and `trigger_params` columns (from patches 20240103000003 and 20240103000004)
|
|
- Execution table with proper identity foreign key
|
|
- Inquiry table with proper identity foreign key
|
|
- All foreign key constraints resolved (no forward references)
|
|
|
|
5. **Supporting Tables** - Combined:
|
|
- Notification table with pg_notify trigger
|
|
- Artifact table
|
|
- All performance optimization indexes from 20240101000012
|
|
|
|
### Forward Reference Handling
|
|
|
|
The old migrations had circular dependencies between tables. The new structure properly handles this:
|
|
|
|
- **Migration 2** creates policy and key tables with forward references to action/sensor (without FK constraints)
|
|
- **Migration 3** adds sensor foreign key to key table
|
|
- **Migration 4** adds action foreign key to policy and key tables, plus rule foreign key to enforcement table
|
|
|
|
### Old Migrations
|
|
|
|
All 18 old migration files were moved to `migrations/old_migrations_backup/` for reference and can be safely deleted after verification.
|
|
|
|
## Benefits
|
|
|
|
1. **Easier to understand** - 5 files vs 18 files, clear logical grouping
|
|
2. **Cleaner history** - All patches incorporated into base migrations
|
|
3. **Better documentation** - Each file has clear sections with comments
|
|
4. **Reduced complexity** - No need to track patch dependencies
|
|
5. **Fresh start** - Since there are no production deployments yet, this is the perfect time to consolidate
|
|
|
|
## Files Modified
|
|
|
|
- Created: `migrations/20250101000001_initial_setup.sql`
|
|
- Created: `migrations/20250101000002_core_tables.sql`
|
|
- Created: `migrations/20250101000003_event_system.sql`
|
|
- Created: `migrations/20250101000004_execution_system.sql`
|
|
- Created: `migrations/20250101000005_supporting_tables.sql`
|
|
- Updated: `migrations/README.md` (comprehensive rewrite with new structure)
|
|
- Moved: All 18 old migrations to `migrations/old_migrations_backup/`
|
|
|
|
## Tables Created (18 total)
|
|
|
|
### Core (7)
|
|
- pack
|
|
- runtime
|
|
- worker
|
|
- identity
|
|
- permission_set
|
|
- permission_assignment
|
|
- policy
|
|
- key
|
|
|
|
### Event System (4)
|
|
- trigger
|
|
- sensor
|
|
- event
|
|
- enforcement
|
|
|
|
### Execution System (4)
|
|
- action
|
|
- rule
|
|
- execution
|
|
- inquiry
|
|
|
|
### Supporting (2)
|
|
- notification
|
|
- artifact
|
|
|
|
## Testing Status
|
|
|
|
- [ ] Test fresh database creation with new migrations
|
|
- [ ] Verify all foreign key constraints are correct
|
|
- [ ] Verify all indexes are created
|
|
- [ ] Test that SQLx compile-time checking still works
|
|
- [ ] Run existing integration tests (if any)
|
|
- [x] Fix sensor service compilation error with missing `trigger_params` field
|
|
- [x] Fix sensor service test compilation error with missing `trigger_params` field
|
|
|
|
### Issues Fixed During Consolidation
|
|
|
|
#### 1. Sensor Service Rule Query
|
|
**Issue:** `attune-sensor` failed to compile after migration consolidation due to missing `trigger_params` field in Rule query.
|
|
|
|
**Error:**
|
|
```
|
|
error[E0063]: missing field `trigger_params` in initializer of `attune_common::models::Rule`
|
|
--> crates/sensor/src/rule_matcher.rs:114:13
|
|
```
|
|
|
|
**Root Cause:** The `find_matching_rules()` query in `rule_matcher.rs` was selecting Rule columns but missing the newly added `trigger_params` field.
|
|
|
|
**Fix:** Added `trigger_params` to the SELECT clause in `crates/sensor/src/rule_matcher.rs:129`:
|
|
```rust
|
|
// Before:
|
|
conditions,
|
|
action_params,
|
|
enabled,
|
|
|
|
// After:
|
|
conditions,
|
|
action_params,
|
|
trigger_params,
|
|
enabled,
|
|
```
|
|
|
|
**Status:** ✅ Fixed - Compilation error resolved
|
|
|
|
#### 2. Sensor Service Test Rule Creation
|
|
**Issue:** Test code failed to compile due to missing `trigger_params` field in Rule struct initialization.
|
|
|
|
**Error:**
|
|
```
|
|
error[E0063]: missing field `trigger_params` in initializer of `attune_common::models::Rule`
|
|
--> crates/sensor/src/rule_matcher.rs:498:9
|
|
```
|
|
|
|
**Root Cause:** The `test_rule()` helper function in tests was creating a Rule instance but missing the newly added `trigger_params` field.
|
|
|
|
**Fix:** Added `trigger_params` field to test Rule initialization in `crates/sensor/src/rule_matcher.rs:499`:
|
|
```rust
|
|
fn test_rule() -> Rule {
|
|
Rule {
|
|
action_params: serde_json::json!({}),
|
|
trigger_params: serde_json::json!({}), // <-- Added this line
|
|
id: 1,
|
|
// ... rest of fields
|
|
}
|
|
}
|
|
```
|
|
|
|
**Status:** ✅ Fixed - Test compilation error resolved
|
|
|
|
## Next Steps
|
|
|
|
1. Test the new migrations on a clean database:
|
|
```bash
|
|
dropdb attune_test && createdb attune_test
|
|
DATABASE_URL="postgresql://postgres:postgres@localhost/attune_test" sqlx migrate run
|
|
```
|
|
|
|
2. Verify all tables, indexes, and constraints:
|
|
```bash
|
|
psql -U postgres -d attune_test -c "\dt attune.*"
|
|
psql -U postgres -d attune_test -c "\di attune.*"
|
|
```
|
|
|
|
3. Update any documentation that references old migration files
|
|
|
|
4. After successful verification, delete `migrations/old_migrations_backup/`
|
|
|
|
## Notes
|
|
|
|
- All enum types (12 total) preserved exactly
|
|
- All constraints and validation triggers preserved
|
|
- All indexes (including GIN indexes) preserved
|
|
- All comments preserved and enhanced
|
|
- pg_notify trigger for notifications preserved
|
|
- Key ownership validation trigger preserved
|
|
- Proper CASCADE and SET NULL behaviors maintained
|
|
|
|
## Post-Consolidation Fixes
|
|
|
|
### Files Modified
|
|
- `crates/sensor/src/rule_matcher.rs` - Added missing `trigger_params` field to Rule query (line 129)
|
|
- `crates/sensor/src/rule_matcher.rs` - Added missing `trigger_params` field to test Rule initialization (line 499)
|
|
|
|
### Compilation Status
|
|
- ✅ Sensor service compilation error fixed (Rule query)
|
|
- ✅ Sensor service test compilation error fixed (test_rule helper)
|
|
- ✅ No more missing field errors in workspace
|
|
- ⏳ SQLx cache needs update (`cargo sqlx prepare`)
|
|
- ⏳ Full workspace compilation pending database setup |