re-uploading work
This commit is contained in:
217
work-summary/sessions/2025-01-16_migration_consolidation.md
Normal file
217
work-summary/sessions/2025-01-16_migration_consolidation.md
Normal file
@@ -0,0 +1,217 @@
|
||||
# Migration Consolidation - January 16, 2025
|
||||
|
||||
## Summary
|
||||
|
||||
Consolidated 18 separate database migration files into 5 logically organized migrations for better maintainability and clarity in this early-stage project.
|
||||
|
||||
## Changes Made
|
||||
|
||||
### Migration Structure
|
||||
|
||||
**Before:** 18 migration files
|
||||
- 12 initial table creation files (20240101 series)
|
||||
- 6 patch/fix files (20240102-20240103 series)
|
||||
|
||||
**After:** 5 consolidated migration files
|
||||
1. `20250101000001_initial_setup.sql` - Schema, enums, shared functions
|
||||
2. `20250101000002_core_tables.sql` - Pack, runtime, worker, identity, permissions, policy, key (7 tables)
|
||||
3. `20250101000003_event_system.sql` - Trigger, sensor, event, enforcement (4 tables)
|
||||
4. `20250101000004_execution_system.sql` - Action, rule, execution, inquiry (4 tables)
|
||||
5. `20250101000005_supporting_tables.sql` - Notification, artifact (2 tables)
|
||||
|
||||
### What Was Consolidated
|
||||
|
||||
Each new migration incorporates all patches and fixes:
|
||||
|
||||
1. **Initial Setup** - Combined:
|
||||
- Schema creation
|
||||
- All 12 enum type definitions
|
||||
- Shared `update_updated_column()` function
|
||||
|
||||
2. **Core Tables** - Combined:
|
||||
- Pack table with all indexes (including performance indexes)
|
||||
- Runtime and worker tables
|
||||
- Identity table with `password_hash` column (from patch 20240102000001)
|
||||
- Permission_set, permission_assignment, policy tables
|
||||
- Key table with ownership validation
|
||||
|
||||
3. **Event System** - Combined:
|
||||
- Trigger table
|
||||
- Sensor table with `config` column (from patch 20240103000001) and CASCADE foreign keys (from patch 20240102000002)
|
||||
- Event table
|
||||
- Enforcement table (with forward reference to rule)
|
||||
|
||||
4. **Execution System** - Combined:
|
||||
- Action table
|
||||
- Rule table with `action_params` and `trigger_params` columns (from patches 20240103000003 and 20240103000004)
|
||||
- Execution table with proper identity foreign key
|
||||
- Inquiry table with proper identity foreign key
|
||||
- All foreign key constraints resolved (no forward references)
|
||||
|
||||
5. **Supporting Tables** - Combined:
|
||||
- Notification table with pg_notify trigger
|
||||
- Artifact table
|
||||
- All performance optimization indexes from 20240101000012
|
||||
|
||||
### Forward Reference Handling
|
||||
|
||||
The old migrations had circular dependencies between tables. The new structure properly handles this:
|
||||
|
||||
- **Migration 2** creates policy and key tables with forward references to action/sensor (without FK constraints)
|
||||
- **Migration 3** adds sensor foreign key to key table
|
||||
- **Migration 4** adds action foreign key to policy and key tables, plus rule foreign key to enforcement table
|
||||
|
||||
### Old Migrations
|
||||
|
||||
All 18 old migration files were moved to `migrations/old_migrations_backup/` for reference and can be safely deleted after verification.
|
||||
|
||||
## Benefits
|
||||
|
||||
1. **Easier to understand** - 5 files vs 18 files, clear logical grouping
|
||||
2. **Cleaner history** - All patches incorporated into base migrations
|
||||
3. **Better documentation** - Each file has clear sections with comments
|
||||
4. **Reduced complexity** - No need to track patch dependencies
|
||||
5. **Fresh start** - Since there are no production deployments yet, this is the perfect time to consolidate
|
||||
|
||||
## Files Modified
|
||||
|
||||
- Created: `migrations/20250101000001_initial_setup.sql`
|
||||
- Created: `migrations/20250101000002_core_tables.sql`
|
||||
- Created: `migrations/20250101000003_event_system.sql`
|
||||
- Created: `migrations/20250101000004_execution_system.sql`
|
||||
- Created: `migrations/20250101000005_supporting_tables.sql`
|
||||
- Updated: `migrations/README.md` (comprehensive rewrite with new structure)
|
||||
- Moved: All 18 old migrations to `migrations/old_migrations_backup/`
|
||||
|
||||
## Tables Created (18 total)
|
||||
|
||||
### Core (7)
|
||||
- pack
|
||||
- runtime
|
||||
- worker
|
||||
- identity
|
||||
- permission_set
|
||||
- permission_assignment
|
||||
- policy
|
||||
- key
|
||||
|
||||
### Event System (4)
|
||||
- trigger
|
||||
- sensor
|
||||
- event
|
||||
- enforcement
|
||||
|
||||
### Execution System (4)
|
||||
- action
|
||||
- rule
|
||||
- execution
|
||||
- inquiry
|
||||
|
||||
### Supporting (2)
|
||||
- notification
|
||||
- artifact
|
||||
|
||||
## Testing Status
|
||||
|
||||
- [ ] Test fresh database creation with new migrations
|
||||
- [ ] Verify all foreign key constraints are correct
|
||||
- [ ] Verify all indexes are created
|
||||
- [ ] Test that SQLx compile-time checking still works
|
||||
- [ ] Run existing integration tests (if any)
|
||||
- [x] Fix sensor service compilation error with missing `trigger_params` field
|
||||
- [x] Fix sensor service test compilation error with missing `trigger_params` field
|
||||
|
||||
### Issues Fixed During Consolidation
|
||||
|
||||
#### 1. Sensor Service Rule Query
|
||||
**Issue:** `attune-sensor` failed to compile after migration consolidation due to missing `trigger_params` field in Rule query.
|
||||
|
||||
**Error:**
|
||||
```
|
||||
error[E0063]: missing field `trigger_params` in initializer of `attune_common::models::Rule`
|
||||
--> crates/sensor/src/rule_matcher.rs:114:13
|
||||
```
|
||||
|
||||
**Root Cause:** The `find_matching_rules()` query in `rule_matcher.rs` was selecting Rule columns but missing the newly added `trigger_params` field.
|
||||
|
||||
**Fix:** Added `trigger_params` to the SELECT clause in `crates/sensor/src/rule_matcher.rs:129`:
|
||||
```rust
|
||||
// Before:
|
||||
conditions,
|
||||
action_params,
|
||||
enabled,
|
||||
|
||||
// After:
|
||||
conditions,
|
||||
action_params,
|
||||
trigger_params,
|
||||
enabled,
|
||||
```
|
||||
|
||||
**Status:** ✅ Fixed - Compilation error resolved
|
||||
|
||||
#### 2. Sensor Service Test Rule Creation
|
||||
**Issue:** Test code failed to compile due to missing `trigger_params` field in Rule struct initialization.
|
||||
|
||||
**Error:**
|
||||
```
|
||||
error[E0063]: missing field `trigger_params` in initializer of `attune_common::models::Rule`
|
||||
--> crates/sensor/src/rule_matcher.rs:498:9
|
||||
```
|
||||
|
||||
**Root Cause:** The `test_rule()` helper function in tests was creating a Rule instance but missing the newly added `trigger_params` field.
|
||||
|
||||
**Fix:** Added `trigger_params` field to test Rule initialization in `crates/sensor/src/rule_matcher.rs:499`:
|
||||
```rust
|
||||
fn test_rule() -> Rule {
|
||||
Rule {
|
||||
action_params: serde_json::json!({}),
|
||||
trigger_params: serde_json::json!({}), // <-- Added this line
|
||||
id: 1,
|
||||
// ... rest of fields
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Status:** ✅ Fixed - Test compilation error resolved
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Test the new migrations on a clean database:
|
||||
```bash
|
||||
dropdb attune_test && createdb attune_test
|
||||
DATABASE_URL="postgresql://postgres:postgres@localhost/attune_test" sqlx migrate run
|
||||
```
|
||||
|
||||
2. Verify all tables, indexes, and constraints:
|
||||
```bash
|
||||
psql -U postgres -d attune_test -c "\dt attune.*"
|
||||
psql -U postgres -d attune_test -c "\di attune.*"
|
||||
```
|
||||
|
||||
3. Update any documentation that references old migration files
|
||||
|
||||
4. After successful verification, delete `migrations/old_migrations_backup/`
|
||||
|
||||
## Notes
|
||||
|
||||
- All enum types (12 total) preserved exactly
|
||||
- All constraints and validation triggers preserved
|
||||
- All indexes (including GIN indexes) preserved
|
||||
- All comments preserved and enhanced
|
||||
- pg_notify trigger for notifications preserved
|
||||
- Key ownership validation trigger preserved
|
||||
- Proper CASCADE and SET NULL behaviors maintained
|
||||
|
||||
## Post-Consolidation Fixes
|
||||
|
||||
### Files Modified
|
||||
- `crates/sensor/src/rule_matcher.rs` - Added missing `trigger_params` field to Rule query (line 129)
|
||||
- `crates/sensor/src/rule_matcher.rs` - Added missing `trigger_params` field to test Rule initialization (line 499)
|
||||
|
||||
### Compilation Status
|
||||
- ✅ Sensor service compilation error fixed (Rule query)
|
||||
- ✅ Sensor service test compilation error fixed (test_rule helper)
|
||||
- ✅ No more missing field errors in workspace
|
||||
- ⏳ SQLx cache needs update (`cargo sqlx prepare`)
|
||||
- ⏳ Full workspace compilation pending database setup
|
||||
Reference in New Issue
Block a user