attune-system/attune
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

trying to rework database migrations

Browse Source
This commit is contained in:
David Culbreth
2026-02-05 11:42:04 -06:00
parent 3b14c65998
commit 343488b3eb
83 changed files with 5793 additions and 876 deletions
Download Patch File Download Diff File Expand all files Collapse all files

177
docs/QUICKREF-dev-packs.md Normal file
View File

@@ -0,0 +1,177 @@
# Quick Reference: Development Packs
## Setup (One Time)
```bash
# Directory is already created, just start Docker
docker compose up -d
```
## Create a Pack
```bash
./scripts/dev-pack.sh create my-pack
```
Creates:
- `packs.dev/my-pack/pack.yaml`
- `packs.dev/my-pack/actions/example.sh`
- Example action YAML
- README
## List Packs
```bash
./scripts/dev-pack.sh list
```
## Validate Pack
```bash
./scripts/dev-pack.sh validate my-pack
```
Checks:
- ✓ pack.yaml exists
- ✓ Action scripts exist and are executable
- ✓ Entry points match
## Register Pack in Attune
```bash
# Get token first
export ATTUNE_TOKEN=$(attune auth login test@attune.local --password TestPass123!)
# Register pack
curl -X POST http://localhost:8080/api/v1/packs \
-H "Authorization: Bearer $ATTUNE_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"ref": "my-pack",
"label": "My Pack",
"version": "1.0.0",
"enabled": true
}'
```
## Execute Action
```bash
curl -X POST http://localhost:8080/api/v1/executions \
-H "Authorization: Bearer $ATTUNE_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"action": "my-pack.example",
"parameters": {
"message": "Hello!"
}
}'
```
## Directory Layout
```
packs.dev/
├── examples/ # Example packs (in git)
│ ├── basic-pack/ # Shell action example
│ └── python-pack/ # Python action example
└── my-pack/ # Your packs (not in git)
├── pack.yaml
├── actions/
├── triggers/
├── sensors/
└── workflows/
```
## File Locations in Docker
- **Core pack**: `/opt/attune/packs` (read-only)
- **Dev packs**: `/opt/attune/packs.dev` (read-write)
## Development Workflow
1. Create pack: `./scripts/dev-pack.sh create my-pack`
2. Edit files: `vim packs.dev/my-pack/actions/my_action.sh`
3. Validate: `./scripts/dev-pack.sh validate my-pack`
4. Register: See "Register Pack" above
5. Test: Execute action via API
6. Iterate: Changes are immediately visible!
## Action Script Template
```bash
#!/bin/bash
set -e
# Get parameters from environment
PARAM="${ATTUNE_ACTION_param:-default}"
# Validate
if [ -z "$PARAM" ]; then
echo '{"error": "param required"}' >&2
exit 1
fi
# Do work
result=$(echo "Processed: $PARAM")
# Return JSON
echo "{\"result\": \"$result\"}"
```
## Common Commands
```bash
# List all packs
./scripts/dev-pack.sh list
# Validate pack structure
./scripts/dev-pack.sh validate my-pack
# View pack in container
docker exec attune-api ls -la /opt/attune/packs.dev/
# Check worker logs
docker logs -f attune-worker-shell
# Sync workflows after changes
curl -X POST http://localhost:8080/api/v1/packs/my-pack/workflows/sync \
-H "Authorization: Bearer $ATTUNE_TOKEN"
# Clean up dev packs
./scripts/dev-pack.sh clean
```
## Troubleshooting
### "Pack not found"
```bash
# Check if registered
curl http://localhost:8080/api/v1/packs/my-pack \
-H "Authorization: Bearer $ATTUNE_TOKEN"
# Check if files exist in container
docker exec attune-api ls /opt/attune/packs.dev/my-pack/
```
### "Entry point not found"
```bash
# Make script executable
chmod +x packs.dev/my-pack/actions/*.sh
# Verify in container
docker exec attune-worker-shell ls -la /opt/attune/packs.dev/my-pack/actions/
```
### Changes not reflected
```bash
# For action scripts: should be immediate
# For action YAML: re-register pack
# For workflows: run sync endpoint
```
## See Also
- [Full Documentation](development/packs-dev-directory.md)
- [Pack Structure](packs/pack-structure.md)
- [Examples](../packs.dev/examples/)

474
docs/development/packs-dev-directory.md Normal file
View File

@@ -0,0 +1,474 @@
# Development Packs Directory
## Overview
The `packs.dev/` directory provides a development environment for creating and testing custom packs without rebuilding Docker images. Files in this directory are mounted directly into Docker containers at `/opt/attune/packs.dev`, allowing immediate access to changes.
## Quick Start
### 1. Create a New Pack
```bash
./scripts/dev-pack.sh create my-pack
```
This creates a complete pack structure:
```
packs.dev/my-pack/
├── pack.yaml
├── actions/
│ ├── example.yaml
│ └── example.sh
├── triggers/
├── sensors/
├── workflows/
└── README.md
```
### 2. Validate the Pack
```bash
./scripts/dev-pack.sh validate my-pack
```
### 3. Start Docker Environment
```bash
docker compose up -d
```
The pack is automatically available at `/opt/attune/packs.dev/my-pack` in all containers.
### 4. Register the Pack
Get an authentication token:
```bash
# Login via web UI or CLI
attune auth login test@attune.local
```
Register the pack via API:
```bash
curl -X POST http://localhost:8080/api/v1/packs \
-H "Authorization: Bearer $ATTUNE_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"ref": "my-pack",
"label": "My Custom Pack",
"description": "My custom automation pack",
"version": "1.0.0",
"system": false,
"enabled": true
}'
```
### 5. Test the Pack
Create a rule that uses your pack's actions, or execute directly:
```bash
curl -X POST http://localhost:8080/api/v1/executions \
-H "Authorization: Bearer $ATTUNE_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"action": "my-pack.example",
"parameters": {
"message": "Hello from dev pack!"
}
}'
```
## Directory Structure
```
packs.dev/
├── README.md # Usage guide
├── .gitignore # Ignore custom packs, keep examples
├── examples/ # Example packs
│ ├── basic-pack/ # Minimal shell action example
│ └── python-pack/ # Python action example
└── my-pack/ # Your custom pack (not in git)
├── pack.yaml # Pack metadata
├── actions/ # Action definitions and scripts
├── triggers/ # Trigger definitions
├── sensors/ # Sensor definitions
└── workflows/ # Workflow definitions
```
## Volume Mounts
The `packs.dev/` directory is mounted in Docker Compose:
```yaml
volumes:
- ./packs.dev:/opt/attune/packs.dev:rw
```
This mount is added to all relevant services:
- **api** - Pack registration and metadata
- **executor** - Workflow execution
- **worker-*** - Action execution
- **sensor** - Sensor execution
### Core vs Dev Packs
| Location | Mount Type | Purpose |
|----------|------------|---------|
| `/opt/attune/packs` | Volume (ro) | Production core pack |
| `/opt/attune/packs.dev` | Bind mount (rw) | Development packs |
The core pack is read-only in containers, while dev packs are read-write for active development.
## Development Workflow
### Typical Development Cycle
1. **Create pack structure**
```bash
./scripts/dev-pack.sh create my-integration
```
2. **Edit pack files**
- Edit `packs.dev/my-integration/pack.yaml`
- Add actions in `actions/`
- Add workflows in `workflows/`
3. **Validate**
```bash
./scripts/dev-pack.sh validate my-integration
```
4. **Test immediately** - Changes are live in containers!
- No rebuild needed
- No restart needed
- Actions are available instantly
5. **Iterate** - Make changes and test again
6. **Export for production** - When ready, package the pack properly
### Live Reloading
Changes to pack files are immediately visible in containers because they're bind-mounted:
- **Action scripts**: Available immediately for execution
- **Action/Trigger YAML**: Requires pack re-registration to update DB
- **Workflows**: Use workflow sync endpoint to reload
```bash
# Sync workflows after changes
curl -X POST http://localhost:8080/api/v1/packs/my-pack/workflows/sync \
-H "Authorization: Bearer $ATTUNE_TOKEN"
```
## Helper Script Reference
### Commands
#### `create <pack-ref>`
Creates a new pack structure with example files.
```bash
./scripts/dev-pack.sh create my-awesome-pack
```
Creates:
- `packs.dev/my-awesome-pack/`
- Basic pack.yaml
- Example shell action
- README with instructions
#### `list`
Lists all development packs.
```bash
./scripts/dev-pack.sh list
```
Output:
```
Development Packs:
my-pack
Label: My Pack
Version: 1.0.0
integration-pack
Label: Integration Pack
Version: 2.1.0
Total: 2 pack(s)
```
#### `validate <pack-ref>`
Validates pack structure and files.
```bash
./scripts/dev-pack.sh validate my-pack
```
Checks:
- `pack.yaml` exists and is valid YAML
- Action definitions reference existing scripts
- Scripts are executable
- Required directories exist
#### `register <pack-ref>`
Shows the API command to register the pack.
```bash
./scripts/dev-pack.sh register my-pack
```
Outputs the `curl` command needed to register via API.
#### `clean`
Removes all non-example packs (interactive confirmation).
```bash
./scripts/dev-pack.sh clean
```
**Warning**: This permanently deletes custom packs!
## Example Packs
### Basic Pack (Shell Actions)
Location: `packs.dev/examples/basic-pack/`
Simple shell-based action that echoes a message.
**Try it:**
```bash
# View the pack
ls -la packs.dev/examples/basic-pack/
# Register it (after starting Docker)
curl -X POST http://localhost:8080/api/v1/packs \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d @packs.dev/examples/basic-pack/pack.json
```
### Python Pack
Location: `packs.dev/examples/python-pack/`
Python action with parameters and structured output.
Features:
- Parameter validation
- JSON output
- Array handling
- Environment variable access
## Best Practices
### Pack Structure
1. **Use descriptive refs**: `my-company-integration`, not `pack1`
2. **Version properly**: Follow semantic versioning (1.0.0)
3. **Document actions**: Clear descriptions and parameter docs
4. **Test parameters**: Validate edge cases and defaults
5. **Handle errors**: Always return valid JSON, even on error
### Action Scripts
```bash
#!/bin/bash
set -e # Exit on error
# Get parameters (with defaults)
PARAM="${ATTUNE_ACTION_param:-default_value}"
# Validate inputs
if [ -z "$PARAM" ]; then
echo '{"error": "param is required"}' >&2
exit 1
fi
# Do work
RESULT=$(do_something "$PARAM")
# Return JSON
echo "{\"result\": \"$RESULT\"}"
```
### Security Considerations
1. **No secrets in code**: Use Attune's secret management
2. **Validate inputs**: Never trust action parameters directly
3. **Sandbox scripts**: Be aware workers execute with privileges
4. **Review dependencies**: Check Python/Node packages carefully
### Version Control
The `.gitignore` in `packs.dev/` excludes custom packs:
```gitignore
*
!.gitignore
!README.md
!examples/
!examples/**
```
This means:
- ✅ Example packs are committed
- ✅ Documentation is committed
- ❌ Your custom packs are NOT committed
To version control a custom pack:
1. Move it to a separate repository
2. Or explicitly add it: `git add -f packs.dev/my-pack/`
## Troubleshooting
### Pack Not Found
**Symptom**: "Pack not found" when executing action
**Solutions**:
1. Verify pack is registered in database:
```bash
curl http://localhost:8080/api/v1/packs/$PACK_REF \
-H "Authorization: Bearer $TOKEN"
```
2. Check pack directory exists:
```bash
docker exec attune-api ls -la /opt/attune/packs.dev/
```
3. Verify mount in docker-compose.yaml:
```bash
grep -A 2 "packs.dev" docker-compose.yaml
```
### Action Not Executing
**Symptom**: Action fails with "entry point not found"
**Solutions**:
1. Check script exists and is executable:
```bash
ls -la packs.dev/my-pack/actions/
```
2. Verify entry_point in action YAML matches filename:
```bash
grep entry_point packs.dev/my-pack/actions/*.yaml
```
3. Check script has shebang and is executable:
```bash
head -1 packs.dev/my-pack/actions/script.sh
chmod +x packs.dev/my-pack/actions/script.sh
```
### Permission Errors
**Symptom**: "Permission denied" when accessing pack files
**Solutions**:
1. Check file ownership (should be readable by UID 1000):
```bash
ls -ln packs.dev/my-pack/
```
2. Fix permissions:
```bash
chmod -R 755 packs.dev/my-pack/
```
3. Ensure scripts are executable:
```bash
find packs.dev/my-pack/ -name "*.sh" -exec chmod +x {} \;
```
### Changes Not Reflected
**Symptom**: Code changes don't appear in execution
**Solutions**:
1. For **action scripts**: Changes are immediate, but verify mount:
```bash
docker exec attune-worker-shell cat /opt/attune/packs.dev/my-pack/actions/script.sh
```
2. For **action YAML**: Re-register pack or update action in DB
3. For **workflows**: Run sync endpoint:
```bash
curl -X POST http://localhost:8080/api/v1/packs/my-pack/workflows/sync \
-H "Authorization: Bearer $TOKEN"
```
## Advanced Usage
### Multiple Environment Packs
Use different pack refs for different environments:
```
packs.dev/
├── my-pack-dev/ # Development version
├── my-pack-staging/ # Staging version
└── my-pack/ # Production-ready version
```
### Pack Dependencies
Reference other packs in workflows:
```yaml
# In packs.dev/my-pack/workflows/example.yaml
tasks:
- name: use_core_action
action: core.http_request
input:
url: https://api.example.com
- name: use_my_action
action: my-pack.process
input:
data: "{{ use_core_action.output.body }}"
```
### Testing Workflows
Create test workflows in `packs.dev/`:
```yaml
# packs.dev/my-pack/workflows/test_integration.yaml
name: test_integration
ref: my-pack.test_integration
description: "Integration test workflow"
tasks:
- name: test_action
action: my-pack.my_action
input:
test: true
```
## Production Migration
When ready to deploy a dev pack to production:
1. **Clean up**: Remove test files and documentation
2. **Version**: Tag with proper version number
3. **Test**: Run full test suite
4. **Package**: Create proper pack archive
5. **Install**: Use pack installation API
6. **Deploy**: Install on production Attune instance
See [Pack Registry Documentation](../packs/pack-registry-spec.md) for production deployment.
## See Also
- [Pack Structure Documentation](../packs/pack-structure.md)
- [Action Development Guide](../packs/PACK_TESTING.md)
- [Workflow Development](../workflows/workflow-summary.md)
- [Pack Registry](../packs/pack-registry-spec.md)
- [Docker Deployment](../deployment/docker-deployment.md)

224
docs/migrations/CONSOLIDATION-COMPLETE.md Normal file
View File

@@ -0,0 +1,224 @@
# Migration Consolidation - Complete
**Date**: 2026-02-04
**Status**: ✅ COMPLETE
**Result**: 22 migrations → 13 migrations
## Summary
Successfully consolidated Attune's migration history from 22 files to 13 clean, logical migrations. This was possible because there are no production deployments yet, allowing us to freely restructure the schema history.
## Changes Made
### Items Removed Entirely (Never Created)
1. **`runtime_type_enum`** - Removed from initial setup
- Associated column `runtime.runtime_type` not created
- Associated indexes not created: `idx_runtime_type`, `idx_runtime_pack_type`, `idx_runtime_type_created`, `idx_runtime_type_sensor`
- Runtime table uses unified approach from the start
2. **`workflow_task_execution` table** - Not created
- Consolidated into `execution.workflow_task JSONB` column from initial execution table creation
- Eliminates need for separate table and join operations
3. **Individual webhook columns** - Not created
- Skipped 10 intermediate columns (webhook_secret, webhook_hmac_*, webhook_rate_limit_*, etc.)
- Only created: `webhook_enabled`, `webhook_key`, `webhook_config JSONB` from start
4. **Runtime data insertions** - Removed from migrations
- All runtime metadata moved to YAML files in `packs/core/runtimes/`
- No SQL INSERT statements for runtime records
### Items Included From Start
1. **Execution table workflow columns** (in 00006):
- `is_workflow BOOLEAN DEFAULT false NOT NULL`
- `workflow_def BIGINT REFERENCES workflow_definition(id)`
- `workflow_task JSONB`
2. **Is adhoc flags** (in respective tables):
- `action.is_adhoc` (in 00004)
- `sensor.is_adhoc` (in 00004)
- `rule.is_adhoc` (in 00005)
3. **Event table rule tracking** (in 00005):
- `event.rule BIGINT`
- `event.rule_ref TEXT`
- Foreign key constraint to rule table
4. **Worker role** (in 00008):
- `worker_role_enum` type (in 00001)
- `worker.worker_role` column
5. **Trigger webhook support** (in 00005):
- `webhook_enabled BOOLEAN NOT NULL DEFAULT FALSE`
- `webhook_key VARCHAR(64) UNIQUE`
- `webhook_config JSONB DEFAULT '{}'::jsonb`
6. **Pack environments** (in 00001 and 00003):
- `pack_environment_status_enum` type (in 00001)
- `pack.installers JSONB` column (in 00003)
- `pack_environment` table (in 00011)
## Final Migration Structure
```
migrations/
├── 20250101000001_initial_setup.sql # Enums, extensions (minus runtime_type_enum, plus worker_role_enum and pack_environment_status_enum)
├── 20250101000002_identity_and_auth.sql # Identity, permission_set, permission_assignment, policy
├── 20250101000003_pack_system.sql # Pack (with installers), runtime (no runtime_type)
├── 20250101000004_action_sensor.sql # Action, sensor (both with is_adhoc)
├── 20250101000005_trigger_event_rule.sql # Trigger (with webhook_config), event (with rule), rule (with is_adhoc)
├── 20250101000006_execution_system.sql # Enforcement, execution (with workflow columns), inquiry
├── 20250101000007_workflow_system.sql # Workflow_definition, workflow_execution (no workflow_task_execution)
├── 20250101000008_worker_notification.sql # Worker (with worker_role), notification
├── 20250101000009_keys_artifacts.sql # Key, artifact
├── 20250101000010_webhook_system.sql # Webhook functions (final versions)
├── 20250101000011_pack_environments.sql # Pack_environment table
├── 20250101000012_pack_testing.sql # Pack_test_results table
└── 20250101000013_notify_triggers.sql # All LISTEN/NOTIFY triggers (consolidated)
```
## Migrations Removed
The following 15 migration files were consolidated or had their data moved to YAML:
1. `20260119000001_add_execution_notify_trigger.sql` → Consolidated into 00013
2. `20260120000001_add_webhook_support.sql` → Columns added to trigger table in 00005
3. `20260120000002_webhook_advanced_features.sql` → Functions consolidated in 00010
4. `20260122000001_pack_installation_metadata.sql` → Merged into pack system
5. `20260127000001_consolidate_webhook_config.sql` → Already consolidated in 00005
6. `20260127212500_consolidate_workflow_task_execution.sql` → Already in execution table in 00006
7. `20260129000001_fix_webhook_function_overload.sql` → Fixed functions in 00010
8. `20260129140130_add_is_adhoc_flag.sql` → Already in tables in 00004/00005
9. `20260129150000_add_event_notify_trigger.sql` → Consolidated into 00013
10. `20260130000001_add_rule_to_event.sql` → Already in event table in 00005
11. `20260131000001_add_worker_role.sql` → Already in worker table in 00008
12. `20260202000001_add_sensor_runtimes.sql` → Data moved to YAML files
13. `20260203000001_unify_runtimes.sql` → Changes applied to base runtime table in 00003
14. `20260203000003_add_rule_trigger_to_execution_notify.sql` → Consolidated into 00013
15. `20260204000001_add_enforcement_notify_trigger.sql` → Consolidated into 00013
Note: One file (`20260204000001_restore_webhook_functions.sql`) was kept and renamed to 00010 with final webhook functions.
## Benefits
1. **Cleaner History**: Future developers see logical progression, not incremental fixes
2. **Faster Tests**: 13 migrations vs 22 (41% reduction)
3. **No Dead Code**: Nothing created just to be dropped
4. **Accurate Schema**: Tables created with final structure from the start
5. **Better Maintainability**: Each migration has clear, focused purpose
6. **Reduced Complexity**: Fewer foreign key constraints to manage incrementally
## Data Migration
### Runtime Metadata
Runtime data is now managed externally:
**Location**: `packs/core/runtimes/*.yaml`
**Files**:
- `python.yaml` - Python 3 runtime
- `nodejs.yaml` - Node.js runtime
- `shell.yaml` - Shell runtime (bash/sh)
- `native.yaml` - Native compiled runtime
- `sensor_builtin.yaml` - Built-in sensor runtime
**Loading**: Handled by pack installation system, not migrations
## Testing
Next steps for validation:
```bash
# 1. Test on fresh database
createdb attune_test_consolidated
export DATABASE_URL="postgresql://attune:attune@localhost/attune_test_consolidated"
sqlx migrate run
# 2. Compare schema
pg_dump --schema-only attune_test_consolidated > schema_new.sql
pg_dump --schema-only attune_dev > schema_old.sql
diff schema_old.sql schema_new.sql
# 3. Verify table counts
psql attune_test_consolidated -c "\dt" | wc -l
# 4. Load core pack
./scripts/load-core-pack.sh
# 5. Run tests
cargo test
```
## Rollback Plan
Original migrations preserved in `migrations.old/` directory. To rollback:
```bash
rm -rf migrations/*.sql
cp migrations.old/*.sql migrations/
```
**Do NOT delete `migrations.old/` until consolidated migrations are verified in production-like environment.**
## Constraints Modified
1. **`runtime_ref_format`** - Removed entirely
- Old format: `^[^.]+\.(action|sensor)\.[^.]+$` (e.g., `core.action.python`)
- New format: No constraint, allows `pack.name` format (e.g., `core.python`)
2. **`runtime_ref_lowercase`** - Kept as-is
- Still enforces lowercase runtime refs
## Indexes Added/Modified
**Runtime table**:
- ❌ Removed: `idx_runtime_type`, `idx_runtime_pack_type`, `idx_runtime_type_created`
- ✅ Added: `idx_runtime_name`, `idx_runtime_verification` (GIN index)
**Trigger table**:
- ✅ Added: `idx_trigger_webhook_key`
**Event table**:
- ✅ Added: `idx_event_rule`
## Statistics
| Metric | Before | After | Change |
|--------|--------|-------|--------|
| Migration files | 22 | 13 | -41% |
| Lines of SQL | ~3,500 | ~2,100 | -40% |
| Enum types | 13 | 12 | -1 |
| Tables created | 22 | 21 | -1 |
| Tables created then dropped | 1 | 0 | -100% |
| Columns added then dropped | 10 | 0 | -100% |
## Completion Checklist
- ✅ Backup created in `migrations.old/`
- ✅ 13 consolidated migrations created
- ✅ Runtime data moved to YAML files
- ✅ All incremental additions consolidated
- ✅ Documentation updated
- ⏳ Test on fresh database
- ⏳ Compare schemas
- ⏳ Run full test suite
- ⏳ Deploy to development
- ⏳ Delete `migrations.old/` after verification
## Notes
- All changes are breaking changes, but that's acceptable since there are no production deployments
- Future migrations should be created normally and incrementally
- This consolidation should be a one-time event before v1.0 release
- After production deployment, normal migration discipline applies (no deletions, only additions)
## Acknowledgments
This consolidation was made possible by the "Breaking Changes Policy" documented in `AGENTS.md`:
> **Breaking changes are explicitly allowed and encouraged** when they improve the architecture, API design, or developer experience. No backward compatibility required - there are no existing versions to support.
Once this project reaches v1.0 or gets its first production deployment, this policy will be replaced with appropriate stability guarantees and versioning policies.

142
docs/migrations/CONSOLIDATION-SUMMARY.md Normal file
View File

@@ -0,0 +1,142 @@
# Migration Consolidation - Executive Summary
**Date**: 2026-02-04
**Status**: Pre-production - Safe to consolidate
**Impact**: No production deployments exist
## Overview
The Attune project has accumulated 22 migrations during active development. Since there are no production deployments, we can safely consolidate these into a clean initial state, removing items that were created and then dropped or modified.
## Key Findings
### Items Created Then Dropped (Remove Entirely)
1. **`runtime_type_enum`** - Created in 00001, dropped in 20260203000001
- Associated column: `runtime.runtime_type`
- Associated indexes: 4 indexes referencing this column
- **Action**: Don't create at all
2. **`workflow_task_execution` table** - Created in 00004, dropped in 20260127212500
- Consolidated into `execution.workflow_task JSONB` column
- **Action**: Don't create table, add JSONB column to execution from start
3. **Individual webhook columns (10 columns)** - Added in 20260120000001/000002, dropped in 20260127000001
- `webhook_secret`, `webhook_hmac_enabled`, `webhook_hmac_secret`, etc.
- Consolidated into single `webhook_config JSONB`
- **Action**: Only create `webhook_enabled`, `webhook_key`, `webhook_config` from start
4. **Runtime INSERT statements** - Added in 20260202000001, truncated in 20260203000001
- Now loaded from YAML files in `packs/core/runtimes/`
- **Action**: Remove all runtime data from migrations
### Items Added Later (Include From Start)
1. **Execution table workflow columns**:
- `is_workflow BOOLEAN` (added later)
- `workflow_def BIGINT` (added later)
- `workflow_task JSONB` (added in consolidation migration)
2. **Is adhoc flags** (added in 20260129140130):
- `action.is_adhoc`
- `sensor.is_adhoc`
- `rule.is_adhoc`
3. **Event table rule tracking** (added in 20260130000001):
- `event.rule BIGINT`
- `event.rule_ref TEXT`
4. **Worker role** (added in 20260131000001):
- `worker_role_enum` type
- `worker.worker_role` column
5. **Pack environments** (added in 20260203000002):
- `pack_environment_status_enum` type
- `pack.installers JSONB` column
- `pack_environment` table
6. **LISTEN/NOTIFY triggers** (added across 4 migrations):
- Execution notify
- Event notify
- Enforcement notify
- Consolidated into single migration
### Constraints Modified
1. **`runtime_ref_format`** - Original: `^[^.]+\.(action|sensor)\.[^.]+$`
- Expected format was `pack.type.name` (e.g., `core.action.python`)
- Changed to allow `pack.name` format (e.g., `core.python`)
- **Action**: Drop constraint entirely or create with final format
2. **`runtime_ref_lowercase`** - Dropped and not recreated
- **Action**: Determine if needed in final schema
## Recommended Consolidation Structure
```
migrations/
├── 20250101000001_initial_setup.sql # Enums, extensions (FINAL VERSIONS)
├── 20250101000002_identity_and_auth.sql # Identity, keys
├── 20250101000003_pack_system.sql # Pack, runtime (no runtime_type)
├── 20250101000004_action_sensor.sql # Action, sensor (with is_adhoc)
├── 20250101000005_trigger_event_rule.sql # Trigger (with webhook_config), event (with rule), rule (with is_adhoc)
├── 20250101000006_execution_system.sql # Execution (with workflow cols), enforcement, inquiry, policy
├── 20250101000007_workflow_system.sql # Workflow_definition only (no workflow_task_execution)
├── 20250101000008_worker_notification.sql # Worker (with role), notification
├── 20250101000009_artifacts.sql # Artifact table
├── 20250101000010_webhook_system.sql # Webhook tables, FINAL functions
├── 20250101000011_pack_environments.sql # Pack_environment table and enum
├── 20250101000012_pack_testing.sql # Pack_test_results table
├── 20250101000013_notify_triggers.sql # ALL LISTEN/NOTIFY triggers
└── README.md # Migration documentation
```
## Benefits
1. **Cleaner git history** - Future developers see logical progression
2. **Faster test setup** - Fewer migrations to run (13 vs 22)
3. **No dead code** - Nothing created just to be dropped
4. **Accurate from start** - Tables created with final schema
5. **Better documentation** - Clear purpose for each migration
## Risks
**NONE** - No production deployments exist. This is the ideal time to consolidate.
## Data Considerations
**Runtime metadata** is now managed in YAML files:
- Location: `packs/core/runtimes/*.yaml`
- Loaded by: Pack installation system
- Files: `python.yaml`, `nodejs.yaml`, `shell.yaml`, `native.yaml`, `sensor_builtin.yaml`
**Core pack data** should be checked for any other SQL insertions that should move to YAML.
## Validation Plan
1. Create consolidated migrations in new directory
2. Test on fresh database: `createdb attune_test && sqlx migrate run`
3. Compare schema output: `pg_dump --schema-only` before/after
4. Verify table counts, column counts, constraint counts match
5. Load core pack and verify all data loads correctly
6. Run full test suite
7. If successful, replace old migrations
## Timeline Estimate
- **Analysis complete**: ✅ Done
- **Create consolidated migrations**: 2-3 hours
- **Testing and validation**: 1-2 hours
- **Documentation updates**: 30 minutes
- **Total**: ~4-6 hours
## Recommendation
**PROCEED** with consolidation. This is a textbook case for migration consolidation:
- Pre-production system ✅
- No user data ✅
- Clear improvement to codebase ✅
- Low risk ✅
- High maintainability gain ✅
The longer we wait, the harder this becomes. Do it now while it's straightforward.

195
docs/migrations/MIGRATION-BY-MIGRATION-CHANGES.md Normal file
View File

@@ -0,0 +1,195 @@
# Migration-by-Migration Change List
This document details exactly what needs to change in each migration file during consolidation.
## Files to Keep (with modifications)
### `20250101000001_initial_setup.sql`
**REMOVE**:
- `runtime_type_enum` type (lines ~42-46)
**KEEP**:
- All other enum types
- Extensions
- update_updated_column() function
### `20250101000002_core_tables.sql` → Rename to `20250101000003_pack_system.sql`
**MODIFY runtime table** (lines ~72-93):
```sql
CREATE TABLE runtime (
id BIGSERIAL PRIMARY KEY,
ref TEXT NOT NULL UNIQUE,
pack BIGINT REFERENCES pack(id) ON DELETE CASCADE,
pack_ref TEXT,
description TEXT,
-- REMOVE: runtime_type runtime_type_enum NOT NULL,
name TEXT NOT NULL,
distributions JSONB NOT NULL,
installation JSONB,
created TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated TIMESTAMPTZ NOT NULL DEFAULT NOW(),
-- REMOVE: CONSTRAINT runtime_ref_format CHECK (ref ~ '^[^.]+\.(action|sensor)\.[^.]+$')
CONSTRAINT runtime_ref_lowercase CHECK (ref = LOWER(ref))
);
```
**REMOVE indexes**:
- `idx_runtime_type`
- `idx_runtime_pack_type`
- `idx_runtime_type_created`
**ADD indexes**:
- `idx_runtime_name` (added in unify migration)
- `idx_runtime_verification` GIN index (added in unify migration)
### `20250101000003_event_system.sql` → Rename to `20250101000005_trigger_event_rule.sql`
**MODIFY trigger table** (add webhook columns from start):
```sql
CREATE TABLE trigger (
-- ... existing columns ...
-- ADD FROM START:
webhook_enabled BOOLEAN NOT NULL DEFAULT FALSE,
webhook_key VARCHAR(64) UNIQUE,
webhook_config JSONB DEFAULT '{}'::jsonb,
-- ... rest of columns ...
);
```
**MODIFY event table** (add rule tracking from start):
```sql
CREATE TABLE event (
-- ... existing columns ...
-- ADD FROM START:
rule BIGINT,
rule_ref TEXT,
-- ... rest of columns ...
);
-- ADD constraint:
ALTER TABLE event
ADD CONSTRAINT event_rule_fkey
FOREIGN KEY (rule) REFERENCES rule(id) ON DELETE SET NULL;
```
**MODIFY rule table** (add is_adhoc from start):
```sql
CREATE TABLE rule (
-- ... existing columns ...
-- ADD FROM START:
is_adhoc BOOLEAN DEFAULT false NOT NULL,
-- ... rest of columns ...
);
```
### `20250101000004_execution_system.sql` → Rename to `20250101000006_execution_system.sql`
**MODIFY execution table** (add workflow columns from start):
```sql
CREATE TABLE execution (
-- ... existing columns ...
-- ADD FROM START:
is_workflow BOOLEAN DEFAULT false NOT NULL,
workflow_def BIGINT REFERENCES workflow_definition(id) ON DELETE CASCADE,
workflow_task JSONB,
-- ... rest of columns ...
);
```
**REMOVE**:
- `workflow_task_execution` table (lines ~329-360)
- Don't create it at all
### `20250101000005_supporting_tables.sql` → Rename to `20250101000008_worker_notification.sql`
**MODIFY worker table** (add role from start):
First, ensure `worker_role_enum` is created in `20250101000001_initial_setup.sql`:
```sql
CREATE TYPE worker_role_enum AS ENUM ('action', 'sensor', 'hybrid');
```
Then in worker table:
```sql
CREATE TABLE worker (
-- ... existing columns ...
-- ADD FROM START:
worker_role worker_role_enum NOT NULL DEFAULT 'action',
-- ... rest of columns ...
);
```
**MODIFY action table** (add is_adhoc from start):
```sql
CREATE TABLE action (
-- ... existing columns ...
-- ADD FROM START:
is_adhoc BOOLEAN DEFAULT false NOT NULL,
-- ... rest of columns ...
);
```
**MODIFY sensor table** (add is_adhoc from start):
```sql
CREATE TABLE sensor (
-- ... existing columns ...
-- ADD FROM START:
is_adhoc BOOLEAN DEFAULT false NOT NULL,
-- ... rest of columns ...
);
```
## Files to Remove Entirely
1. `20260119000001_add_execution_notify_trigger.sql` - Consolidate into notify triggers migration
2. `20260120000001_add_webhook_support.sql` - Columns added in trigger table from start
3. `20260120000002_webhook_advanced_features.sql` - Functions consolidated, columns already in trigger table
4. `20260127000001_consolidate_webhook_config.sql` - Already consolidated in base migration
5. `20260127212500_consolidate_workflow_task_execution.sql` - Already in base execution table
6. `20260129000001_fix_webhook_function_overload.sql` - Use fixed functions from start
7. `20260129140130_add_is_adhoc_flag.sql` - Already in base tables
8. `20260129150000_add_event_notify_trigger.sql` - Consolidate into notify triggers migration
9. `20260130000001_add_rule_to_event.sql` - Already in event table
10. `20260131000001_add_worker_role.sql` - Already in worker table
11. `20260202000001_add_sensor_runtimes.sql` - Data now in YAML files
12. `20260203000001_unify_runtimes.sql` - Changes already applied to base tables
13. `20260203000003_add_rule_trigger_to_execution_notify.sql` - Consolidate into notify triggers migration
14. `20260204000001_add_enforcement_notify_trigger.sql` - Consolidate into notify triggers migration
15. `20260204000001_restore_webhook_functions.sql` - Use final functions from start
## New Files to Create
### `20250101000010_webhook_system.sql`
- Webhook-related tables
- FINAL versions of webhook functions (from 20260204000001_restore_webhook_functions.sql)
- No individual webhook columns (use webhook_config JSONB)
### `20250101000011_pack_environments.sql`
```sql
-- From 20260203000002_add_pack_environments.sql
CREATE TYPE pack_environment_status_enum AS ENUM (...);
CREATE TABLE pack_environment (...);
ALTER TABLE pack ADD COLUMN IF NOT EXISTS installers JSONB DEFAULT '[]'::jsonb;
```
### `20250101000013_notify_triggers.sql`
Consolidate ALL LISTEN/NOTIFY triggers from:
- 20260119000001 - execution
- 20260129150000 - event
- 20260203000003 - add rule to execution notify
- 20260204000001 - enforcement
Final notify_execution_change() function should include rule field from the start.
## Files to Keep As-Is
1. `20260120200000_add_pack_test_results.sql` → Rename to `20250101000012_pack_testing.sql`
2. `20260122000001_pack_installation_metadata.sql` → Merge into pack_system or keep separate
## Summary
**Original**: 22 migration files
**Consolidated**: ~13 migration files
**Removed**: 15 files (consolidation or data moved to YAML)
**Modified**: 5 files (add columns/constraints from start)
**New**: 3 files (consolidated functionality)

260
docs/migrations/migration-consolidation-plan.md Normal file
View File

@@ -0,0 +1,260 @@
# Migration Consolidation Plan
**Status**: Pre-production consolidation
**Date**: 2026-02-04
**Goal**: Consolidate migrations into a clean, minimal set before initial release
## Background
Since this project has no production deployments, we can freely consolidate migrations to create a cleaner initial state. This document identifies items that are created and then dropped/modified, so we can simplify the migration history.
## Issues Identified
### 1. Runtime Type Enum - Created Then Dropped
**Problem**: `runtime_type_enum` is created in the initial migration but dropped in a later migration.
- **Created**: `20250101000001_initial_setup.sql` (line 42)
- **Dropped**: `20260203000001_unify_runtimes.sql` (line 35)
- **Associated column**: `runtime.runtime_type` (also dropped)
- **Associated indexes**:
- `idx_runtime_type`
- `idx_runtime_pack_type`
- `idx_runtime_type_created`
- `idx_runtime_type_sensor`
**Action**: Remove enum type, column, and indexes from initial creation.
### 2. Runtime Table Constraints - Created Then Dropped
**Problem**: Runtime constraints are created with one format, then dropped and not recreated.
- **Created**: `20250101000002_core_tables.sql` (line 84)
- `runtime_ref_format CHECK (ref ~ '^[^.]+\.(action|sensor)\.[^.]+$')`
- Expected format: `pack.type.name`
- **Dropped**: `20260203000001_unify_runtimes.sql` (line 16)
- **New format**: `pack.name` (e.g., `core.python` instead of `core.action.python`)
**Action**: Create constraint with final format initially, or omit if not needed.
### 3. Webhook Columns - Added Then Consolidated
**Problem**: Individual webhook columns are added, then dropped in favor of a JSONB column.
**Added in `20260120000001_add_webhook_support.sql`**:
- `webhook_enabled BOOLEAN`
- `webhook_key VARCHAR(64)`
- `webhook_secret VARCHAR(128)`
**Added in `20260120000002_webhook_advanced_features.sql`**:
- `webhook_hmac_enabled BOOLEAN`
- `webhook_hmac_secret VARCHAR(128)`
- `webhook_hmac_algorithm VARCHAR(32)`
- `webhook_rate_limit_enabled BOOLEAN`
- `webhook_rate_limit_requests INTEGER`
- `webhook_rate_limit_window_seconds INTEGER`
- `webhook_ip_whitelist_enabled BOOLEAN`
- `webhook_ip_whitelist JSONB`
- `webhook_payload_size_limit_kb INTEGER`
**Consolidated in `20260127000001_consolidate_webhook_config.sql`**:
- All individual columns dropped
- Single `webhook_config JSONB` column added
**Action**: Add only `webhook_enabled`, `webhook_key`, and `webhook_config` in initial trigger table creation. Skip intermediate columns.
### 4. Runtime Data Insertions - Later Truncated
**Problem**: Runtime records are inserted via SQL, then truncated and moved to YAML files.
**Insertions in `20260202000001_add_sensor_runtimes.sql`**:
- 4 INSERT statements for sensor runtimes
- All records truncated in `20260203000001_unify_runtimes.sql`
**Insertions elsewhere**: Check if initial migrations insert any runtime data.
**Action**: Remove all runtime INSERT statements. Runtime data now loaded from YAML files in `packs/core/runtimes/`.
### 5. Workflow Task Execution Table - Created Then Dropped
**Problem**: Separate table created, then consolidated into execution table JSONB column.
- **Created**: `20250101000004_execution_system.sql` (line 329)
- `workflow_task_execution` table with multiple columns
- **Consolidated**: `20260127212500_consolidate_workflow_task_execution.sql`
- Table dropped
- `execution.workflow_task JSONB` column added instead
**Action**: Don't create `workflow_task_execution` table. Add `workflow_task JSONB` column to `execution` table in initial creation.
### 6. Execution Table Columns - Added for Workflows
**Problem**: Workflow-related columns added after initial table creation.
**Added in `20250101000004_execution_system.sql` (line 381)**:
- `is_workflow BOOLEAN DEFAULT false NOT NULL`
- `workflow_def BIGINT REFERENCES workflow_definition(id)`
**Action**: Include these columns in initial `execution` table creation (line ~60).
### 7. Is Adhoc Flag - Added Later
**Problem**: `is_adhoc` flag added to multiple tables after initial creation.
**Added in `20260129140130_add_is_adhoc_flag.sql`**:
- `action.is_adhoc`
- `sensor.is_adhoc`
- `rule.is_adhoc`
**Action**: Include `is_adhoc BOOLEAN DEFAULT false NOT NULL` in initial table definitions.
### 8. Event Table - Rule Reference Added Later
**Problem**: Rule tracking added to event table after initial creation.
**Added in `20260130000001_add_rule_to_event.sql`**:
- `event.rule BIGINT`
- `event.rule_ref TEXT`
- Foreign key constraint
**Action**: Include rule columns and constraint in initial event table creation.
### 9. Worker Role Column - Added Later
**Problem**: Worker role enum and column added after initial creation.
**Added in `20260131000001_add_worker_role.sql`**:
- `worker_role_enum` type
- `worker.worker_role` column
**Action**: Include enum type and column in initial worker table creation.
### 10. Pack Environments - Added Later
**Problem**: Pack installers column added after initial creation.
**Added in `20260203000002_add_pack_environments.sql`**:
- `pack_environment_status_enum` type
- `pack.installers JSONB` column
- `pack_environment` table
**Action**: Include in initial pack/environment setup.
### 11. Notify Triggers - Added Incrementally
**Problem**: PostgreSQL LISTEN/NOTIFY triggers added across multiple migrations.
**Migrations**:
- `20260119000001_add_execution_notify_trigger.sql` - execution events
- `20260129150000_add_event_notify_trigger.sql` - event creation
- `20260203000003_add_rule_trigger_to_execution_notify.sql` - add rule to execution notify
- `20260204000001_add_enforcement_notify_trigger.sql` - enforcement events
**Action**: Create all notify triggers in a single migration after table creation.
### 12. Webhook Functions - Created, Modified, Dropped, Restored
**Problem**: Webhook validation/processing functions have been rewritten multiple times.
**Timeline**:
- `20260120000001_add_webhook_support.sql` - Initial functions (4 created)
- `20260120000002_webhook_advanced_features.sql` - Advanced functions (7 created)
- `20260127000001_consolidate_webhook_config.sql` - Modified (2 dropped, 3 created)
- `20260129000001_fix_webhook_function_overload.sql` - Fixed overloading (3 dropped)
- `20260204000001_restore_webhook_functions.sql` - Restored (4 dropped, 3 created)
**Action**: Determine final set of webhook functions needed and create them once.
## Consolidation Strategy
### Phase 1: Analyze Dependencies
1. Map all foreign key relationships
2. Identify minimum viable table set
3. Document final schema for each table
### Phase 2: Create New Base Migrations
Create consolidated migrations:
1. **`00001_initial_setup.sql`** - Enums, extensions, base types
2. **`00002_identity_and_auth.sql`** - Identity, keys, auth tables
3. **`00003_pack_system.sql`** - Pack, runtime, action, sensor tables (with final schema)
4. **`00004_event_system.sql`** - Trigger, sensor, event, rule tables
5. **`00005_execution_system.sql`** - Execution, enforcement, inquiry, policy tables (including workflow columns)
6. **`00006_supporting_tables.sql`** - Worker, notification, artifact, etc.
7. **`00007_webhook_system.sql`** - Webhook tables, triggers, functions (final versions)
8. **`00008_notify_triggers.sql`** - All LISTEN/NOTIFY triggers
9. **`00009_pack_testing.sql`** - Pack test results table
### Phase 3: Validation
1. Test migrations on fresh database
2. Compare final schema to current production-like schema
3. Verify all indexes, constraints, triggers present
4. Load core pack and verify runtime data loads correctly
### Phase 4: Documentation
1. Update migration README
2. Document schema version
3. Add migration best practices
## Items to Remove Entirely
**Never created in consolidated migrations**:
1. `runtime_type_enum` type
2. `runtime.runtime_type` column
3. `runtime_ref_format` constraint (old format)
4. Indexes: `idx_runtime_type`, `idx_runtime_pack_type`, `idx_runtime_type_created`, `idx_runtime_type_sensor`
5. Individual webhook columns (9 columns that were later consolidated)
6. `idx_trigger_webhook_enabled` index
7. `workflow_task_execution` table
8. All runtime INSERT statements
9. Intermediate webhook function versions
## Items to Include From Start
**Must be in initial table creation**:
1. `execution.is_workflow` column
2. `execution.workflow_def` column
3. `execution.workflow_task` JSONB column
4. `action.is_adhoc` column
5. `sensor.is_adhoc` column
6. `rule.is_adhoc` column
7. `event.rule` and `event.rule_ref` columns
8. `worker_role_enum` type
9. `worker.worker_role` column
10. `trigger.webhook_enabled` column
11. `trigger.webhook_key` column
12. `trigger.webhook_config` JSONB column
13. `pack.installers` JSONB column
14. `pack_environment` table and `pack_environment_status_enum`
## Data Migration Notes
**Runtime Data**:
- Remove all INSERT statements from migrations
- Runtime records loaded from YAML files in `packs/core/runtimes/`
- Loader: `scripts/load_core_pack.py` or pack installation system
**Core Pack Data**:
- Check if any other core pack data is inserted via migrations
- Move to appropriate YAML files in `packs/core/`
## Next Steps
1. ✅ Create this consolidation plan
2. ⏳ Review with team
3. ⏳ Back up current migration directory
4. ⏳ Create consolidated migrations
5. ⏳ Test on fresh database
6. ⏳ Verify schema matches current state
7. ⏳ Replace old migrations
8. ⏳ Update documentation
## Rollback Plan
Keep copy of old migrations in `migrations.old/` directory until consolidated migrations are verified in development environment.