attune/docker/distributable/migrations/README.md

Attune Database Migrations

This directory contains SQL migrations for the Attune automation platform database schema.

Overview

Migrations are numbered and executed in order. Each migration file is named with a timestamp prefix to ensure proper ordering:

YYYYMMDDHHMMSS_description.sql

Migration Files

The schema is organized into 5 logical migration files:

File	Description
20250101000001_initial_setup.sql	Creates schema, service role, all enum types, and shared functions
20250101000002_core_tables.sql	Creates pack, runtime, worker, identity, permission_set, permission_assignment, policy, and key tables
20250101000003_event_system.sql	Creates trigger, sensor, event, and enforcement tables
20250101000004_execution_system.sql	Creates action, rule, execution, inquiry, workflow orchestration tables (workflow_definition, workflow_execution, workflow_task_execution), and workflow views
20250101000005_supporting_tables.sql	Creates notification, artifact, and queue_stats tables with performance indexes

Migration Dependencies

The migrations must be run in order due to foreign key dependencies:

1. Initial Setup - Foundation (schema, enums, functions)
2. Core Tables - Base entities (pack, runtime, worker, identity, permissions, policy, key)
3. Event System - Event monitoring (trigger, sensor, event, enforcement)
4. Execution System - Action execution (action, rule, execution, inquiry)
5. Supporting Tables - Auxiliary features (notification, artifact)

Running Migrations

Using SQLx CLI

# Install sqlx-cli if not already installed
cargo install sqlx-cli --no-default-features --features postgres

# Run all pending migrations
sqlx migrate run

# Check migration status
sqlx migrate info

# Revert last migration (if needed)
sqlx migrate revert

Manual Execution

You can also run migrations manually using psql:

# Run all migrations in order
for file in migrations/202501*.sql; do
    psql -U postgres -d attune -f "$file"
done

Or individually:

psql -U postgres -d attune -f migrations/20250101000001_initial_setup.sql
psql -U postgres -d attune -f migrations/20250101000002_core_tables.sql
# ... etc

Database Setup

Prerequisites

1. PostgreSQL 14 or later installed
2. Create the database:

createdb attune

3. Set environment variable:

export DATABASE_URL="postgresql://postgres:postgres@localhost:5432/attune"

Initial Setup

# Navigate to workspace root
cd /path/to/attune

# Run migrations
sqlx migrate run

# Verify tables were created
psql -U postgres -d attune -c "\dt attune.*"

Schema Overview

The Attune schema includes 22 tables organized into logical groups:

Core Tables (Migration 2)

• pack: Automation component bundles
• runtime: Execution environments (Python, Node.js, containers)
• worker: Execution workers
• identity: Users and service accounts
• permission_set: Permission groups (like roles)
• permission_assignment: Identity-permission links (many-to-many)
• policy: Execution policies (rate limiting, concurrency)
• key: Secure configuration and secrets storage

Event System (Migration 3)

• trigger: Event type definitions
• sensor: Event monitors that watch for triggers
• event: Event instances (trigger firings)
• enforcement: Rule activation instances

Execution System (Migration 4)

• action: Executable operations (can be workflows)
• rule: Trigger-to-action automation logic
• execution: Action execution instances (supports workflows)
• inquiry: Human-in-the-loop interactions (approvals, inputs)
• workflow_definition: YAML-based workflow definitions (composable action graphs)
• workflow_execution: Runtime state tracking for workflow executions
• workflow_task_execution: Individual task executions within workflows

Supporting Tables (Migration 5)

• notification: Real-time system notifications (uses PostgreSQL LISTEN/NOTIFY)
• artifact: Execution outputs (files, logs, progress data)
• queue_stats: Real-time execution queue statistics for FIFO ordering

Key Features

Automatic Timestamps

All tables include created and updated timestamps that are automatically managed by the update_updated_column() trigger function.

Reference Preservation

Tables use both ID foreign keys and *_ref text columns. The ref columns preserve string references even when the referenced entity is deleted, maintaining complete audit trails.

Soft Deletes

Foreign keys strategically use:

• ON DELETE CASCADE - For dependent data that should be removed
• ON DELETE SET NULL - To preserve historical records while breaking the link

Validation Constraints

• Reference format validation - Lowercase, specific patterns (e.g., pack.name)
• Semantic version validation - For pack versions
• Ownership validation - Custom trigger for key table ownership rules
• Range checks - Port numbers, positive thresholds, etc.

Performance Optimization

• B-tree indexes - On frequently queried columns (IDs, refs, status, timestamps)
• Partial indexes - For filtered queries (e.g., enabled = TRUE)
• GIN indexes - On JSONB and array columns for fast containment queries
• Composite indexes - For common multi-column query patterns

PostgreSQL Features

• JSONB - Flexible schema storage for configurations, payloads, results
• Array types - Multi-value fields (tags, parameters, dependencies)
• Custom enum types - Constrained string values with type safety
• Triggers - Data validation, timestamp management, notifications
• pg_notify - Real-time notifications via PostgreSQL's LISTEN/NOTIFY

Service Role

The migrations create a svc_attune role with appropriate permissions. Change the password in production:

ALTER ROLE svc_attune WITH PASSWORD 'secure_password_here';

The default password is attune_service_password (only for development).

Rollback Strategy

Complete Reset

To completely reset the database:

# Drop and recreate
dropdb attune
createdb attune
sqlx migrate run

Or drop just the schema:

psql -U postgres -d attune -c "DROP SCHEMA attune CASCADE;"

Then re-run migrations.

Individual Migration Revert

With SQLx CLI:

sqlx migrate revert

Or manually remove from tracking:

DELETE FROM _sqlx_migrations WHERE version = 20250101000001;

Best Practices

1. Never edit existing migrations - Create new migrations to modify schema
2. Test migrations - Always test on a copy of production data first
3. Backup before migrating - Backup production database before applying migrations
4. Review changes - Review all migrations before applying to production
5. Version control - Keep migrations in version control (they are!)
6. Document changes - Add comments to complex migrations

Development Workflow

1. Create new migration file with timestamp:

   touch migrations/$(date +%Y%m%d%H%M%S)_description.sql
   

2. Write migration SQL (follow existing patterns)

3. Test migration:

   sqlx migrate run
   

4. Verify changes:

   psql -U postgres -d attune
   \d+ attune.table_name
   

5. Commit to version control

Production Deployment

1. Backup production database
2. Review all pending migrations
3. Test migrations on staging environment with production data copy
4. Schedule maintenance window if needed
5. Apply migrations:

   sqlx migrate run
   

6. Verify application functionality
7. Monitor for errors in logs

Troubleshooting

Migration already applied

If you need to re-run a migration:

# Remove from migration tracking (SQLx)
psql -U postgres -d attune -c "DELETE FROM _sqlx_migrations WHERE version = 20250101000001;"

# Then re-run
sqlx migrate run

Permission denied

Ensure the PostgreSQL user has sufficient permissions:

GRANT ALL PRIVILEGES ON DATABASE attune TO postgres;
GRANT ALL PRIVILEGES ON SCHEMA attune TO postgres;

Connection refused

Check PostgreSQL is running:

# Linux/macOS
pg_ctl status
sudo systemctl status postgresql

# Check if listening
psql -U postgres -c "SELECT version();"

Foreign key constraint violations

Ensure migrations run in correct order. The consolidated migrations handle forward references correctly:

• Migration 2 creates tables with forward references (commented as such)
• Migration 3 and 4 add the foreign key constraints back

Schema Diagram

┌─────────────┐
│    pack     │◄──┐
└─────────────┘   │
       ▲          │
       │          │
┌──────┴──────────┴──────┐
│ runtime │ trigger │ ... │  (Core entities reference pack)
└─────────┴─────────┴─────┘
       ▲          ▲
       │          │
┌──────┴──────┐  │
│   sensor    │──┘  (Sensors reference both runtime and trigger)
└─────────────┘
       │
       ▼
┌─────────────┐     ┌──────────────┐
│    event    │────►│ enforcement  │  (Events trigger enforcements)
└─────────────┘     └──────────────┘
                           │
                           ▼
                    ┌──────────────┐
                    │  execution   │  (Enforcements create executions)
                    └──────────────┘

Workflow Orchestration

Migration 4 includes comprehensive workflow orchestration support:

• workflow_definition: Stores parsed YAML workflow definitions with tasks, variables, and transitions
• workflow_execution: Tracks runtime state including current/completed/failed tasks and variables
• workflow_task_execution: Individual task execution tracking with retry and timeout support
• Action table extensions: is_workflow and workflow_def columns link actions to workflows
• Helper views: Three views for querying workflow state (summary, task detail, action links)

Queue Statistics

Migration 5 includes the queue_stats table for execution ordering:

• Tracks per-action queue length, active executions, and concurrency limits
• Enables FIFO queue management with database persistence
• Supports monitoring and API visibility of execution queues

Additional Resources

• SQLx Documentation
• PostgreSQL Documentation
• Attune Architecture Documentation
• Attune Data Model Documentation