distributable, please

docker/distributable/migrations/README.md

@@ -0,0 +1,348 @@
+# Attune Database Migrations
+
+This directory contains SQL migrations for the Attune automation platform database schema.
+
+## Overview
+
+Migrations are numbered and executed in order. Each migration file is named with a timestamp prefix to ensure proper ordering:
+
+```
+YYYYMMDDHHMMSS_description.sql
+```
+
+## Migration Files
+
+The schema is organized into 5 logical migration files:
+
+| File | Description |
+|------|-------------|
+| `20250101000001_initial_setup.sql` | Creates schema, service role, all enum types, and shared functions |
+| `20250101000002_core_tables.sql` | Creates pack, runtime, worker, identity, permission_set, permission_assignment, policy, and key tables |
+| `20250101000003_event_system.sql` | Creates trigger, sensor, event, and enforcement tables |
+| `20250101000004_execution_system.sql` | Creates action, rule, execution, inquiry, workflow orchestration tables (workflow_definition, workflow_execution, workflow_task_execution), and workflow views |
+| `20250101000005_supporting_tables.sql` | Creates notification, artifact, and queue_stats tables with performance indexes |
+
+### Migration Dependencies
+
+The migrations must be run in order due to foreign key dependencies:
+
+1. **Initial Setup** - Foundation (schema, enums, functions)
+2. **Core Tables** - Base entities (pack, runtime, worker, identity, permissions, policy, key)
+3. **Event System** - Event monitoring (trigger, sensor, event, enforcement)
+4. **Execution System** - Action execution (action, rule, execution, inquiry)
+5. **Supporting Tables** - Auxiliary features (notification, artifact)
+
+## Running Migrations
+
+### Using SQLx CLI
+
+```bash
+# Install sqlx-cli if not already installed
+cargo install sqlx-cli --no-default-features --features postgres
+
+# Run all pending migrations
+sqlx migrate run
+
+# Check migration status
+sqlx migrate info
+
+# Revert last migration (if needed)
+sqlx migrate revert
+```
+
+### Manual Execution
+
+You can also run migrations manually using `psql`:
+
+```bash
+# Run all migrations in order
+for file in migrations/202501*.sql; do
+    psql -U postgres -d attune -f "$file"
+done
+```
+
+Or individually:
+
+```bash
+psql -U postgres -d attune -f migrations/20250101000001_initial_setup.sql
+psql -U postgres -d attune -f migrations/20250101000002_core_tables.sql
+# ... etc
+```
+
+## Database Setup
+
+### Prerequisites
+
+1. PostgreSQL 14 or later installed
+2. Create the database:
+
+```bash
+createdb attune
+```
+
+3. Set environment variable:
+
+```bash
+export DATABASE_URL="postgresql://postgres:postgres@localhost:5432/attune"
+```
+
+### Initial Setup
+
+```bash
+# Navigate to workspace root
+cd /path/to/attune
+
+# Run migrations
+sqlx migrate run
+
+# Verify tables were created
+psql -U postgres -d attune -c "\dt attune.*"
+```
+
+## Schema Overview
+
+The Attune schema includes 22 tables organized into logical groups:
+
+### Core Tables (Migration 2)
+- **pack**: Automation component bundles
+- **runtime**: Execution environments (Python, Node.js, containers)
+- **worker**: Execution workers
+- **identity**: Users and service accounts
+- **permission_set**: Permission groups (like roles)
+- **permission_assignment**: Identity-permission links (many-to-many)
+- **policy**: Execution policies (rate limiting, concurrency)
+- **key**: Secure configuration and secrets storage
+
+### Event System (Migration 3)
+- **trigger**: Event type definitions
+- **sensor**: Event monitors that watch for triggers
+- **event**: Event instances (trigger firings)
+- **enforcement**: Rule activation instances
+
+### Execution System (Migration 4)
+- **action**: Executable operations (can be workflows)
+- **rule**: Trigger-to-action automation logic
+- **execution**: Action execution instances (supports workflows)
+- **inquiry**: Human-in-the-loop interactions (approvals, inputs)
+- **workflow_definition**: YAML-based workflow definitions (composable action graphs)
+- **workflow_execution**: Runtime state tracking for workflow executions
+- **workflow_task_execution**: Individual task executions within workflows
+
+### Supporting Tables (Migration 5)
+- **notification**: Real-time system notifications (uses PostgreSQL LISTEN/NOTIFY)
+- **artifact**: Execution outputs (files, logs, progress data)
+- **queue_stats**: Real-time execution queue statistics for FIFO ordering
+
+## Key Features
+
+### Automatic Timestamps
+All tables include `created` and `updated` timestamps that are automatically managed by the `update_updated_column()` trigger function.
+
+### Reference Preservation
+Tables use both ID foreign keys and `*_ref` text columns. The ref columns preserve string references even when the referenced entity is deleted, maintaining complete audit trails.
+
+### Soft Deletes
+Foreign keys strategically use:
+- `ON DELETE CASCADE` - For dependent data that should be removed
+- `ON DELETE SET NULL` - To preserve historical records while breaking the link
+
+### Validation Constraints
+- **Reference format validation** - Lowercase, specific patterns (e.g., `pack.name`)
+- **Semantic version validation** - For pack versions
+- **Ownership validation** - Custom trigger for key table ownership rules
+- **Range checks** - Port numbers, positive thresholds, etc.
+
+### Performance Optimization
+- **B-tree indexes** - On frequently queried columns (IDs, refs, status, timestamps)
+- **Partial indexes** - For filtered queries (e.g., `enabled = TRUE`)
+- **GIN indexes** - On JSONB and array columns for fast containment queries
+- **Composite indexes** - For common multi-column query patterns
+
+### PostgreSQL Features
+- **JSONB** - Flexible schema storage for configurations, payloads, results
+- **Array types** - Multi-value fields (tags, parameters, dependencies)
+- **Custom enum types** - Constrained string values with type safety
+- **Triggers** - Data validation, timestamp management, notifications
+- **pg_notify** - Real-time notifications via PostgreSQL's LISTEN/NOTIFY
+
+## Service Role
+
+The migrations create a `svc_attune` role with appropriate permissions. **Change the password in production:**
+
+```sql
+ALTER ROLE svc_attune WITH PASSWORD 'secure_password_here';
+```
+
+The default password is `attune_service_password` (only for development).
+
+## Rollback Strategy
+
+### Complete Reset
+
+To completely reset the database:
+
+```bash
+# Drop and recreate
+dropdb attune
+createdb attune
+sqlx migrate run
+```
+
+Or drop just the schema:
+
+```sql
+psql -U postgres -d attune -c "DROP SCHEMA attune CASCADE;"
+```
+
+Then re-run migrations.
+
+### Individual Migration Revert
+
+With SQLx CLI:
+
+```bash
+sqlx migrate revert
+```
+
+Or manually remove from tracking:
+
+```sql
+DELETE FROM _sqlx_migrations WHERE version = 20250101000001;
+```
+
+## Best Practices
+
+1. **Never edit existing migrations** - Create new migrations to modify schema
+2. **Test migrations** - Always test on a copy of production data first
+3. **Backup before migrating** - Backup production database before applying migrations
+4. **Review changes** - Review all migrations before applying to production
+5. **Version control** - Keep migrations in version control (they are!)
+6. **Document changes** - Add comments to complex migrations
+
+## Development Workflow
+
+1. Create new migration file with timestamp:
+   ```bash
+   touch migrations/$(date +%Y%m%d%H%M%S)_description.sql
+   ```
+
+2. Write migration SQL (follow existing patterns)
+
+3. Test migration:
+   ```bash
+   sqlx migrate run
+   ```
+
+4. Verify changes:
+   ```bash
+   psql -U postgres -d attune
+   \d+ attune.table_name
+   ```
+
+5. Commit to version control
+
+## Production Deployment
+
+1. **Backup** production database
+2. **Review** all pending migrations
+3. **Test** migrations on staging environment with production data copy
+4. **Schedule** maintenance window if needed
+5. **Apply** migrations:
+   ```bash
+   sqlx migrate run
+   ```
+6. **Verify** application functionality
+7. **Monitor** for errors in logs
+
+## Troubleshooting
+
+### Migration already applied
+
+If you need to re-run a migration:
+
+```bash
+# Remove from migration tracking (SQLx)
+psql -U postgres -d attune -c "DELETE FROM _sqlx_migrations WHERE version = 20250101000001;"
+
+# Then re-run
+sqlx migrate run
+```
+
+### Permission denied
+
+Ensure the PostgreSQL user has sufficient permissions:
+
+```sql
+GRANT ALL PRIVILEGES ON DATABASE attune TO postgres;
+GRANT ALL PRIVILEGES ON SCHEMA attune TO postgres;
+```
+
+### Connection refused
+
+Check PostgreSQL is running:
+
+```bash
+# Linux/macOS
+pg_ctl status
+sudo systemctl status postgresql
+
+# Check if listening
+psql -U postgres -c "SELECT version();"
+```
+
+### Foreign key constraint violations
+
+Ensure migrations run in correct order. The consolidated migrations handle forward references correctly:
+- Migration 2 creates tables with forward references (commented as such)
+- Migration 3 and 4 add the foreign key constraints back
+
+## Schema Diagram
+
+```
+┌─────────────┐
+│    pack     │◄──┐
+└─────────────┘   │
+       ▲          │
+       │          │
+┌──────┴──────────┴──────┐
+│ runtime │ trigger │ ... │  (Core entities reference pack)
+└─────────┴─────────┴─────┘
+       ▲          ▲
+       │          │
+┌──────┴──────┐  │
+│   sensor    │──┘  (Sensors reference both runtime and trigger)
+└─────────────┘
+       │
+       ▼
+┌─────────────┐     ┌──────────────┐
+│    event    │────►│ enforcement  │  (Events trigger enforcements)
+└─────────────┘     └──────────────┘
+                           │
+                           ▼
+                    ┌──────────────┐
+                    │  execution   │  (Enforcements create executions)
+                    └──────────────┘
+```
+
+## Workflow Orchestration
+
+Migration 4 includes comprehensive workflow orchestration support:
+- **workflow_definition**: Stores parsed YAML workflow definitions with tasks, variables, and transitions
+- **workflow_execution**: Tracks runtime state including current/completed/failed tasks and variables
+- **workflow_task_execution**: Individual task execution tracking with retry and timeout support
+- **Action table extensions**: `is_workflow` and `workflow_def` columns link actions to workflows
+- **Helper views**: Three views for querying workflow state (summary, task detail, action links)
+
+## Queue Statistics
+
+Migration 5 includes the queue_stats table for execution ordering:
+- Tracks per-action queue length, active executions, and concurrency limits
+- Enables FIFO queue management with database persistence
+- Supports monitoring and API visibility of execution queues
+
+## Additional Resources
+
+- [SQLx Documentation](https://github.com/launchbadge/sqlx)
+- [PostgreSQL Documentation](https://www.postgresql.org/docs/)
+- [Attune Architecture Documentation](../docs/architecture.md)
+- [Attune Data Model Documentation](../docs/data-model.md)