# Attune Database Migrations

This directory contains SQL migrations for the Attune automation platform database schema.

## Overview

Migrations are numbered and executed in order. Each migration file is named with a timestamp prefix to ensure proper ordering:

```
YYYYMMDDHHMMSS_description.sql
```

## Migration Files

The schema is organized into 5 logical migration files:

| File | Description |
|------|-------------|
| `20250101000001_initial_setup.sql` | Creates schema, service role, all enum types, and shared functions |
| `20250101000002_core_tables.sql` | Creates pack, runtime, worker, identity, permission_set, permission_assignment, policy, and key tables |
| `20250101000003_event_system.sql` | Creates trigger, sensor, event, and enforcement tables |
| `20250101000004_execution_system.sql` | Creates action, rule, execution, inquiry, workflow orchestration tables (workflow_definition, workflow_execution, workflow_task_execution), and workflow views |
| `20250101000005_supporting_tables.sql` | Creates notification, artifact, and queue_stats tables with performance indexes |

### Migration Dependencies

The migrations must be run in order due to foreign key dependencies:

1. **Initial Setup** - Foundation (schema, enums, functions)
2. **Core Tables** - Base entities (pack, runtime, worker, identity, permissions, policy, key)
3. **Event System** - Event monitoring (trigger, sensor, event, enforcement)
4. **Execution System** - Action execution (action, rule, execution, inquiry)
5. **Supporting Tables** - Auxiliary features (notification, artifact)

## Running Migrations

### Using SQLx CLI

```bash
# Install sqlx-cli if not already installed
cargo install sqlx-cli --no-default-features --features postgres

# Run all pending migrations
sqlx migrate run

# Check migration status
sqlx migrate info

# Revert last migration (if needed)
sqlx migrate revert
```

### Manual Execution

You can also run migrations manually using `psql`:

```bash
# Run all migrations in order
for file in migrations/202501*.sql; do
    psql -U postgres -d attune -f "$file"
done
```

Or individually:

```bash
psql -U postgres -d attune -f migrations/20250101000001_initial_setup.sql
psql -U postgres -d attune -f migrations/20250101000002_core_tables.sql
# ... etc
```

## Database Setup

### Prerequisites

1. PostgreSQL 14 or later installed
2. Create the database:

```bash
createdb attune
```

3. Set environment variable:

```bash
export DATABASE_URL="postgresql://postgres:postgres@localhost:5432/attune"
```

### Initial Setup

```bash
# Navigate to workspace root
cd /path/to/attune

# Run migrations
sqlx migrate run

# Verify tables were created
psql -U postgres -d attune -c "\dt attune.*"
```

## Schema Overview

The Attune schema includes 22 tables organized into logical groups:

### Core Tables (Migration 2)
- **pack**: Automation component bundles
- **runtime**: Execution environments (Python, Node.js, containers)
- **worker**: Execution workers
- **identity**: Users and service accounts
- **permission_set**: Permission groups (like roles)
- **permission_assignment**: Identity-permission links (many-to-many)
- **policy**: Execution policies (rate limiting, concurrency)
- **key**: Secure configuration and secrets storage

### Event System (Migration 3)
- **trigger**: Event type definitions
- **sensor**: Event monitors that watch for triggers
- **event**: Event instances (trigger firings)
- **enforcement**: Rule activation instances

### Execution System (Migration 4)
- **action**: Executable operations (can be workflows)
- **rule**: Trigger-to-action automation logic
- **execution**: Action execution instances (supports workflows)
- **inquiry**: Human-in-the-loop interactions (approvals, inputs)
- **workflow_definition**: YAML-based workflow definitions (composable action graphs)
- **workflow_execution**: Runtime state tracking for workflow executions
- **workflow_task_execution**: Individual task executions within workflows

### Supporting Tables (Migration 5)
- **notification**: Real-time system notifications (uses PostgreSQL LISTEN/NOTIFY)
- **artifact**: Execution outputs (files, logs, progress data)
- **queue_stats**: Real-time execution queue statistics for FIFO ordering

## Key Features

### Automatic Timestamps
All tables include `created` and `updated` timestamps that are automatically managed by the `update_updated_column()` trigger function.

### Reference Preservation
Tables use both ID foreign keys and `*_ref` text columns. The ref columns preserve string references even when the referenced entity is deleted, maintaining complete audit trails.

### Soft Deletes
Foreign keys strategically use:
- `ON DELETE CASCADE` - For dependent data that should be removed
- `ON DELETE SET NULL` - To preserve historical records while breaking the link

### Validation Constraints
- **Reference format validation** - Lowercase, specific patterns (e.g., `pack.name`)
- **Semantic version validation** - For pack versions
- **Ownership validation** - Custom trigger for key table ownership rules
- **Range checks** - Port numbers, positive thresholds, etc.

### Performance Optimization
- **B-tree indexes** - On frequently queried columns (IDs, refs, status, timestamps)
- **Partial indexes** - For filtered queries (e.g., `enabled = TRUE`)
- **GIN indexes** - On JSONB and array columns for fast containment queries
- **Composite indexes** - For common multi-column query patterns

### PostgreSQL Features
- **JSONB** - Flexible schema storage for configurations, payloads, results
- **Array types** - Multi-value fields (tags, parameters, dependencies)
- **Custom enum types** - Constrained string values with type safety
- **Triggers** - Data validation, timestamp management, notifications
- **pg_notify** - Real-time notifications via PostgreSQL's LISTEN/NOTIFY

## Service Role

The migrations create a `svc_attune` role with appropriate permissions. **Change the password in production:**

```sql
ALTER ROLE svc_attune WITH PASSWORD 'secure_password_here';
```

The default password is `attune_service_password` (only for development).

## Rollback Strategy

### Complete Reset

To completely reset the database:

```bash
# Drop and recreate
dropdb attune
createdb attune
sqlx migrate run
```

Or drop just the schema:

```sql
psql -U postgres -d attune -c "DROP SCHEMA attune CASCADE;"
```

Then re-run migrations.

### Individual Migration Revert

With SQLx CLI:

```bash
sqlx migrate revert
```

Or manually remove from tracking:

```sql
DELETE FROM _sqlx_migrations WHERE version = 20250101000001;
```

## Best Practices

1. **Never edit existing migrations** - Create new migrations to modify schema
2. **Test migrations** - Always test on a copy of production data first
3. **Backup before migrating** - Backup production database before applying migrations
4. **Review changes** - Review all migrations before applying to production
5. **Version control** - Keep migrations in version control (they are!)
6. **Document changes** - Add comments to complex migrations

## Development Workflow

1. Create new migration file with timestamp:
   ```bash
   touch migrations/$(date +%Y%m%d%H%M%S)_description.sql
   ```

2. Write migration SQL (follow existing patterns)

3. Test migration:
   ```bash
   sqlx migrate run
   ```

4. Verify changes:
   ```bash
   psql -U postgres -d attune
   \d+ attune.table_name
   ```

5. Commit to version control

## Production Deployment

1. **Backup** production database
2. **Review** all pending migrations
3. **Test** migrations on staging environment with production data copy
4. **Schedule** maintenance window if needed
5. **Apply** migrations:
   ```bash
   sqlx migrate run
   ```
6. **Verify** application functionality
7. **Monitor** for errors in logs

## Troubleshooting

### Migration already applied

If you need to re-run a migration:

```bash
# Remove from migration tracking (SQLx)
psql -U postgres -d attune -c "DELETE FROM _sqlx_migrations WHERE version = 20250101000001;"

# Then re-run
sqlx migrate run
```

### Permission denied

Ensure the PostgreSQL user has sufficient permissions:

```sql
GRANT ALL PRIVILEGES ON DATABASE attune TO postgres;
GRANT ALL PRIVILEGES ON SCHEMA attune TO postgres;
```

### Connection refused

Check PostgreSQL is running:

```bash
# Linux/macOS
pg_ctl status
sudo systemctl status postgresql

# Check if listening
psql -U postgres -c "SELECT version();"
```

### Foreign key constraint violations

Ensure migrations run in correct order. The consolidated migrations handle forward references correctly:
- Migration 2 creates tables with forward references (commented as such)
- Migration 3 and 4 add the foreign key constraints back

## Schema Diagram

```
┌─────────────┐
│    pack     │◄──┐
└─────────────┘   │
       ▲          │
       │          │
┌──────┴──────────┴──────┐
│ runtime │ trigger │ ... │  (Core entities reference pack)
└─────────┴─────────┴─────┘
       ▲          ▲
       │          │
┌──────┴──────┐  │
│   sensor    │──┘  (Sensors reference both runtime and trigger)
└─────────────┘
       │
       ▼
┌─────────────┐     ┌──────────────┐
│    event    │────►│ enforcement  │  (Events trigger enforcements)
└─────────────┘     └──────────────┘
                           │
                           ▼
                    ┌──────────────┐
                    │  execution   │  (Enforcements create executions)
                    └──────────────┘
```

## Workflow Orchestration

Migration 4 includes comprehensive workflow orchestration support:
- **workflow_definition**: Stores parsed YAML workflow definitions with tasks, variables, and transitions
- **workflow_execution**: Tracks runtime state including current/completed/failed tasks and variables
- **workflow_task_execution**: Individual task execution tracking with retry and timeout support
- **Action table extensions**: `is_workflow` and `workflow_def` columns link actions to workflows
- **Helper views**: Three views for querying workflow state (summary, task detail, action links)

## Queue Statistics

Migration 5 includes the queue_stats table for execution ordering:
- Tracks per-action queue length, active executions, and concurrency limits
- Enables FIFO queue management with database persistence
- Supports monitoring and API visibility of execution queues

## Additional Resources

- [SQLx Documentation](https://github.com/launchbadge/sqlx)
- [PostgreSQL Documentation](https://www.postgresql.org/docs/)
- [Attune Architecture Documentation](../docs/architecture.md)
- [Attune Data Model Documentation](../docs/data-model.md)