re-uploading work
This commit is contained in:
348
migrations/README.md
Normal file
348
migrations/README.md
Normal file
@@ -0,0 +1,348 @@
|
||||
# Attune Database Migrations
|
||||
|
||||
This directory contains SQL migrations for the Attune automation platform database schema.
|
||||
|
||||
## Overview
|
||||
|
||||
Migrations are numbered and executed in order. Each migration file is named with a timestamp prefix to ensure proper ordering:
|
||||
|
||||
```
|
||||
YYYYMMDDHHMMSS_description.sql
|
||||
```
|
||||
|
||||
## Migration Files
|
||||
|
||||
The schema is organized into 5 logical migration files:
|
||||
|
||||
| File | Description |
|
||||
|------|-------------|
|
||||
| `20250101000001_initial_setup.sql` | Creates schema, service role, all enum types, and shared functions |
|
||||
| `20250101000002_core_tables.sql` | Creates pack, runtime, worker, identity, permission_set, permission_assignment, policy, and key tables |
|
||||
| `20250101000003_event_system.sql` | Creates trigger, sensor, event, and enforcement tables |
|
||||
| `20250101000004_execution_system.sql` | Creates action, rule, execution, inquiry, workflow orchestration tables (workflow_definition, workflow_execution, workflow_task_execution), and workflow views |
|
||||
| `20250101000005_supporting_tables.sql` | Creates notification, artifact, and queue_stats tables with performance indexes |
|
||||
|
||||
### Migration Dependencies
|
||||
|
||||
The migrations must be run in order due to foreign key dependencies:
|
||||
|
||||
1. **Initial Setup** - Foundation (schema, enums, functions)
|
||||
2. **Core Tables** - Base entities (pack, runtime, worker, identity, permissions, policy, key)
|
||||
3. **Event System** - Event monitoring (trigger, sensor, event, enforcement)
|
||||
4. **Execution System** - Action execution (action, rule, execution, inquiry)
|
||||
5. **Supporting Tables** - Auxiliary features (notification, artifact)
|
||||
|
||||
## Running Migrations
|
||||
|
||||
### Using SQLx CLI
|
||||
|
||||
```bash
|
||||
# Install sqlx-cli if not already installed
|
||||
cargo install sqlx-cli --no-default-features --features postgres
|
||||
|
||||
# Run all pending migrations
|
||||
sqlx migrate run
|
||||
|
||||
# Check migration status
|
||||
sqlx migrate info
|
||||
|
||||
# Revert last migration (if needed)
|
||||
sqlx migrate revert
|
||||
```
|
||||
|
||||
### Manual Execution
|
||||
|
||||
You can also run migrations manually using `psql`:
|
||||
|
||||
```bash
|
||||
# Run all migrations in order
|
||||
for file in migrations/202501*.sql; do
|
||||
psql -U postgres -d attune -f "$file"
|
||||
done
|
||||
```
|
||||
|
||||
Or individually:
|
||||
|
||||
```bash
|
||||
psql -U postgres -d attune -f migrations/20250101000001_initial_setup.sql
|
||||
psql -U postgres -d attune -f migrations/20250101000002_core_tables.sql
|
||||
# ... etc
|
||||
```
|
||||
|
||||
## Database Setup
|
||||
|
||||
### Prerequisites
|
||||
|
||||
1. PostgreSQL 14 or later installed
|
||||
2. Create the database:
|
||||
|
||||
```bash
|
||||
createdb attune
|
||||
```
|
||||
|
||||
3. Set environment variable:
|
||||
|
||||
```bash
|
||||
export DATABASE_URL="postgresql://postgres:postgres@localhost:5432/attune"
|
||||
```
|
||||
|
||||
### Initial Setup
|
||||
|
||||
```bash
|
||||
# Navigate to workspace root
|
||||
cd /path/to/attune
|
||||
|
||||
# Run migrations
|
||||
sqlx migrate run
|
||||
|
||||
# Verify tables were created
|
||||
psql -U postgres -d attune -c "\dt attune.*"
|
||||
```
|
||||
|
||||
## Schema Overview
|
||||
|
||||
The Attune schema includes 22 tables organized into logical groups:
|
||||
|
||||
### Core Tables (Migration 2)
|
||||
- **pack**: Automation component bundles
|
||||
- **runtime**: Execution environments (Python, Node.js, containers)
|
||||
- **worker**: Execution workers
|
||||
- **identity**: Users and service accounts
|
||||
- **permission_set**: Permission groups (like roles)
|
||||
- **permission_assignment**: Identity-permission links (many-to-many)
|
||||
- **policy**: Execution policies (rate limiting, concurrency)
|
||||
- **key**: Secure configuration and secrets storage
|
||||
|
||||
### Event System (Migration 3)
|
||||
- **trigger**: Event type definitions
|
||||
- **sensor**: Event monitors that watch for triggers
|
||||
- **event**: Event instances (trigger firings)
|
||||
- **enforcement**: Rule activation instances
|
||||
|
||||
### Execution System (Migration 4)
|
||||
- **action**: Executable operations (can be workflows)
|
||||
- **rule**: Trigger-to-action automation logic
|
||||
- **execution**: Action execution instances (supports workflows)
|
||||
- **inquiry**: Human-in-the-loop interactions (approvals, inputs)
|
||||
- **workflow_definition**: YAML-based workflow definitions (composable action graphs)
|
||||
- **workflow_execution**: Runtime state tracking for workflow executions
|
||||
- **workflow_task_execution**: Individual task executions within workflows
|
||||
|
||||
### Supporting Tables (Migration 5)
|
||||
- **notification**: Real-time system notifications (uses PostgreSQL LISTEN/NOTIFY)
|
||||
- **artifact**: Execution outputs (files, logs, progress data)
|
||||
- **queue_stats**: Real-time execution queue statistics for FIFO ordering
|
||||
|
||||
## Key Features
|
||||
|
||||
### Automatic Timestamps
|
||||
All tables include `created` and `updated` timestamps that are automatically managed by the `update_updated_column()` trigger function.
|
||||
|
||||
### Reference Preservation
|
||||
Tables use both ID foreign keys and `*_ref` text columns. The ref columns preserve string references even when the referenced entity is deleted, maintaining complete audit trails.
|
||||
|
||||
### Soft Deletes
|
||||
Foreign keys strategically use:
|
||||
- `ON DELETE CASCADE` - For dependent data that should be removed
|
||||
- `ON DELETE SET NULL` - To preserve historical records while breaking the link
|
||||
|
||||
### Validation Constraints
|
||||
- **Reference format validation** - Lowercase, specific patterns (e.g., `pack.name`)
|
||||
- **Semantic version validation** - For pack versions
|
||||
- **Ownership validation** - Custom trigger for key table ownership rules
|
||||
- **Range checks** - Port numbers, positive thresholds, etc.
|
||||
|
||||
### Performance Optimization
|
||||
- **B-tree indexes** - On frequently queried columns (IDs, refs, status, timestamps)
|
||||
- **Partial indexes** - For filtered queries (e.g., `enabled = TRUE`)
|
||||
- **GIN indexes** - On JSONB and array columns for fast containment queries
|
||||
- **Composite indexes** - For common multi-column query patterns
|
||||
|
||||
### PostgreSQL Features
|
||||
- **JSONB** - Flexible schema storage for configurations, payloads, results
|
||||
- **Array types** - Multi-value fields (tags, parameters, dependencies)
|
||||
- **Custom enum types** - Constrained string values with type safety
|
||||
- **Triggers** - Data validation, timestamp management, notifications
|
||||
- **pg_notify** - Real-time notifications via PostgreSQL's LISTEN/NOTIFY
|
||||
|
||||
## Service Role
|
||||
|
||||
The migrations create a `svc_attune` role with appropriate permissions. **Change the password in production:**
|
||||
|
||||
```sql
|
||||
ALTER ROLE svc_attune WITH PASSWORD 'secure_password_here';
|
||||
```
|
||||
|
||||
The default password is `attune_service_password` (only for development).
|
||||
|
||||
## Rollback Strategy
|
||||
|
||||
### Complete Reset
|
||||
|
||||
To completely reset the database:
|
||||
|
||||
```bash
|
||||
# Drop and recreate
|
||||
dropdb attune
|
||||
createdb attune
|
||||
sqlx migrate run
|
||||
```
|
||||
|
||||
Or drop just the schema:
|
||||
|
||||
```sql
|
||||
psql -U postgres -d attune -c "DROP SCHEMA attune CASCADE;"
|
||||
```
|
||||
|
||||
Then re-run migrations.
|
||||
|
||||
### Individual Migration Revert
|
||||
|
||||
With SQLx CLI:
|
||||
|
||||
```bash
|
||||
sqlx migrate revert
|
||||
```
|
||||
|
||||
Or manually remove from tracking:
|
||||
|
||||
```sql
|
||||
DELETE FROM _sqlx_migrations WHERE version = 20250101000001;
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Never edit existing migrations** - Create new migrations to modify schema
|
||||
2. **Test migrations** - Always test on a copy of production data first
|
||||
3. **Backup before migrating** - Backup production database before applying migrations
|
||||
4. **Review changes** - Review all migrations before applying to production
|
||||
5. **Version control** - Keep migrations in version control (they are!)
|
||||
6. **Document changes** - Add comments to complex migrations
|
||||
|
||||
## Development Workflow
|
||||
|
||||
1. Create new migration file with timestamp:
|
||||
```bash
|
||||
touch migrations/$(date +%Y%m%d%H%M%S)_description.sql
|
||||
```
|
||||
|
||||
2. Write migration SQL (follow existing patterns)
|
||||
|
||||
3. Test migration:
|
||||
```bash
|
||||
sqlx migrate run
|
||||
```
|
||||
|
||||
4. Verify changes:
|
||||
```bash
|
||||
psql -U postgres -d attune
|
||||
\d+ attune.table_name
|
||||
```
|
||||
|
||||
5. Commit to version control
|
||||
|
||||
## Production Deployment
|
||||
|
||||
1. **Backup** production database
|
||||
2. **Review** all pending migrations
|
||||
3. **Test** migrations on staging environment with production data copy
|
||||
4. **Schedule** maintenance window if needed
|
||||
5. **Apply** migrations:
|
||||
```bash
|
||||
sqlx migrate run
|
||||
```
|
||||
6. **Verify** application functionality
|
||||
7. **Monitor** for errors in logs
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Migration already applied
|
||||
|
||||
If you need to re-run a migration:
|
||||
|
||||
```bash
|
||||
# Remove from migration tracking (SQLx)
|
||||
psql -U postgres -d attune -c "DELETE FROM _sqlx_migrations WHERE version = 20250101000001;"
|
||||
|
||||
# Then re-run
|
||||
sqlx migrate run
|
||||
```
|
||||
|
||||
### Permission denied
|
||||
|
||||
Ensure the PostgreSQL user has sufficient permissions:
|
||||
|
||||
```sql
|
||||
GRANT ALL PRIVILEGES ON DATABASE attune TO postgres;
|
||||
GRANT ALL PRIVILEGES ON SCHEMA attune TO postgres;
|
||||
```
|
||||
|
||||
### Connection refused
|
||||
|
||||
Check PostgreSQL is running:
|
||||
|
||||
```bash
|
||||
# Linux/macOS
|
||||
pg_ctl status
|
||||
sudo systemctl status postgresql
|
||||
|
||||
# Check if listening
|
||||
psql -U postgres -c "SELECT version();"
|
||||
```
|
||||
|
||||
### Foreign key constraint violations
|
||||
|
||||
Ensure migrations run in correct order. The consolidated migrations handle forward references correctly:
|
||||
- Migration 2 creates tables with forward references (commented as such)
|
||||
- Migration 3 and 4 add the foreign key constraints back
|
||||
|
||||
## Schema Diagram
|
||||
|
||||
```
|
||||
┌─────────────┐
|
||||
│ pack │◄──┐
|
||||
└─────────────┘ │
|
||||
▲ │
|
||||
│ │
|
||||
┌──────┴──────────┴──────┐
|
||||
│ runtime │ trigger │ ... │ (Core entities reference pack)
|
||||
└─────────┴─────────┴─────┘
|
||||
▲ ▲
|
||||
│ │
|
||||
┌──────┴──────┐ │
|
||||
│ sensor │──┘ (Sensors reference both runtime and trigger)
|
||||
└─────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────┐ ┌──────────────┐
|
||||
│ event │────►│ enforcement │ (Events trigger enforcements)
|
||||
└─────────────┘ └──────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────┐
|
||||
│ execution │ (Enforcements create executions)
|
||||
└──────────────┘
|
||||
```
|
||||
|
||||
## Workflow Orchestration
|
||||
|
||||
Migration 4 includes comprehensive workflow orchestration support:
|
||||
- **workflow_definition**: Stores parsed YAML workflow definitions with tasks, variables, and transitions
|
||||
- **workflow_execution**: Tracks runtime state including current/completed/failed tasks and variables
|
||||
- **workflow_task_execution**: Individual task execution tracking with retry and timeout support
|
||||
- **Action table extensions**: `is_workflow` and `workflow_def` columns link actions to workflows
|
||||
- **Helper views**: Three views for querying workflow state (summary, task detail, action links)
|
||||
|
||||
## Queue Statistics
|
||||
|
||||
Migration 5 includes the queue_stats table for execution ordering:
|
||||
- Tracks per-action queue length, active executions, and concurrency limits
|
||||
- Enables FIFO queue management with database persistence
|
||||
- Supports monitoring and API visibility of execution queues
|
||||
|
||||
## Additional Resources
|
||||
|
||||
- [SQLx Documentation](https://github.com/launchbadge/sqlx)
|
||||
- [PostgreSQL Documentation](https://www.postgresql.org/docs/)
|
||||
- [Attune Architecture Documentation](../docs/architecture.md)
|
||||
- [Attune Data Model Documentation](../docs/data-model.md)
|
||||
Reference in New Issue
Block a user