attune/work-summary/migrations/2026-01-27-workflow-migration.md

Work Summary: Workflow Database Migration

Date: 2026-01-27 Session Focus: Phase 1.1 - Database migration for workflow orchestration Status: Complete ✅

---

Objective

Implement the database schema changes required for workflow orchestration support in Attune, including 3 new tables and modifications to the existing action table.

---

Accomplishments

1. Migration File Created ✅

File: migrations/20250127000002_workflow_orchestration.sql (268 lines)

Created comprehensive migration including:

• 3 new tables with full schema definitions
• 2 new columns on existing action table
• 12 indexes for query optimization
• 3 triggers for timestamp management
• 3 helper views for querying
• Extensive comments and documentation

2. Database Schema Additions ✅

New Tables

1. attune.workflow_definition

• Stores parsed workflow YAML as JSON
• Links to pack table
• Contains parameter and output schemas
• Tracks version and metadata
• Columns: 14 (id, ref, pack, pack_ref, label, description, version, param_schema, out_schema, definition, tags, enabled, created, updated)

2. attune.workflow_execution

• Tracks runtime state of workflow executions
• Stores variable context (JSONB)
• Maintains task completion tracking (text arrays)
• Links to parent execution
• Supports pause/resume functionality
• Columns: 13 (id, execution, workflow_def, current_tasks, completed_tasks, failed_tasks, skipped_tasks, variables, task_graph, status, error_message, paused, pause_reason, created, updated)

3. attune.workflow_task_execution

• Individual task execution records
• Supports iteration (task_index, task_batch)
• Tracks retry attempts and timeouts
• Stores results and errors
• Columns: 16 (id, workflow_execution, execution, task_name, task_index, task_batch, status, started_at, completed_at, duration_ms, result, error, retry_count, max_retries, next_retry_at, timeout_seconds, timed_out, created, updated)

Modified Tables

attune.action - Added 2 columns:

• is_workflow BOOLEAN - Flags workflow actions
• workflow_def BIGINT - Foreign key to workflow_definition

3. Indexes Created ✅

Total: 12 indexes for performance optimization

• 4 indexes on workflow_definition (pack, enabled, ref, tags)
• 4 indexes on workflow_execution (execution, workflow_def, status, paused)
• 6 indexes on workflow_task_execution (workflow, execution, status, task_name, retry, timeout)
• 2 indexes on action (is_workflow, workflow_def)

4. Helper Views Created ✅

1. workflow_execution_summary

• Aggregates workflow execution state with task counts
• Joins workflow_definition for metadata
• Useful for monitoring dashboards

2. workflow_task_detail

• Detailed view of individual task executions
• Includes workflow context
• Useful for debugging and tracing

3. workflow_action_link

• Links workflow definitions to action records
• Shows synthetic action created for each workflow
• Useful for pack management

5. Migration Applied Successfully ✅

$ sqlx migrate run
Applied 20250127000002/migrate workflow orchestration (20.900297ms)

6. Schema Verified ✅

Verified all tables, columns, indexes, triggers, and views created correctly:

• ✅ 3 tables created in attune schema
• ✅ 14 columns in workflow_definition
• ✅ 13 columns in workflow_execution
• ✅ 16 columns in workflow_task_execution
• ✅ 2 columns added to action table
• ✅ 12 indexes created
• ✅ 3 triggers created
• ✅ 3 views created

---

Technical Details

Foreign Key Relationships

workflow_definition
  ↑ (FK: pack)
  └─ pack

workflow_execution
  ↑ (FK: execution)
  └─ execution
  ↑ (FK: workflow_def)
  └─ workflow_definition

workflow_task_execution
  ↑ (FK: workflow_execution)
  └─ workflow_execution
  ↑ (FK: execution)
  └─ execution

action
  ↑ (FK: workflow_def) [optional]
  └─ workflow_definition

Cascade Behavior

• ON DELETE CASCADE: Deleting a pack removes all its workflow definitions
• ON DELETE CASCADE: Deleting a workflow_definition removes its executions and action link
• ON DELETE CASCADE: Deleting a workflow_execution removes all task executions

JSONB Columns

Three tables use JSONB for flexible data storage:

• workflow_definition.definition - Full workflow spec (tasks, vars, transitions)
• workflow_execution.variables - Workflow-scoped variable context
• workflow_execution.task_graph - Adjacency list graph representation

Array Columns

workflow_execution uses text arrays for tracking:

• current_tasks - Currently executing task names
• completed_tasks - Successfully completed task names
• failed_tasks - Failed task names
• skipped_tasks - Skipped due to conditions

---

Migration Statistics

• Lines of SQL: 268
• Tables Added: 3
• Columns Added: 43 (14 + 13 + 16)
• Columns Modified: 2 (action table)
• Indexes Created: 12
• Triggers Created: 3
• Views Created: 3
• Comments Added: 15+
• Migration Time: ~21ms

---

Next Steps

Phase 1 continues with:

1. Add workflow models to common/src/models.rs

   • WorkflowDefinition struct
   • WorkflowExecution struct
   • WorkflowTaskExecution struct
   • Derive FromRow for SQLx

2. Create repositories in common/src/repositories/

   • workflow_definition.rs (CRUD operations)
   • workflow_execution.rs (state management)
   • workflow_task_execution.rs (task tracking)

3. Implement YAML parser in executor/src/workflow/parser.rs

   • Parse workflow YAML to struct
   • Validate workflow structure
   • Support all task types

4. Integrate template engine (Tera)

   • Add dependency to executor Cargo.toml
   • Create template context
   • Implement variable scoping

5. Create variable context manager in executor/src/workflow/context.rs

   • 6-scope variable system
   • Template rendering
   • Variable publishing

---

References

• Migration File: migrations/20250127000002_workflow_orchestration.sql
• Design Doc: docs/workflow-orchestration.md
• Implementation Plan: docs/workflow-implementation-plan.md
• Quick Start: docs/workflow-quickstart.md
• TODO Tasks: work-summary/TODO.md Phase 8.1

---

Notes

• Migration completed without issues
• All schema changes aligned with design specification
• Database ready for model and repository implementation
• No breaking changes to existing tables (only additions)
• Performance indexes included from the start

---

Status: ✅ Phase 1.1 Complete - Database migration successful Next: Phase 1.2 - Add workflow models to common crate