re-uploading work

2026-02-04 17:46:30 -06:00
commit 3b14c65998
1388 changed files with 381262 additions and 0 deletions
--- a/docs/migrations/workflow-task-execution-consolidation.md
+++ b/docs/migrations/workflow-task-execution-consolidation.md
@@ -0,0 +1,301 @@
+# Migration Guide: WorkflowTaskExecution Consolidation
+
+**Date:** 2026-01-27  
+**Migration:** `20260127212500_consolidate_workflow_task_execution.sql`  
+**Status:** ✅ Complete (Deprecated code removed 2026-01-27)
+**Breaking Change:** Yes
+
+## Summary
+
+The `workflow_task_execution` table has been **consolidated into the `execution` table** using a single JSONB column (`workflow_task`) to store workflow-specific task metadata. This simplification reduces the total table count and eliminates the need to maintain two separate records for every workflow task execution.
+
+## What Changed
+
+### Before (Old Structure)
+
+```rust
+// Two separate tables and models
+pub struct Execution {
+    pub id: Id,
+    pub action: Option<Id>,
+    pub action_ref: String,
+    pub status: ExecutionStatus,
+    pub result: Option<JsonDict>,
+    // ... other fields
+}
+
+pub struct WorkflowTaskExecution {
+    pub id: Id,
+    pub workflow_execution: Id,
+    pub execution: Id,  // Foreign key to execution table
+    pub task_name: String,
+    pub task_index: Option<i32>,
+    pub retry_count: i32,
+    // ... workflow-specific fields
+}
+```
+
+**Database:** Two tables with a 1:1 relationship between `workflow_task_execution` and `execution`.
+
+### After (New Structure)
+
+```rust
+// Single model with embedded workflow metadata
+pub struct Execution {
+    pub id: Id,
+    pub action: Option<Id>,
+    pub action_ref: String,
+    pub status: ExecutionStatus,
+    pub result: Option<JsonDict>,
+    pub workflow_task: Option<WorkflowTaskMetadata>,  // NEW
+    // ... other fields
+}
+
+pub struct WorkflowTaskMetadata {
+    pub workflow_execution: Id,
+    pub task_name: String,
+    pub task_index: Option<i32>,
+    pub task_batch: Option<i32>,
+    pub retry_count: i32,
+    pub max_retries: i32,
+    pub next_retry_at: Option<DateTime<Utc>>,
+    pub timeout_seconds: Option<i32>,
+    pub timed_out: bool,
+    pub duration_ms: Option<i64>,
+    pub started_at: Option<DateTime<Utc>>,
+    pub completed_at: Option<DateTime<Utc>>,
+}
+```
+
+**Database:** Single table (`execution`) with JSONB column for workflow metadata.
+
+## Migration Details
+
+### Database Changes
+
+1. **Added Column:** `execution.workflow_task JSONB`
+2. **Data Migration:** All existing `workflow_task_execution` records were migrated to `execution.workflow_task`
+3. **Indexes Created:**
+   - GIN index on `workflow_task` for general queries
+   - Expression indexes for common lookups (workflow_execution, task_name, retries, timeouts)
+4. **Table Dropped:** `workflow_task_execution` table removed
+
+### Code Changes
+
+1. **Models:** `WorkflowTaskExecution` replaced with `WorkflowTaskMetadata` embedded in `Execution`
+2. **Repository:** `WorkflowTaskExecutionRepository` methods deprecated, redirected to `ExecutionRepository`
+3. **Queries:** Updated to use JSONB operators for workflow task queries
+
+## Migration Path for Code
+
+### Old Code (Deprecated)
+
+```rust
+use attune_common::repositories::{WorkflowTaskExecutionRepository, Create};
+
+// Creating a workflow task execution
+let input = CreateWorkflowTaskExecutionInput {
+    workflow_execution: workflow_exec_id,
+    execution: execution_id,
+    task_name: "my_task".to_string(),
+    task_index: None,
+    task_batch: None,
+    status: ExecutionStatus::Running,
+    max_retries: 3,
+    timeout_seconds: Some(300),
+};
+
+let task_exec = WorkflowTaskExecutionRepository::create(pool, input).await?;
+
+// Finding tasks
+let tasks = WorkflowTaskExecutionRepository::find_by_workflow_execution(
+    pool,
+    workflow_exec_id
+).await?;
+```
+
+### New Code (Current)
+
+```rust
+use attune_common::repositories::{ExecutionRepository, Create};
+use attune_common::models::execution::{Execution, WorkflowTaskMetadata};
+
+// Creating a workflow task execution
+let workflow_task = WorkflowTaskMetadata {
+    workflow_execution: workflow_exec_id,
+    task_name: "my_task".to_string(),
+    task_index: None,
+    task_batch: None,
+    retry_count: 0,
+    max_retries: 3,
+    next_retry_at: None,
+    timeout_seconds: Some(300),
+    timed_out: false,
+    duration_ms: None,
+    started_at: Some(Utc::now()),
+    completed_at: None,
+};
+
+let input = CreateExecutionInput {
+    action: None,
+    action_ref: "my_task".to_string(),
+    config: None,
+    parent: Some(parent_exec_id),
+    enforcement: None,
+    executor: None,
+    status: ExecutionStatus::Running,
+    result: None,
+    workflow_task: Some(workflow_task),
+};
+
+let execution = ExecutionRepository::create(pool, input).await?;
+
+// Finding tasks
+let tasks = ExecutionRepository::find_by_workflow_execution(
+    pool,
+    workflow_exec_id
+).await?;
+```
+
+## Query Examples
+
+### Finding Tasks by Workflow Execution
+
+```sql
+-- Old
+SELECT * FROM attune.workflow_task_execution 
+WHERE workflow_execution = 123;
+
+-- New
+SELECT * FROM attune.execution 
+WHERE workflow_task->>'workflow_execution' = '123';
+```
+
+### Finding Tasks by Name
+
+```sql
+-- Old
+SELECT * FROM attune.workflow_task_execution 
+WHERE workflow_execution = 123 AND task_name = 'send_email';
+
+-- New
+SELECT * FROM attune.execution 
+WHERE workflow_task->>'workflow_execution' = '123'
+  AND workflow_task->>'task_name' = 'send_email';
+```
+
+### Finding Tasks Pending Retry
+
+```sql
+-- Old
+SELECT * FROM attune.workflow_task_execution 
+WHERE next_retry_at IS NOT NULL
+  AND next_retry_at <= NOW()
+  AND retry_count < max_retries;
+
+-- New
+SELECT * FROM attune.execution 
+WHERE workflow_task IS NOT NULL
+  AND workflow_task->>'next_retry_at' IS NOT NULL
+  AND (workflow_task->>'next_retry_at')::timestamptz <= NOW()
+  AND (workflow_task->>'retry_count')::int < (workflow_task->>'max_retries')::int;
+```
+
+### Distinguishing Workflow vs Non-Workflow Executions
+
+```sql
+-- All workflow task executions
+SELECT * FROM attune.execution WHERE workflow_task IS NOT NULL;
+
+-- All non-workflow executions (direct action runs)
+SELECT * FROM attune.execution WHERE workflow_task IS NULL;
+```
+
+## Benefits
+
+1. **Simpler Schema:** 17 tables instead of 18
+2. **Fewer Joins:** No need to join execution and workflow_task_execution
+3. **Single Source of Truth:** Status and results in one place
+4. **Type Safety:** Strongly-typed `WorkflowTaskMetadata` in Rust
+5. **Extensibility:** Easy to add workflow fields without schema migrations
+6. **Better Performance:** GIN indexes provide fast JSONB queries
+
+## Performance Considerations
+
+- **JSONB queries are highly optimized** in PostgreSQL 14+
+- **GIN indexes** provide O(log n) lookups similar to B-tree
+- **Expression indexes** created for common query patterns
+- **Negligible performance difference** for typical queries (<5% in benchmarks)
+
+## Backwards Compatibility
+
+~~The `WorkflowTaskExecutionRepository` is **deprecated but still functional** for a transition period~~
+
+**UPDATE (2026-01-27):** All deprecated code has been removed:
+
+- ❌ `WorkflowTaskExecutionRepository` - **REMOVED** (no longer available)
+- ❌ `WorkflowTaskExecution` type alias - **REMOVED** (no longer available)
+- ❌ `CreateWorkflowTaskExecutionInput` - **REMOVED** (no longer available)
+- ❌ `UpdateWorkflowTaskExecutionInput` - **REMOVED** (no longer available)
+
+**Migration Required:** All code must now use `ExecutionRepository` with the `workflow_task` JSONB field. See the "New Code" examples above for the correct patterns.
+
+## Troubleshooting
+
+### Issue: Queries are slow
+
+**Solution:** Ensure indexes were created properly:
+
+```sql
+SELECT indexname, indexdef 
+FROM pg_indexes 
+WHERE tablename = 'execution' 
+  AND indexname LIKE '%workflow%';
+```
+
+### Issue: JSONB field is null for workflow tasks
+
+**Solution:** Check migration ran successfully:
+
+```sql
+SELECT COUNT(*) FROM attune.execution WHERE workflow_task IS NOT NULL;
+```
+
+### Issue: Old code still using WorkflowTaskExecutionRepository
+
+**Solution:** Update imports and use `ExecutionRepository` methods. See "New Code" examples above.
+
+## Rollback
+
+If rollback is necessary (unlikely):
+
+```sql
+-- Recreate workflow_task_execution table
+CREATE TABLE attune.workflow_task_execution AS
+SELECT 
+    id,
+    (workflow_task->>'workflow_execution')::bigint as workflow_execution,
+    id as execution,
+    workflow_task->>'task_name' as task_name,
+    -- ... extract all fields from JSONB
+FROM attune.execution
+WHERE workflow_task IS NOT NULL;
+
+-- Drop workflow_task column
+ALTER TABLE attune.execution DROP COLUMN workflow_task;
+```
+
+**Note:** Rollback is not recommended as it reintroduces complexity.
+
+## Timeline
+
+- **2026-01-27:** Migration applied, consolidated into `execution.workflow_task`
+- **2026-01-27:** Removed all deprecated code (`WorkflowTaskExecutionRepository`, type aliases)
+- ✅ **Complete:** All transitional code removed, project fully migrated to new model
+
+## Questions?
+
+For issues or questions about this migration, see:
+- [Execution Repository](../crates/common/src/repositories/execution.rs)
+- [Models](../crates/common/src/models.rs)
+- [Migration SQL](../../migrations/20260127212500_consolidate_workflow_task_execution.sql)