attune/docs/workflows/workflow-summary.md

Workflow Orchestration - Summary

Overview

Attune's workflow orchestration system enables the composition of multiple actions into complex, conditional execution graphs. Workflows are themselves actions that can be triggered by rules, invoked by other workflows, or executed directly via API.

Key Concepts

1. Workflows are Actions

• Workflows are first-class actions with is_workflow=true
• Can be triggered by rules (event-driven)
• Can invoke other workflows (composable)
• Can be executed directly via API

2. YAML-Based Definitions

• Workflows defined in packs/{pack}/workflows/*.yaml
• Declarative, version-controlled
• Parsed and stored as JSON in database
• Portable across environments

3. Task Graph Execution

• Each task is a node in an execution graph
• Tasks transition based on success/failure/completion
• Supports sequential, parallel, and conditional execution
• Fully asynchronous via message queue

4. Multi-Scope Variables

Variables accessible from 6 scopes (precedence order):

1. task.* - Results from completed tasks

   • {{ task.build_image.result.image_uri }}
   • {{ task.health_check.status }}

2. vars.* - Workflow-scoped variables

   • {{ vars.deployment_id }}
   • Set via publish directives

3. parameters.* - Input parameters

   • {{ parameters.app_name }}
   • {{ parameters.environment }}

4. pack.config.* - Pack configuration

   • {{ pack.config.api_key }}
   • {{ pack.config.base_url }}

5. system.* - System variables

   • {{ system.execution_id }}
   • {{ system.timestamp }}
   • {{ system.identity.login }}

6. kv.* - Key-value datastore

   • {{ kv.get('feature.flags.enabled') }}
   • {{ kv.get_secret('api.token') }}

Core Features

Sequential Execution

tasks:
  - name: task1
    action: pack.action1
    on_success: task2
  - name: task2
    action: pack.action2
    on_success: task3

Parallel Execution

tasks:
  - name: parallel_checks
    type: parallel
    tasks:
      - name: check_database
        action: db.health_check
      - name: check_cache
        action: redis.ping
      - name: check_queue
        action: rabbitmq.status
    on_success: deploy_app

Conditional Branching

tasks:
  - name: check_environment
    action: core.noop
    decision:
      - when: "{{ parameters.environment == 'production' }}"
        next: require_approval
      - when: "{{ parameters.environment == 'staging' }}"
        next: run_tests
      - default: deploy_directly

Iteration (with-items)

tasks:
  - name: deploy_to_regions
    action: cloud.deploy
    with_items: "{{ parameters.regions }}"
    batch_size: 5  # Process 5 regions at a time
    input:
      region: "{{ item }}"
      version: "{{ parameters.version }}"

Variable Publishing

tasks:
  - name: create_deployment
    action: deployments.create
    input:
      app_name: "{{ parameters.app_name }}"
    publish:
      - deployment_id: "{{ task.create_deployment.result.id }}"
      - health_url: "{{ task.create_deployment.result.url }}/health"

Error Handling & Retry

tasks:
  - name: flaky_api_call
    action: http.post
    input:
      url: "{{ vars.api_endpoint }}"
    retry:
      count: 5
      delay: 10
      backoff: exponential
      max_delay: 60
    on_success: process_response
    on_failure: log_error_and_continue

Human-in-the-Loop

tasks:
  - name: require_approval
    action: core.inquiry
    input:
      prompt: "Approve production deployment?"
      schema:
        type: object
        properties:
          approved:
            type: boolean
      timeout: 3600  # 1 hour
    decision:
      - when: "{{ task.require_approval.result.approved == true }}"
        next: deploy_to_production
      - default: cancel_deployment

Nested Workflows

tasks:
  - name: provision_infrastructure
    action: infra.full_stack_workflow  # This is also a workflow
    input:
      environment: "{{ parameters.environment }}"
      region: "{{ parameters.region }}"
    on_success: deploy_application

Template System

Tera Template Engine

Jinja2-like syntax for variable interpolation:

# String operations
message: "{{ parameters.app_name | upper | trim }}"

# List operations
first_region: "{{ parameters.regions | first }}"
region_count: "{{ parameters.regions | length }}"

# JSON operations
config: "{{ vars.json_string | from_json }}"

# Batching helper
batches: "{{ vars.large_list | batch(size=100) }}"

# Conditionals
status: "{{ vars.success ? 'deployed' : 'failed' }}"

# Key-value store
api_key: "{{ kv.get_secret('service.api_key') }}"
feature_enabled: "{{ kv.get('flags.new_feature', default=false) }}"

Architecture

┌─────────────────────────────────────────────────┐
│              Workflow Execution                  │
├─────────────────────────────────────────────────┤
│                                                  │
│  1. Rule/API triggers workflow action           │
│  2. Executor loads workflow definition           │
│  3. Create workflow_execution record             │
│  4. Initialize variable context                  │
│  5. Build task dependency graph                  │
│  6. Schedule initial tasks                       │
│  7. For each task:                               │
│     a. Template inputs using context             │
│     b. Create child execution                    │
│     c. Worker executes action                    │
│     d. Update task result                        │
│     e. Publish variables                         │
│     f. Evaluate transitions                      │
│     g. Schedule next tasks                       │
│  8. Complete when all tasks done                 │
│                                                  │
└─────────────────────────────────────────────────┘

Database Schema

New Tables

1. workflow_definition - Stores workflow YAML as JSON
2. workflow_execution - Tracks workflow runtime state
3. workflow_task_execution - Individual task executions

Modified Tables

• action table: Add is_workflow and workflow_def columns

API Endpoints

Workflow Management

POST   /api/v1/packs/{pack_ref}/workflows          - Create workflow
GET    /api/v1/packs/{pack_ref}/workflows          - List workflows
GET    /api/v1/workflows/{workflow_ref}            - Get workflow
PUT    /api/v1/workflows/{workflow_ref}            - Update workflow
DELETE /api/v1/workflows/{workflow_ref}            - Delete workflow
POST   /api/v1/workflows/{workflow_ref}/execute    - Execute workflow

Execution Management

GET    /api/v1/workflow-executions/{id}            - Get execution
GET    /api/v1/workflow-executions/{id}/tasks      - List tasks
GET    /api/v1/workflow-executions/{id}/graph      - Get graph
POST   /api/v1/workflow-executions/{id}/pause      - Pause
POST   /api/v1/workflow-executions/{id}/resume     - Resume
POST   /api/v1/workflow-executions/{id}/cancel     - Cancel

Implementation Timeline

Phase 1: Foundation (2 weeks)

• Database schema and migration
• Data models and repositories
• YAML parser
• Template engine integration

Phase 2: Execution Engine (2 weeks)

• Task graph builder
• Workflow executor
• Message handlers
• State management

Phase 3: Advanced Features (2 weeks)

• Iteration support (with-items)
• Parallel execution
• Retry logic
• Conditional branching

Phase 4: API & Tools (2 weeks)

• Workflow CRUD endpoints
• Execution monitoring API
• Control operations
• Validation tools

Phase 5: Testing & Docs (1 week)

• Comprehensive tests
• Example workflows
• User documentation

Total: 9 weeks

Example Workflow Structure

ref: my_pack.deploy_workflow
label: "Deploy Application"
description: "Deploys application with health checks"
version: "1.0.0"

parameters:
  app_name:
    type: string
    required: true
  version:
    type: string
    required: true
  environment:
    type: string
    enum: [dev, staging, production]

vars:
  deployment_id: null
  health_url: null

tasks:
  - name: create_deployment
    action: deployments.create
    input:
      app_name: "{{ parameters.app_name }}"
      version: "{{ parameters.version }}"
    publish:
      - deployment_id: "{{ task.create_deployment.result.id }}"
    on_success: build_image

  - name: build_image
    action: docker.build
    input:
      app: "{{ parameters.app_name }}"
      tag: "{{ parameters.version }}"
    on_success: deploy
    on_failure: cleanup

  - name: deploy
    action: kubernetes.deploy
    input:
      image: "{{ task.build_image.result.image }}"
    on_success: health_check
    on_failure: rollback

  - name: health_check
    action: http.get
    input:
      url: "{{ task.deploy.result.health_url }}"
    retry:
      count: 5
      delay: 10
    on_success: notify_success
    on_failure: rollback

  - name: rollback
    action: kubernetes.rollback
    on_complete: notify_failure

  - name: notify_success
    action: slack.post
    input:
      message: "✅ Deployed {{ parameters.app_name }} v{{ parameters.version }}"

  - name: notify_failure
    action: slack.post
    input:
      message: "❌ Deployment failed"

output_map:
  deployment_id: "{{ vars.deployment_id }}"
  status: "success"

Pack Structure

packs/my_pack/
├── pack.yaml
├── config.yaml
├── actions/
│   ├── action1.py
│   └── action.yaml
├── sensors/
│   └── sensor.yaml
├── workflows/           # NEW
│   ├── deploy.yaml
│   ├── backup.yaml
│   └── rollback.yaml
└── rules/
    └── on_push.yaml

Benefits

1. Composability - Build complex workflows from simple actions
2. Reusability - Share workflows across packs and organizations
3. Maintainability - YAML definitions are easy to read and version
4. Observability - Full execution tracking and tracing
5. Flexibility - Conditional logic, iteration, parallel execution
6. Reliability - Built-in retry, error handling, rollback
7. Human Control - Inquiry tasks for approval workflows
8. Event-Driven - Fully async, no blocking or polling

Resources

• Full Design: docs/workflow-orchestration.md
• Implementation Plan: docs/workflow-implementation-plan.md
• Simple Example: docs/examples/simple-workflow.yaml
• Complex Example: docs/examples/complete-workflow.yaml
• Migration SQL: docs/examples/workflow-migration.sql