attune/tests/README.md

End-to-End Integration Testing

Status: 🔄 In Progress - Tier 3 (62% Complete)
Last Updated: 2026-01-24
Purpose: Comprehensive integration testing across all 5 Attune services

🆕 API Client Migration: Tests now use auto-generated OpenAPI client for improved type safety and maintainability. See MIGRATION_TO_GENERATED_CLIENT.md for details.

Test Coverage:

• ✅ Tier 1: Complete (8 scenarios, 33 tests) - Core automation flows
• ✅ Tier 2: Complete (13 scenarios, 37 tests) - Orchestration & data flow
• 🔄 Tier 3: 62% Complete (13/21 scenarios, 40 tests) - Advanced features & edge cases

---

Overview

This directory contains end-to-end integration tests that verify the complete Attune automation platform works correctly when all services are running together.

API Client

Tests use an auto-generated Python client created from the Attune API's OpenAPI specification:

• Generated Client: tests/generated_client/ - 71 endpoints, 200+ Pydantic models
• Wrapper Client: tests/helpers/client_wrapper.py - Backward-compatible interface
• Benefits: Type safety, automatic schema sync, reduced maintenance

For migration details and usage examples, see MIGRATION_TO_GENERATED_CLIENT.md.

Test Scope

Services Under Test:

1. API Service (attune-api) - REST API gateway
2. Executor Service (attune-executor) - Orchestration & scheduling
3. Worker Service (attune-worker) - Action execution
4. Sensor Service (attune-sensor) - Event monitoring
5. Notifier Service (attune-notifier) - Real-time notifications

External Dependencies:

• PostgreSQL (database)
• RabbitMQ (message queue)
• Redis (optional cache)

---

Test Organization

Tests are organized into three tiers based on priority and complexity:

Tier 1: Core Automation Flows ✅ COMPLETE

Essential MVP functionality - timer, webhook, workflow, datastore, multi-tenancy, failure handling.

• Location: tests/e2e/tier1/
• Count: 8 scenarios, 33 test functions
• Duration: ~4 minutes total

Tier 2: Orchestration & Data Flow ✅ COMPLETE

Advanced orchestration - nested workflows, datastore writes, criteria, inquiries, retry policies.

• Location: tests/e2e/tier2/
• Count: 13 scenarios, 37 test functions
• Duration: ~6 minutes total

Tier 3: Advanced Features & Edge Cases 🔄 IN PROGRESS (62%)

Security, edge cases, notifications, container runner, log limits, crash recovery.

• Location: tests/e2e/tier3/
• Count: 13/21 scenarios complete, 40 test functions
• Duration: ~8 minutes (when complete)

Completed T3 Scenarios:

• T3.1: Date Timer with Past Date (3 tests) ⏱️
• T3.2: Timer Cancellation (3 tests) ⏱️
• T3.3: Multiple Concurrent Timers (3 tests) ⏱️
• T3.4: Webhook with Multiple Rules (2 tests) 🔗
• T3.5: Webhook with Rule Criteria Filtering (4 tests) 🎯
• T3.10: RBAC Permission Checks (4 tests) 🔒
• T3.11: System vs User Packs (4 tests) 🔒
• T3.13: Invalid Action Parameters (4 tests) ⚠️
• T3.14: Execution Completion Notifications (4 tests) 🔔
• T3.15: Inquiry Creation Notifications (4 tests) 🔔
• T3.17: Container Runner Execution (4 tests) 🐳
• T3.18: HTTP Runner Execution (4 tests) 🌐
• T3.20: Secret Injection Security (4 tests) 🔐
• T3.21: Action Log Size Limits (4 tests) 📝

Remaining T3 Scenarios:

• T3.6: Sensor-generated custom events
• T3.7: Complex workflow orchestration
• T3.8: Chained webhook triggers
• T3.9: Multi-step approval workflow
• T3.12: Worker crash recovery
• T3.16: Rule trigger notifications
• T3.19: Dependency conflict isolation

---

Running Tests

Quick Start

# Run all tests
./tests/run_e2e_tests.sh

# Run specific tier
pytest tests/e2e/tier1/
pytest tests/e2e/tier2/
pytest tests/e2e/tier3/

# Run by marker
pytest -m "tier1"
pytest -m "tier3 and notifications"
pytest -m "container"

Prerequisites

1. Start all services:

   docker-compose up -d postgres rabbitmq redis
   cargo run --bin attune-api &
   cargo run --bin attune-executor &
   cargo run --bin attune-worker &
   cargo run --bin attune-sensor &
   cargo run --bin attune-notifier &
   

2. Install test dependencies:

   cd tests
   pip install -r requirements.txt
   

3. Verify services are healthy:

   curl http://localhost:8080/health
   

---

Example Test Scenarios

Scenario 1: Basic Timer Automation

Duration: ~30 seconds
Flow: Timer → Event → Rule → Enforcement → Execution → Completion

Steps:

1. Create a pack via API
2. Create a timer trigger (fires every 10 seconds)
3. Create a simple echo action
4. Create a rule linking trigger to action
5. Sensor detects timer and generates event
6. Rule evaluates and creates enforcement
7. Executor schedules execution
8. Worker executes action
9. Verify execution completed successfully

Success Criteria:

• ✅ Event created within 10 seconds
• ✅ Enforcement created with correct rule_id
• ✅ Execution scheduled with correct action_ref
• ✅ Execution status progresses: requested → scheduled → running → succeeded
• ✅ Worker logs action output
• ✅ Completion notification sent back to executor
• ✅ No errors in any service logs

---

Scenario 2: Workflow Execution

Duration: ~45 seconds
Flow: Manual trigger → Workflow with 3 tasks → All tasks complete

Steps:

1. Create a workflow with sequential tasks:
   • Task 1: Echo "Starting workflow"
   • Task 2: Wait 2 seconds
   • Task 3: Echo "Workflow complete"
2. Trigger workflow execution via API
3. Monitor task execution order
4. Verify task outputs and variables

Success Criteria:

• ✅ Workflow execution created
• ✅ Tasks execute in correct order (sequential)
• ✅ Task 1 completes before Task 2 starts
• ✅ Task 2 completes before Task 3 starts
• ✅ Workflow variables propagate correctly
• ✅ Workflow status becomes 'succeeded'
• ✅ All task outputs captured

---

Scenario 3: FIFO Queue Ordering

Duration: ~20 seconds
Flow: Multiple executions with concurrency limit

Steps:

1. Create action with concurrency policy (max=1)
2. Submit 5 execution requests rapidly
3. Monitor execution order
4. Verify FIFO ordering maintained

Success Criteria:

• ✅ Executions enqueued in submission order
• ✅ Only 1 execution runs at a time
• ✅ Next execution starts after previous completes
• ✅ Queue stats accurate (queue_length, active_count)
• ✅ All 5 executions complete successfully
• ✅ Order preserved: exec1 → exec2 → exec3 → exec4 → exec5

---

Scenario 4: Secret Management

Duration: ~15 seconds
Flow: Action uses secrets securely

Steps:

1. Create a secret/key via API
2. Create action that uses the secret
3. Execute action
4. Verify secret injected via stdin (not env vars)
5. Check process environment doesn't contain secret

Success Criteria:

• ✅ Secret created and stored encrypted
• ✅ Worker retrieves secret for execution
• ✅ Secret passed via stdin to action
• ✅ Secret NOT in process environment
• ✅ Secret NOT in execution logs
• ✅ Action can access secret via get_secret() helper

---

Scenario 5: Human-in-the-Loop (Inquiry)

Duration: ~30 seconds
Flow: Action requests user input → Execution pauses → User responds → Execution resumes

Steps:

1. Create action that creates an inquiry
2. Execute action
3. Verify execution pauses with status 'paused'
4. Submit inquiry response via API
5. Verify execution resumes and completes

Success Criteria:

• ✅ Inquiry created with correct prompt
• ✅ Execution status changes to 'paused'
• ✅ Inquiry status is 'pending'
• ✅ Response submission updates inquiry
• ✅ Execution resumes after response
• ✅ Action receives response data
• ✅ Execution completes successfully

---

Scenario 6: Error Handling & Recovery

Duration: ~25 seconds
Flow: Action fails → Retry logic → Final failure

Steps:

1. Create action that always fails
2. Configure retry policy (max_retries=2)
3. Execute action
4. Monitor retry attempts
5. Verify final failure status

Success Criteria:

• ✅ Action fails on first attempt
• ✅ Executor retries execution
• ✅ Action fails on second attempt
• ✅ Executor retries again
• ✅ Action fails on third attempt
• ✅ Execution status becomes 'failed'
• ✅ Retry count accurate (3 total attempts)
• ✅ Error message captured

---

Scenario 7: Real-Time Notifications

Duration: ~20 seconds
Flow: Execution state changes → Notifications sent → WebSocket clients receive updates

Steps:

1. Connect WebSocket client to notifier
2. Create and execute action
3. Monitor notifications for state changes
4. Verify notification delivery

Success Criteria:

• ✅ WebSocket connection established
• ✅ Notification on execution created
• ✅ Notification on execution scheduled
• ✅ Notification on execution running
• ✅ Notification on execution succeeded
• ✅ All notifications contain correct entity_id
• ✅ Notifications delivered in real-time (<100ms)

---

Scenario 8: Dependency Isolation

Duration: ~40 seconds
Flow: Two packs with conflicting dependencies execute correctly

Steps:

1. Create Pack A with Python dependency: requests==2.25.0
2. Create Pack B with Python dependency: requests==2.28.0
3. Create actions in both packs
4. Execute both actions
5. Verify correct dependency versions used

Success Criteria:

• ✅ Pack A venv created with requests 2.25.0
• ✅ Pack B venv created with requests 2.28.0
• ✅ Pack A action uses correct venv
• ✅ Pack B action uses correct venv
• ✅ Both executions succeed
• ✅ No dependency conflicts

---

Test Infrastructure

Prerequisites

Required Services:

# PostgreSQL
docker run -d --name postgres \
  -e POSTGRES_PASSWORD=postgres \
  -p 5432:5432 \
  postgres:14

# RabbitMQ
docker run -d --name rabbitmq \
  -p 5672:5672 \
  -p 15672:15672 \
  rabbitmq:3-management

# Optional: Redis
docker run -d --name redis \
  -p 6379:6379 \
  redis:7

Database Setup:

# Create test database
createdb attune_e2e

# Run migrations
export DATABASE_URL="postgresql://postgres:postgres@localhost:5432/attune_e2e"
sqlx migrate run

Service Configuration

Config File: config.e2e.yaml

environment: test
packs_base_dir: ./tests/fixtures/packs

database:
  url: "postgresql://postgres:postgres@localhost:5432/attune_e2e"
  max_connections: 5

message_queue:
  url: "amqp://guest:guest@localhost:5672/%2F"
  
security:
  jwt_secret: "test-secret-for-e2e-testing-only"
  
server:
  host: "127.0.0.1"
  port: 18080  # Different port for E2E tests

worker:
  runtimes:
    - name: "python3"
      type: "python"
      python_path: "/usr/bin/python3"
    - name: "shell"
      type: "shell"
      shell_path: "/bin/bash"

executor:
  default_execution_timeout: 300
  
sensor:
  poll_interval_seconds: 5
  timer_precision_seconds: 1

---

Running Tests

Option 1: Manual Service Start

Terminal 1 - API:

cd crates/api
ATTUNE__CONFIG_FILE=../../config.e2e.yaml cargo run

Terminal 2 - Executor:

cd crates/executor
ATTUNE__CONFIG_FILE=../../config.e2e.yaml cargo run

Terminal 3 - Worker:

cd crates/worker
ATTUNE__CONFIG_FILE=../../config.e2e.yaml cargo run

Terminal 4 - Sensor:

cd crates/sensor
ATTUNE__CONFIG_FILE=../../config.e2e.yaml cargo run

Terminal 5 - Notifier:

cd crates/notifier
ATTUNE__CONFIG_FILE=../../config.e2e.yaml cargo run

Terminal 6 - Run Tests:

cd tests
cargo test --test e2e_*

---

Option 2: Automated Test Runner (TODO)

# Start all services in background
./tests/scripts/start-services.sh

# Run tests
./tests/scripts/run-e2e-tests.sh

# Stop services
./tests/scripts/stop-services.sh

---

Option 3: Docker Compose (TODO)

# Start all services
docker-compose -f docker-compose.e2e.yaml up -d

# Run tests
docker-compose -f docker-compose.e2e.yaml run --rm test

# Cleanup
docker-compose -f docker-compose.e2e.yaml down

---

Test Implementation

Test Structure

tests/
├── README.md                    # This file
├── config.e2e.yaml             # E2E test configuration
├── fixtures/                   # Test data
│   ├── packs/                  # Test packs
│   │   ├── test_pack/
│   │   │   ├── pack.yaml
│   │   │   ├── actions/
│   │   │   │   ├── echo.yaml
│   │   │   │   └── echo.py
│   │   │   └── workflows/
│   │   │       └── simple.yaml
│   └── seed_data.sql          # Initial test data
├── helpers/                    # Test utilities
│   ├── mod.rs
│   ├── api_client.rs          # API client wrapper
│   ├── service_manager.rs     # Start/stop services
│   └── assertions.rs          # Custom assertions
└── integration/               # Test files
    ├── test_timer_automation.rs
    ├── test_workflow_execution.rs
    ├── test_fifo_ordering.rs
    ├── test_secret_management.rs
    ├── test_inquiry_flow.rs
    ├── test_error_handling.rs
    ├── test_notifications.rs
    └── test_dependency_isolation.rs

---

Debugging Failed Tests

Check Service Logs

# API logs
tail -f logs/api.log

# Executor logs
tail -f logs/executor.log

# Worker logs
tail -f logs/worker.log

# Sensor logs
tail -f logs/sensor.log

# Notifier logs
tail -f logs/notifier.log

Check Database State

-- Check executions
SELECT id, action_ref, status, created, updated 
FROM attune.execution 
ORDER BY created DESC 
LIMIT 10;

-- Check events
SELECT id, trigger, payload, created 
FROM attune.event 
ORDER BY created DESC 
LIMIT 10;

-- Check enforcements
SELECT id, rule, event, status, created 
FROM attune.enforcement 
ORDER BY created DESC 
LIMIT 10;

-- Check queue stats
SELECT action_id, queue_length, active_count, max_concurrent
FROM attune.queue_stats;

Check Message Queue

# RabbitMQ Management UI
open http://localhost:15672
# Login: guest/guest

# Check queues
rabbitmqadmin list queues name messages

# Purge queue (if needed)
rabbitmqadmin purge queue name=executor.enforcement

---

Common Issues

Issue: Services can't connect to database

Solution:

• Verify PostgreSQL is running: psql -U postgres -c "SELECT 1"
• Check DATABASE_URL in config
• Ensure migrations ran: sqlx migrate info

Issue: Services can't connect to RabbitMQ

Solution:

• Verify RabbitMQ is running: rabbitmqctl status
• Check message_queue URL in config
• Verify RabbitMQ user/vhost exists

Issue: Worker can't execute actions

Solution:

• Check Python path in config
• Verify test pack exists: ls tests/fixtures/packs/test_pack
• Check worker logs for runtime errors

Issue: Tests timeout

Solution:

• Increase timeout in test
• Check if services are actually running
• Verify message queue messages are being consumed

Issue: Timer doesn't fire

Solution:

• Verify sensor service is running
• Check sensor poll interval in config
• Look for timer trigger in database: SELECT * FROM attune.trigger WHERE type = 'timer'

---

Success Criteria

A successful integration test run should show:

✅ All services start without errors
✅ Services establish database connections
✅ Services connect to message queue
✅ API endpoints respond correctly
✅ Timer triggers fire on schedule
✅ Events generate from triggers
✅ Rules evaluate correctly
✅ Enforcements create executions
✅ Executions reach workers
✅ Workers execute actions successfully
✅ Results propagate back through system
✅ Notifications delivered in real-time
✅ All 8 test scenarios pass
✅ No errors in service logs
✅ Clean shutdown of all services

---

Next Steps

Phase 1: Setup (Current)

• Document test plan
• Create config.e2e.yaml
• Create test fixtures
• Set up test infrastructure

Phase 2: Basic Tests

• Implement timer automation test
• Implement workflow execution test
• Implement FIFO ordering test

Phase 3: Advanced Tests

• Implement secret management test
• Implement inquiry flow test
• Implement error handling test

Phase 4: Real-time & Performance

• Implement notification test
• Implement dependency isolation test
• Add performance benchmarks

Phase 5: Automation

• Create service start/stop scripts
• Create automated test runner
• Set up CI/CD integration

---

Contributing

When adding new integration tests:

1. Document the scenario in this README
2. Create test fixtures if needed
3. Write the test with clear assertions
4. Test locally with all services running
5. Update CI configuration if needed

---

Resources

• Architecture Documentation
• Service Documentation
• API Documentation
• Workflow Documentation
• Queue Documentation

---

Status: 🔄 In Progress
Current Phase: Phase 1 - Setup
Next Milestone: First test scenario passing