Status: 🔄 In Progress - Tier 3 (62% Complete)
Last Updated: 2026-01-24
Purpose: Comprehensive integration testing across all 5 Attune services

🆕 API Client Migration: Tests now use auto-generated OpenAPI client for improved type safety and maintainability. See MIGRATION_TO_GENERATED_CLIENT.md for details.

Test Coverage:

✅ Tier 1: Complete (8 scenarios, 33 tests) - Core automation flows
✅ Tier 2: Complete (13 scenarios, 37 tests) - Orchestration & data flow
🔄 Tier 3: 62% Complete (13/21 scenarios, 40 tests) - Advanced features & edge cases

Overview

This directory contains end-to-end integration tests that verify the complete Attune automation platform works correctly when all services are running together.

API Client

Tests use an auto-generated Python client created from the Attune API's OpenAPI specification:

Generated Client: tests/generated_client/ - 71 endpoints, 200+ Pydantic models
Wrapper Client: tests/helpers/client_wrapper.py - Backward-compatible interface
Benefits: Type safety, automatic schema sync, reduced maintenance

For migration details and usage examples, see MIGRATION_TO_GENERATED_CLIENT.md.

Test Scope

Services Under Test:

API Service (attune-api) - REST API gateway
Executor Service (attune-executor) - Orchestration & scheduling
Worker Service (attune-worker) - Action execution
Sensor Service (attune-sensor) - Event monitoring
Notifier Service (attune-notifier) - Real-time notifications

External Dependencies:

PostgreSQL (database)
RabbitMQ (message queue)
Redis (optional cache)

Test Organization

Tests are organized into three tiers based on priority and complexity:

Tier 1: Core Automation Flows ✅ COMPLETE

Essential MVP functionality - timer, webhook, workflow, datastore, multi-tenancy, failure handling.

Location: tests/e2e/tier1/
Count: 8 scenarios, 33 test functions
Duration: ~4 minutes total

Tier 2: Orchestration & Data Flow ✅ COMPLETE

Advanced orchestration - nested workflows, datastore writes, criteria, inquiries, retry policies.

Location: tests/e2e/tier2/
Count: 13 scenarios, 37 test functions
Duration: ~6 minutes total

Tier 3: Advanced Features & Edge Cases 🔄 IN PROGRESS (62%)

Security, edge cases, notifications, container runner, log limits, crash recovery.

Location: tests/e2e/tier3/
Count: 13/21 scenarios complete, 40 test functions
Duration: ~8 minutes (when complete)

Completed T3 Scenarios:

T3.1: Date Timer with Past Date (3 tests) ⏱️
T3.2: Timer Cancellation (3 tests) ⏱️
T3.3: Multiple Concurrent Timers (3 tests) ⏱️
T3.4: Webhook with Multiple Rules (2 tests) 🔗
T3.5: Webhook with Rule Criteria Filtering (4 tests) 🎯
T3.10: RBAC Permission Checks (4 tests) 🔒
T3.11: System vs User Packs (4 tests) 🔒
T3.13: Invalid Action Parameters (4 tests) ⚠️
T3.14: Execution Completion Notifications (4 tests) 🔔
T3.15: Inquiry Creation Notifications (4 tests) 🔔
T3.17: Container Runner Execution (4 tests) 🐳
T3.18: HTTP Runner Execution (4 tests) 🌐
T3.20: Secret Injection Security (4 tests) 🔐
T3.21: Action Log Size Limits (4 tests) 📝

Remaining T3 Scenarios:

T3.6: Sensor-generated custom events
T3.7: Complex workflow orchestration
T3.8: Chained webhook triggers
T3.9: Multi-step approval workflow
T3.12: Worker crash recovery
T3.16: Rule trigger notifications
T3.19: Dependency conflict isolation

Running Tests

Quick Start

# Run all tests
./tests/run_e2e_tests.sh

# Run specific tier
pytest tests/e2e/tier1/
pytest tests/e2e/tier2/
pytest tests/e2e/tier3/

# Run by marker
pytest -m "tier1"
pytest -m "tier3 and notifications"
pytest -m "container"

Prerequisites

Start all services:

docker-compose up -d postgres rabbitmq redis
cargo run --bin attune-api &
cargo run --bin attune-executor &
cargo run --bin attune-worker &
cargo run --bin attune-sensor &
cargo run --bin attune-notifier &

Install test dependencies:

cd tests
pip install -r requirements.txt

Verify services are healthy:
```
curl http://localhost:8080/health
```

Example Test Scenarios

Scenario 1: Basic Timer Automation

Duration: ~30 seconds
Flow: Timer → Event → Rule → Enforcement → Execution → Completion

Steps:

Create a pack via API
Create a timer trigger (fires every 10 seconds)
Create a simple echo action
Create a rule linking trigger to action
Sensor detects timer and generates event
Rule evaluates and creates enforcement
Executor schedules execution
Worker executes action
Verify execution completed successfully

Success Criteria:

✅ Event created within 10 seconds
✅ Enforcement created with correct rule_id
✅ Execution scheduled with correct action_ref
✅ Execution status progresses: requested → scheduled → running → succeeded
✅ Worker logs action output
✅ Completion notification sent back to executor
✅ No errors in any service logs

Scenario 2: Workflow Execution

Duration: ~45 seconds
Flow: Manual trigger → Workflow with 3 tasks → All tasks complete

Steps:

Create a workflow with sequential tasks:
- Task 1: Echo "Starting workflow"
- Task 2: Wait 2 seconds
- Task 3: Echo "Workflow complete"
Trigger workflow execution via API
Monitor task execution order
Verify task outputs and variables

Success Criteria:

✅ Workflow execution created
✅ Tasks execute in correct order (sequential)
✅ Task 1 completes before Task 2 starts
✅ Task 2 completes before Task 3 starts
✅ Workflow variables propagate correctly
✅ Workflow status becomes 'succeeded'
✅ All task outputs captured

Scenario 3: FIFO Queue Ordering

Duration: ~20 seconds
Flow: Multiple executions with concurrency limit

Steps:

Create action with concurrency policy (max=1)
Submit 5 execution requests rapidly
Monitor execution order
Verify FIFO ordering maintained

Success Criteria:

✅ Executions enqueued in submission order
✅ Only 1 execution runs at a time
✅ Next execution starts after previous completes
✅ Queue stats accurate (queue_length, active_count)
✅ All 5 executions complete successfully
✅ Order preserved: exec1 → exec2 → exec3 → exec4 → exec5

Scenario 4: Secret Management

Duration: ~15 seconds
Flow: Action uses secrets securely

Steps:

Create a secret/key via API
Create action that uses the secret
Execute action
Verify secret injected via stdin (not env vars)
Check process environment doesn't contain secret

Success Criteria:

✅ Secret created and stored encrypted
✅ Worker retrieves secret for execution
✅ Secret passed via stdin to action
✅ Secret NOT in process environment
✅ Secret NOT in execution logs
✅ Action can access secret via get_secret() helper

Scenario 5: Human-in-the-Loop (Inquiry)

Duration: ~30 seconds
Flow: Action requests user input → Execution pauses → User responds → Execution resumes

Steps:

Create action that creates an inquiry
Execute action
Verify execution pauses with status 'paused'
Submit inquiry response via API
Verify execution resumes and completes

Success Criteria:

✅ Inquiry created with correct prompt
✅ Execution status changes to 'paused'
✅ Inquiry status is 'pending'
✅ Response submission updates inquiry
✅ Execution resumes after response
✅ Action receives response data
✅ Execution completes successfully

Scenario 6: Error Handling & Recovery

Duration: ~25 seconds
Flow: Action fails → Retry logic → Final failure

Steps:

Create action that always fails
Configure retry policy (max_retries=2)
Execute action
Monitor retry attempts
Verify final failure status

Success Criteria:

✅ Action fails on first attempt
✅ Executor retries execution
✅ Action fails on second attempt
✅ Executor retries again
✅ Action fails on third attempt
✅ Execution status becomes 'failed'
✅ Retry count accurate (3 total attempts)
✅ Error message captured

Scenario 7: Real-Time Notifications

Duration: ~20 seconds
Flow: Execution state changes → Notifications sent → WebSocket clients receive updates

Steps:

Connect WebSocket client to notifier
Create and execute action
Monitor notifications for state changes
Verify notification delivery

Success Criteria:

✅ WebSocket connection established
✅ Notification on execution created
✅ Notification on execution scheduled
✅ Notification on execution running
✅ Notification on execution succeeded
✅ All notifications contain correct entity_id
✅ Notifications delivered in real-time (<100ms)

Scenario 8: Dependency Isolation

Duration: ~40 seconds
Flow: Two packs with conflicting dependencies execute correctly

Steps:

Create Pack A with Python dependency: requests==2.25.0
Create Pack B with Python dependency: requests==2.28.0
Create actions in both packs
Execute both actions
Verify correct dependency versions used

Success Criteria:

✅ Pack A venv created with requests 2.25.0
✅ Pack B venv created with requests 2.28.0
✅ Pack A action uses correct venv
✅ Pack B action uses correct venv
✅ Both executions succeed
✅ No dependency conflicts

Test Infrastructure

Prerequisites

Required Services:

# PostgreSQL
docker run -d --name postgres \
  -e POSTGRES_PASSWORD=postgres \
  -p 5432:5432 \
  postgres:14

# RabbitMQ
docker run -d --name rabbitmq \
  -p 5672:5672 \
  -p 15672:15672 \
  rabbitmq:3-management

# Optional: Redis
docker run -d --name redis \
  -p 6379:6379 \
  redis:7

Database Setup:

# Create test database
createdb attune_e2e

# Run migrations
export DATABASE_URL="postgresql://postgres:postgres@localhost:5432/attune_e2e"
sqlx migrate run

Service Configuration

Config File: config.e2e.yaml

environment: test
packs_base_dir: ./tests/fixtures/packs

database:
  url: "postgresql://postgres:postgres@localhost:5432/attune_e2e"
  max_connections: 5

message_queue:
  url: "amqp://guest:guest@localhost:5672/%2F"
  
security:
  jwt_secret: "test-secret-for-e2e-testing-only"
  
server:
  host: "127.0.0.1"
  port: 18080  # Different port for E2E tests

worker:
  runtimes:
    - name: "python3"
      type: "python"
      python_path: "/usr/bin/python3"
    - name: "shell"
      type: "shell"
      shell_path: "/bin/bash"

executor:
  default_execution_timeout: 300
  
sensor:
  poll_interval_seconds: 5
  timer_precision_seconds: 1

Running Tests

Option 1: Manual Service Start

Terminal 1 - API:

cd crates/api
ATTUNE__CONFIG_FILE=../../config.e2e.yaml cargo run

Terminal 2 - Executor:

cd crates/executor
ATTUNE__CONFIG_FILE=../../config.e2e.yaml cargo run

Terminal 3 - Worker:

cd crates/worker
ATTUNE__CONFIG_FILE=../../config.e2e.yaml cargo run

Terminal 4 - Sensor:

cd crates/sensor
ATTUNE__CONFIG_FILE=../../config.e2e.yaml cargo run

Terminal 5 - Notifier:

cd crates/notifier
ATTUNE__CONFIG_FILE=../../config.e2e.yaml cargo run

Terminal 6 - Run Tests:

cd tests
cargo test --test e2e_*

Option 2: Automated Test Runner (TODO)

# Start all services in background
./tests/scripts/start-services.sh

# Run tests
./tests/scripts/run-e2e-tests.sh

# Stop services
./tests/scripts/stop-services.sh

Option 3: Docker Compose (TODO)

# Start all services
docker-compose -f docker-compose.e2e.yaml up -d

# Run tests
docker-compose -f docker-compose.e2e.yaml run --rm test

# Cleanup
docker-compose -f docker-compose.e2e.yaml down

Test Implementation

Test Structure

tests/
├── README.md                    # This file
├── config.e2e.yaml             # E2E test configuration
├── fixtures/                   # Test data
│   ├── packs/                  # Test packs
│   │   ├── test_pack/
│   │   │   ├── pack.yaml
│   │   │   ├── actions/
│   │   │   │   ├── echo.yaml
│   │   │   │   └── echo.py
│   │   │   └── workflows/
│   │   │       └── simple.yaml
│   └── seed_data.sql          # Initial test data
├── helpers/                    # Test utilities
│   ├── mod.rs
│   ├── api_client.rs          # API client wrapper
│   ├── service_manager.rs     # Start/stop services
│   └── assertions.rs          # Custom assertions
└── integration/               # Test files
    ├── test_timer_automation.rs
    ├── test_workflow_execution.rs
    ├── test_fifo_ordering.rs
    ├── test_secret_management.rs
    ├── test_inquiry_flow.rs
    ├── test_error_handling.rs
    ├── test_notifications.rs
    └── test_dependency_isolation.rs

Debugging Failed Tests

Check Service Logs

# API logs
tail -f logs/api.log

# Executor logs
tail -f logs/executor.log

# Worker logs
tail -f logs/worker.log

# Sensor logs
tail -f logs/sensor.log

# Notifier logs
tail -f logs/notifier.log

Check Database State

-- Check executions
SELECT id, action_ref, status, created, updated 
FROM attune.execution 
ORDER BY created DESC 
LIMIT 10;

-- Check events
SELECT id, trigger, payload, created 
FROM attune.event 
ORDER BY created DESC 
LIMIT 10;

-- Check enforcements
SELECT id, rule, event, status, created 
FROM attune.enforcement 
ORDER BY created DESC 
LIMIT 10;

-- Check queue stats
SELECT action_id, queue_length, active_count, max_concurrent
FROM attune.queue_stats;

Check Message Queue

# RabbitMQ Management UI
open http://localhost:15672
# Login: guest/guest

# Check queues
rabbitmqadmin list queues name messages

# Purge queue (if needed)
rabbitmqadmin purge queue name=executor.enforcement

Common Issues

Issue: Services can't connect to database

Solution:

Verify PostgreSQL is running: psql -U postgres -c "SELECT 1"
Check DATABASE_URL in config
Ensure migrations ran: sqlx migrate info

Issue: Services can't connect to RabbitMQ

Solution:

Verify RabbitMQ is running: rabbitmqctl status
Check message_queue URL in config
Verify RabbitMQ user/vhost exists

Issue: Worker can't execute actions

Solution:

Check Python path in config
Verify test pack exists: ls tests/fixtures/packs/test_pack
Check worker logs for runtime errors

Issue: Tests timeout

Solution:

Increase timeout in test
Check if services are actually running
Verify message queue messages are being consumed

Issue: Timer doesn't fire

Solution:

Verify sensor service is running
Check sensor poll interval in config
Look for timer trigger in database: SELECT * FROM attune.trigger WHERE type = 'timer'

Success Criteria

A successful integration test run should show:

✅ All services start without errors
✅ Services establish database connections
✅ Services connect to message queue
✅ API endpoints respond correctly
✅ Timer triggers fire on schedule
✅ Events generate from triggers
✅ Rules evaluate correctly
✅ Enforcements create executions
✅ Executions reach workers
✅ Workers execute actions successfully
✅ Results propagate back through system
✅ Notifications delivered in real-time
✅ All 8 test scenarios pass
✅ No errors in service logs
✅ Clean shutdown of all services

Next Steps

Phase 1: Setup (Current)

Document test plan
Create config.e2e.yaml
Create test fixtures
Set up test infrastructure

Phase 2: Basic Tests

Implement timer automation test
Implement workflow execution test
Implement FIFO ordering test

Phase 3: Advanced Tests

Implement secret management test
Implement inquiry flow test
Implement error handling test

Phase 4: Real-time & Performance

Implement notification test
Implement dependency isolation test
Add performance benchmarks

Phase 5: Automation

Create service start/stop scripts
Create automated test runner
Set up CI/CD integration

Contributing

When adding new integration tests:

Document the scenario in this README
Create test fixtures if needed
Write the test with clear assertions
Test locally with all services running
Update CI configuration if needed

Resources

Status: 🔄 In Progress
Current Phase: Phase 1 - Setup
Next Milestone: First test scenario passing