attune/docker/README.worker.md

Attune Worker Containers

This directory contains Docker configurations for building Attune worker containers with different runtime capabilities.

Overview

Attune workers can run in containers with specialized runtime environments. Workers automatically declare their capabilities when they register with the system, enabling intelligent action scheduling based on runtime requirements.

Worker Variants

Base Worker (worker-base)

• Runtimes: shell
• Base Image: Debian Bookworm Slim
• Size: ~580 MB
• Use Case: Lightweight workers for shell scripts and basic automation
• Build: make docker-build-worker-base

Python Worker (worker-python)

• Runtimes: shell, python
• Base Image: Python 3.11 Slim
• Size: ~1.2 GB
• Includes: pip, virtualenv, common Python libraries (requests, pyyaml, jinja2, python-dateutil)
• Use Case: Python actions and scripts with dependencies
• Build: make docker-build-worker-python

Node.js Worker (worker-node)

• Runtimes: shell, node
• Base Image: Node 20 Slim
• Size: ~760 MB
• Includes: npm, yarn
• Use Case: JavaScript/TypeScript actions and npm packages
• Build: make docker-build-worker-node

Full Worker (worker-full)

• Runtimes: shell, python, node, native
• Base Image: Debian Bookworm
• Size: ~1.6 GB
• Includes: Python 3.x, Node.js 20, build tools
• Use Case: General-purpose automation requiring multiple runtimes
• Build: make docker-build-worker-full

Building Worker Images

Build All Variants

make docker-build-workers

Build Individual Variants

# Base worker (shell only)
make docker-build-worker-base

# Python worker
make docker-build-worker-python

# Node.js worker
make docker-build-worker-node

# Full worker (all runtimes)
make docker-build-worker-full

Direct Docker Build

# Using Docker directly with BuildKit
DOCKER_BUILDKIT=1 docker build \
  --target worker-python \
  -t attune-worker:python \
  -f docker/Dockerfile.worker \
  .

Running Workers

Using Docker Compose

# Start specific worker type
docker-compose up -d worker-python

# Start all workers
docker-compose up -d worker-shell worker-python worker-node worker-full

# Scale workers
docker-compose up -d --scale worker-python=3

Using Docker Run

docker run -d \
  --name worker-python-01 \
  --network attune_attune-network \
  -e ATTUNE_WORKER_NAME=worker-python-01 \
  -e ATTUNE_WORKER_RUNTIMES=shell,python \
  -e ATTUNE__DATABASE__URL=postgresql://attune:attune@postgres:5432/attune \
  -e ATTUNE__MESSAGE_QUEUE__URL=amqp://attune:attune@rabbitmq:5672 \
  -v $(pwd)/packs:/opt/attune/packs:ro \
  attune-worker:python

Runtime Capability Declaration

Workers declare their capabilities in three ways (in order of precedence):

1. Environment Variable (Highest Priority)

ATTUNE_WORKER_RUNTIMES="shell,python,custom"

2. Configuration File

worker:
  capabilities:
    runtimes: ["shell", "python"]

3. Auto-Detection (Fallback)

Workers automatically detect available runtimes by checking for binaries:

• python3 or python → adds python
• node → adds node
• Always includes shell and native

Configuration

Key Environment Variables

Variable	Description	Example
ATTUNE_WORKER_NAME	Unique worker identifier	worker-python-01
ATTUNE_WORKER_RUNTIMES	Comma-separated runtime list	shell,python
ATTUNE_WORKER_TYPE	Worker type	container
ATTUNE__DATABASE__URL	PostgreSQL connection	postgresql://...
ATTUNE__MESSAGE_QUEUE__URL	RabbitMQ connection	amqp://...
RUST_LOG	Log level	info, debug, trace

Resource Limits

Set CPU and memory limits in docker-compose.override.yml:

services:
  worker-python:
    deploy:
      resources:
        limits:
          cpus: '2.0'
          memory: 2G
        reservations:
          cpus: '0.5'
          memory: 512M

Custom Worker Images

Extend Python Worker

Create a custom worker with additional packages:

# Dockerfile.worker.ml
FROM attune-worker:python

USER root

# Install ML packages
RUN pip install --no-cache-dir \
    pandas \
    numpy \
    scikit-learn \
    torch

USER attune

ENV ATTUNE_WORKER_RUNTIMES="shell,python,ml"

Build and run:

docker build -t attune-worker:ml -f Dockerfile.worker.ml .
docker run -d --name worker-ml-01 ... attune-worker:ml

Add New Runtime

Example: Adding Ruby support

FROM attune-worker:base

USER root

RUN apt-get update && apt-get install -y \
    ruby-full \
    && rm -rf /var/lib/apt/lists/*

USER attune

ENV ATTUNE_WORKER_RUNTIMES="shell,ruby"

Architecture

Multi-stage Build

The Dockerfile.worker uses a multi-stage build pattern:

1. Builder Stage: Compiles the Rust worker binary

   • Uses BuildKit cache mounts for fast incremental builds
   • Shared across all worker variants

2. Runtime Stages: Creates specialized worker images

   • worker-base: Minimal shell runtime
   • worker-python: Python runtime
   • worker-node: Node.js runtime
   • worker-full: All runtimes

Build Cache

BuildKit cache mounts dramatically speed up builds:

• First build: ~5-6 minutes
• Incremental builds: ~30-60 seconds

Cache is shared across builds using sharing=locked to prevent race conditions.

Security

Non-root Execution

All workers run as user attune (UID 1000)

Read-only Packs

Pack files are mounted read-only to prevent modification:

volumes:
  - ./packs:/opt/attune/packs:ro  # :ro = read-only

Network Isolation

Workers run in isolated Docker network with only necessary service access

Secret Management

Use environment variables for sensitive data; never hardcode in images

Monitoring

Check Worker Registration

docker-compose exec postgres psql -U attune -d attune -c \
  "SELECT name, worker_type, status, capabilities->>'runtimes' as runtimes FROM worker;"

View Logs

docker-compose logs -f worker-python

Check Resource Usage

docker stats attune-worker-python

Verify Health

docker-compose ps | grep worker

Troubleshooting

Worker Not Registering

Check database connectivity:

docker-compose logs worker-python | grep -i database

Verify environment:

docker-compose exec worker-python env | grep ATTUNE

Runtime Not Detected

Check runtime availability:

docker-compose exec worker-python python3 --version
docker-compose exec worker-python node --version

Force runtime declaration:

ATTUNE_WORKER_RUNTIMES=shell,python

Actions Not Scheduled

Verify runtime match:

-- Check action runtime requirement
SELECT a.ref, r.name as runtime 
FROM action a 
JOIN runtime r ON a.runtime = r.id 
WHERE a.ref = 'core.my_action';

-- Check worker capabilities
SELECT name, capabilities->>'runtimes' 
FROM worker 
WHERE status = 'active';

Performance

Image Sizes

Image	Size	Build Time (Cold)	Build Time (Cached)
worker-base	~580 MB	~5 min	~30 sec
worker-python	~1.2 GB	~6 min	~45 sec
worker-node	~760 MB	~6 min	~45 sec
worker-full	~1.6 GB	~7 min	~60 sec

Optimization Tips

1. Use specific variants: Don't use worker-full if you only need Python
2. Enable BuildKit: Dramatically speeds up builds
3. Layer caching: Order Dockerfile commands from least to most frequently changed
4. Multi-stage builds: Keeps runtime images small

Files

• Dockerfile.worker - Multi-stage worker Dockerfile with all variants
• README.worker.md - This file
• ../docker-compose.yaml - Service definitions for all workers

References

• Worker Containerization Design
• Quick Start Guide
• Worker Service Architecture
• Production Deployment

Quick Commands

# Build all workers
make docker-build-workers

# Start all workers
docker-compose up -d worker-shell worker-python worker-node worker-full

# Check worker status
docker-compose exec postgres psql -U attune -d attune -c \
  "SELECT name, status, capabilities FROM worker;"

# View Python worker logs
docker-compose logs -f worker-python

# Restart worker
docker-compose restart worker-python

# Scale Python workers
docker-compose up -d --scale worker-python=3

# Stop all workers
docker-compose stop worker-shell worker-python worker-node worker-full