attune/docs/docker-deployment.md

Docker Deployment Guide

This guide explains how to deploy Attune using Docker and Docker Compose.

Table of Contents

• Overview
• Prerequisites
• Quick Start
• Configuration
• Building Images
• Running Services
• Monitoring
• Troubleshooting
• Production Considerations

Overview

Attune uses Docker Compose to orchestrate multiple services:

• Infrastructure Services:

  • PostgreSQL 16 (database)
  • RabbitMQ 3.13 (message queue)
  • Redis 7 (cache)

• Attune Services:

  • attune-api - REST API gateway (port 8080)
  • attune-executor - Execution orchestration
  • attune-worker - Action execution
  • attune-sensor - Event monitoring
  • attune-notifier - WebSocket notifications (port 8081)
  • attune-web - React web UI (port 3000)

All services communicate via a dedicated Docker network and use persistent volumes for data storage.

Prerequisites

• Docker Engine 20.10+ or Docker Desktop
• Docker Compose 2.0+
• At least 4GB RAM available for Docker
• At least 10GB free disk space (15-20GB recommended with BuildKit cache)
• Docker BuildKit enabled (recommended for fast incremental builds)

Install Docker

Linux:

curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo usermod -aG docker $USER

macOS/Windows: Download and install Docker Desktop from https://www.docker.com/products/docker-desktop/

Quick Start

1. Clone the Repository

git clone <repository-url>
cd attune

2. Configure Environment Variables

Copy the example environment file and customize it:

cp env.docker.example .env

IMPORTANT: Edit .env and set secure values for:

• JWT_SECRET - Generate with: openssl rand -base64 32
• ENCRYPTION_KEY - Generate with: openssl rand -base64 32

3. Enable BuildKit (Recommended)

BuildKit dramatically speeds up incremental builds:

# Enable for current session
export DOCKER_BUILDKIT=1
export COMPOSE_DOCKER_CLI_BUILD=1

# Or enable globally using the helper script
./docker/enable-buildkit.sh

# Or manually add to ~/.bashrc or ~/.zshrc
echo 'export DOCKER_BUILDKIT=1' >> ~/.bashrc
echo 'export COMPOSE_DOCKER_CLI_BUILD=1' >> ~/.bashrc
source ~/.bashrc

Build time comparison:

• With BuildKit: First build ~5-6 min, incremental ~30-60 sec
• Without BuildKit: Every build ~5-6 min

4. Start All Services

docker-compose up -d

This will:

1. Pull/build all required images (first build takes ~5-6 minutes with BuildKit)
2. Create volumes and networks
3. Start all services in the correct order
4. Run database migrations

5. Verify Services

Check that all services are running:

docker-compose ps

All services should show Up status with healthy health checks.

6. Access the Application

• Web UI: http://localhost:3000
• API Documentation: http://localhost:8080/api-spec/swagger-ui/
• RabbitMQ Management: http://localhost:15672 (user: attune, pass: attune)

7. Create Initial Admin User

docker-compose exec api attune-service --create-admin-user

Or use the CLI:

# Install CLI
cargo install --path crates/cli

# Create user
attune auth register --username admin --email admin@example.com --password <password>

Configuration

Environment Variables

The .env file contains all configurable settings. Key variables:

Security (Required)

• JWT_SECRET - Secret for JWT token signing
• ENCRYPTION_KEY - Key for encrypting secrets (min 32 chars)

Service URLs

• API_URL - External URL for API (default: http://localhost:8080)
• WS_URL - External URL for WebSocket (default: ws://localhost:8081)

Infrastructure

• POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_DB - Database credentials
• RABBITMQ_DEFAULT_USER, RABBITMQ_DEFAULT_PASS - Message queue credentials

Application

• ENVIRONMENT - Environment name (docker, production, etc.)
• RUST_LOG - Log level (trace, debug, info, warn, error)

Configuration Files

The Docker deployment uses config.docker.yaml which overrides base settings:

• Database URL: postgresql://attune:attune@postgres:5432/attune
• Message Queue: amqp://attune:attune@rabbitmq:5672
• Cache: redis://redis:6379
• Packs directory: /opt/attune/packs

Custom Configuration

To override specific settings, use environment variables with the ATTUNE__ prefix:

# In .env file
ATTUNE__SERVER__PORT=9090
ATTUNE__LOG__LEVEL=debug
ATTUNE__EXECUTOR__MAX_CONCURRENT_EXECUTIONS=100

Building Images

Build All Images

docker-compose build

Build Specific Service

docker-compose build api
docker-compose build worker
docker-compose build web

Build Arguments

The Rust services Dockerfile accepts build arguments:

# Always use DOCKER_BUILDKIT=1 for cache mounts
DOCKER_BUILDKIT=1 docker build \
  --build-arg SERVICE=api \
  --build-arg RUST_VERSION=1.92 \
  -f docker/Dockerfile \
  -t attune-api .

Multi-Stage Build & BuildKit Cache Mounts

The Dockerfiles use multi-stage builds combined with BuildKit cache mounts:

Multi-stage benefits:

1. Compile with full Rust toolchain in builder stage
2. Minimize final image size (~140-180MB per service)
3. Separate build-time and runtime dependencies

BuildKit cache mounts:

• Persists Cargo registry (~1-2GB)
• Persists Git dependencies
• Persists incremental compilation artifacts (~5-10GB)

Build Time:

• First build: ~5-6 minutes
• Code-only changes (with BuildKit): ~30-60 seconds
• Dependency changes (with BuildKit): ~2-3 minutes
• Without BuildKit: ~5-6 minutes every time

Enable BuildKit:

export DOCKER_BUILDKIT=1
export COMPOSE_DOCKER_CLI_BUILD=1

Running Services

Start Services

# Start all services
docker-compose up -d

# Start specific services
docker-compose up -d postgres rabbitmq redis
docker-compose up -d api executor worker

Stop Services

# Stop all services
docker-compose down

# Stop and remove volumes (WARNING: deletes data)
docker-compose down -v

Restart Services

# Restart all services
docker-compose restart

# Restart specific service
docker-compose restart api

View Logs

# All services
docker-compose logs -f

# Specific service
docker-compose logs -f api

# Last 100 lines
docker-compose logs --tail=100 api

Scale Workers

You can run multiple worker instances:

docker-compose up -d --scale worker=3

Monitoring

Health Checks

All services have health checks configured. View status:

docker-compose ps

Service Status

Check if a specific service is healthy:

# API health check
curl http://localhost:8080/health

# Web UI health check
curl http://localhost:3000/health

# Database
docker-compose exec postgres pg_isready -U attune

# RabbitMQ
docker-compose exec rabbitmq rabbitmq-diagnostics ping

Resource Usage

# Container stats
docker stats

# Disk usage
docker system df

Access Container Shell

# API service
docker-compose exec api /bin/sh

# Database
docker-compose exec postgres psql -U attune

Troubleshooting

Services Won't Start

1. Check logs:

   docker-compose logs <service-name>
   

2. Verify dependencies:

   docker-compose ps
   

   Ensure postgres, rabbitmq, and redis are healthy before starting application services.

3. Check resource availability:

   docker system df
   docker stats
   

Database Connection Issues

# Test database connection
docker-compose exec postgres psql -U attune -c "SELECT version();"

# Check database logs
docker-compose logs postgres

# Verify migrations ran
docker-compose exec postgres psql -U attune -c "\dt attune.*"

Message Queue Issues

# Check RabbitMQ status
docker-compose exec rabbitmq rabbitmqctl status

# List queues
docker-compose exec rabbitmq rabbitmqctl list_queues

# Access management UI
open http://localhost:15672

Port Conflicts

If ports are already in use, modify docker-compose.yaml or use environment variables:

# In docker-compose.yaml
ports:
  - "${API_PORT:-8080}:8080"

Then in .env:

API_PORT=9090

Permission Issues

The services run as non-root user attune (UID 1000). If you have permission issues with volumes:

# Fix ownership
sudo chown -R 1000:1000 ./packs ./logs

Reset Everything

To completely reset the environment:

# Stop and remove containers, networks, volumes
docker-compose down -v

# Remove images
docker-compose down --rmi all

# Clear build cache (if using BuildKit)
docker builder prune

# Remove all Docker data (CAREFUL!)
docker system prune -a --volumes

BuildKit Cache Management

If using BuildKit (recommended), manage the build cache:

# View cache size
docker system df

# Clear build cache
docker builder prune

# Clear all unused cache
docker builder prune -a

# View what will be removed
docker builder prune --dry-run

BuildKit cache can grow to 5-10GB but dramatically speeds up rebuilds.

Production Considerations

Security

1. Change default credentials:

   • Generate strong JWT_SECRET and ENCRYPTION_KEY
   • Update database, RabbitMQ, and Redis passwords
   • Never commit .env to version control

2. Use secrets management:

   • Docker Swarm secrets
   • Kubernetes secrets
   • HashiCorp Vault
   • AWS Secrets Manager

3. Network security:

   • Use TLS/SSL for all external connections
   • Restrict network access with firewall rules
   • Use reverse proxy (Nginx, Traefik) for TLS termination

4. Container security:

   • Run containers as non-root (already configured)
   • Keep images updated
   • Scan images for vulnerabilities

Performance

1. Resource limits:

   api:
     deploy:
       resources:
         limits:
           cpus: '2'
           memory: 2G
         reservations:
           cpus: '1'
           memory: 1G
   

2. Database tuning:

   • Adjust max_connections, shared_buffers
   • Enable connection pooling
   • Use read replicas for heavy read loads

3. Scaling:

   • Scale worker services horizontally
   • Use load balancer for API instances
   • Separate read/write database connections

Backup and Recovery

1. Database backups:

   # Backup
   docker-compose exec postgres pg_dump -U attune > backup.sql
   
   # Restore
   docker-compose exec -T postgres psql -U attune < backup.sql
   

2. Volume backups:

   # Backup volumes
   docker run --rm -v attune_postgres_data:/data -v $(pwd):/backup \
     alpine tar czf /backup/postgres_backup.tar.gz /data
   

3. Automated backups:

   • Use backup solutions (Velero, Restic)
   • Schedule regular backups with cron
   • Test restore procedures regularly

Logging

1. Centralized logging:

   • Configure Docker logging driver
   • Use ELK stack, Loki, or CloudWatch
   • Set log retention policies

2. Structured logging:

   • Services already use JSON logging in Docker mode
   • Parse and index logs for analysis

Monitoring

1. Metrics collection:

   • Prometheus + Grafana
   • DataDog, New Relic
   • Container metrics (cAdvisor)

2. Alerting:

   • Set up alerts for service failures
   • Monitor resource usage
   • Track error rates

High Availability

1. Database:

   • PostgreSQL streaming replication
   • Automatic failover (Patroni, Stolon)
   • Regular backups

2. Message Queue:

   • RabbitMQ clustering
   • Mirrored queues
   • Load balancing

3. Application:

   • Run multiple instances
   • Use container orchestration (Kubernetes, Swarm)
   • Implement circuit breakers

Makefile Integration

The project Makefile includes Docker commands:

# Build images
make docker-build

# Start services
make docker-up

# Stop services
make docker-down

# View logs
make docker-logs

Next Steps

• Review Production Deployment Guide for production best practices
• Configure Monitoring and Alerting
• Set up Backup and Recovery
• Review Security Guide

Support

For issues and questions:

• Check Troubleshooting section
• Review service logs: docker-compose logs <service>
• Open an issue on GitHub
• Consult the Documentation Index