re-uploading work
This commit is contained in:
595
docs/docker-deployment.md
Normal file
595
docs/docker-deployment.md
Normal file
@@ -0,0 +1,595 @@
|
||||
# Docker Deployment Guide
|
||||
|
||||
This guide explains how to deploy Attune using Docker and Docker Compose.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Overview](#overview)
|
||||
- [Prerequisites](#prerequisites)
|
||||
- [Quick Start](#quick-start)
|
||||
- [Configuration](#configuration)
|
||||
- [Building Images](#building-images)
|
||||
- [Running Services](#running-services)
|
||||
- [Monitoring](#monitoring)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
- [Production Considerations](#production-considerations)
|
||||
|
||||
## Overview
|
||||
|
||||
Attune uses Docker Compose to orchestrate multiple services:
|
||||
|
||||
- **Infrastructure Services**:
|
||||
- PostgreSQL 16 (database)
|
||||
- RabbitMQ 3.13 (message queue)
|
||||
- Redis 7 (cache)
|
||||
|
||||
- **Attune Services**:
|
||||
- `attune-api` - REST API gateway (port 8080)
|
||||
- `attune-executor` - Execution orchestration
|
||||
- `attune-worker` - Action execution
|
||||
- `attune-sensor` - Event monitoring
|
||||
- `attune-notifier` - WebSocket notifications (port 8081)
|
||||
- `attune-web` - React web UI (port 3000)
|
||||
|
||||
All services communicate via a dedicated Docker network and use persistent volumes for data storage.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Docker Engine 20.10+ or Docker Desktop
|
||||
- Docker Compose 2.0+
|
||||
- At least 4GB RAM available for Docker
|
||||
- At least 10GB free disk space (15-20GB recommended with BuildKit cache)
|
||||
- Docker BuildKit enabled (recommended for fast incremental builds)
|
||||
|
||||
### Install Docker
|
||||
|
||||
**Linux:**
|
||||
```bash
|
||||
curl -fsSL https://get.docker.com -o get-docker.sh
|
||||
sudo sh get-docker.sh
|
||||
sudo usermod -aG docker $USER
|
||||
```
|
||||
|
||||
**macOS/Windows:**
|
||||
Download and install Docker Desktop from https://www.docker.com/products/docker-desktop/
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Clone the Repository
|
||||
|
||||
```bash
|
||||
git clone <repository-url>
|
||||
cd attune
|
||||
```
|
||||
|
||||
### 2. Configure Environment Variables
|
||||
|
||||
Copy the example environment file and customize it:
|
||||
|
||||
```bash
|
||||
cp env.docker.example .env
|
||||
```
|
||||
|
||||
**IMPORTANT**: Edit `.env` and set secure values for:
|
||||
- `JWT_SECRET` - Generate with: `openssl rand -base64 32`
|
||||
- `ENCRYPTION_KEY` - Generate with: `openssl rand -base64 32`
|
||||
|
||||
### 3. Enable BuildKit (Recommended)
|
||||
|
||||
BuildKit dramatically speeds up incremental builds:
|
||||
|
||||
```bash
|
||||
# Enable for current session
|
||||
export DOCKER_BUILDKIT=1
|
||||
export COMPOSE_DOCKER_CLI_BUILD=1
|
||||
|
||||
# Or enable globally using the helper script
|
||||
./docker/enable-buildkit.sh
|
||||
|
||||
# Or manually add to ~/.bashrc or ~/.zshrc
|
||||
echo 'export DOCKER_BUILDKIT=1' >> ~/.bashrc
|
||||
echo 'export COMPOSE_DOCKER_CLI_BUILD=1' >> ~/.bashrc
|
||||
source ~/.bashrc
|
||||
```
|
||||
|
||||
**Build time comparison:**
|
||||
- With BuildKit: First build ~5-6 min, incremental ~30-60 sec
|
||||
- Without BuildKit: Every build ~5-6 min
|
||||
|
||||
### 4. Start All Services
|
||||
|
||||
```bash
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
This will:
|
||||
1. Pull/build all required images (first build takes ~5-6 minutes with BuildKit)
|
||||
2. Create volumes and networks
|
||||
3. Start all services in the correct order
|
||||
4. Run database migrations
|
||||
|
||||
### 5. Verify Services
|
||||
|
||||
Check that all services are running:
|
||||
|
||||
```bash
|
||||
docker-compose ps
|
||||
```
|
||||
|
||||
All services should show `Up` status with healthy health checks.
|
||||
|
||||
### 6. Access the Application
|
||||
|
||||
- **Web UI**: http://localhost:3000
|
||||
- **API Documentation**: http://localhost:8080/api-spec/swagger-ui/
|
||||
- **RabbitMQ Management**: http://localhost:15672 (user: attune, pass: attune)
|
||||
|
||||
### 7. Create Initial Admin User
|
||||
|
||||
```bash
|
||||
docker-compose exec api attune-service --create-admin-user
|
||||
```
|
||||
|
||||
Or use the CLI:
|
||||
|
||||
```bash
|
||||
# Install CLI
|
||||
cargo install --path crates/cli
|
||||
|
||||
# Create user
|
||||
attune auth register --username admin --email admin@example.com --password <password>
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
The `.env` file contains all configurable settings. Key variables:
|
||||
|
||||
#### Security (Required)
|
||||
- `JWT_SECRET` - Secret for JWT token signing
|
||||
- `ENCRYPTION_KEY` - Key for encrypting secrets (min 32 chars)
|
||||
|
||||
#### Service URLs
|
||||
- `API_URL` - External URL for API (default: http://localhost:8080)
|
||||
- `WS_URL` - External URL for WebSocket (default: ws://localhost:8081)
|
||||
|
||||
#### Infrastructure
|
||||
- `POSTGRES_USER`, `POSTGRES_PASSWORD`, `POSTGRES_DB` - Database credentials
|
||||
- `RABBITMQ_DEFAULT_USER`, `RABBITMQ_DEFAULT_PASS` - Message queue credentials
|
||||
|
||||
#### Application
|
||||
- `ENVIRONMENT` - Environment name (docker, production, etc.)
|
||||
- `RUST_LOG` - Log level (trace, debug, info, warn, error)
|
||||
|
||||
### Configuration Files
|
||||
|
||||
The Docker deployment uses `config.docker.yaml` which overrides base settings:
|
||||
|
||||
- Database URL: `postgresql://attune:attune@postgres:5432/attune`
|
||||
- Message Queue: `amqp://attune:attune@rabbitmq:5672`
|
||||
- Cache: `redis://redis:6379`
|
||||
- Packs directory: `/opt/attune/packs`
|
||||
|
||||
### Custom Configuration
|
||||
|
||||
To override specific settings, use environment variables with the `ATTUNE__` prefix:
|
||||
|
||||
```bash
|
||||
# In .env file
|
||||
ATTUNE__SERVER__PORT=9090
|
||||
ATTUNE__LOG__LEVEL=debug
|
||||
ATTUNE__EXECUTOR__MAX_CONCURRENT_EXECUTIONS=100
|
||||
```
|
||||
|
||||
## Building Images
|
||||
|
||||
### Build All Images
|
||||
|
||||
```bash
|
||||
docker-compose build
|
||||
```
|
||||
|
||||
### Build Specific Service
|
||||
|
||||
```bash
|
||||
docker-compose build api
|
||||
docker-compose build worker
|
||||
docker-compose build web
|
||||
```
|
||||
|
||||
### Build Arguments
|
||||
|
||||
The Rust services Dockerfile accepts build arguments:
|
||||
|
||||
```bash
|
||||
# Always use DOCKER_BUILDKIT=1 for cache mounts
|
||||
DOCKER_BUILDKIT=1 docker build \
|
||||
--build-arg SERVICE=api \
|
||||
--build-arg RUST_VERSION=1.92 \
|
||||
-f docker/Dockerfile \
|
||||
-t attune-api .
|
||||
```
|
||||
|
||||
### Multi-Stage Build & BuildKit Cache Mounts
|
||||
|
||||
The Dockerfiles use multi-stage builds combined with BuildKit cache mounts:
|
||||
|
||||
**Multi-stage benefits:**
|
||||
1. Compile with full Rust toolchain in builder stage
|
||||
2. Minimize final image size (~140-180MB per service)
|
||||
3. Separate build-time and runtime dependencies
|
||||
|
||||
**BuildKit cache mounts:**
|
||||
- Persists Cargo registry (~1-2GB)
|
||||
- Persists Git dependencies
|
||||
- Persists incremental compilation artifacts (~5-10GB)
|
||||
|
||||
**Build Time:**
|
||||
- First build: ~5-6 minutes
|
||||
- Code-only changes (with BuildKit): ~30-60 seconds
|
||||
- Dependency changes (with BuildKit): ~2-3 minutes
|
||||
- Without BuildKit: ~5-6 minutes every time
|
||||
|
||||
**Enable BuildKit:**
|
||||
```bash
|
||||
export DOCKER_BUILDKIT=1
|
||||
export COMPOSE_DOCKER_CLI_BUILD=1
|
||||
```
|
||||
|
||||
## Running Services
|
||||
|
||||
### Start Services
|
||||
|
||||
```bash
|
||||
# Start all services
|
||||
docker-compose up -d
|
||||
|
||||
# Start specific services
|
||||
docker-compose up -d postgres rabbitmq redis
|
||||
docker-compose up -d api executor worker
|
||||
```
|
||||
|
||||
### Stop Services
|
||||
|
||||
```bash
|
||||
# Stop all services
|
||||
docker-compose down
|
||||
|
||||
# Stop and remove volumes (WARNING: deletes data)
|
||||
docker-compose down -v
|
||||
```
|
||||
|
||||
### Restart Services
|
||||
|
||||
```bash
|
||||
# Restart all services
|
||||
docker-compose restart
|
||||
|
||||
# Restart specific service
|
||||
docker-compose restart api
|
||||
```
|
||||
|
||||
### View Logs
|
||||
|
||||
```bash
|
||||
# All services
|
||||
docker-compose logs -f
|
||||
|
||||
# Specific service
|
||||
docker-compose logs -f api
|
||||
|
||||
# Last 100 lines
|
||||
docker-compose logs --tail=100 api
|
||||
```
|
||||
|
||||
### Scale Workers
|
||||
|
||||
You can run multiple worker instances:
|
||||
|
||||
```bash
|
||||
docker-compose up -d --scale worker=3
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Health Checks
|
||||
|
||||
All services have health checks configured. View status:
|
||||
|
||||
```bash
|
||||
docker-compose ps
|
||||
```
|
||||
|
||||
### Service Status
|
||||
|
||||
Check if a specific service is healthy:
|
||||
|
||||
```bash
|
||||
# API health check
|
||||
curl http://localhost:8080/health
|
||||
|
||||
# Web UI health check
|
||||
curl http://localhost:3000/health
|
||||
|
||||
# Database
|
||||
docker-compose exec postgres pg_isready -U attune
|
||||
|
||||
# RabbitMQ
|
||||
docker-compose exec rabbitmq rabbitmq-diagnostics ping
|
||||
```
|
||||
|
||||
### Resource Usage
|
||||
|
||||
```bash
|
||||
# Container stats
|
||||
docker stats
|
||||
|
||||
# Disk usage
|
||||
docker system df
|
||||
```
|
||||
|
||||
### Access Container Shell
|
||||
|
||||
```bash
|
||||
# API service
|
||||
docker-compose exec api /bin/sh
|
||||
|
||||
# Database
|
||||
docker-compose exec postgres psql -U attune
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Services Won't Start
|
||||
|
||||
1. **Check logs**:
|
||||
```bash
|
||||
docker-compose logs <service-name>
|
||||
```
|
||||
|
||||
2. **Verify dependencies**:
|
||||
```bash
|
||||
docker-compose ps
|
||||
```
|
||||
Ensure postgres, rabbitmq, and redis are healthy before starting application services.
|
||||
|
||||
3. **Check resource availability**:
|
||||
```bash
|
||||
docker system df
|
||||
docker stats
|
||||
```
|
||||
|
||||
### Database Connection Issues
|
||||
|
||||
```bash
|
||||
# Test database connection
|
||||
docker-compose exec postgres psql -U attune -c "SELECT version();"
|
||||
|
||||
# Check database logs
|
||||
docker-compose logs postgres
|
||||
|
||||
# Verify migrations ran
|
||||
docker-compose exec postgres psql -U attune -c "\dt attune.*"
|
||||
```
|
||||
|
||||
### Message Queue Issues
|
||||
|
||||
```bash
|
||||
# Check RabbitMQ status
|
||||
docker-compose exec rabbitmq rabbitmqctl status
|
||||
|
||||
# List queues
|
||||
docker-compose exec rabbitmq rabbitmqctl list_queues
|
||||
|
||||
# Access management UI
|
||||
open http://localhost:15672
|
||||
```
|
||||
|
||||
### Port Conflicts
|
||||
|
||||
If ports are already in use, modify `docker-compose.yaml` or use environment variables:
|
||||
|
||||
```yaml
|
||||
# In docker-compose.yaml
|
||||
ports:
|
||||
- "${API_PORT:-8080}:8080"
|
||||
```
|
||||
|
||||
Then in `.env`:
|
||||
```bash
|
||||
API_PORT=9090
|
||||
```
|
||||
|
||||
### Permission Issues
|
||||
|
||||
The services run as non-root user `attune` (UID 1000). If you have permission issues with volumes:
|
||||
|
||||
```bash
|
||||
# Fix ownership
|
||||
sudo chown -R 1000:1000 ./packs ./logs
|
||||
```
|
||||
|
||||
### Reset Everything
|
||||
|
||||
To completely reset the environment:
|
||||
|
||||
```bash
|
||||
# Stop and remove containers, networks, volumes
|
||||
docker-compose down -v
|
||||
|
||||
# Remove images
|
||||
docker-compose down --rmi all
|
||||
|
||||
# Clear build cache (if using BuildKit)
|
||||
docker builder prune
|
||||
|
||||
# Remove all Docker data (CAREFUL!)
|
||||
docker system prune -a --volumes
|
||||
```
|
||||
|
||||
### BuildKit Cache Management
|
||||
|
||||
If using BuildKit (recommended), manage the build cache:
|
||||
|
||||
```bash
|
||||
# View cache size
|
||||
docker system df
|
||||
|
||||
# Clear build cache
|
||||
docker builder prune
|
||||
|
||||
# Clear all unused cache
|
||||
docker builder prune -a
|
||||
|
||||
# View what will be removed
|
||||
docker builder prune --dry-run
|
||||
```
|
||||
|
||||
BuildKit cache can grow to 5-10GB but dramatically speeds up rebuilds.
|
||||
|
||||
## Production Considerations
|
||||
|
||||
### Security
|
||||
|
||||
1. **Change default credentials**:
|
||||
- Generate strong `JWT_SECRET` and `ENCRYPTION_KEY`
|
||||
- Update database, RabbitMQ, and Redis passwords
|
||||
- Never commit `.env` to version control
|
||||
|
||||
2. **Use secrets management**:
|
||||
- Docker Swarm secrets
|
||||
- Kubernetes secrets
|
||||
- HashiCorp Vault
|
||||
- AWS Secrets Manager
|
||||
|
||||
3. **Network security**:
|
||||
- Use TLS/SSL for all external connections
|
||||
- Restrict network access with firewall rules
|
||||
- Use reverse proxy (Nginx, Traefik) for TLS termination
|
||||
|
||||
4. **Container security**:
|
||||
- Run containers as non-root (already configured)
|
||||
- Keep images updated
|
||||
- Scan images for vulnerabilities
|
||||
|
||||
### Performance
|
||||
|
||||
1. **Resource limits**:
|
||||
```yaml
|
||||
api:
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '2'
|
||||
memory: 2G
|
||||
reservations:
|
||||
cpus: '1'
|
||||
memory: 1G
|
||||
```
|
||||
|
||||
2. **Database tuning**:
|
||||
- Adjust `max_connections`, `shared_buffers`
|
||||
- Enable connection pooling
|
||||
- Use read replicas for heavy read loads
|
||||
|
||||
3. **Scaling**:
|
||||
- Scale worker services horizontally
|
||||
- Use load balancer for API instances
|
||||
- Separate read/write database connections
|
||||
|
||||
### Backup and Recovery
|
||||
|
||||
1. **Database backups**:
|
||||
```bash
|
||||
# Backup
|
||||
docker-compose exec postgres pg_dump -U attune > backup.sql
|
||||
|
||||
# Restore
|
||||
docker-compose exec -T postgres psql -U attune < backup.sql
|
||||
```
|
||||
|
||||
2. **Volume backups**:
|
||||
```bash
|
||||
# Backup volumes
|
||||
docker run --rm -v attune_postgres_data:/data -v $(pwd):/backup \
|
||||
alpine tar czf /backup/postgres_backup.tar.gz /data
|
||||
```
|
||||
|
||||
3. **Automated backups**:
|
||||
- Use backup solutions (Velero, Restic)
|
||||
- Schedule regular backups with cron
|
||||
- Test restore procedures regularly
|
||||
|
||||
### Logging
|
||||
|
||||
1. **Centralized logging**:
|
||||
- Configure Docker logging driver
|
||||
- Use ELK stack, Loki, or CloudWatch
|
||||
- Set log retention policies
|
||||
|
||||
2. **Structured logging**:
|
||||
- Services already use JSON logging in Docker mode
|
||||
- Parse and index logs for analysis
|
||||
|
||||
### Monitoring
|
||||
|
||||
1. **Metrics collection**:
|
||||
- Prometheus + Grafana
|
||||
- DataDog, New Relic
|
||||
- Container metrics (cAdvisor)
|
||||
|
||||
2. **Alerting**:
|
||||
- Set up alerts for service failures
|
||||
- Monitor resource usage
|
||||
- Track error rates
|
||||
|
||||
### High Availability
|
||||
|
||||
1. **Database**:
|
||||
- PostgreSQL streaming replication
|
||||
- Automatic failover (Patroni, Stolon)
|
||||
- Regular backups
|
||||
|
||||
2. **Message Queue**:
|
||||
- RabbitMQ clustering
|
||||
- Mirrored queues
|
||||
- Load balancing
|
||||
|
||||
3. **Application**:
|
||||
- Run multiple instances
|
||||
- Use container orchestration (Kubernetes, Swarm)
|
||||
- Implement circuit breakers
|
||||
|
||||
## Makefile Integration
|
||||
|
||||
The project Makefile includes Docker commands:
|
||||
|
||||
```bash
|
||||
# Build images
|
||||
make docker-build
|
||||
|
||||
# Start services
|
||||
make docker-up
|
||||
|
||||
# Stop services
|
||||
make docker-down
|
||||
|
||||
# View logs
|
||||
make docker-logs
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
- Review [Production Deployment Guide](production-deployment.md) for production best practices
|
||||
- Configure [Monitoring and Alerting](../README.md#monitoring)
|
||||
- Set up [Backup and Recovery](../README.md#backup)
|
||||
- Review [Security Guide](authentication/security-review-2024-01-02.md)
|
||||
|
||||
## Support
|
||||
|
||||
For issues and questions:
|
||||
- Check [Troubleshooting](#troubleshooting) section
|
||||
- Review service logs: `docker-compose logs <service>`
|
||||
- Open an issue on GitHub
|
||||
- Consult the [Documentation Index](../AGENTS.md)
|
||||
Reference in New Issue
Block a user