attune/docs/QUICKREF-packs-volumes.md

Quick Reference: Packs Volume Architecture

TL;DR

Packs are NOT copied into Docker images. They are mounted as volumes.

# Build pack binaries (one-time or when updated)
./scripts/build-pack-binaries.sh

# Start services - init-packs copies packs to volume
docker compose up -d

# Update pack files - no image rebuild needed!
vim packs/core/actions/my_action.yaml
docker compose restart

Architecture Overview

Host Filesystem          Docker Volumes           Service Containers
─────────────────        ───────────────          ──────────────────

./packs/                                          
  ├── core/                                       
  │   ├── actions/                                
  │   ├── sensors/                                
  │   └── pack.yaml                               
  │                                               
  │                      ┌─────────────┐          
  │   (copy during       │ packs_data  │──────────> /opt/attune/packs (api)
  │    init-packs)       │   volume    │          
  │         └────────────>│             │──────────> /opt/attune/packs (executor)
  │                      │             │          
  │                      │             │──────────> /opt/attune/packs (worker)
  │                      │             │          
  │                      │             │──────────> /opt/attune/packs (sensor)
  │                      └─────────────┘          
  │                                               
./packs.dev/                                      
  └── custom-pack/       ┌────────────────────────> /opt/attune/packs.dev (all)
       (bind mount)      │ (read-write for dev)
                         │
                         └─ (mounted directly)

Why Volumes Instead of COPY?

Aspect	COPY into Image	Volume Mount
Update packs	Rebuild image (~5 min)	Restart service (~5 sec)
Image size	Larger (+packs)	Smaller (no packs)
Development	Slow iteration	Fast iteration
Consistency	Each service separate	All services share
Pack binaries	Baked into image	Updateable

docker-compose.yaml Configuration

volumes:
  packs_data:
    driver: local

services:
  # Step 1: init-packs runs once to populate packs_data volume
  init-packs:
    image: python:3.11-alpine
    volumes:
      - ./packs:/source/packs:ro              # Host packs (read-only)
      - packs_data:/opt/attune/packs          # Target volume
    command: ["/bin/sh", "/init-packs.sh"]
    restart: on-failure

  # Step 2: Services mount packs_data as read-only
  api:
    volumes:
      - packs_data:/opt/attune/packs:ro       # Production packs (RO)
      - ./packs.dev:/opt/attune/packs.dev:rw  # Dev packs (RW)
    depends_on:
      init-packs:
        condition: service_completed_successfully

  worker-shell:
    volumes:
      - packs_data:/opt/attune/packs:ro       # Same volume
      - ./packs.dev:/opt/attune/packs.dev:rw

  # ... all services follow same pattern

Pack Binaries (Native Code)

Some packs contain compiled binaries (e.g., sensors written in Rust).

Building Pack Binaries

Option 1: Use the script (recommended)

./scripts/build-pack-binaries.sh

Option 2: Manual build

# Build in Docker with GLIBC compatibility
docker build -f docker/Dockerfile.pack-binaries -t attune-pack-builder .

# Extract binaries
docker create --name pack-tmp attune-pack-builder
docker cp pack-tmp:/pack-binaries/. ./packs/
docker rm pack-tmp

Option 3: Native build (if GLIBC matches)

cargo build --release --bin attune-core-timer-sensor
cp target/release/attune-core-timer-sensor packs/core/sensors/

When to Rebuild Pack Binaries

• ✅ After git pull that updates pack binary source
• ✅ After modifying sensor source code (e.g., crates/core-timer-sensor)
• ✅ When setting up development environment for first time
• ❌ NOT needed for YAML/script changes in packs

Development Workflow

Editing Pack YAML Files

# 1. Edit pack files
vim packs/core/actions/echo.yaml

# 2. Restart services (no rebuild!)
docker compose restart

# 3. Test changes
curl -X POST http://localhost:8080/api/v1/executions \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"action_ref": "core.echo", "parameters": {"message": "hello"}}'

Time: ~5 seconds

Editing Pack Scripts (Python/Shell)

# 1. Edit script
vim packs/core/actions/http_request.py

# 2. Restart services
docker compose restart worker-python

# 3. Test
# (run execution)

Time: ~5 seconds

Editing Pack Binaries (Native Sensors)

# 1. Edit source
vim crates/core-timer-sensor/src/main.rs

# 2. Rebuild binary
./scripts/build-pack-binaries.sh

# 3. Restart services
docker compose restart sensor

# 4. Test
# (check sensor registration)

Time: ~2 minutes (compile + restart)

Development Packs (packs.dev)

For rapid development, use the packs.dev directory:

# Create a dev pack
mkdir -p packs.dev/mypack/actions

# Create action
cat > packs.dev/mypack/actions/test.yaml <<EOF
name: test
description: Test action
runner_type: Shell
entry_point: echo.sh
parameters:
  message:
    type: string
    required: true
EOF

cat > packs.dev/mypack/actions/echo.sh <<'EOF'
#!/bin/bash
echo "Message: $ATTUNE_MESSAGE"
EOF

chmod +x packs.dev/mypack/actions/echo.sh

# Restart to pick up changes
docker compose restart

# Test immediately - no rebuild needed!

Benefits of packs.dev:

• ✅ Direct bind mount (changes visible immediately)
• ✅ Read-write access (can modify from container)
• ✅ No init-packs step needed
• ✅ Perfect for iteration

Optimized Dockerfiles and Packs

The optimized Dockerfiles (docker/Dockerfile.optimized) do NOT copy packs:

# ❌ OLD: Packs copied into image
COPY packs/ ./packs/

# ✅ NEW: Only create mount point
RUN mkdir -p /opt/attune/packs /opt/attune/logs

# Packs mounted at runtime from packs_data volume

Result:

• Service images contain only binaries + configs
• Packs updated independently
• Faster builds (no pack layer invalidation)

Troubleshooting

"Pack not found" errors

Symptom: API returns 404 for pack/action Cause: Packs not loaded into volume

Fix:

# Check if packs exist in volume
docker compose exec api ls -la /opt/attune/packs/

# If empty, restart init-packs
docker compose restart init-packs
docker compose logs init-packs

Pack changes not visible

Symptom: Updated pack.yaml but changes not reflected Cause: Changes made to host ./packs/ after init-packs ran

Fix:

# Option 1: Use packs.dev for development
mv packs/mypack packs.dev/mypack
docker compose restart

# Option 2: Recreate packs_data volume
docker compose down
docker volume rm attune_packs_data
docker compose up -d

Pack binary "exec format error"

Symptom: Sensor binary fails with exec format error Cause: Binary compiled for wrong architecture or GLIBC version

Fix:

# Rebuild with Docker (ensures compatibility)
./scripts/build-pack-binaries.sh

# Restart sensor service
docker compose restart sensor

Pack binary "permission denied"

Symptom: Binary exists but can't execute Cause: Binary not executable

Fix:

chmod +x packs/core/sensors/attune-core-timer-sensor
docker compose restart init-packs sensor

Best Practices

DO:

• ✅ Use ./scripts/build-pack-binaries.sh for pack binaries
• ✅ Put development packs in packs.dev/
• ✅ Keep production packs in packs/
• ✅ Commit pack YAML/scripts to git
• ✅ Use .gitignore for compiled pack binaries
• ✅ Restart services after pack changes
• ✅ Use init-packs logs to debug loading issues

DON'T:

• ❌ Don't copy packs into Dockerfiles
• ❌ Don't edit packs inside running containers
• ❌ Don't commit compiled pack binaries to git
• ❌ Don't expect instant updates to packs/ (need restart)
• ❌ Don't rebuild service images for pack changes
• ❌ Don't modify packs_data volume directly

Migration from Old Dockerfiles

If your old Dockerfiles copied packs:

# OLD Dockerfile
COPY packs/ ./packs/
COPY --from=pack-builder /build/pack-binaries/ ./packs/

Migration steps:

1. Build pack binaries separately:

   ./scripts/build-pack-binaries.sh
   

2. Update to optimized Dockerfile:

   # docker-compose.yaml
   api:
     build:
       dockerfile: docker/Dockerfile.optimized
   

3. Rebuild service images:

   docker compose build --no-cache
   

4. Start services (init-packs will populate volume):

   docker compose up -d
   

Summary

Architecture: Packs → Volume → Services

• Host ./packs/ copied to packs_data volume by init-packs
• Services mount packs_data as read-only
• Dev packs in packs.dev/ bind-mounted directly

Benefits:

• 90% faster pack updates (restart vs rebuild)
• Smaller service images
• Consistent packs across all services
• Clear separation: services = code, packs = content

Key Commands:

./scripts/build-pack-binaries.sh   # Build native pack binaries
docker compose restart              # Pick up pack changes
docker compose logs init-packs      # Debug pack loading

Remember: Packs are content, not code. Treat them as configuration, not part of the service image.