attune-system/attune
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

trying to make the pipeline builds work, desperately.

Browse Source
This commit is contained in:
David Culbreth
2026-03-20 20:15:44 -05:00
parent 1c16f65476
commit 5a18c73572
13 changed files with 497 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
.dockerignore
View File

@@ -50,8 +50,8 @@ web/node_modules/
web/dist/
web/.vite/
# SQLx offline data (generated at build time)
#.sqlx/
# SQLx offline data (generated when using `cargo sqlx prepare`)
# .sqlx/
# Configuration files (copied selectively)
config.development.yaml

48
.gitea/workflows/ci.yml
View File

@@ -9,10 +9,12 @@ on:
env:
CARGO_TERM_COLOR: always
RUST_MIN_STACK: 16777216
RUST_MIN_STACK: 67108864
CARGO_INCREMENTAL: 0
CARGO_NET_RETRY: 10
RUSTUP_MAX_RETRIES: 10
# Gitea Actions runner tool cache. Actions like setup-node/setup-python can reuse this.
RUNNER_TOOL_CACHE: /toolcache
jobs:
rust-fmt:
@@ -22,6 +24,17 @@ jobs:
- name: Checkout
uses: actions/checkout@v4
- name: Cache Rust toolchain
uses: actions/cache@v4
with:
path: |
~/.rustup/toolchains
~/.rustup/update-hashes
key: rustup-rustfmt-${{ runner.os }}-stable-v1
restore-keys: |
rustup-${{ runner.os }}-stable-v1
rustup-
- name: Setup Rust
uses: dtolnay/rust-toolchain@stable
with:
@@ -37,6 +50,17 @@ jobs:
- name: Checkout
uses: actions/checkout@v4
- name: Cache Rust toolchain
uses: actions/cache@v4
with:
path: |
~/.rustup/toolchains
~/.rustup/update-hashes
key: rustup-clippy-${{ runner.os }}-stable-v1
restore-keys: |
rustup-${{ runner.os }}-stable-v1
rustup-
- name: Setup Rust
uses: dtolnay/rust-toolchain@stable
with:
@@ -72,6 +96,17 @@ jobs:
- name: Checkout
uses: actions/checkout@v4
- name: Cache Rust toolchain
uses: actions/cache@v4
with:
path: |
~/.rustup/toolchains
~/.rustup/update-hashes
key: rustup-test-${{ runner.os }}-stable-v1
restore-keys: |
rustup-${{ runner.os }}-stable-v1
rustup-
- name: Setup Rust
uses: dtolnay/rust-toolchain@stable
@@ -105,6 +140,17 @@ jobs:
- name: Checkout
uses: actions/checkout@v4
- name: Cache Rust toolchain
uses: actions/cache@v4
with:
path: |
~/.rustup/toolchains
~/.rustup/update-hashes
key: rustup-audit-${{ runner.os }}-stable-v1
restore-keys: |
rustup-${{ runner.os }}-stable-v1
rustup-
- name: Setup Rust
uses: dtolnay/rust-toolchain@stable

2
AGENTS.md
View File

@@ -102,7 +102,7 @@ docker compose logs -f <svc> # View logs
- **BuildKit cache mounts**: Persist cargo registry and compilation artifacts between builds
- **Cache strategy**: `sharing=shared` for registry/git (concurrent-safe), service-specific IDs for target caches
- **Parallel builds**: 4x faster than old `sharing=locked` strategy - no serialization overhead
- **Rustc stack size**: All Rust Dockerfiles set `ENV RUST_MIN_STACK=16777216` (16 MiB) in the build stage to prevent `rustc` SIGSEGV crashes during release compilation. The `Makefile` also exports this variable for local builds.
- **Rustc stack size**: All Rust Dockerfiles set `ENV RUST_MIN_STACK=67108864` (64 MiB) in the build stage to prevent `rustc` SIGSEGV crashes during release compilation. The `Makefile` also exports this variable for local builds.
- **Documentation**: See `docs/docker-layer-optimization.md`, `docs/QUICKREF-docker-optimization.md`, `docs/QUICKREF-buildkit-cache-strategy.md`
### Docker Runtime Standardization

2
Makefile
View File

@@ -69,7 +69,7 @@ help:
@echo ""
# Increase rustc stack size to prevent SIGSEGV during compilation
export RUST_MIN_STACK := 16777216
export RUST_MIN_STACK:=67108864
# Building
build:

148
docker/Dockerfile Normal file
View File

@@ -0,0 +1,148 @@
# Multi-stage Dockerfile for Attune Rust services
# This Dockerfile can build any of the Attune services by specifying a build argument
# Usage: DOCKER_BUILDKIT=1 docker build --build-arg SERVICE=api -f docker/Dockerfile -t attune-api .
#
# BuildKit cache mounts are used to speed up incremental builds by persisting:
# - Cargo registry and git cache (with sharing=locked to prevent race conditions)
# - Rust incremental compilation artifacts
#
# This dramatically reduces rebuild times from ~5 minutes to ~30 seconds for code-only changes.
ARG RUST_VERSION=1.92
ARG DEBIAN_VERSION=bookworm
# ============================================================================
# Stage 1: Builder - Compile the Rust services
# ============================================================================
FROM rust:${RUST_VERSION}-${DEBIAN_VERSION} AS builder
# Install build dependencies
RUN apt-get update && apt-get install -y \
pkg-config \
libssl-dev \
ca-certificates \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /build
# Increase rustc stack size to prevent SIGSEGV during release builds
ENV RUST_MIN_STACK=67108864
# Copy workspace manifests and source code
COPY Cargo.toml Cargo.lock ./
COPY crates/ ./crates/
COPY migrations/ ./migrations/
COPY .sqlx/ ./.sqlx/
# Build argument to specify which service to build
ARG SERVICE=api
# Build the specified service with BuildKit cache mounts
# Cache mount sharing modes prevent race conditions during parallel builds:
# - sharing=locked: Only one build can access the cache at a time (prevents file conflicts)
# - cargo registry/git: Locked to prevent "File exists" errors when extracting dependencies
# - target: Locked to prevent compilation artifact conflicts
#
# This is slower than parallel builds but eliminates race conditions.
# Alternative: Use docker-compose --build with --no-parallel flag, or build sequentially.
#
# First build: ~5-6 minutes
# Incremental builds (code changes only): ~30-60 seconds
RUN --mount=type=cache,target=/usr/local/cargo/registry,sharing=locked \
--mount=type=cache,target=/usr/local/cargo/git,sharing=locked \
--mount=type=cache,target=/build/target,sharing=locked \
cargo build --release --bin attune-${SERVICE} && \
cp /build/target/release/attune-${SERVICE} /build/attune-service-binary
# ============================================================================
# Stage 2: Pack Binaries Builder - Build native pack binaries with GLIBC 2.36
# ============================================================================
FROM rust:${RUST_VERSION}-${DEBIAN_VERSION} AS pack-builder
# Install build dependencies
RUN apt-get update && apt-get install -y \
pkg-config \
libssl-dev \
ca-certificates \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /build
# Increase rustc stack size to prevent SIGSEGV during release builds
ENV RUST_MIN_STACK=67108864
# Copy workspace files
COPY Cargo.toml Cargo.lock ./
COPY crates/ ./crates/
COPY .sqlx/ ./.sqlx/
# Build pack binaries (sensors, etc.) with GLIBC 2.36 for maximum compatibility
# These binaries will work on any system with GLIBC 2.36 or newer
# IMPORTANT: Copy binaries WITHIN the cache mount, before it's unmounted
RUN --mount=type=cache,target=/usr/local/cargo/registry,sharing=locked \
--mount=type=cache,target=/usr/local/cargo/git,sharing=locked \
--mount=type=cache,target=/build/target,sharing=locked \
mkdir -p /build/pack-binaries && \
cargo build --release --bin attune-core-timer-sensor && \
cp /build/target/release/attune-core-timer-sensor /build/pack-binaries/attune-core-timer-sensor && \
ls -lh /build/pack-binaries/
# Verify binaries were copied successfully (after cache unmount)
RUN ls -lah /build/pack-binaries/ && \
test -f /build/pack-binaries/attune-core-timer-sensor && \
echo "Timer sensor binary built successfully"
# ============================================================================
# Stage 3: Runtime - Create minimal runtime image
# ============================================================================
FROM debian:${DEBIAN_VERSION}-slim AS runtime
# Install runtime dependencies
RUN apt-get update && apt-get install -y \
ca-certificates \
libssl3 \
curl \
&& rm -rf /var/lib/apt/lists/*
# Create non-root user
RUN useradd -m -u 1000 attune && \
mkdir -p /opt/attune/packs /opt/attune/logs /opt/attune/config && \
chown -R attune:attune /opt/attune
WORKDIR /opt/attune
# Copy the service binary from builder
# Note: We copy from /build/attune-service-binary because the cache mount is not available in COPY
COPY --from=builder /build/attune-service-binary /usr/local/bin/attune-service
# Copy migrations for services that need them.
# Runtime config is mounted via Docker Compose.
COPY migrations/ ./migrations/
# Copy packs directory (excluding binaries that will be overwritten)
COPY packs/ ./packs/
# Overwrite pack binaries with ones built with compatible GLIBC from pack-builder stage
# Copy individual files to ensure they overwrite existing ones
COPY --from=pack-builder /build/pack-binaries/attune-core-timer-sensor ./packs/core/sensors/attune-core-timer-sensor
# Make binaries executable and set ownership
RUN chmod +x ./packs/core/sensors/attune-core-timer-sensor && \
chown -R attune:attune /opt/attune
# Switch to non-root user
USER attune
# Environment variables (can be overridden at runtime)
ENV RUST_LOG=info
ENV ATTUNE_CONFIG=/opt/attune/config/config.yaml
# Health check (will be overridden per service in docker-compose)
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
CMD curl -f http://localhost:8080/health || exit 1
# Expose default port (override per service)
EXPOSE 8080
# Run the service
CMD ["/usr/local/bin/attune-service"]

5
docker/Dockerfile.optimized
View File

@@ -28,7 +28,7 @@ RUN apt-get update && apt-get install -y \
WORKDIR /build
# Increase rustc stack size to prevent SIGSEGV during release builds
ENV RUST_MIN_STACK=33554432
ENV RUST_MIN_STACK=67108864
# Copy dependency metadata first so `cargo fetch` layer is cached
# when only source code changes (Cargo.toml/Cargo.lock stay the same)
@@ -60,8 +60,7 @@ RUN --mount=type=cache,target=/usr/local/cargo/registry,sharing=shared \
--mount=type=cache,target=/usr/local/cargo/git,sharing=shared \
cargo fetch
# Now copy the real source code, SQLx metadata, and migrations
COPY .sqlx/ ./.sqlx/
# Now copy the real source code and migrations
COPY migrations/ ./migrations/
COPY crates/ ./crates/

5
docker/Dockerfile.pack-binaries
View File

@@ -30,7 +30,7 @@ RUN apt-get update && apt-get install -y \
WORKDIR /build
# Increase rustc stack size to prevent SIGSEGV during release builds
ENV RUST_MIN_STACK=16777216
ENV RUST_MIN_STACK=67108864
# Copy workspace configuration
COPY Cargo.toml Cargo.lock ./
@@ -54,9 +54,6 @@ RUN mkdir -p crates/worker/src && echo "fn main() {}" > crates/worker/src/main.r
RUN mkdir -p crates/notifier/src && echo "fn main() {}" > crates/notifier/src/main.rs
RUN mkdir -p crates/cli/src && echo "fn main() {}" > crates/cli/src/main.rs
# Copy SQLx metadata for compile-time query checking
COPY .sqlx/ ./.sqlx/
# Copy only the source code needed for pack binaries
COPY crates/common/ ./crates/common/
COPY crates/core-timer-sensor/ ./crates/core-timer-sensor/

3
docker/Dockerfile.pack-builder
View File

@@ -31,12 +31,11 @@ RUN apt-get update && apt-get install -y \
WORKDIR /build
# Increase rustc stack size to prevent SIGSEGV during release builds
ENV RUST_MIN_STACK=16777216
ENV RUST_MIN_STACK=67108864
# Copy workspace files
COPY Cargo.toml Cargo.lock ./
COPY crates/ ./crates/
COPY .sqlx/ ./.sqlx/
# Build all pack binaries in release mode with BuildKit cache
RUN --mount=type=cache,target=/usr/local/cargo/registry,sharing=locked \

5
docker/Dockerfile.sensor.optimized
View File

@@ -31,7 +31,7 @@ RUN apt-get update && apt-get install -y \
WORKDIR /build
# Increase rustc stack size to prevent SIGSEGV during release builds
ENV RUST_MIN_STACK=33554432
ENV RUST_MIN_STACK=67108864
# Copy dependency metadata first so `cargo fetch` layer is cached
# when only source code changes (Cargo.toml/Cargo.lock stay the same)
@@ -62,8 +62,7 @@ RUN --mount=type=cache,target=/usr/local/cargo/registry,sharing=shared \
--mount=type=cache,target=/usr/local/cargo/git,sharing=shared \
cargo fetch
# Now copy the real source code, SQLx metadata, and migrations
COPY .sqlx/ ./.sqlx/
# Now copy the real source code and migrations
COPY migrations/ ./migrations/
COPY crates/ ./crates/

288
docker/Dockerfile.worker Normal file
View File

@@ -0,0 +1,288 @@
# Multi-stage Dockerfile for Attune workers
# Supports building different worker variants with different runtime capabilities
#
# Usage:
# docker build --target worker-base -t attune-worker:base -f docker/Dockerfile.worker .
# docker build --target worker-python -t attune-worker:python -f docker/Dockerfile.worker .
# docker build --target worker-node -t attune-worker:node -f docker/Dockerfile.worker .
# docker build --target worker-full -t attune-worker:full -f docker/Dockerfile.worker .
#
# BuildKit cache mounts are used to speed up incremental builds.
ARG RUST_VERSION=1.92
ARG DEBIAN_VERSION=bookworm
ARG NODE_VERSION=20
# ============================================================================
# Stage 1: Builder - Compile the worker binary
# ============================================================================
FROM rust:${RUST_VERSION}-${DEBIAN_VERSION} AS builder
# Install build dependencies
RUN apt-get update && apt-get install -y \
pkg-config \
libssl-dev \
ca-certificates \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /build
# Increase rustc stack size to prevent SIGSEGV during release builds
ENV RUST_MIN_STACK=67108864
# Copy workspace manifests and source code
COPY Cargo.toml Cargo.lock ./
COPY crates/ ./crates/
COPY migrations/ ./migrations/
COPY .sqlx/ ./.sqlx/
# Build the worker binary with BuildKit cache mounts
# sharing=locked prevents race conditions during parallel builds
RUN --mount=type=cache,target=/usr/local/cargo/registry,sharing=locked \
--mount=type=cache,target=/usr/local/cargo/git,sharing=locked \
--mount=type=cache,target=/build/target,sharing=locked \
cargo build --release --bin attune-worker && \
cp /build/target/release/attune-worker /build/attune-worker
# Verify the binary was built
RUN ls -lh /build/attune-worker && \
file /build/attune-worker && \
/build/attune-worker --version || echo "Version check skipped"
# ============================================================================
# Stage 2a: Base Worker (Shell only)
# Runtime capabilities: shell
# Use case: Lightweight workers for shell scripts and basic automation
# ============================================================================
FROM debian:${DEBIAN_VERSION}-slim AS worker-base
# Install runtime dependencies
RUN apt-get update && apt-get install -y \
ca-certificates \
libssl3 \
curl \
bash \
procps \
&& rm -rf /var/lib/apt/lists/*
# Create worker user and directories
RUN useradd -m -u 1000 attune && \
mkdir -p /opt/attune/packs /opt/attune/logs /opt/attune/config && \
chown -R attune:attune /opt/attune
WORKDIR /opt/attune
# Copy worker binary from builder
COPY --from=builder /build/attune-worker /usr/local/bin/attune-worker
# Copy packs directory
COPY packs/ ./packs/
# Set ownership
RUN chown -R attune:attune /opt/attune
# Switch to non-root user
USER attune
# Environment variables
ENV ATTUNE_WORKER_RUNTIMES="shell"
ENV ATTUNE_WORKER_TYPE="container"
ENV RUST_LOG=info
ENV ATTUNE_CONFIG=/opt/attune/config/config.yaml
# Health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
CMD pgrep -f attune-worker || exit 1
# Run the worker
CMD ["/usr/local/bin/attune-worker"]
# ============================================================================
# Stage 2b: Python Worker (Shell + Python)
# Runtime capabilities: shell, python
# Use case: Python actions and scripts with dependencies
#
# Uses debian-slim + apt python3 (NOT the python: Docker image) so that
# python3 lives at /usr/bin/python3 — the same path as worker-full.
# This avoids broken venv symlinks when multiple workers share the
# runtime_envs volume.
# ============================================================================
FROM debian:${DEBIAN_VERSION}-slim AS worker-python
# Install system dependencies including Python
RUN apt-get update && apt-get install -y \
ca-certificates \
libssl3 \
curl \
build-essential \
python3 \
python3-pip \
python3-venv \
procps \
&& rm -rf /var/lib/apt/lists/*
# Create python symlink for convenience
RUN ln -sf /usr/bin/python3 /usr/bin/python
# Install common Python packages
# Use --break-system-packages for Debian 12+ pip-in-system-python restrictions
RUN pip3 install --no-cache-dir --break-system-packages \
requests>=2.31.0 \
pyyaml>=6.0 \
jinja2>=3.1.0 \
python-dateutil>=2.8.0
# Create worker user and directories
RUN useradd -m -u 1000 attune && \
mkdir -p /opt/attune/packs /opt/attune/logs /opt/attune/runtime_envs /opt/attune/config && \
chown -R attune:attune /opt/attune
WORKDIR /opt/attune
# Copy worker binary from builder
COPY --from=builder /build/attune-worker /usr/local/bin/attune-worker
# Copy packs directory
COPY packs/ ./packs/
# Set ownership
RUN chown -R attune:attune /opt/attune
# Switch to non-root user
USER attune
# Environment variables
ENV ATTUNE_WORKER_RUNTIMES="shell,python"
ENV ATTUNE_WORKER_TYPE="container"
ENV RUST_LOG=info
ENV ATTUNE_CONFIG=/opt/attune/config/config.yaml
# Health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
CMD pgrep -f attune-worker || exit 1
# Run the worker
CMD ["/usr/local/bin/attune-worker"]
# ============================================================================
# Stage 2c: Node Worker (Shell + Node.js)
# Runtime capabilities: shell, node
# Use case: JavaScript/TypeScript actions and npm packages
#
# Uses debian-slim + NodeSource apt repo (NOT the node: Docker image) so that
# node lives at /usr/bin/node — the same path as worker-full.
# This avoids path mismatches when multiple workers share volumes.
# ============================================================================
FROM debian:${DEBIAN_VERSION}-slim AS worker-node
# Install system dependencies
RUN apt-get update && apt-get install -y \
ca-certificates \
libssl3 \
curl \
procps \
&& rm -rf /var/lib/apt/lists/*
# Install Node.js from NodeSource (same method as worker-full)
RUN curl -fsSL https://deb.nodesource.com/setup_${NODE_VERSION}.x | bash - && \
apt-get install -y nodejs && \
rm -rf /var/lib/apt/lists/*
# Create worker user and directories
RUN useradd -m -u 1000 attune && \
mkdir -p /opt/attune/packs /opt/attune/logs /opt/attune/runtime_envs /opt/attune/config && \
chown -R attune:attune /opt/attune
WORKDIR /opt/attune
# Copy worker binary from builder
COPY --from=builder /build/attune-worker /usr/local/bin/attune-worker
# Copy packs directory
COPY packs/ ./packs/
# Set ownership
RUN chown -R attune:attune /opt/attune
# Switch to non-root user
USER attune
# Environment variables
ENV ATTUNE_WORKER_RUNTIMES="shell,node"
ENV ATTUNE_WORKER_TYPE="container"
ENV RUST_LOG=info
ENV ATTUNE_CONFIG=/opt/attune/config/config.yaml
# Health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
CMD pgrep -f attune-worker || exit 1
# Run the worker
CMD ["/usr/local/bin/attune-worker"]
# ============================================================================
# Stage 2d: Full Worker (All runtimes)
# Runtime capabilities: shell, python, node, native
# Use case: General-purpose automation with multi-language support
# ============================================================================
FROM debian:${DEBIAN_VERSION} AS worker-full
# Install system dependencies including Python and Node.js
RUN apt-get update && apt-get install -y \
ca-certificates \
libssl3 \
curl \
build-essential \
python3 \
python3-pip \
python3-venv \
procps \
&& rm -rf /var/lib/apt/lists/*
# Install Node.js from NodeSource (same method and version as worker-node)
RUN curl -fsSL https://deb.nodesource.com/setup_${NODE_VERSION}.x | bash - && \
apt-get install -y nodejs && \
rm -rf /var/lib/apt/lists/*
# Create python symlink for convenience
RUN ln -sf /usr/bin/python3 /usr/bin/python
# Install common Python packages
# Use --break-system-packages for Debian 12+ pip-in-system-python restrictions
RUN pip3 install --no-cache-dir --break-system-packages \
requests>=2.31.0 \
pyyaml>=6.0 \
jinja2>=3.1.0 \
python-dateutil>=2.8.0
# Create worker user and directories
RUN useradd -m -u 1000 attune && \
mkdir -p /opt/attune/packs /opt/attune/logs /opt/attune/runtime_envs /opt/attune/config && \
chown -R attune:attune /opt/attune
WORKDIR /opt/attune
# Copy worker binary from builder
COPY --from=builder /build/attune-worker /usr/local/bin/attune-worker
# Copy packs directory
COPY packs/ ./packs/
# Set ownership
RUN chown -R attune:attune /opt/attune
# Switch to non-root user
USER attune
# Environment variables
ENV ATTUNE_WORKER_RUNTIMES="shell,python,node,native"
ENV ATTUNE_WORKER_TYPE="container"
ENV RUST_LOG=info
ENV ATTUNE_CONFIG=/opt/attune/config/config.yaml
# Health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
CMD pgrep -f attune-worker || exit 1
# Run the worker
CMD ["/usr/local/bin/attune-worker"]

5
docker/Dockerfile.worker.optimized
View File

@@ -36,7 +36,7 @@ RUN apt-get update && apt-get install -y \
WORKDIR /build
# Increase rustc stack size to prevent SIGSEGV during release builds
ENV RUST_MIN_STACK=33554432
ENV RUST_MIN_STACK=67108864
# Copy dependency metadata first so `cargo fetch` layer is cached
# when only source code changes (Cargo.toml/Cargo.lock stay the same)
@@ -67,8 +67,7 @@ RUN --mount=type=cache,target=/usr/local/cargo/registry,sharing=shared \
--mount=type=cache,target=/usr/local/cargo/git,sharing=shared \
cargo fetch
# Now copy the real source code, SQLx metadata, and migrations
COPY .sqlx/ ./.sqlx/
# Now copy the real source code and migrations
COPY migrations/ ./migrations/
COPY crates/ ./crates/

4
docker/README.md
View File

@@ -425,10 +425,6 @@ docker builder prune -a
docker compose build --no-cache
```
**SQLx compile-time verification fails:**
- Ensure `.sqlx/` directory is present
- Regenerate with `cargo sqlx prepare` if needed
### Runtime Issues
**Service won't start:**

4
work-summary/2026-02-05-rust-min-stack-and-workflow-file-separation.md
View File

@@ -12,7 +12,7 @@ Docker Compose builds were failing with `rustc interrupted by SIGSEGV` during re
### Fix
Added `ENV RUST_MIN_STACK=16777216` to the build stage of all 7 Rust Dockerfiles:
Added `ENV RUST_MIN_STACK=67108864` to the build stage of all 7 Rust Dockerfiles:
- `docker/Dockerfile` (both build stages)
- `docker/Dockerfile.optimized`
@@ -117,4 +117,4 @@ The visual workflow builder and API save endpoints now produce the action-linked
- All 598 workspace lib tests pass
- Zero TypeScript errors
- Zero compiler warnings
- Zero build errors
- Zero build errors