Compare commits

5 Commits

Author SHA1 Message Date
af5175b96a removing no-longer-used dockerfiles.
Some checks failed
CI / Cargo Audit & Deny (push) Successful in 1m10s
CI / Security Blocking Checks (push) Successful in 10s
CI / Web Advisory Checks (push) Successful in 1m13s
CI / Clippy (push) Failing after 2m50s
Publish Images And Chart / Resolve Publish Metadata (push) Successful in 1s
CI / Security Advisory Checks (push) Successful in 1m24s
Publish Images And Chart / Publish init-packs (push) Failing after 12s
CI / Rustfmt (push) Successful in 4m22s
Publish Images And Chart / Publish web (push) Successful in 45s
Publish Images And Chart / Publish worker (push) Failing after 54s
Publish Images And Chart / Publish agent (push) Successful in 4m14s
CI / Web Blocking Checks (push) Successful in 9m31s
CI / Tests (push) Successful in 9m41s
Publish Images And Chart / Publish migrations (push) Failing after 13s
Publish Images And Chart / Publish sensor (push) Failing after 12s
Publish Images And Chart / Publish init-user (push) Failing after 2m3s
Publish Images And Chart / Publish api (push) Successful in 8m55s
Publish Images And Chart / Publish notifier (push) Successful in 8m53s
Publish Images And Chart / Publish executor (push) Successful in 1h16m29s
Publish Images And Chart / Publish Helm Chart (push) Has been skipped
2026-03-23 13:05:53 -05:00
8af8c1af9c first iteration of agent-style worker and sensor containers. 2026-03-23 12:49:15 -05:00
d4c6240485 agent workers 2026-03-21 10:05:02 -05:00
4d5a3b1bf5 agent-style workers 2026-03-21 08:27:20 -05:00
8ba7e3bb84 [wip] universal workers 2026-03-21 07:32:11 -05:00
87 changed files with 6510 additions and 1362 deletions

View File

@@ -163,6 +163,12 @@ jobs:
context: . context: .
target: "" target: ""
build_args: "" build_args: ""
- name: agent
repository: attune-agent
dockerfile: docker/Dockerfile.agent
context: .
target: agent-init
build_args: ""
steps: steps:
- name: Checkout - name: Checkout
uses: actions/checkout@v4 uses: actions/checkout@v4

File diff suppressed because one or more lines are too long

1
Cargo.lock generated
View File

@@ -490,6 +490,7 @@ dependencies = [
"sha1", "sha1",
"sha2", "sha2",
"sqlx", "sqlx",
"subtle",
"tar", "tar",
"tempfile", "tempfile",
"thiserror 2.0.18", "thiserror 2.0.18",

View File

@@ -21,7 +21,7 @@ repository = "https://git.rdrx.app/attune-system/attune"
[workspace.dependencies] [workspace.dependencies]
# Async runtime # Async runtime
tokio = { version = "1.50", features = ["full"] } tokio = { version = "1.50", features = ["full"] }
tokio-util = "0.7" tokio-util = { version = "0.7", features = ["io"] }
tokio-stream = { version = "0.1", features = ["sync"] } tokio-stream = { version = "0.1", features = ["sync"] }
# Web framework # Web framework

View File

@@ -4,7 +4,9 @@
docker-build-workers docker-build-worker-base docker-build-worker-python \ docker-build-workers docker-build-worker-base docker-build-worker-python \
docker-build-worker-node docker-build-worker-full deny ci-rust ci-web-blocking ci-web-advisory \ docker-build-worker-node docker-build-worker-full deny ci-rust ci-web-blocking ci-web-advisory \
ci-security-blocking ci-security-advisory ci-blocking ci-advisory \ ci-security-blocking ci-security-advisory ci-blocking ci-advisory \
fmt-check pre-commit install-git-hooks fmt-check pre-commit install-git-hooks \
build-agent docker-build-agent run-agent run-agent-release \
docker-up-agent docker-down-agent
# Default target # Default target
help: help:
@@ -60,6 +62,14 @@ help:
@echo " make docker-up - Start services with docker compose" @echo " make docker-up - Start services with docker compose"
@echo " make docker-down - Stop services" @echo " make docker-down - Stop services"
@echo "" @echo ""
@echo "Agent (Universal Worker):"
@echo " make build-agent - Build statically-linked agent binary (musl)"
@echo " make docker-build-agent - Build agent Docker image"
@echo " make run-agent - Run agent in development mode"
@echo " make run-agent-release - Run agent in release mode"
@echo " make docker-up-agent - Start all services + agent workers (ruby, etc.)"
@echo " make docker-down-agent - Stop agent stack"
@echo ""
@echo "Development:" @echo "Development:"
@echo " make watch - Watch and rebuild on changes" @echo " make watch - Watch and rebuild on changes"
@echo " make install-tools - Install development tools" @echo " make install-tools - Install development tools"
@@ -227,38 +237,53 @@ docker-build-api:
docker-build-web: docker-build-web:
docker compose build web docker compose build web
# Build worker images # Agent binary (statically-linked for injection into any container)
docker-build-workers: docker-build-worker-base docker-build-worker-python docker-build-worker-node docker-build-worker-full build-agent:
@echo "✅ All worker images built successfully" @echo "Installing musl target (if not already installed)..."
rustup target add x86_64-unknown-linux-musl 2>/dev/null || true
@echo "Building statically-linked worker and sensor agent binaries..."
SQLX_OFFLINE=true cargo build --release --target x86_64-unknown-linux-musl --bin attune-agent --bin attune-sensor-agent
strip target/x86_64-unknown-linux-musl/release/attune-agent
strip target/x86_64-unknown-linux-musl/release/attune-sensor-agent
@echo "✅ Agent binaries built:"
@echo " - target/x86_64-unknown-linux-musl/release/attune-agent"
@echo " - target/x86_64-unknown-linux-musl/release/attune-sensor-agent"
@ls -lh target/x86_64-unknown-linux-musl/release/attune-agent
@ls -lh target/x86_64-unknown-linux-musl/release/attune-sensor-agent
docker-build-worker-base: docker-build-agent:
@echo "Building base worker (shell only)..." @echo "Building agent Docker image (statically-linked binary)..."
DOCKER_BUILDKIT=1 docker build --target worker-base -t attune-worker:base -f docker/Dockerfile.worker.optimized . DOCKER_BUILDKIT=1 docker buildx build --target agent-init -f docker/Dockerfile.agent -t attune-agent:latest .
@echo "✅ Base worker image built: attune-worker:base" @echo "✅ Agent image built: attune-agent:latest"
docker-build-worker-python: run-agent:
@echo "Building Python worker (shell + python)..." cargo run --bin attune-agent
DOCKER_BUILDKIT=1 docker build --target worker-python -t attune-worker:python -f docker/Dockerfile.worker.optimized .
@echo "✅ Python worker image built: attune-worker:python"
docker-build-worker-node: run-agent-release:
@echo "Building Node.js worker (shell + node)..." cargo run --bin attune-agent --release
DOCKER_BUILDKIT=1 docker build --target worker-node -t attune-worker:node -f docker/Dockerfile.worker.optimized .
@echo "✅ Node.js worker image built: attune-worker:node"
docker-build-worker-full: run-sensor-agent:
@echo "Building full worker (all runtimes)..." cargo run --bin attune-sensor-agent
DOCKER_BUILDKIT=1 docker build --target worker-full -t attune-worker:full -f docker/Dockerfile.worker.optimized .
@echo "✅ Full worker image built: attune-worker:full" run-sensor-agent-release:
cargo run --bin attune-sensor-agent --release
docker-up: docker-up:
@echo "Starting all services with Docker Compose..." @echo "Starting all services with Docker Compose..."
docker compose up -d docker compose up -d
docker-up-agent:
@echo "Starting all services + agent-based workers..."
docker compose -f docker-compose.yaml -f docker-compose.agent.yaml up -d
docker-down: docker-down:
@echo "Stopping all services..." @echo "Stopping all services..."
docker compose down docker compose down
docker-down-agent:
@echo "Stopping all services (including agent workers)..."
docker compose -f docker-compose.yaml -f docker-compose.agent.yaml down
docker-down-volumes: docker-down-volumes:
@echo "Stopping all services and removing volumes (WARNING: deletes data)..." @echo "Stopping all services and removing volumes (WARNING: deletes data)..."
docker compose down -v docker compose down -v

View File

@@ -1,3 +1,26 @@
1. Set `global.imageRegistry`, `global.imageNamespace`, and `global.imageTag` so the chart pulls the images published by the Gitea workflow. 1. Set `global.imageRegistry`, `global.imageNamespace`, and `global.imageTag` so the chart pulls the images published by the Gitea workflow.
2. Set `web.config.apiUrl` and `web.config.wsUrl` to browser-reachable endpoints before exposing the web UI. 2. Set `web.config.apiUrl` and `web.config.wsUrl` to browser-reachable endpoints before exposing the web UI.
3. The shared `packs`, `runtime_envs`, and `artifacts` PVCs default to `ReadWriteMany`; your cluster storage class must support RWX or you need to override those claims. 3. The shared `packs`, `runtime_envs`, and `artifacts` PVCs default to `ReadWriteMany`; your cluster storage class must support RWX or you need to override those claims.
{{- if .Values.agentWorkers }}
Agent-based workers enabled:
{{- range .Values.agentWorkers }}
- {{ .name }}: image={{ .image }}, replicas={{ .replicas | default 1 }}
{{- if .runtimes }} runtimes={{ join "," .runtimes }}{{ else }} runtimes=auto-detect{{ end }}
{{- end }}
Each agent worker uses an init container to copy the statically-linked
attune-agent binary into the worker pod via an emptyDir volume. The agent
auto-detects available runtimes in the container and registers with Attune.
The default sensor deployment also uses the same injection pattern, copying
`attune-sensor-agent` into the pod before starting a stock runtime image.
To add more agent workers, append entries to `agentWorkers` in your values:
agentWorkers:
- name: my-runtime
image: my-org/my-image:latest
replicas: 1
runtimes: [] # auto-detect
{{- end }}

View File

@@ -0,0 +1,137 @@
{{- range .Values.agentWorkers }}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "attune.fullname" $ }}-agent-worker-{{ .name }}
labels:
{{- include "attune.labels" $ | nindent 4 }}
app.kubernetes.io/component: agent-worker-{{ .name }}
spec:
replicas: {{ .replicas | default 1 }}
selector:
matchLabels:
{{- include "attune.selectorLabels" $ | nindent 6 }}
app.kubernetes.io/component: agent-worker-{{ .name }}
template:
metadata:
labels:
{{- include "attune.selectorLabels" $ | nindent 8 }}
app.kubernetes.io/component: agent-worker-{{ .name }}
spec:
{{- if $.Values.global.imagePullSecrets }}
imagePullSecrets:
{{- toYaml $.Values.global.imagePullSecrets | nindent 8 }}
{{- end }}
{{- if .runtimeClassName }}
runtimeClassName: {{ .runtimeClassName }}
{{- end }}
{{- if .nodeSelector }}
nodeSelector:
{{- toYaml .nodeSelector | nindent 8 }}
{{- end }}
{{- if .tolerations }}
tolerations:
{{- toYaml .tolerations | nindent 8 }}
{{- end }}
{{- if .stopGracePeriod }}
terminationGracePeriodSeconds: {{ .stopGracePeriod }}
{{- else }}
terminationGracePeriodSeconds: 45
{{- end }}
initContainers:
- name: agent-loader
image: {{ include "attune.image" (dict "root" $ "image" $.Values.images.agent) }}
imagePullPolicy: {{ $.Values.images.agent.pullPolicy }}
command: ["cp", "/usr/local/bin/attune-agent", "/opt/attune/agent/attune-agent"]
volumeMounts:
- name: agent-bin
mountPath: /opt/attune/agent
- name: wait-for-schema
image: postgres:16-alpine
command: ["/bin/sh", "-ec"]
args:
- |
until PGPASSWORD="$DB_PASSWORD" psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" -tAc "SELECT to_regclass('${DB_SCHEMA}.identity')" | grep -q identity; do
echo "waiting for schema";
sleep 2;
done
envFrom:
- secretRef:
name: {{ include "attune.secretName" $ }}
- name: wait-for-packs
image: busybox:1.36
command: ["/bin/sh", "-ec"]
args:
- |
until [ -f /opt/attune/packs/core/pack.yaml ]; do
echo "waiting for packs";
sleep 2;
done
volumeMounts:
- name: packs
mountPath: /opt/attune/packs
containers:
- name: worker
image: {{ .image }}
{{- if .imagePullPolicy }}
imagePullPolicy: {{ .imagePullPolicy }}
{{- end }}
command: ["/opt/attune/agent/attune-agent"]
envFrom:
- secretRef:
name: {{ include "attune.secretName" $ }}
env:
- name: ATTUNE_CONFIG
value: /opt/attune/config.yaml
- name: ATTUNE__DATABASE__SCHEMA
value: {{ $.Values.database.schema | quote }}
- name: ATTUNE_WORKER_TYPE
value: container
- name: ATTUNE_WORKER_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: ATTUNE_API_URL
value: http://{{ include "attune.apiServiceName" $ }}:{{ $.Values.api.service.port }}
- name: RUST_LOG
value: {{ .logLevel | default "info" }}
{{- if .runtimes }}
- name: ATTUNE_WORKER_RUNTIMES
value: {{ join "," .runtimes | quote }}
{{- end }}
{{- if .env }}
{{- toYaml .env | nindent 12 }}
{{- end }}
resources:
{{- toYaml (.resources | default dict) | nindent 12 }}
volumeMounts:
- name: agent-bin
mountPath: /opt/attune/agent
readOnly: true
- name: config
mountPath: /opt/attune/config.yaml
subPath: config.yaml
- name: packs
mountPath: /opt/attune/packs
readOnly: true
- name: runtime-envs
mountPath: /opt/attune/runtime_envs
- name: artifacts
mountPath: /opt/attune/artifacts
volumes:
- name: agent-bin
emptyDir: {}
- name: config
configMap:
name: {{ include "attune.fullname" $ }}-config
- name: packs
persistentVolumeClaim:
claimName: {{ include "attune.fullname" $ }}-packs
- name: runtime-envs
persistentVolumeClaim:
claimName: {{ include "attune.fullname" $ }}-runtime-envs
- name: artifacts
persistentVolumeClaim:
claimName: {{ include "attune.fullname" $ }}-artifacts
{{- end }}

View File

@@ -304,7 +304,15 @@ spec:
imagePullSecrets: imagePullSecrets:
{{- toYaml .Values.global.imagePullSecrets | nindent 8 }} {{- toYaml .Values.global.imagePullSecrets | nindent 8 }}
{{- end }} {{- end }}
terminationGracePeriodSeconds: 45
initContainers: initContainers:
- name: sensor-agent-loader
image: {{ include "attune.image" (dict "root" . "image" .Values.images.agent) }}
imagePullPolicy: {{ .Values.images.agent.pullPolicy }}
command: ["cp", "/usr/local/bin/attune-sensor-agent", "/opt/attune/agent/attune-sensor-agent"]
volumeMounts:
- name: agent-bin
mountPath: /opt/attune/agent
- name: wait-for-schema - name: wait-for-schema
image: postgres:16-alpine image: postgres:16-alpine
command: ["/bin/sh", "-ec"] command: ["/bin/sh", "-ec"]
@@ -333,6 +341,7 @@ spec:
- name: sensor - name: sensor
image: {{ include "attune.image" (dict "root" . "image" .Values.images.sensor) }} image: {{ include "attune.image" (dict "root" . "image" .Values.images.sensor) }}
imagePullPolicy: {{ .Values.images.sensor.pullPolicy }} imagePullPolicy: {{ .Values.images.sensor.pullPolicy }}
command: ["/opt/attune/agent/attune-sensor-agent"]
envFrom: envFrom:
- secretRef: - secretRef:
name: {{ include "attune.secretName" . }} name: {{ include "attune.secretName" . }}
@@ -343,23 +352,33 @@ spec:
value: {{ .Values.database.schema | quote }} value: {{ .Values.database.schema | quote }}
- name: ATTUNE__WORKER__WORKER_TYPE - name: ATTUNE__WORKER__WORKER_TYPE
value: container value: container
- name: ATTUNE_SENSOR_RUNTIMES
value: {{ .Values.sensor.runtimes | quote }}
- name: ATTUNE_API_URL - name: ATTUNE_API_URL
value: http://{{ include "attune.apiServiceName" . }}:{{ .Values.api.service.port }} value: http://{{ include "attune.apiServiceName" . }}:{{ .Values.api.service.port }}
- name: ATTUNE_MQ_URL - name: ATTUNE_MQ_URL
value: {{ include "attune.rabbitmqUrl" . | quote }} value: {{ include "attune.rabbitmqUrl" . | quote }}
- name: ATTUNE_PACKS_BASE_DIR - name: ATTUNE_PACKS_BASE_DIR
value: /opt/attune/packs value: /opt/attune/packs
- name: RUST_LOG
value: {{ .Values.sensor.logLevel | quote }}
resources: resources:
{{- toYaml .Values.sensor.resources | nindent 12 }} {{- toYaml .Values.sensor.resources | nindent 12 }}
volumeMounts: volumeMounts:
- name: agent-bin
mountPath: /opt/attune/agent
readOnly: true
- name: config - name: config
mountPath: /opt/attune/config.yaml mountPath: /opt/attune/config.yaml
subPath: config.yaml subPath: config.yaml
- name: packs - name: packs
mountPath: /opt/attune/packs mountPath: /opt/attune/packs
readOnly: true
- name: runtime-envs - name: runtime-envs
mountPath: /opt/attune/runtime_envs mountPath: /opt/attune/runtime_envs
volumes: volumes:
- name: agent-bin
emptyDir: {}
- name: config - name: config
configMap: configMap:
name: {{ include "attune.fullname" . }}-config name: {{ include "attune.fullname" . }}-config

View File

@@ -108,8 +108,8 @@ images:
tag: "" tag: ""
pullPolicy: IfNotPresent pullPolicy: IfNotPresent
sensor: sensor:
repository: attune-sensor repository: nikolaik/python-nodejs
tag: "" tag: python3.12-nodejs22-slim
pullPolicy: IfNotPresent pullPolicy: IfNotPresent
notifier: notifier:
repository: attune-notifier repository: attune-notifier
@@ -131,6 +131,10 @@ images:
repository: attune-init-packs repository: attune-init-packs
tag: "" tag: ""
pullPolicy: IfNotPresent pullPolicy: IfNotPresent
agent:
repository: attune-agent
tag: ""
pullPolicy: IfNotPresent
jobs: jobs:
migrations: migrations:
@@ -162,6 +166,8 @@ worker:
sensor: sensor:
replicaCount: 1 replicaCount: 1
runtimes: shell,python,node,native
logLevel: debug
resources: {} resources: {}
notifier: notifier:
@@ -191,3 +197,57 @@ web:
- path: / - path: /
pathType: Prefix pathType: Prefix
tls: [] tls: []
# Agent-based workers
# These deploy the universal worker agent into any container image.
# The agent auto-detects available runtimes (python, ruby, node, etc.)
# and registers with the Attune platform.
#
# Each entry creates a separate Deployment with an init container that
# copies the statically-linked agent binary into the worker container.
#
# Supported fields per worker:
# name (required) - Unique name for this worker (used in resource names)
# image (required) - Container image with your desired runtime(s)
# replicas (optional) - Number of pod replicas (default: 1)
# runtimes (optional) - List of runtimes to expose; [] = auto-detect
# resources (optional) - Kubernetes resource requests/limits
# env (optional) - Extra environment variables (list of {name, value})
# imagePullPolicy (optional) - Pull policy for the worker image
# logLevel (optional) - RUST_LOG level (default: "info")
# runtimeClassName (optional) - Kubernetes RuntimeClass (e.g., "nvidia" for GPU)
# nodeSelector (optional) - Node selector map for pod scheduling
# tolerations (optional) - Tolerations list for pod scheduling
# stopGracePeriod (optional) - Termination grace period in seconds (default: 45)
#
# Examples:
# agentWorkers:
# - name: ruby
# image: ruby:3.3
# replicas: 2
# runtimes: [] # auto-detect
# resources: {}
#
# - name: python-gpu
# image: nvidia/cuda:12.3.1-runtime-ubuntu22.04
# replicas: 1
# runtimes: [python, shell]
# runtimeClassName: nvidia
# nodeSelector:
# gpu: "true"
# tolerations:
# - key: nvidia.com/gpu
# operator: Exists
# effect: NoSchedule
# resources:
# limits:
# nvidia.com/gpu: 1
#
# - name: custom
# image: my-org/my-custom-image:latest
# replicas: 1
# runtimes: []
# env:
# - name: MY_CUSTOM_VAR
# value: my-value
agentWorkers: []

View File

@@ -125,3 +125,8 @@ executor:
scheduled_timeout: 120 # 2 minutes (faster feedback in dev) scheduled_timeout: 120 # 2 minutes (faster feedback in dev)
timeout_check_interval: 30 # Check every 30 seconds timeout_check_interval: 30 # Check every 30 seconds
enable_timeout_monitor: true enable_timeout_monitor: true
# Agent binary distribution (optional - for local development)
# Binary is built via: make build-agent
# agent:
# binary_dir: ./target/x86_64-unknown-linux-musl/release

View File

@@ -89,6 +89,7 @@ hmac = "0.12"
sha1 = "0.10" sha1 = "0.10"
sha2 = { workspace = true } sha2 = { workspace = true }
hex = "0.4" hex = "0.4"
subtle = "2.6"
# OpenAPI/Swagger # OpenAPI/Swagger
utoipa = { workspace = true, features = ["axum_extras"] } utoipa = { workspace = true, features = ["axum_extras"] }

View File

@@ -199,6 +199,10 @@ use crate::dto::{
crate::routes::webhooks::disable_webhook, crate::routes::webhooks::disable_webhook,
crate::routes::webhooks::regenerate_webhook_key, crate::routes::webhooks::regenerate_webhook_key,
crate::routes::webhooks::receive_webhook, crate::routes::webhooks::receive_webhook,
// Agent
crate::routes::agent::download_agent_binary,
crate::routes::agent::agent_info,
), ),
components( components(
schemas( schemas(
@@ -341,6 +345,10 @@ use crate::dto::{
WebhookReceiverRequest, WebhookReceiverRequest,
WebhookReceiverResponse, WebhookReceiverResponse,
ApiResponse<WebhookReceiverResponse>, ApiResponse<WebhookReceiverResponse>,
// Agent DTOs
crate::routes::agent::AgentBinaryInfo,
crate::routes::agent::AgentArchInfo,
) )
), ),
modifiers(&SecurityAddon), modifiers(&SecurityAddon),
@@ -359,6 +367,7 @@ use crate::dto::{
(name = "secrets", description = "Secret management endpoints"), (name = "secrets", description = "Secret management endpoints"),
(name = "workflows", description = "Workflow management endpoints"), (name = "workflows", description = "Workflow management endpoints"),
(name = "webhooks", description = "Webhook management and receiver endpoints"), (name = "webhooks", description = "Webhook management and receiver endpoints"),
(name = "agent", description = "Agent binary distribution endpoints"),
) )
)] )]
pub struct ApiDoc; pub struct ApiDoc;
@@ -441,14 +450,14 @@ mod tests {
// We have 57 unique paths with 81 total operations (HTTP methods) // We have 57 unique paths with 81 total operations (HTTP methods)
// This test ensures we don't accidentally remove endpoints // This test ensures we don't accidentally remove endpoints
assert!( assert!(
path_count >= 57, path_count >= 59,
"Expected at least 57 unique API paths, found {}", "Expected at least 59 unique API paths, found {}",
path_count path_count
); );
assert!( assert!(
operation_count >= 81, operation_count >= 83,
"Expected at least 81 API operations, found {}", "Expected at least 83 API operations, found {}",
operation_count operation_count
); );

View File

@@ -0,0 +1,482 @@
//! Agent binary download endpoints
//!
//! Provides endpoints for downloading the attune-agent binary for injection
//! into arbitrary containers. This supports deployments where shared Docker
//! volumes are impractical (Kubernetes, ECS, remote Docker hosts).
use axum::{
body::Body,
extract::{Query, State},
http::{header, HeaderMap, StatusCode},
response::IntoResponse,
routing::get,
Json, Router,
};
use serde::{Deserialize, Serialize};
use std::sync::Arc;
use subtle::ConstantTimeEq;
use tokio::fs;
use tokio_util::io::ReaderStream;
use utoipa::{IntoParams, ToSchema};
use crate::state::AppState;
/// Query parameters for the binary download endpoint
#[derive(Debug, Deserialize, IntoParams)]
pub struct BinaryDownloadParams {
/// Target architecture (x86_64, aarch64). Defaults to x86_64.
#[param(example = "x86_64")]
pub arch: Option<String>,
/// Optional bootstrap token for authentication
pub token: Option<String>,
}
/// Agent binary metadata
#[derive(Debug, Serialize, ToSchema)]
pub struct AgentBinaryInfo {
/// Available architectures
pub architectures: Vec<AgentArchInfo>,
/// Agent version (from build)
pub version: String,
}
/// Per-architecture binary info
#[derive(Debug, Serialize, ToSchema)]
pub struct AgentArchInfo {
/// Architecture name
pub arch: String,
/// Binary size in bytes
pub size_bytes: u64,
/// Whether this binary is available
pub available: bool,
}
/// Validate that the architecture name is safe (no path traversal) and normalize it.
fn validate_arch(arch: &str) -> Result<&str, (StatusCode, Json<serde_json::Value>)> {
match arch {
"x86_64" | "aarch64" => Ok(arch),
// Accept arm64 as an alias for aarch64
"arm64" => Ok("aarch64"),
_ => Err((
StatusCode::BAD_REQUEST,
Json(serde_json::json!({
"error": "Invalid architecture",
"message": format!("Unsupported architecture '{}'. Supported: x86_64, aarch64", arch),
})),
)),
}
}
/// Validate bootstrap token if configured.
///
/// If the agent config has a `bootstrap_token` set, the request must provide it
/// via the `X-Agent-Token` header or the `token` query parameter. If no token
/// is configured, access is unrestricted.
fn validate_token(
config: &attune_common::config::Config,
headers: &HeaderMap,
query_token: &Option<String>,
) -> Result<(), (StatusCode, Json<serde_json::Value>)> {
let expected_token = config
.agent
.as_ref()
.and_then(|ac| ac.bootstrap_token.as_ref());
let expected_token = match expected_token {
Some(t) => t,
None => {
use std::sync::Once;
static WARN_ONCE: Once = Once::new();
WARN_ONCE.call_once(|| {
tracing::warn!(
"Agent binary download endpoint has no bootstrap_token configured. \
Anyone with network access to the API can download the agent binary. \
Set agent.bootstrap_token in config to restrict access."
);
});
return Ok(());
}
};
// Check X-Agent-Token header first, then query param
let provided_token = headers
.get("x-agent-token")
.and_then(|v| v.to_str().ok())
.map(|s| s.to_string())
.or_else(|| query_token.clone());
match provided_token {
Some(ref t) if bool::from(t.as_bytes().ct_eq(expected_token.as_bytes())) => Ok(()),
Some(_) => Err((
StatusCode::UNAUTHORIZED,
Json(serde_json::json!({
"error": "Invalid token",
"message": "The provided bootstrap token is invalid",
})),
)),
None => Err((
StatusCode::UNAUTHORIZED,
Json(serde_json::json!({
"error": "Token required",
"message": "A bootstrap token is required. Provide via X-Agent-Token header or token query parameter.",
})),
)),
}
}
/// Download the agent binary
///
/// Returns the statically-linked attune-agent binary for the requested architecture.
/// The binary can be injected into any container to turn it into an Attune worker.
#[utoipa::path(
get,
path = "/api/v1/agent/binary",
params(BinaryDownloadParams),
responses(
(status = 200, description = "Agent binary", content_type = "application/octet-stream"),
(status = 400, description = "Invalid architecture"),
(status = 401, description = "Invalid or missing bootstrap token"),
(status = 404, description = "Agent binary not found"),
(status = 503, description = "Agent binary distribution not configured"),
),
tag = "agent"
)]
pub async fn download_agent_binary(
State(state): State<Arc<AppState>>,
headers: HeaderMap,
Query(params): Query<BinaryDownloadParams>,
) -> Result<impl IntoResponse, (StatusCode, Json<serde_json::Value>)> {
// Validate bootstrap token if configured
validate_token(&state.config, &headers, &params.token)?;
let agent_config = state.config.agent.as_ref().ok_or_else(|| {
(
StatusCode::SERVICE_UNAVAILABLE,
Json(serde_json::json!({
"error": "Not configured",
"message": "Agent binary distribution is not configured. Set agent.binary_dir in config.",
})),
)
})?;
let arch = params.arch.as_deref().unwrap_or("x86_64");
let arch = validate_arch(arch)?;
let binary_dir = std::path::Path::new(&agent_config.binary_dir);
// Try arch-specific binary first, then fall back to generic name.
// IMPORTANT: The generic `attune-agent` binary is only safe to serve for
// x86_64 requests, because the current build pipeline produces an
// x86_64-unknown-linux-musl binary. Serving it for aarch64/arm64 would
// give the caller an incompatible executable (exec format error).
let arch_specific = binary_dir.join(format!("attune-agent-{}", arch));
let generic = binary_dir.join("attune-agent");
let binary_path = if arch_specific.exists() {
arch_specific
} else if arch == "x86_64" && generic.exists() {
tracing::debug!(
"Arch-specific binary not found at {:?}, falling back to generic {:?} (safe for x86_64)",
arch_specific,
generic
);
generic
} else {
tracing::warn!(
"Agent binary not found. Checked: {:?} and {:?}",
arch_specific,
generic
);
return Err((
StatusCode::NOT_FOUND,
Json(serde_json::json!({
"error": "Not found",
"message": format!(
"Agent binary not found for architecture '{}'. Ensure the agent binary is built and placed in '{}'.",
arch,
agent_config.binary_dir
),
})),
));
};
// Get file metadata for Content-Length
let metadata = fs::metadata(&binary_path).await.map_err(|e| {
tracing::error!(
"Failed to read agent binary metadata at {:?}: {}",
binary_path,
e
);
(
StatusCode::INTERNAL_SERVER_ERROR,
Json(serde_json::json!({
"error": "Internal error",
"message": "Failed to read agent binary",
})),
)
})?;
// Open file for streaming
let file = fs::File::open(&binary_path).await.map_err(|e| {
tracing::error!("Failed to open agent binary at {:?}: {}", binary_path, e);
(
StatusCode::INTERNAL_SERVER_ERROR,
Json(serde_json::json!({
"error": "Internal error",
"message": "Failed to open agent binary",
})),
)
})?;
let stream = ReaderStream::new(file);
let body = Body::from_stream(stream);
let headers_response = [
(header::CONTENT_TYPE, "application/octet-stream".to_string()),
(
header::CONTENT_DISPOSITION,
"attachment; filename=\"attune-agent\"".to_string(),
),
(header::CONTENT_LENGTH, metadata.len().to_string()),
(header::CACHE_CONTROL, "public, max-age=3600".to_string()),
];
tracing::info!(
arch = arch,
size_bytes = metadata.len(),
path = ?binary_path,
"Serving agent binary download"
);
Ok((headers_response, body))
}
/// Get agent binary metadata
///
/// Returns information about available agent binaries, including
/// supported architectures and binary sizes.
#[utoipa::path(
get,
path = "/api/v1/agent/info",
responses(
(status = 200, description = "Agent binary info", body = AgentBinaryInfo),
(status = 503, description = "Agent binary distribution not configured"),
),
tag = "agent"
)]
pub async fn agent_info(
State(state): State<Arc<AppState>>,
) -> Result<impl IntoResponse, (StatusCode, Json<serde_json::Value>)> {
let agent_config = state.config.agent.as_ref().ok_or_else(|| {
(
StatusCode::SERVICE_UNAVAILABLE,
Json(serde_json::json!({
"error": "Not configured",
"message": "Agent binary distribution is not configured.",
})),
)
})?;
let binary_dir = std::path::Path::new(&agent_config.binary_dir);
let architectures = ["x86_64", "aarch64"];
let mut arch_infos = Vec::new();
for arch in &architectures {
let arch_specific = binary_dir.join(format!("attune-agent-{}", arch));
let generic = binary_dir.join("attune-agent");
// Only fall back to the generic binary for x86_64, since the build
// pipeline currently produces x86_64-only generic binaries.
let (available, size_bytes) = if arch_specific.exists() {
match fs::metadata(&arch_specific).await {
Ok(m) => (true, m.len()),
Err(_) => (false, 0),
}
} else if *arch == "x86_64" && generic.exists() {
match fs::metadata(&generic).await {
Ok(m) => (true, m.len()),
Err(_) => (false, 0),
}
} else {
(false, 0)
};
arch_infos.push(AgentArchInfo {
arch: arch.to_string(),
size_bytes,
available,
});
}
Ok(Json(AgentBinaryInfo {
architectures: arch_infos,
version: env!("CARGO_PKG_VERSION").to_string(),
}))
}
/// Create agent routes
pub fn routes() -> Router<Arc<AppState>> {
Router::new()
.route("/agent/binary", get(download_agent_binary))
.route("/agent/info", get(agent_info))
}
#[cfg(test)]
mod tests {
use super::*;
use attune_common::config::AgentConfig;
use axum::http::{HeaderMap, HeaderValue};
// ── validate_arch tests ─────────────────────────────────────────
#[test]
fn test_validate_arch_valid_x86_64() {
let result = validate_arch("x86_64");
assert!(result.is_ok());
assert_eq!(result.unwrap(), "x86_64");
}
#[test]
fn test_validate_arch_valid_aarch64() {
let result = validate_arch("aarch64");
assert!(result.is_ok());
assert_eq!(result.unwrap(), "aarch64");
}
#[test]
fn test_validate_arch_arm64_alias() {
// "arm64" is an alias for "aarch64"
let result = validate_arch("arm64");
assert!(result.is_ok());
assert_eq!(result.unwrap(), "aarch64");
}
#[test]
fn test_validate_arch_invalid() {
let result = validate_arch("mips");
assert!(result.is_err());
let (status, body) = result.unwrap_err();
assert_eq!(status, StatusCode::BAD_REQUEST);
assert_eq!(body.0["error"], "Invalid architecture");
}
// ── validate_token tests ────────────────────────────────────────
/// Helper: build a minimal Config with the given agent config.
/// Only the `agent` field is relevant for `validate_token`.
fn test_config(agent: Option<AgentConfig>) -> attune_common::config::Config {
let manifest_dir = std::env::var("CARGO_MANIFEST_DIR").unwrap_or_else(|_| ".".to_string());
let config_path = format!("{}/../../config.test.yaml", manifest_dir);
let mut config = attune_common::config::Config::load_from_file(&config_path)
.expect("Failed to load test config");
config.agent = agent;
config
}
#[test]
fn test_validate_token_no_config() {
// When no agent config is set at all, no token is required.
let config = test_config(None);
let headers = HeaderMap::new();
let query_token = None;
let result = validate_token(&config, &headers, &query_token);
assert!(result.is_ok());
}
#[test]
fn test_validate_token_no_bootstrap_token_configured() {
// Agent config exists but bootstrap_token is None → no token required.
let config = test_config(Some(AgentConfig {
binary_dir: "/tmp/test".to_string(),
bootstrap_token: None,
}));
let headers = HeaderMap::new();
let query_token = None;
let result = validate_token(&config, &headers, &query_token);
assert!(result.is_ok());
}
#[test]
fn test_validate_token_valid_from_header() {
let config = test_config(Some(AgentConfig {
binary_dir: "/tmp/test".to_string(),
bootstrap_token: Some("s3cret-bootstrap".to_string()),
}));
let mut headers = HeaderMap::new();
headers.insert(
"x-agent-token",
HeaderValue::from_static("s3cret-bootstrap"),
);
let query_token = None;
let result = validate_token(&config, &headers, &query_token);
assert!(result.is_ok());
}
#[test]
fn test_validate_token_valid_from_query() {
let config = test_config(Some(AgentConfig {
binary_dir: "/tmp/test".to_string(),
bootstrap_token: Some("s3cret-bootstrap".to_string()),
}));
let headers = HeaderMap::new();
let query_token = Some("s3cret-bootstrap".to_string());
let result = validate_token(&config, &headers, &query_token);
assert!(result.is_ok());
}
#[test]
fn test_validate_token_invalid() {
let config = test_config(Some(AgentConfig {
binary_dir: "/tmp/test".to_string(),
bootstrap_token: Some("correct-token".to_string()),
}));
let mut headers = HeaderMap::new();
headers.insert("x-agent-token", HeaderValue::from_static("wrong-token"));
let query_token = None;
let result = validate_token(&config, &headers, &query_token);
assert!(result.is_err());
let (status, body) = result.unwrap_err();
assert_eq!(status, StatusCode::UNAUTHORIZED);
assert_eq!(body.0["error"], "Invalid token");
}
#[test]
fn test_validate_token_missing_when_required() {
// bootstrap_token is configured but caller provides nothing.
let config = test_config(Some(AgentConfig {
binary_dir: "/tmp/test".to_string(),
bootstrap_token: Some("required-token".to_string()),
}));
let headers = HeaderMap::new();
let query_token = None;
let result = validate_token(&config, &headers, &query_token);
assert!(result.is_err());
let (status, body) = result.unwrap_err();
assert_eq!(status, StatusCode::UNAUTHORIZED);
assert_eq!(body.0["error"], "Token required");
}
#[test]
fn test_validate_token_header_takes_precedence_over_query() {
// When both header and query provide a token, the header value is
// checked first (it appears first in the or_else chain). Provide a
// valid token in the header and an invalid one in the query — should
// succeed because the header matches.
let config = test_config(Some(AgentConfig {
binary_dir: "/tmp/test".to_string(),
bootstrap_token: Some("the-real-token".to_string()),
}));
let mut headers = HeaderMap::new();
headers.insert("x-agent-token", HeaderValue::from_static("the-real-token"));
let query_token = Some("wrong-token".to_string());
let result = validate_token(&config, &headers, &query_token);
assert!(result.is_ok());
}
}

View File

@@ -1,6 +1,7 @@
//! API route modules //! API route modules
pub mod actions; pub mod actions;
pub mod agent;
pub mod analytics; pub mod analytics;
pub mod artifacts; pub mod artifacts;
pub mod auth; pub mod auth;
@@ -19,6 +20,7 @@ pub mod webhooks;
pub mod workflows; pub mod workflows;
pub use actions::routes as action_routes; pub use actions::routes as action_routes;
pub use agent::routes as agent_routes;
pub use analytics::routes as analytics_routes; pub use analytics::routes as analytics_routes;
pub use artifacts::routes as artifact_routes; pub use artifacts::routes as artifact_routes;
pub use auth::routes as auth_routes; pub use auth::routes as auth_routes;

View File

@@ -176,9 +176,12 @@ pub async fn create_runtime(
pack_ref, pack_ref,
description: request.description, description: request.description,
name: request.name, name: request.name,
aliases: vec![],
distributions: request.distributions, distributions: request.distributions,
installation: request.installation, installation: request.installation,
execution_config: request.execution_config, execution_config: request.execution_config,
auto_detected: false,
detection_config: serde_json::json!({}),
}, },
) )
.await?; .await?;
@@ -232,6 +235,7 @@ pub async fn update_runtime(
NullableJsonPatch::Clear => Patch::Clear, NullableJsonPatch::Clear => Patch::Clear,
}), }),
execution_config: request.execution_config, execution_config: request.execution_config,
..Default::default()
}, },
) )
.await?; .await?;

View File

@@ -60,6 +60,7 @@ impl Server {
.merge(routes::history_routes()) .merge(routes::history_routes())
.merge(routes::analytics_routes()) .merge(routes::analytics_routes())
.merge(routes::artifact_routes()) .merge(routes::artifact_routes())
.merge(routes::agent_routes())
.with_state(self.state.clone()); .with_state(self.state.clone());
// Auth routes at root level (not versioned for frontend compatibility) // Auth routes at root level (not versioned for frontend compatibility)

View File

@@ -0,0 +1,138 @@
//! Integration tests for agent binary distribution endpoints
//!
//! The agent endpoints (`/api/v1/agent/binary` and `/api/v1/agent/info`) are
//! intentionally unauthenticated — the agent needs to download its binary
//! before it has JWT credentials. An optional `bootstrap_token` can restrict
//! access, but that is validated inside the handler, not via RequireAuth
//! middleware.
//!
//! The test configuration (`config.test.yaml`) does NOT include an `agent`
//! section, so both endpoints return 503 Service Unavailable. This is the
//! correct behaviour: the endpoints are reachable (no 401/404 from middleware)
//! but the feature is not configured.
use axum::http::StatusCode;
#[allow(dead_code)]
mod helpers;
use helpers::TestContext;
// ── /api/v1/agent/info ──────────────────────────────────────────────
#[tokio::test]
#[ignore = "integration test — requires database"]
async fn test_agent_info_not_configured() {
let ctx = TestContext::new()
.await
.expect("Failed to create test context");
let response = ctx
.get("/api/v1/agent/info", None)
.await
.expect("Failed to make request");
// Agent config is not set in config.test.yaml, so the handler returns 503.
assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE);
let body: serde_json::Value = response.json().await.expect("Failed to parse JSON");
assert_eq!(body["error"], "Not configured");
}
#[tokio::test]
#[ignore = "integration test — requires database"]
async fn test_agent_info_no_auth_required() {
// Verify that the endpoint is reachable WITHOUT any JWT token.
// If RequireAuth middleware were applied, this would return 401.
// Instead we expect 503 (not configured) — proving the endpoint
// is publicly accessible.
let ctx = TestContext::new()
.await
.expect("Failed to create test context");
let response = ctx
.get("/api/v1/agent/info", None)
.await
.expect("Failed to make request");
// Must NOT be 401 Unauthorized — the endpoint has no auth middleware.
assert_ne!(
response.status(),
StatusCode::UNAUTHORIZED,
"agent/info should not require authentication"
);
// Should be 503 because agent config is absent.
assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE);
}
// ── /api/v1/agent/binary ────────────────────────────────────────────
#[tokio::test]
#[ignore = "integration test — requires database"]
async fn test_agent_binary_not_configured() {
let ctx = TestContext::new()
.await
.expect("Failed to create test context");
let response = ctx
.get("/api/v1/agent/binary", None)
.await
.expect("Failed to make request");
// Agent config is not set in config.test.yaml, so the handler returns 503.
assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE);
let body: serde_json::Value = response.json().await.expect("Failed to parse JSON");
assert_eq!(body["error"], "Not configured");
}
#[tokio::test]
#[ignore = "integration test — requires database"]
async fn test_agent_binary_no_auth_required() {
// Same reasoning as test_agent_info_no_auth_required: the binary
// download endpoint must be publicly accessible (no RequireAuth).
// When no bootstrap_token is configured, any caller can reach the
// handler. We still get 503 because the agent feature itself is
// not configured in the test environment.
let ctx = TestContext::new()
.await
.expect("Failed to create test context");
let response = ctx
.get("/api/v1/agent/binary", None)
.await
.expect("Failed to make request");
// Must NOT be 401 Unauthorized — the endpoint has no auth middleware.
assert_ne!(
response.status(),
StatusCode::UNAUTHORIZED,
"agent/binary should not require authentication when no bootstrap_token is configured"
);
// Should be 503 because agent config is absent.
assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE);
}
#[tokio::test]
#[ignore = "integration test — requires database"]
async fn test_agent_binary_invalid_arch() {
// Architecture validation (`validate_arch`) rejects unsupported values
// with 400 Bad Request. However, in the handler the execution order is:
// 1. validate_token (passes — no bootstrap_token configured)
// 2. check agent config (fails with 503 — not configured)
// 3. validate_arch (never reached)
//
// So even with an invalid arch like "mips", we get 503 from the config
// check before the arch is ever validated. The arch validation is covered
// by unit tests in routes/agent.rs instead.
let ctx = TestContext::new()
.await
.expect("Failed to create test context");
let response = ctx
.get("/api/v1/agent/binary?arch=mips", None)
.await
.expect("Failed to make request");
// 503 from the agent-config-not-set check, NOT 400 from arch validation.
assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE);
}

View File

@@ -0,0 +1,107 @@
//! Shared bootstrap helpers for injected agent binaries.
use crate::agent_runtime_detection::{
detect_runtimes, format_as_env_value, print_detection_report_for_env, DetectedRuntime,
};
use tracing::{info, warn};
#[derive(Debug, Clone)]
pub struct RuntimeBootstrapResult {
pub runtimes_override: Option<String>,
pub detected_runtimes: Option<Vec<DetectedRuntime>>,
}
/// Detect runtimes and populate the agent runtime environment variable when needed.
///
/// This must run before the Tokio runtime starts because it may mutate process
/// environment variables.
pub fn bootstrap_runtime_env(env_var_name: &str) -> RuntimeBootstrapResult {
let runtimes_override = std::env::var(env_var_name).ok();
let mut detected_runtimes = None;
if let Some(ref override_value) = runtimes_override {
info!(
"{} already set (override): {}",
env_var_name, override_value
);
info!("Running auto-detection for override-specified runtimes...");
let detected = detect_runtimes();
let override_names: Vec<&str> = override_value.split(',').map(|s| s.trim()).collect();
let filtered: Vec<_> = detected
.into_iter()
.filter(|rt| {
let lower_name = rt.name.to_ascii_lowercase();
override_names
.iter()
.any(|ov| ov.to_ascii_lowercase() == lower_name)
})
.collect();
if filtered.is_empty() {
warn!(
"None of the override runtimes ({}) were found on this system",
override_value
);
} else {
info!(
"Matched {} override runtime(s) to detected interpreters:",
filtered.len()
);
for rt in &filtered {
match &rt.version {
Some(ver) => info!(" ✓ {} — {} ({})", rt.name, rt.path, ver),
None => info!(" ✓ {} — {}", rt.name, rt.path),
}
}
detected_runtimes = Some(filtered);
}
} else {
info!("No {} override — running auto-detection...", env_var_name);
let detected = detect_runtimes();
if detected.is_empty() {
warn!("No runtimes detected! The agent may not be able to execute any work.");
} else {
info!("Detected {} runtime(s):", detected.len());
for rt in &detected {
match &rt.version {
Some(ver) => info!(" ✓ {} — {} ({})", rt.name, rt.path, ver),
None => info!(" ✓ {} — {}", rt.name, rt.path),
}
}
let runtime_csv = format_as_env_value(&detected);
info!("Setting {}={}", env_var_name, runtime_csv);
std::env::set_var(env_var_name, &runtime_csv);
detected_runtimes = Some(detected);
}
}
RuntimeBootstrapResult {
runtimes_override,
detected_runtimes,
}
}
pub fn print_detect_only_report(env_var_name: &str, result: &RuntimeBootstrapResult) {
if result.runtimes_override.is_some() {
info!("--detect-only: re-running detection to show what is available on this system...");
println!(
"NOTE: {} is set — auto-detection was skipped during normal startup.",
env_var_name
);
println!(" Showing what auto-detection would find on this system:");
println!();
let detected = detect_runtimes();
print_detection_report_for_env(env_var_name, &detected);
} else if let Some(ref detected) = result.detected_runtimes {
print_detection_report_for_env(env_var_name, detected);
} else {
let detected = detect_runtimes();
print_detection_report_for_env(env_var_name, &detected);
}
}

View File

@@ -0,0 +1,306 @@
//! Runtime auto-detection for injected Attune agent binaries.
//!
//! This module probes the local system directly for well-known interpreters,
//! without requiring database access.
use serde::{Deserialize, Serialize};
use std::fmt;
use std::process::Command;
use tracing::{debug, info};
/// A runtime interpreter discovered on the local system.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct DetectedRuntime {
/// Canonical runtime name (for example, "python" or "node").
pub name: String,
/// Absolute path to the interpreter binary.
pub path: String,
/// Version string if the version command succeeded.
pub version: Option<String>,
}
impl fmt::Display for DetectedRuntime {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match &self.version {
Some(v) => write!(f, "{} ({}, v{})", self.name, self.path, v),
None => write!(f, "{} ({})", self.name, self.path),
}
}
}
struct RuntimeCandidate {
name: &'static str,
binaries: &'static [&'static str],
version_args: &'static [&'static str],
version_parser: VersionParser,
}
enum VersionParser {
SemverLike,
JavaStyle,
}
fn candidates() -> Vec<RuntimeCandidate> {
vec![
RuntimeCandidate {
name: "shell",
binaries: &["bash", "sh"],
version_args: &["--version"],
version_parser: VersionParser::SemverLike,
},
RuntimeCandidate {
name: "python",
binaries: &["python3", "python"],
version_args: &["--version"],
version_parser: VersionParser::SemverLike,
},
RuntimeCandidate {
name: "node",
binaries: &["node", "nodejs"],
version_args: &["--version"],
version_parser: VersionParser::SemverLike,
},
RuntimeCandidate {
name: "ruby",
binaries: &["ruby"],
version_args: &["--version"],
version_parser: VersionParser::SemverLike,
},
RuntimeCandidate {
name: "go",
binaries: &["go"],
version_args: &["version"],
version_parser: VersionParser::SemverLike,
},
RuntimeCandidate {
name: "java",
binaries: &["java"],
version_args: &["-version"],
version_parser: VersionParser::JavaStyle,
},
RuntimeCandidate {
name: "r",
binaries: &["Rscript"],
version_args: &["--version"],
version_parser: VersionParser::SemverLike,
},
RuntimeCandidate {
name: "perl",
binaries: &["perl"],
version_args: &["--version"],
version_parser: VersionParser::SemverLike,
},
]
}
/// Detect available runtimes by probing the local system.
pub fn detect_runtimes() -> Vec<DetectedRuntime> {
info!("Starting runtime auto-detection...");
let mut detected = Vec::new();
for candidate in candidates() {
match detect_single_runtime(&candidate) {
Some(runtime) => {
info!(" ✓ Detected: {}", runtime);
detected.push(runtime);
}
None => {
debug!(" ✗ Not found: {}", candidate.name);
}
}
}
info!(
"Runtime auto-detection complete: found {} runtime(s): [{}]",
detected.len(),
detected
.iter()
.map(|r| r.name.as_str())
.collect::<Vec<_>>()
.join(", ")
);
detected
}
fn detect_single_runtime(candidate: &RuntimeCandidate) -> Option<DetectedRuntime> {
for binary in candidate.binaries {
if let Some(path) = which_binary(binary) {
let version = get_version(&path, candidate.version_args, &candidate.version_parser);
return Some(DetectedRuntime {
name: candidate.name.to_string(),
path,
version,
});
}
}
None
}
fn which_binary(binary: &str) -> Option<String> {
if binary == "bash" || binary == "sh" {
let absolute_path = format!("/bin/{}", binary);
if std::path::Path::new(&absolute_path).exists() {
return Some(absolute_path);
}
}
match Command::new("which").arg(binary).output() {
Ok(output) if output.status.success() => {
let path = String::from_utf8_lossy(&output.stdout).trim().to_string();
if path.is_empty() {
None
} else {
Some(path)
}
}
Ok(_) => None,
Err(e) => {
debug!("'which' command failed ({}), trying 'command -v'", e);
match Command::new("sh")
.args(["-c", &format!("command -v {}", binary)])
.output()
{
Ok(output) if output.status.success() => {
let path = String::from_utf8_lossy(&output.stdout).trim().to_string();
if path.is_empty() {
None
} else {
Some(path)
}
}
_ => None,
}
}
}
}
fn get_version(binary_path: &str, version_args: &[&str], parser: &VersionParser) -> Option<String> {
let output = match Command::new(binary_path).args(version_args).output() {
Ok(output) => output,
Err(e) => {
debug!("Failed to run version command for {}: {}", binary_path, e);
return None;
}
};
let stdout = String::from_utf8_lossy(&output.stdout);
let stderr = String::from_utf8_lossy(&output.stderr);
let combined = format!("{}{}", stdout, stderr);
match parser {
VersionParser::SemverLike => parse_semver_like(&combined),
VersionParser::JavaStyle => parse_java_version(&combined),
}
}
fn parse_semver_like(output: &str) -> Option<String> {
let re = regex::Regex::new(r"(?:v|go)?(\d+\.\d+(?:\.\d+)?)").ok()?;
re.captures(output)
.and_then(|captures| captures.get(1).map(|m| m.as_str().to_string()))
}
fn parse_java_version(output: &str) -> Option<String> {
let quoted_re = regex::Regex::new(r#"version\s+"([^"]+)""#).ok()?;
if let Some(captures) = quoted_re.captures(output) {
return captures.get(1).map(|m| m.as_str().to_string());
}
parse_semver_like(output)
}
pub fn format_as_env_value(runtimes: &[DetectedRuntime]) -> String {
runtimes
.iter()
.map(|r| r.name.as_str())
.collect::<Vec<_>>()
.join(",")
}
pub fn print_detection_report_for_env(env_var_name: &str, runtimes: &[DetectedRuntime]) {
println!("=== Attune Agent Runtime Detection Report ===");
println!();
if runtimes.is_empty() {
println!("No runtimes detected!");
println!();
println!("The agent could not find any supported interpreter binaries.");
println!("Ensure at least one of the following is installed and on PATH:");
println!(" - bash / sh (shell scripts)");
println!(" - python3 / python (Python scripts)");
println!(" - node / nodejs (Node.js scripts)");
println!(" - ruby (Ruby scripts)");
println!(" - go (Go programs)");
println!(" - java (Java programs)");
println!(" - Rscript (R scripts)");
println!(" - perl (Perl scripts)");
} else {
println!("Detected {} runtime(s):", runtimes.len());
println!();
for rt in runtimes {
let version_str = rt.version.as_deref().unwrap_or("unknown version");
println!("{:<10} {} ({})", rt.name, rt.path, version_str);
}
}
println!();
println!("{}={}", env_var_name, format_as_env_value(runtimes));
println!();
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_parse_semver_like_python() {
assert_eq!(
parse_semver_like("Python 3.12.1"),
Some("3.12.1".to_string())
);
}
#[test]
fn test_parse_semver_like_node() {
assert_eq!(parse_semver_like("v20.11.0"), Some("20.11.0".to_string()));
}
#[test]
fn test_parse_semver_like_go() {
assert_eq!(
parse_semver_like("go version go1.22.0 linux/amd64"),
Some("1.22.0".to_string())
);
}
#[test]
fn test_parse_java_version_openjdk() {
assert_eq!(
parse_java_version(r#"openjdk version "21.0.1" 2023-10-17"#),
Some("21.0.1".to_string())
);
}
#[test]
fn test_format_as_env_value_multiple() {
let runtimes = vec![
DetectedRuntime {
name: "shell".to_string(),
path: "/bin/bash".to_string(),
version: Some("5.2.15".to_string()),
},
DetectedRuntime {
name: "python".to_string(),
path: "/usr/bin/python3".to_string(),
version: Some("3.12.1".to_string()),
},
];
assert_eq!(format_as_env_value(&runtimes), "shell,python");
}
}

View File

@@ -677,6 +677,15 @@ impl Default for PackRegistryConfig {
} }
} }
/// Agent binary distribution configuration
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AgentConfig {
/// Directory containing agent binary files
pub binary_dir: String,
/// Optional bootstrap token for authenticating agent binary downloads
pub bootstrap_token: Option<String>,
}
/// Executor service configuration /// Executor service configuration
#[derive(Debug, Clone, Serialize, Deserialize)] #[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ExecutorConfig { pub struct ExecutorConfig {
@@ -770,6 +779,9 @@ pub struct Config {
/// Executor configuration (optional, for executor service) /// Executor configuration (optional, for executor service)
pub executor: Option<ExecutorConfig>, pub executor: Option<ExecutorConfig>,
/// Agent configuration (optional, for agent binary distribution)
pub agent: Option<AgentConfig>,
} }
fn default_service_name() -> String { fn default_service_name() -> String {
@@ -1066,6 +1078,7 @@ mod tests {
notifier: None, notifier: None,
pack_registry: PackRegistryConfig::default(), pack_registry: PackRegistryConfig::default(),
executor: None, executor: None,
agent: None,
}; };
assert_eq!(config.service_name, "attune"); assert_eq!(config.service_name, "attune");
@@ -1144,6 +1157,7 @@ mod tests {
notifier: None, notifier: None,
pack_registry: PackRegistryConfig::default(), pack_registry: PackRegistryConfig::default(),
executor: None, executor: None,
agent: None,
}; };
assert!(config.validate().is_ok()); assert!(config.validate().is_ok());

View File

@@ -6,6 +6,8 @@
//! - Configuration //! - Configuration
//! - Utilities //! - Utilities
pub mod agent_bootstrap;
pub mod agent_runtime_detection;
pub mod auth; pub mod auth;
pub mod config; pub mod config;
pub mod crypto; pub mod crypto;

View File

@@ -776,10 +776,13 @@ pub mod runtime {
pub pack_ref: Option<String>, pub pack_ref: Option<String>,
pub description: Option<String>, pub description: Option<String>,
pub name: String, pub name: String,
pub aliases: Vec<String>,
pub distributions: JsonDict, pub distributions: JsonDict,
pub installation: Option<JsonDict>, pub installation: Option<JsonDict>,
pub installers: JsonDict, pub installers: JsonDict,
pub execution_config: JsonDict, pub execution_config: JsonDict,
pub auto_detected: bool,
pub detection_config: JsonDict,
pub created: DateTime<Utc>, pub created: DateTime<Utc>,
pub updated: DateTime<Utc>, pub updated: DateTime<Utc>,
} }

View File

@@ -10,7 +10,7 @@ use crate::config::Config;
use crate::error::{Error, Result}; use crate::error::{Error, Result};
use crate::models::Runtime; use crate::models::Runtime;
use crate::repositories::action::ActionRepository; use crate::repositories::action::ActionRepository;
use crate::repositories::runtime::RuntimeRepository; use crate::repositories::runtime::{self, RuntimeRepository};
use crate::repositories::FindById as _; use crate::repositories::FindById as _;
use serde_json::Value as JsonValue; use serde_json::Value as JsonValue;
use sqlx::{PgPool, Row}; use sqlx::{PgPool, Row};
@@ -370,19 +370,15 @@ impl PackEnvironmentManager {
// ======================================================================== // ========================================================================
async fn get_runtime(&self, runtime_id: i64) -> Result<Runtime> { async fn get_runtime(&self, runtime_id: i64) -> Result<Runtime> {
sqlx::query_as::<_, Runtime>( let query = format!(
r#" "SELECT {} FROM runtime WHERE id = $1",
SELECT id, ref, pack, pack_ref, description, name, runtime::SELECT_COLUMNS
distributions, installation, installers, execution_config, );
created, updated sqlx::query_as::<_, Runtime>(&query)
FROM runtime .bind(runtime_id)
WHERE id = $1 .fetch_one(&self.pool)
"#, .await
) .map_err(|e| Error::Internal(format!("Failed to fetch runtime: {}", e)))
.bind(runtime_id)
.fetch_one(&self.pool)
.await
.map_err(|e| Error::Internal(format!("Failed to fetch runtime: {}", e)))
} }
fn runtime_requires_environment(&self, runtime: &Runtime) -> Result<bool> { fn runtime_requires_environment(&self, runtime: &Runtime) -> Result<bool> {

View File

@@ -404,6 +404,16 @@ impl<'a> PackComponentLoader<'a> {
.and_then(|v| serde_json::to_value(v).ok()) .and_then(|v| serde_json::to_value(v).ok())
.unwrap_or_else(|| serde_json::json!({})); .unwrap_or_else(|| serde_json::json!({}));
let aliases: Vec<String> = data
.get("aliases")
.and_then(|v| v.as_sequence())
.map(|arr| {
arr.iter()
.filter_map(|v| v.as_str().map(|s| s.to_ascii_lowercase()))
.collect()
})
.unwrap_or_default();
// Check if runtime already exists — update in place if so // Check if runtime already exists — update in place if so
if let Some(existing) = RuntimeRepository::find_by_ref(self.pool, &runtime_ref).await? { if let Some(existing) = RuntimeRepository::find_by_ref(self.pool, &runtime_ref).await? {
let update_input = UpdateRuntimeInput { let update_input = UpdateRuntimeInput {
@@ -418,6 +428,8 @@ impl<'a> PackComponentLoader<'a> {
None => Patch::Clear, None => Patch::Clear,
}), }),
execution_config: Some(execution_config), execution_config: Some(execution_config),
aliases: Some(aliases),
..Default::default()
}; };
match RuntimeRepository::update(self.pool, existing.id, update_input).await { match RuntimeRepository::update(self.pool, existing.id, update_input).await {
@@ -448,6 +460,9 @@ impl<'a> PackComponentLoader<'a> {
distributions, distributions,
installation, installation,
execution_config, execution_config,
aliases,
auto_detected: false,
detection_config: serde_json::json!({}),
}; };
match RuntimeRepository::create(self.pool, input).await { match RuntimeRepository::create(self.pool, input).await {

View File

@@ -23,6 +23,13 @@ impl Repository for RuntimeRepository {
} }
} }
/// Columns selected for all Runtime queries. Centralised here so that
/// schema changes only need one update.
pub const SELECT_COLUMNS: &str = "id, ref, pack, pack_ref, description, name, aliases, \
distributions, installation, installers, execution_config, \
auto_detected, detection_config, \
created, updated";
/// Input for creating a new runtime /// Input for creating a new runtime
#[derive(Debug, Clone)] #[derive(Debug, Clone)]
pub struct CreateRuntimeInput { pub struct CreateRuntimeInput {
@@ -31,9 +38,12 @@ pub struct CreateRuntimeInput {
pub pack_ref: Option<String>, pub pack_ref: Option<String>,
pub description: Option<String>, pub description: Option<String>,
pub name: String, pub name: String,
pub aliases: Vec<String>,
pub distributions: JsonDict, pub distributions: JsonDict,
pub installation: Option<JsonDict>, pub installation: Option<JsonDict>,
pub execution_config: JsonDict, pub execution_config: JsonDict,
pub auto_detected: bool,
pub detection_config: JsonDict,
} }
/// Input for updating a runtime /// Input for updating a runtime
@@ -41,9 +51,12 @@ pub struct CreateRuntimeInput {
pub struct UpdateRuntimeInput { pub struct UpdateRuntimeInput {
pub description: Option<Patch<String>>, pub description: Option<Patch<String>>,
pub name: Option<String>, pub name: Option<String>,
pub aliases: Option<Vec<String>>,
pub distributions: Option<JsonDict>, pub distributions: Option<JsonDict>,
pub installation: Option<Patch<JsonDict>>, pub installation: Option<Patch<JsonDict>>,
pub execution_config: Option<JsonDict>, pub execution_config: Option<JsonDict>,
pub auto_detected: Option<bool>,
pub detection_config: Option<JsonDict>,
} }
#[async_trait::async_trait] #[async_trait::async_trait]
@@ -52,18 +65,11 @@ impl FindById for RuntimeRepository {
where where
E: Executor<'e, Database = Postgres> + 'e, E: Executor<'e, Database = Postgres> + 'e,
{ {
let runtime = sqlx::query_as::<_, Runtime>( let query = format!("SELECT {} FROM runtime WHERE id = $1", SELECT_COLUMNS);
r#" let runtime = sqlx::query_as::<_, Runtime>(&query)
SELECT id, ref, pack, pack_ref, description, name, .bind(id)
distributions, installation, installers, execution_config, .fetch_optional(executor)
created, updated .await?;
FROM runtime
WHERE id = $1
"#,
)
.bind(id)
.fetch_optional(executor)
.await?;
Ok(runtime) Ok(runtime)
} }
@@ -75,18 +81,11 @@ impl FindByRef for RuntimeRepository {
where where
E: Executor<'e, Database = Postgres> + 'e, E: Executor<'e, Database = Postgres> + 'e,
{ {
let runtime = sqlx::query_as::<_, Runtime>( let query = format!("SELECT {} FROM runtime WHERE ref = $1", SELECT_COLUMNS);
r#" let runtime = sqlx::query_as::<_, Runtime>(&query)
SELECT id, ref, pack, pack_ref, description, name, .bind(ref_str)
distributions, installation, installers, execution_config, .fetch_optional(executor)
created, updated .await?;
FROM runtime
WHERE ref = $1
"#,
)
.bind(ref_str)
.fetch_optional(executor)
.await?;
Ok(runtime) Ok(runtime)
} }
@@ -98,17 +97,10 @@ impl List for RuntimeRepository {
where where
E: Executor<'e, Database = Postgres> + 'e, E: Executor<'e, Database = Postgres> + 'e,
{ {
let runtimes = sqlx::query_as::<_, Runtime>( let query = format!("SELECT {} FROM runtime ORDER BY ref ASC", SELECT_COLUMNS);
r#" let runtimes = sqlx::query_as::<_, Runtime>(&query)
SELECT id, ref, pack, pack_ref, description, name, .fetch_all(executor)
distributions, installation, installers, execution_config, .await?;
created, updated
FROM runtime
ORDER BY ref ASC
"#,
)
.fetch_all(executor)
.await?;
Ok(runtimes) Ok(runtimes)
} }
@@ -122,27 +114,29 @@ impl Create for RuntimeRepository {
where where
E: Executor<'e, Database = Postgres> + 'e, E: Executor<'e, Database = Postgres> + 'e,
{ {
let runtime = sqlx::query_as::<_, Runtime>( let query = format!(
r#" "INSERT INTO runtime (ref, pack, pack_ref, description, name, aliases, \
INSERT INTO runtime (ref, pack, pack_ref, description, name, distributions, installation, installers, execution_config, \
distributions, installation, installers, execution_config) auto_detected, detection_config) \
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12) \
RETURNING id, ref, pack, pack_ref, description, name, RETURNING {}",
distributions, installation, installers, execution_config, SELECT_COLUMNS
created, updated );
"#, let runtime = sqlx::query_as::<_, Runtime>(&query)
) .bind(&input.r#ref)
.bind(&input.r#ref) .bind(input.pack)
.bind(input.pack) .bind(&input.pack_ref)
.bind(&input.pack_ref) .bind(&input.description)
.bind(&input.description) .bind(&input.name)
.bind(&input.name) .bind(&input.aliases)
.bind(&input.distributions) .bind(&input.distributions)
.bind(&input.installation) .bind(&input.installation)
.bind(serde_json::json!({})) .bind(serde_json::json!({}))
.bind(&input.execution_config) .bind(&input.execution_config)
.fetch_one(executor) .bind(input.auto_detected)
.await?; .bind(&input.detection_config)
.fetch_one(executor)
.await?;
Ok(runtime) Ok(runtime)
} }
@@ -179,6 +173,15 @@ impl Update for RuntimeRepository {
has_updates = true; has_updates = true;
} }
if let Some(aliases) = &input.aliases {
if has_updates {
query.push(", ");
}
query.push("aliases = ");
query.push_bind(aliases.as_slice());
has_updates = true;
}
if let Some(distributions) = &input.distributions { if let Some(distributions) = &input.distributions {
if has_updates { if has_updates {
query.push(", "); query.push(", ");
@@ -209,6 +212,24 @@ impl Update for RuntimeRepository {
has_updates = true; has_updates = true;
} }
if let Some(auto_detected) = input.auto_detected {
if has_updates {
query.push(", ");
}
query.push("auto_detected = ");
query.push_bind(auto_detected);
has_updates = true;
}
if let Some(detection_config) = &input.detection_config {
if has_updates {
query.push(", ");
}
query.push("detection_config = ");
query.push_bind(detection_config);
has_updates = true;
}
if !has_updates { if !has_updates {
// No updates requested, fetch and return existing entity // No updates requested, fetch and return existing entity
return Self::get_by_id(executor, id).await; return Self::get_by_id(executor, id).await;
@@ -216,10 +237,7 @@ impl Update for RuntimeRepository {
query.push(", updated = NOW() WHERE id = "); query.push(", updated = NOW() WHERE id = ");
query.push_bind(id); query.push_bind(id);
query.push( query.push(&format!(" RETURNING {}", SELECT_COLUMNS));
" RETURNING id, ref, pack, pack_ref, description, name, \
distributions, installation, installers, execution_config, created, updated",
);
let runtime = query let runtime = query
.build_query_as::<Runtime>() .build_query_as::<Runtime>()
@@ -251,19 +269,14 @@ impl RuntimeRepository {
where where
E: Executor<'e, Database = Postgres> + 'e, E: Executor<'e, Database = Postgres> + 'e,
{ {
let runtimes = sqlx::query_as::<_, Runtime>( let query = format!(
r#" "SELECT {} FROM runtime WHERE pack = $1 ORDER BY ref ASC",
SELECT id, ref, pack, pack_ref, description, name, SELECT_COLUMNS
distributions, installation, installers, execution_config, );
created, updated let runtimes = sqlx::query_as::<_, Runtime>(&query)
FROM runtime .bind(pack_id)
WHERE pack = $1 .fetch_all(executor)
ORDER BY ref ASC .await?;
"#,
)
.bind(pack_id)
.fetch_all(executor)
.await?;
Ok(runtimes) Ok(runtimes)
} }
@@ -273,23 +286,35 @@ impl RuntimeRepository {
where where
E: Executor<'e, Database = Postgres> + 'e, E: Executor<'e, Database = Postgres> + 'e,
{ {
let runtime = sqlx::query_as::<_, Runtime>( let query = format!(
r#" "SELECT {} FROM runtime WHERE LOWER(name) = LOWER($1) LIMIT 1",
SELECT id, ref, pack, pack_ref, description, name, SELECT_COLUMNS
distributions, installation, installers, execution_config, );
created, updated let runtime = sqlx::query_as::<_, Runtime>(&query)
FROM runtime .bind(name)
WHERE LOWER(name) = LOWER($1) .fetch_optional(executor)
LIMIT 1 .await?;
"#,
)
.bind(name)
.fetch_optional(executor)
.await?;
Ok(runtime) Ok(runtime)
} }
/// Find a runtime where the given alias appears in its `aliases` array.
/// Uses PostgreSQL's `@>` (array contains) operator with a GIN index.
pub async fn find_by_alias<'e, E>(executor: E, alias: &str) -> Result<Option<Runtime>>
where
E: Executor<'e, Database = Postgres> + 'e,
{
let query = format!(
"SELECT {} FROM runtime WHERE aliases @> ARRAY[$1]::text[] LIMIT 1",
SELECT_COLUMNS
);
let runtime = sqlx::query_as::<_, Runtime>(&query)
.bind(alias)
.fetch_optional(executor)
.await?;
Ok(runtime)
}
/// Delete runtimes belonging to a pack whose refs are NOT in the given set. /// Delete runtimes belonging to a pack whose refs are NOT in the given set.
/// ///
/// Used during pack reinstallation to clean up runtimes that were removed /// Used during pack reinstallation to clean up runtimes that were removed

View File

@@ -6,59 +6,41 @@
//! 2. Config file specification (medium priority) //! 2. Config file specification (medium priority)
//! 3. Database-driven detection with verification (lowest priority) //! 3. Database-driven detection with verification (lowest priority)
//! //!
//! Also provides [`normalize_runtime_name`] for alias-aware runtime name //! Also provides alias-based matching functions ([`runtime_aliases_match_filter`]
//! comparison across the codebase (worker filters, env setup, etc.). //! and [`runtime_aliases_contain`]) for comparing runtime alias lists against
//! worker filters and capability strings. Aliases are declared per-runtime in
//! pack manifests, so no hardcoded alias table is needed here.
use crate::config::Config; use crate::config::Config;
use crate::error::Result; use crate::error::Result;
use crate::models::Runtime; use crate::models::Runtime;
use crate::repositories::runtime::SELECT_COLUMNS;
use serde_json::json; use serde_json::json;
use sqlx::PgPool; use sqlx::PgPool;
use std::collections::HashMap; use std::collections::HashMap;
use std::process::Command; use std::process::Command;
use tracing::{debug, info, warn}; use tracing::{debug, info, warn};
/// Normalize a runtime name to its canonical short form. /// Check if a runtime's aliases overlap with a filter list.
/// ///
/// This ensures that different ways of referring to the same runtime /// The filter list comes from `ATTUNE_WORKER_RUNTIMES` (e.g., `["python", "shell"]`).
/// (e.g., "node", "nodejs", "node.js") all resolve to a single canonical /// A runtime matches if any of its declared aliases appear in the filter list.
/// name. Used by worker runtime filters and environment setup to match /// Comparison is case-insensitive.
/// database runtime names against short filter values. pub fn runtime_aliases_match_filter(aliases: &[String], filter: &[String]) -> bool {
/// aliases.iter().any(|alias| {
/// The canonical names mirror the alias groups in let lower_alias = alias.to_ascii_lowercase();
/// `PackComponentLoader::resolve_runtime`. filter.iter().any(|f| f.to_ascii_lowercase() == lower_alias)
/// })
/// # Examples
/// ```
/// use attune_common::runtime_detection::normalize_runtime_name;
/// assert_eq!(normalize_runtime_name("node.js"), "node");
/// assert_eq!(normalize_runtime_name("nodejs"), "node");
/// assert_eq!(normalize_runtime_name("python3"), "python");
/// assert_eq!(normalize_runtime_name("shell"), "shell");
/// ```
pub fn normalize_runtime_name(name: &str) -> &str {
match name {
"node" | "nodejs" | "node.js" => "node",
"python" | "python3" => "python",
"bash" | "sh" | "shell" => "shell",
"native" | "builtin" | "standalone" => "native",
other => other,
}
} }
/// Check if a runtime name matches a filter entry, supporting common aliases. /// Check if a runtime's aliases contain a specific name.
/// ///
/// Both sides are lowercased and then normalized before comparison so that, /// Used by the scheduler to check if a worker's capability string
/// e.g., a filter value of `"node"` matches a database runtime name `"Node.js"`. /// (e.g., "python") matches a runtime's aliases (e.g., ["python", "python3"]).
pub fn runtime_matches_filter(rt_name: &str, filter_entry: &str) -> bool { /// Comparison is case-insensitive.
let rt_lower = rt_name.to_ascii_lowercase(); pub fn runtime_aliases_contain(aliases: &[String], name: &str) -> bool {
let filter_lower = filter_entry.to_ascii_lowercase(); let lower = name.to_ascii_lowercase();
normalize_runtime_name(&rt_lower) == normalize_runtime_name(&filter_lower) aliases.iter().any(|a| a.to_ascii_lowercase() == lower)
}
/// Check if a runtime name matches any entry in a filter list.
pub fn runtime_in_filter(rt_name: &str, filter: &[String]) -> bool {
filter.iter().any(|f| runtime_matches_filter(rt_name, f))
} }
/// Runtime detection service /// Runtime detection service
@@ -156,17 +138,10 @@ impl RuntimeDetector {
info!("Querying database for runtime definitions..."); info!("Querying database for runtime definitions...");
// Query all runtimes from database // Query all runtimes from database
let runtimes = sqlx::query_as::<_, Runtime>( let query = format!("SELECT {} FROM runtime ORDER BY ref", SELECT_COLUMNS);
r#" let runtimes = sqlx::query_as::<_, Runtime>(&query)
SELECT id, ref, pack, pack_ref, description, name, .fetch_all(&self.pool)
distributions, installation, installers, execution_config, .await?;
created, updated
FROM runtime
ORDER BY ref
"#,
)
.fetch_all(&self.pool)
.await?;
info!("Found {} runtime(s) in database", runtimes.len()); info!("Found {} runtime(s) in database", runtimes.len());
@@ -337,69 +312,46 @@ mod tests {
use serde_json::json; use serde_json::json;
#[test] #[test]
fn test_normalize_runtime_name_node_variants() { fn test_runtime_aliases_match_filter() {
assert_eq!(normalize_runtime_name("node"), "node"); let aliases = vec!["python".to_string(), "python3".to_string()];
assert_eq!(normalize_runtime_name("nodejs"), "node"); let filter = vec!["python".to_string(), "shell".to_string()];
assert_eq!(normalize_runtime_name("node.js"), "node"); assert!(runtime_aliases_match_filter(&aliases, &filter));
let filter_no_match = vec!["node".to_string(), "ruby".to_string()];
assert!(!runtime_aliases_match_filter(&aliases, &filter_no_match));
} }
#[test] #[test]
fn test_normalize_runtime_name_python_variants() { fn test_runtime_aliases_match_filter_case_insensitive() {
assert_eq!(normalize_runtime_name("python"), "python"); let aliases = vec!["Python".to_string(), "python3".to_string()];
assert_eq!(normalize_runtime_name("python3"), "python"); let filter = vec!["python".to_string()];
assert!(runtime_aliases_match_filter(&aliases, &filter));
} }
#[test] #[test]
fn test_normalize_runtime_name_shell_variants() { fn test_runtime_aliases_match_filter_empty() {
assert_eq!(normalize_runtime_name("shell"), "shell"); let aliases: Vec<String> = vec![];
assert_eq!(normalize_runtime_name("bash"), "shell"); let filter = vec!["python".to_string()];
assert_eq!(normalize_runtime_name("sh"), "shell"); assert!(!runtime_aliases_match_filter(&aliases, &filter));
let aliases = vec!["python".to_string()];
let filter: Vec<String> = vec![];
assert!(!runtime_aliases_match_filter(&aliases, &filter));
} }
#[test] #[test]
fn test_normalize_runtime_name_native_variants() { fn test_runtime_aliases_contain() {
assert_eq!(normalize_runtime_name("native"), "native"); let aliases = vec!["ruby".to_string(), "rb".to_string()];
assert_eq!(normalize_runtime_name("builtin"), "native"); assert!(runtime_aliases_contain(&aliases, "ruby"));
assert_eq!(normalize_runtime_name("standalone"), "native"); assert!(runtime_aliases_contain(&aliases, "rb"));
assert!(!runtime_aliases_contain(&aliases, "python"));
} }
#[test] #[test]
fn test_normalize_runtime_name_passthrough() { fn test_runtime_aliases_contain_case_insensitive() {
assert_eq!(normalize_runtime_name("custom_runtime"), "custom_runtime"); let aliases = vec!["ruby".to_string(), "rb".to_string()];
} assert!(runtime_aliases_contain(&aliases, "Ruby"));
assert!(runtime_aliases_contain(&aliases, "RB"));
#[test]
fn test_runtime_matches_filter() {
// Node.js DB name lowercased vs worker filter "node"
assert!(runtime_matches_filter("node.js", "node"));
assert!(runtime_matches_filter("node", "nodejs"));
assert!(runtime_matches_filter("nodejs", "node.js"));
// Exact match
assert!(runtime_matches_filter("shell", "shell"));
// No match
assert!(!runtime_matches_filter("python", "node"));
}
#[test]
fn test_runtime_matches_filter_case_insensitive() {
// Database stores capitalized names (e.g., "Node.js", "Python")
// Worker capabilities store lowercase (e.g., "node", "python")
assert!(runtime_matches_filter("Node.js", "node"));
assert!(runtime_matches_filter("node", "Node.js"));
assert!(runtime_matches_filter("Python", "python"));
assert!(runtime_matches_filter("python", "Python"));
assert!(runtime_matches_filter("Shell", "shell"));
assert!(runtime_matches_filter("NODEJS", "node"));
assert!(!runtime_matches_filter("Python", "node"));
}
#[test]
fn test_runtime_in_filter() {
let filter = vec!["shell".to_string(), "node".to_string()];
assert!(runtime_in_filter("shell", &filter));
assert!(runtime_in_filter("node.js", &filter));
assert!(runtime_in_filter("nodejs", &filter));
assert!(!runtime_in_filter("python", &filter));
} }
#[test] #[test]

View File

@@ -961,9 +961,12 @@ impl RuntimeFixture {
pack_ref: self.pack_ref, pack_ref: self.pack_ref,
description: self.description, description: self.description,
name: self.name, name: self.name,
aliases: vec![],
distributions: self.distributions, distributions: self.distributions,
installation: self.installation, installation: self.installation,
execution_config: self.execution_config, execution_config: self.execution_config,
auto_detected: false,
detection_config: serde_json::json!({}),
}; };
RuntimeRepository::create(pool, input).await RuntimeRepository::create(pool, input).await

View File

@@ -64,6 +64,7 @@ impl RuntimeFixture {
pack_ref: None, pack_ref: None,
description: Some(format!("Test runtime {}", seq)), description: Some(format!("Test runtime {}", seq)),
name, name,
aliases: vec![],
distributions: json!({ distributions: json!({
"linux": { "supported": true, "versions": ["ubuntu20.04", "ubuntu22.04"] }, "linux": { "supported": true, "versions": ["ubuntu20.04", "ubuntu22.04"] },
"darwin": { "supported": true, "versions": ["12", "13"] } "darwin": { "supported": true, "versions": ["12", "13"] }
@@ -79,6 +80,8 @@ impl RuntimeFixture {
"file_extension": ".py" "file_extension": ".py"
} }
}), }),
auto_detected: false,
detection_config: json!({}),
} }
} }
@@ -93,6 +96,7 @@ impl RuntimeFixture {
pack_ref: None, pack_ref: None,
description: None, description: None,
name, name,
aliases: vec![],
distributions: json!({}), distributions: json!({}),
installation: None, installation: None,
execution_config: json!({ execution_config: json!({
@@ -102,6 +106,8 @@ impl RuntimeFixture {
"file_extension": ".sh" "file_extension": ".sh"
} }
}), }),
auto_detected: false,
detection_config: json!({}),
} }
} }
} }
@@ -268,6 +274,7 @@ async fn test_update_runtime() {
"method": "npm" "method": "npm"
}))), }))),
execution_config: None, execution_config: None,
..Default::default()
}; };
let updated = RuntimeRepository::update(&pool, created.id, update_input.clone()) let updated = RuntimeRepository::update(&pool, created.id, update_input.clone())
@@ -299,6 +306,7 @@ async fn test_update_runtime_partial() {
distributions: None, distributions: None,
installation: None, installation: None,
execution_config: None, execution_config: None,
..Default::default()
}; };
let updated = RuntimeRepository::update(&pool, created.id, update_input.clone()) let updated = RuntimeRepository::update(&pool, created.id, update_input.clone())

View File

@@ -574,6 +574,7 @@ async fn test_worker_with_runtime() {
pack_ref: None, pack_ref: None,
description: Some("Test runtime".to_string()), description: Some("Test runtime".to_string()),
name: "test_runtime".to_string(), name: "test_runtime".to_string(),
aliases: vec![],
distributions: json!({}), distributions: json!({}),
installation: None, installation: None,
execution_config: json!({ execution_config: json!({
@@ -583,6 +584,8 @@ async fn test_worker_with_runtime() {
"file_extension": ".sh" "file_extension": ".sh"
} }
}), }),
auto_detected: false,
detection_config: json!({}),
}; };
let runtime = RuntimeRepository::create(&pool, runtime_input) let runtime = RuntimeRepository::create(&pool, runtime_input)

View File

@@ -13,8 +13,11 @@
use anyhow::Result; use anyhow::Result;
use attune_common::{ use attune_common::{
models::{enums::ExecutionStatus, execution::WorkflowTaskMetadata, Action, Execution}, models::{enums::ExecutionStatus, execution::WorkflowTaskMetadata, Action, Execution, Runtime},
mq::{Consumer, ExecutionRequestedPayload, MessageEnvelope, MessageType, Publisher}, mq::{
Consumer, ExecutionCompletedPayload, ExecutionRequestedPayload, MessageEnvelope,
MessageType, Publisher,
},
repositories::{ repositories::{
action::ActionRepository, action::ActionRepository,
execution::{CreateExecutionInput, ExecutionRepository, UpdateExecutionInput}, execution::{CreateExecutionInput, ExecutionRepository, UpdateExecutionInput},
@@ -24,7 +27,7 @@ use attune_common::{
}, },
Create, FindById, FindByRef, Update, Create, FindById, FindByRef, Update,
}, },
runtime_detection::runtime_matches_filter, runtime_detection::runtime_aliases_contain,
workflow::WorkflowDefinition, workflow::WorkflowDefinition,
}; };
use chrono::Utc; use chrono::Utc;
@@ -205,7 +208,23 @@ impl ExecutionScheduler {
} }
// Regular action: select appropriate worker (round-robin among compatible workers) // Regular action: select appropriate worker (round-robin among compatible workers)
let worker = Self::select_worker(pool, &action, round_robin_counter).await?; let worker = match Self::select_worker(pool, &action, round_robin_counter).await {
Ok(worker) => worker,
Err(err) if Self::is_unschedulable_error(&err) => {
Self::fail_unschedulable_execution(
pool,
publisher,
envelope,
execution_id,
action.id,
&action.r#ref,
&err.to_string(),
)
.await?;
return Ok(());
}
Err(err) => return Err(err),
};
info!( info!(
"Selected worker {} for execution {}", "Selected worker {} for execution {}",
@@ -1561,7 +1580,7 @@ impl ExecutionScheduler {
let compatible_workers: Vec<_> = if let Some(ref runtime) = runtime { let compatible_workers: Vec<_> = if let Some(ref runtime) = runtime {
workers workers
.into_iter() .into_iter()
.filter(|w| Self::worker_supports_runtime(w, &runtime.name)) .filter(|w| Self::worker_supports_runtime(w, runtime))
.collect() .collect()
} else { } else {
workers workers
@@ -1619,20 +1638,26 @@ impl ExecutionScheduler {
/// Check if a worker supports a given runtime /// Check if a worker supports a given runtime
/// ///
/// This checks the worker's capabilities.runtimes array for the runtime name. /// This checks the worker's capabilities.runtimes array against the runtime's aliases.
/// Falls back to checking the deprecated runtime column if capabilities are not set. /// If aliases are missing, fall back to the runtime's canonical name.
fn worker_supports_runtime(worker: &attune_common::models::Worker, runtime_name: &str) -> bool { fn worker_supports_runtime(worker: &attune_common::models::Worker, runtime: &Runtime) -> bool {
// First, try to parse capabilities and check runtimes array let runtime_names = Self::runtime_capability_names(runtime);
// Try to parse capabilities and check runtimes array
if let Some(ref capabilities) = worker.capabilities { if let Some(ref capabilities) = worker.capabilities {
if let Some(runtimes) = capabilities.get("runtimes") { if let Some(runtimes) = capabilities.get("runtimes") {
if let Some(runtime_array) = runtimes.as_array() { if let Some(runtime_array) = runtimes.as_array() {
// Check if any runtime in the array matches (alias-aware) // Check if any runtime in the array matches via aliases
for runtime_value in runtime_array { for runtime_value in runtime_array {
if let Some(runtime_str) = runtime_value.as_str() { if let Some(runtime_str) = runtime_value.as_str() {
if runtime_matches_filter(runtime_name, runtime_str) { if runtime_names
.iter()
.any(|candidate| candidate.eq_ignore_ascii_case(runtime_str))
|| runtime_aliases_contain(&runtime.aliases, runtime_str)
{
debug!( debug!(
"Worker {} supports runtime '{}' via capabilities (matched '{}')", "Worker {} supports runtime '{}' via capabilities (matched '{}', candidates: {:?})",
worker.name, runtime_name, runtime_str worker.name, runtime.name, runtime_str, runtime_names
); );
return true; return true;
} }
@@ -1642,25 +1667,90 @@ impl ExecutionScheduler {
} }
} }
// Fallback: check deprecated runtime column
// This is kept for backward compatibility but should be removed in the future
if worker.runtime.is_some() {
debug!(
"Worker {} using deprecated runtime column for matching",
worker.name
);
// Note: This fallback is incomplete because we'd need to look up the runtime name
// from the ID, which would require an async call. Since we're moving to capabilities,
// we'll just return false here and require workers to set capabilities properly.
}
debug!( debug!(
"Worker {} does not support runtime '{}'", "Worker {} does not support runtime '{}' (candidates: {:?})",
worker.name, runtime_name worker.name, runtime.name, runtime_names
); );
false false
} }
fn runtime_capability_names(runtime: &Runtime) -> Vec<String> {
let mut names: Vec<String> = runtime
.aliases
.iter()
.map(|alias| alias.to_ascii_lowercase())
.filter(|alias| !alias.is_empty())
.collect();
let runtime_name = runtime.name.to_ascii_lowercase();
if !runtime_name.is_empty() && !names.iter().any(|name| name == &runtime_name) {
names.push(runtime_name);
}
names
}
fn is_unschedulable_error(error: &anyhow::Error) -> bool {
let message = error.to_string();
message.starts_with("No compatible workers found")
|| message.starts_with("No action workers available")
|| message.starts_with("No active workers available")
|| message.starts_with("No workers with fresh heartbeats available")
}
async fn fail_unschedulable_execution(
pool: &PgPool,
publisher: &Publisher,
envelope: &MessageEnvelope<ExecutionRequestedPayload>,
execution_id: i64,
action_id: i64,
action_ref: &str,
error_message: &str,
) -> Result<()> {
let completed_at = Utc::now();
let result = serde_json::json!({
"error": "Execution is unschedulable",
"message": error_message,
"action_ref": action_ref,
"failed_by": "execution_scheduler",
"failed_at": completed_at.to_rfc3339(),
});
ExecutionRepository::update(
pool,
execution_id,
UpdateExecutionInput {
status: Some(ExecutionStatus::Failed),
result: Some(result.clone()),
..Default::default()
},
)
.await?;
let completed = MessageEnvelope::new(
MessageType::ExecutionCompleted,
ExecutionCompletedPayload {
execution_id,
action_id,
action_ref: action_ref.to_string(),
status: "failed".to_string(),
result: Some(result),
completed_at,
},
)
.with_correlation_id(envelope.correlation_id)
.with_source("attune-executor");
publisher.publish_envelope(&completed).await?;
warn!(
"Execution {} marked failed as unschedulable: {}",
execution_id, error_message
);
Ok(())
}
/// Check if a worker's heartbeat is fresh enough to schedule work /// Check if a worker's heartbeat is fresh enough to schedule work
/// ///
/// A worker is considered fresh if its last heartbeat is within /// A worker is considered fresh if its last heartbeat is within
@@ -1826,6 +1916,70 @@ mod tests {
// Real tests will require database and message queue setup // Real tests will require database and message queue setup
} }
#[test]
fn test_worker_supports_runtime_with_alias_match() {
let worker = create_test_worker("test-worker", 5);
let runtime = Runtime {
id: 1,
r#ref: "core.shell".to_string(),
pack: None,
pack_ref: Some("core".to_string()),
description: Some("Shell runtime".to_string()),
name: "Shell".to_string(),
aliases: vec!["shell".to_string(), "bash".to_string()],
distributions: serde_json::json!({}),
installation: None,
installers: serde_json::json!({}),
execution_config: serde_json::json!({}),
auto_detected: false,
detection_config: serde_json::json!({}),
created: Utc::now(),
updated: Utc::now(),
};
assert!(ExecutionScheduler::worker_supports_runtime(
&worker, &runtime
));
}
#[test]
fn test_worker_supports_runtime_falls_back_to_runtime_name_when_aliases_missing() {
let worker = create_test_worker("test-worker", 5);
let runtime = Runtime {
id: 1,
r#ref: "core.shell".to_string(),
pack: None,
pack_ref: Some("core".to_string()),
description: Some("Shell runtime".to_string()),
name: "Shell".to_string(),
aliases: vec![],
distributions: serde_json::json!({}),
installation: None,
installers: serde_json::json!({}),
execution_config: serde_json::json!({}),
auto_detected: false,
detection_config: serde_json::json!({}),
created: Utc::now(),
updated: Utc::now(),
};
assert!(ExecutionScheduler::worker_supports_runtime(
&worker, &runtime
));
}
#[test]
fn test_unschedulable_error_classification() {
assert!(ExecutionScheduler::is_unschedulable_error(
&anyhow::anyhow!(
"No compatible workers found for action: core.sleep (requires runtime: Shell)"
)
));
assert!(!ExecutionScheduler::is_unschedulable_error(
&anyhow::anyhow!("database temporarily unavailable")
));
}
#[test] #[test]
fn test_concurrency_limit_dispatch_count() { fn test_concurrency_limit_dispatch_count() {
// Verify the dispatch_count calculation used by dispatch_with_items_task // Verify the dispatch_count calculation used by dispatch_with_items_task

View File

@@ -72,6 +72,7 @@ async fn _create_test_runtime(pool: &PgPool, suffix: &str) -> i64 {
pack_ref: None, pack_ref: None,
description: Some(format!("Test runtime {}", suffix)), description: Some(format!("Test runtime {}", suffix)),
name: format!("Python {}", suffix), name: format!("Python {}", suffix),
aliases: vec![],
distributions: json!({"ubuntu": "python3"}), distributions: json!({"ubuntu": "python3"}),
installation: Some(json!({"method": "apt"})), installation: Some(json!({"method": "apt"})),
execution_config: json!({ execution_config: json!({
@@ -81,6 +82,8 @@ async fn _create_test_runtime(pool: &PgPool, suffix: &str) -> i64 {
"file_extension": ".py" "file_extension": ".py"
} }
}), }),
auto_detected: false,
detection_config: json!({}),
}; };
RuntimeRepository::create(pool, runtime_input) RuntimeRepository::create(pool, runtime_input)

View File

@@ -67,6 +67,7 @@ async fn create_test_runtime(pool: &PgPool, suffix: &str) -> i64 {
pack_ref: None, pack_ref: None,
description: Some(format!("Test runtime {}", suffix)), description: Some(format!("Test runtime {}", suffix)),
name: format!("Python {}", suffix), name: format!("Python {}", suffix),
aliases: vec![],
distributions: json!({"ubuntu": "python3"}), distributions: json!({"ubuntu": "python3"}),
installation: Some(json!({"method": "apt"})), installation: Some(json!({"method": "apt"})),
execution_config: json!({ execution_config: json!({
@@ -76,6 +77,8 @@ async fn create_test_runtime(pool: &PgPool, suffix: &str) -> i64 {
"file_extension": ".py" "file_extension": ".py"
} }
}), }),
auto_detected: false,
detection_config: json!({}),
}; };
let runtime = RuntimeRepository::create(pool, runtime_input) let runtime = RuntimeRepository::create(pool, runtime_input)

View File

@@ -14,6 +14,10 @@ path = "src/lib.rs"
name = "attune-sensor" name = "attune-sensor"
path = "src/main.rs" path = "src/main.rs"
[[bin]]
name = "attune-sensor-agent"
path = "src/agent_main.rs"
[dependencies] [dependencies]
attune-common = { path = "../common" } attune-common = { path = "../common" }
tokio = { workspace = true } tokio = { workspace = true }

View File

@@ -0,0 +1,79 @@
//! Attune Universal Sensor Agent.
use anyhow::Result;
use attune_common::agent_bootstrap::{bootstrap_runtime_env, print_detect_only_report};
use attune_common::config::Config;
use attune_sensor::startup::{
apply_sensor_name_override, init_tracing, log_config_details, run_sensor_service,
set_config_path,
};
use clap::Parser;
use tracing::info;
#[derive(Parser, Debug)]
#[command(name = "attune-sensor-agent")]
#[command(
version,
about = "Attune Universal Sensor Agent - Injected into runtime containers to auto-detect sensor runtimes"
)]
struct Args {
/// Path to configuration file (optional)
#[arg(short, long)]
config: Option<String>,
/// Sensor worker name override
#[arg(short, long)]
name: Option<String>,
/// Run runtime detection, print results, and exit
#[arg(long)]
detect_only: bool,
}
fn main() -> Result<()> {
attune_common::auth::install_crypto_provider();
init_tracing(tracing::Level::INFO);
let args = Args::parse();
info!("Starting Attune Universal Sensor Agent");
info!(
"Agent binary: attune-sensor-agent {}",
env!("CARGO_PKG_VERSION")
);
// Safe: no async runtime or worker threads are running yet.
std::env::set_var("ATTUNE_SENSOR_AGENT_MODE", "true");
std::env::set_var("ATTUNE_SENSOR_AGENT_BINARY_NAME", "attune-sensor-agent");
std::env::set_var(
"ATTUNE_SENSOR_AGENT_BINARY_VERSION",
env!("CARGO_PKG_VERSION"),
);
let bootstrap = bootstrap_runtime_env("ATTUNE_SENSOR_RUNTIMES");
if args.detect_only {
print_detect_only_report("ATTUNE_SENSOR_RUNTIMES", &bootstrap);
return Ok(());
}
set_config_path(args.config.as_deref());
let runtime = tokio::runtime::Runtime::new()?;
runtime.block_on(async_main(args))
}
async fn async_main(args: Args) -> Result<()> {
let mut config = Config::load()?;
config.validate()?;
if let Some(name) = args.name {
apply_sensor_name_override(&mut config, name);
}
log_config_details(&config);
run_sensor_service(config, "Attune Sensor Agent is ready").await?;
info!("Attune Sensor Agent shutdown complete");
Ok(())
}

View File

@@ -8,6 +8,7 @@ pub mod rule_lifecycle_listener;
pub mod sensor_manager; pub mod sensor_manager;
pub mod sensor_worker_registration; pub mod sensor_worker_registration;
pub mod service; pub mod service;
pub mod startup;
// Re-export template resolver from common crate // Re-export template resolver from common crate
pub mod template_resolver { pub mod template_resolver {

View File

@@ -1,15 +1,14 @@
//! Attune Sensor Service //! Attune Sensor Service
//! //!
//! The Sensor Service monitors for trigger conditions and generates events. //! The Sensor Service monitors for trigger conditions and generates events.
//! It executes custom sensor code, manages sensor lifecycle, and publishes
//! events to the message queue for rule matching and enforcement creation.
use anyhow::Result; use anyhow::Result;
use attune_common::config::Config; use attune_common::config::Config;
use attune_sensor::service::SensorService; use attune_sensor::startup::{
init_tracing, log_config_details, run_sensor_service, set_config_path,
};
use clap::Parser; use clap::Parser;
use tokio::signal::unix::{signal, SignalKind}; use tracing::info;
use tracing::{error, info};
#[derive(Parser, Debug)] #[derive(Parser, Debug)]
#[command(name = "attune-sensor")] #[command(name = "attune-sensor")]
@@ -26,114 +25,23 @@ struct Args {
#[tokio::main] #[tokio::main]
async fn main() -> Result<()> { async fn main() -> Result<()> {
// Install HMAC-only JWT crypto provider (must be before any token operations)
attune_common::auth::install_crypto_provider(); attune_common::auth::install_crypto_provider();
let args = Args::parse(); let args = Args::parse();
// Initialize tracing with specified log level
let log_level = args.log_level.parse().unwrap_or(tracing::Level::INFO); let log_level = args.log_level.parse().unwrap_or(tracing::Level::INFO);
tracing_subscriber::fmt() init_tracing(log_level);
.with_max_level(log_level)
.with_target(false)
.with_thread_ids(true)
.with_file(true)
.with_line_number(true)
.init();
info!("Starting Attune Sensor Service"); info!("Starting Attune Sensor Service");
info!("Version: {}", env!("CARGO_PKG_VERSION")); info!("Version: {}", env!("CARGO_PKG_VERSION"));
// Load configuration set_config_path(args.config.as_deref());
if let Some(config_path) = args.config {
info!("Loading configuration from: {}", config_path);
std::env::set_var("ATTUNE_CONFIG", config_path);
}
let config = Config::load()?; let config = Config::load()?;
config.validate()?; config.validate()?;
info!("Configuration loaded successfully"); log_config_details(&config);
info!("Environment: {}", config.environment); run_sensor_service(config, "Attune Sensor Service is ready").await?;
info!("Database: {}", mask_connection_string(&config.database.url));
if let Some(ref mq_config) = config.message_queue {
info!("Message Queue: {}", mask_connection_string(&mq_config.url));
}
// Create and start sensor service
let service = SensorService::new(config).await?;
info!("Sensor Service initialized successfully");
// Start the service (spawns background tasks and returns)
info!("Starting Sensor Service components...");
service.start().await?;
info!("Attune Sensor Service is ready");
// Setup signal handlers for graceful shutdown
let mut sigint = signal(SignalKind::interrupt())?;
let mut sigterm = signal(SignalKind::terminate())?;
tokio::select! {
_ = sigint.recv() => {
info!("Received SIGINT signal");
}
_ = sigterm.recv() => {
info!("Received SIGTERM signal");
}
}
info!("Shutting down gracefully...");
// Stop the service: deregister worker, stop sensors, clean up connections
if let Err(e) = service.stop().await {
error!("Error during shutdown: {}", e);
}
info!("Attune Sensor Service shutdown complete"); info!("Attune Sensor Service shutdown complete");
Ok(()) Ok(())
} }
/// Mask sensitive parts of connection strings for logging
fn mask_connection_string(url: &str) -> String {
if let Some(at_pos) = url.find('@') {
if let Some(proto_end) = url.find("://") {
let protocol = &url[..proto_end + 3];
let host_and_path = &url[at_pos..];
return format!("{}***:***{}", protocol, host_and_path);
}
}
"***:***@***".to_string()
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_mask_connection_string() {
let url = "postgresql://user:password@localhost:5432/attune";
let masked = mask_connection_string(url);
assert!(!masked.contains("user"));
assert!(!masked.contains("password"));
assert!(masked.contains("@localhost"));
}
#[test]
fn test_mask_connection_string_no_credentials() {
let url = "postgresql://localhost:5432/attune";
let masked = mask_connection_string(url);
assert_eq!(masked, "***:***@***");
}
#[test]
fn test_mask_rabbitmq_connection() {
let url = "amqp://admin:secret@rabbitmq:5672/%2F";
let masked = mask_connection_string(url);
assert!(!masked.contains("admin"));
assert!(!masked.contains("secret"));
assert!(masked.contains("@rabbitmq"));
}
}

View File

@@ -11,7 +11,7 @@
//! - Monitoring sensor health and restarting failed sensors //! - Monitoring sensor health and restarting failed sensors
use anyhow::{anyhow, Result}; use anyhow::{anyhow, Result};
use attune_common::models::{Id, Sensor, Trigger}; use attune_common::models::{runtime::RuntimeExecutionConfig, Id, Sensor, Trigger};
use attune_common::repositories::{FindById, List, RuntimeRepository}; use attune_common::repositories::{FindById, List, RuntimeRepository};
use sqlx::{PgPool, Row}; use sqlx::{PgPool, Row};
@@ -162,6 +162,127 @@ impl SensorManager {
Ok(enabled_sensors) Ok(enabled_sensors)
} }
async fn ensure_runtime_environment(
&self,
exec_config: &RuntimeExecutionConfig,
pack_dir: &std::path::Path,
env_dir: &std::path::Path,
) -> Result<()> {
let env_cfg = match &exec_config.environment {
Some(cfg) if cfg.env_type != "none" => cfg,
_ => return Ok(()),
};
let vars = exec_config.build_template_vars_with_env(pack_dir, Some(env_dir));
if !env_dir.exists() {
if env_cfg.create_command.is_empty() {
return Err(anyhow!(
"Runtime environment '{}' requires create_command but none is configured",
env_cfg.env_type
));
}
if let Some(parent) = env_dir.parent() {
tokio::fs::create_dir_all(parent).await.map_err(|e| {
anyhow!(
"Failed to create runtime environment parent directory {}: {}",
parent.display(),
e
)
})?;
}
let resolved_cmd =
RuntimeExecutionConfig::resolve_command(&env_cfg.create_command, &vars);
let (program, args) = resolved_cmd
.split_first()
.ok_or_else(|| anyhow!("Empty create_command for runtime environment"))?;
info!(
"Creating sensor runtime environment at {}: {:?}",
env_dir.display(),
resolved_cmd
);
let output = Command::new(program)
.args(args)
.current_dir(pack_dir)
.output()
.await
.map_err(|e| anyhow!("Failed to run create command '{}': {}", program, e))?;
if !output.status.success() {
let stderr = String::from_utf8_lossy(&output.stderr);
return Err(anyhow!(
"Runtime environment creation failed (exit {}): {}",
output.status.code().unwrap_or(-1),
stderr.trim()
));
}
}
let dep_cfg = match &exec_config.dependencies {
Some(cfg) => cfg,
None => return Ok(()),
};
let manifest_path = pack_dir.join(&dep_cfg.manifest_file);
if !manifest_path.exists() || dep_cfg.install_command.is_empty() {
return Ok(());
}
let install_marker = env_dir.join(".attune_sensor_deps_installed");
if install_marker.exists() {
return Ok(());
}
let resolved_cmd = RuntimeExecutionConfig::resolve_command(&dep_cfg.install_command, &vars);
let (program, args) = resolved_cmd
.split_first()
.ok_or_else(|| anyhow!("Empty install_command for runtime dependencies"))?;
info!(
"Installing sensor runtime dependencies for {} using {:?}",
pack_dir.display(),
resolved_cmd
);
let output = Command::new(program)
.args(args)
.current_dir(pack_dir)
.output()
.await
.map_err(|e| {
anyhow!(
"Failed to run dependency install command '{}': {}",
program,
e
)
})?;
if !output.status.success() {
let stderr = String::from_utf8_lossy(&output.stderr);
return Err(anyhow!(
"Runtime dependency installation failed (exit {}): {}",
output.status.code().unwrap_or(-1),
stderr.trim()
));
}
tokio::fs::write(&install_marker, b"ok")
.await
.map_err(|e| {
anyhow!(
"Failed to write dependency install marker {}: {}",
install_marker.display(),
e
)
})?;
Ok(())
}
/// Start a sensor instance /// Start a sensor instance
async fn start_sensor(&self, sensor: Sensor) -> Result<()> { async fn start_sensor(&self, sensor: Sensor) -> Result<()> {
info!("Starting sensor {} ({})", sensor.r#ref, sensor.id); info!("Starting sensor {} ({})", sensor.r#ref, sensor.id);
@@ -231,6 +352,12 @@ impl SensorManager {
let exec_config = runtime.parsed_execution_config(); let exec_config = runtime.parsed_execution_config();
let rt_name = runtime.name.to_lowercase(); let rt_name = runtime.name.to_lowercase();
let runtime_env_suffix = runtime
.r#ref
.rsplit('.')
.next()
.filter(|suffix| !suffix.is_empty())
.unwrap_or(&rt_name);
info!( info!(
"Sensor {} runtime details: id={}, ref='{}', name='{}', execution_config={}", "Sensor {} runtime details: id={}, ref='{}', name='{}', execution_config={}",
@@ -242,7 +369,19 @@ impl SensorManager {
let pack_dir = std::path::PathBuf::from(&self.inner.packs_base_dir).join(pack_ref); let pack_dir = std::path::PathBuf::from(&self.inner.packs_base_dir).join(pack_ref);
let env_dir = std::path::PathBuf::from(&self.inner.runtime_envs_dir) let env_dir = std::path::PathBuf::from(&self.inner.runtime_envs_dir)
.join(pack_ref) .join(pack_ref)
.join(&rt_name); .join(runtime_env_suffix);
if let Err(e) = self
.ensure_runtime_environment(&exec_config, &pack_dir, &env_dir)
.await
{
warn!(
"Failed to ensure sensor runtime environment for {} at {}: {}",
sensor.r#ref,
env_dir.display(),
e
);
}
let env_dir_opt = if env_dir.exists() { let env_dir_opt = if env_dir.exists() {
Some(env_dir.as_path()) Some(env_dir.as_path())
} else { } else {
@@ -354,15 +493,31 @@ impl SensorManager {
// Start the standalone sensor with token and configuration // Start the standalone sensor with token and configuration
// Pass sensor ref (e.g., "core.interval_timer_sensor") for proper identification // Pass sensor ref (e.g., "core.interval_timer_sensor") for proper identification
let mut child = cmd cmd.env("ATTUNE_API_URL", &self.inner.api_url)
.env("ATTUNE_API_URL", &self.inner.api_url)
.env("ATTUNE_API_TOKEN", &token_response.token) .env("ATTUNE_API_TOKEN", &token_response.token)
.env("ATTUNE_SENSOR_ID", sensor.id.to_string()) .env("ATTUNE_SENSOR_ID", sensor.id.to_string())
.env("ATTUNE_SENSOR_REF", &sensor.r#ref) .env("ATTUNE_SENSOR_REF", &sensor.r#ref)
.env("ATTUNE_SENSOR_TRIGGERS", &trigger_instances_json) .env("ATTUNE_SENSOR_TRIGGERS", &trigger_instances_json)
.env("ATTUNE_MQ_URL", &self.inner.mq_url) .env("ATTUNE_MQ_URL", &self.inner.mq_url)
.env("ATTUNE_MQ_EXCHANGE", "attune.events") .env("ATTUNE_MQ_EXCHANGE", "attune.events")
.env("ATTUNE_LOG_LEVEL", "info") .env("ATTUNE_LOG_LEVEL", "info");
if !exec_config.env_vars.is_empty() {
let vars = exec_config.build_template_vars_with_env(&pack_dir, env_dir_opt);
for (key, value_template) in &exec_config.env_vars {
let resolved = attune_common::models::RuntimeExecutionConfig::resolve_template(
value_template,
&vars,
);
debug!(
"Setting sensor runtime env var: {}={} (template: {})",
key, resolved, value_template
);
cmd.env(key, resolved);
}
}
let mut child = cmd
.stdin(Stdio::null()) .stdin(Stdio::null())
.stdout(Stdio::piped()) .stdout(Stdio::piped())
.stderr(Stdio::piped()) .stderr(Stdio::piped())
@@ -371,13 +526,14 @@ impl SensorManager {
anyhow!( anyhow!(
"Failed to start sensor process for '{}': {} \ "Failed to start sensor process for '{}': {} \
(binary='{}', is_native={}, runtime_ref='{}', \ (binary='{}', is_native={}, runtime_ref='{}', \
interpreter_config='{}')", interpreter_config='{}', env_dir='{}')",
sensor.r#ref, sensor.r#ref,
e, e,
spawn_binary, spawn_binary,
is_native, is_native,
runtime.r#ref, runtime.r#ref,
interpreter_binary interpreter_binary,
env_dir.display()
) )
})?; })?;

View File

@@ -15,6 +15,10 @@ use sqlx::{PgPool, Row};
use std::collections::HashMap; use std::collections::HashMap;
use tracing::{debug, info}; use tracing::{debug, info};
const ATTUNE_SENSOR_AGENT_MODE_ENV: &str = "ATTUNE_SENSOR_AGENT_MODE";
const ATTUNE_SENSOR_AGENT_BINARY_NAME_ENV: &str = "ATTUNE_SENSOR_AGENT_BINARY_NAME";
const ATTUNE_SENSOR_AGENT_BINARY_VERSION_ENV: &str = "ATTUNE_SENSOR_AGENT_BINARY_VERSION";
/// Sensor worker registration manager /// Sensor worker registration manager
pub struct SensorWorkerRegistration { pub struct SensorWorkerRegistration {
pool: PgPool, pool: PgPool,
@@ -25,6 +29,33 @@ pub struct SensorWorkerRegistration {
} }
impl SensorWorkerRegistration { impl SensorWorkerRegistration {
fn env_truthy(name: &str) -> bool {
std::env::var(name)
.ok()
.map(|value| matches!(value.trim().to_ascii_lowercase().as_str(), "1" | "true"))
.unwrap_or(false)
}
fn inject_agent_capabilities(capabilities: &mut HashMap<String, serde_json::Value>) {
if Self::env_truthy(ATTUNE_SENSOR_AGENT_MODE_ENV) {
capabilities.insert("agent_mode".to_string(), json!(true));
}
if let Ok(binary_name) = std::env::var(ATTUNE_SENSOR_AGENT_BINARY_NAME_ENV) {
let binary_name = binary_name.trim();
if !binary_name.is_empty() {
capabilities.insert("agent_binary_name".to_string(), json!(binary_name));
}
}
if let Ok(binary_version) = std::env::var(ATTUNE_SENSOR_AGENT_BINARY_VERSION_ENV) {
let binary_version = binary_version.trim();
if !binary_version.is_empty() {
capabilities.insert("agent_binary_version".to_string(), json!(binary_version));
}
}
}
/// Create a new sensor worker registration manager /// Create a new sensor worker registration manager
pub fn new(pool: PgPool, config: &Config) -> Self { pub fn new(pool: PgPool, config: &Config) -> Self {
let worker_name = config let worker_name = config
@@ -67,6 +98,8 @@ impl SensorWorkerRegistration {
json!(env!("CARGO_PKG_VERSION")), json!(env!("CARGO_PKG_VERSION")),
); );
Self::inject_agent_capabilities(&mut capabilities);
// Placeholder for runtimes (will be detected asynchronously) // Placeholder for runtimes (will be detected asynchronously)
capabilities.insert("runtimes".to_string(), json!(Vec::<String>::new())); capabilities.insert("runtimes".to_string(), json!(Vec::<String>::new()));
@@ -351,4 +384,28 @@ mod tests {
registration.deregister().await.unwrap(); registration.deregister().await.unwrap();
} }
#[test]
fn test_inject_agent_capabilities_from_env() {
std::env::set_var(ATTUNE_SENSOR_AGENT_MODE_ENV, "1");
std::env::set_var(ATTUNE_SENSOR_AGENT_BINARY_NAME_ENV, "attune-sensor-agent");
std::env::set_var(ATTUNE_SENSOR_AGENT_BINARY_VERSION_ENV, "1.2.3");
let mut capabilities = HashMap::new();
SensorWorkerRegistration::inject_agent_capabilities(&mut capabilities);
assert_eq!(capabilities.get("agent_mode"), Some(&json!(true)));
assert_eq!(
capabilities.get("agent_binary_name"),
Some(&json!("attune-sensor-agent"))
);
assert_eq!(
capabilities.get("agent_binary_version"),
Some(&json!("1.2.3"))
);
std::env::remove_var(ATTUNE_SENSOR_AGENT_MODE_ENV);
std::env::remove_var(ATTUNE_SENSOR_AGENT_BINARY_NAME_ENV);
std::env::remove_var(ATTUNE_SENSOR_AGENT_BINARY_VERSION_ENV);
}
} }

View File

@@ -0,0 +1,119 @@
use crate::service::SensorService;
use anyhow::Result;
use attune_common::config::{Config, SensorConfig};
use tokio::signal::unix::{signal, SignalKind};
use tracing::{error, info};
pub fn init_tracing(log_level: tracing::Level) {
tracing_subscriber::fmt()
.with_max_level(log_level)
.with_target(false)
.with_thread_ids(true)
.with_file(true)
.with_line_number(true)
.init();
}
pub fn set_config_path(config_path: Option<&str>) {
if let Some(config_path) = config_path {
info!("Loading configuration from: {}", config_path);
std::env::set_var("ATTUNE_CONFIG", config_path);
}
}
pub fn apply_sensor_name_override(config: &mut Config, name: String) {
if let Some(ref mut sensor_config) = config.sensor {
sensor_config.worker_name = Some(name);
} else {
config.sensor = Some(SensorConfig {
worker_name: Some(name),
host: None,
capabilities: None,
max_concurrent_sensors: None,
heartbeat_interval: 30,
poll_interval: 30,
sensor_timeout: 30,
shutdown_timeout: 30,
});
}
}
pub fn log_config_details(config: &Config) {
info!("Configuration loaded successfully");
info!("Environment: {}", config.environment);
info!("Database: {}", mask_connection_string(&config.database.url));
if let Some(ref mq_config) = config.message_queue {
info!("Message Queue: {}", mask_connection_string(&mq_config.url));
}
}
pub async fn run_sensor_service(config: Config, ready_message: &str) -> Result<()> {
let service = SensorService::new(config).await?;
info!("Sensor Service initialized successfully");
info!("Starting Sensor Service components...");
service.start().await?;
info!("{}", ready_message);
let mut sigint = signal(SignalKind::interrupt())?;
let mut sigterm = signal(SignalKind::terminate())?;
tokio::select! {
_ = sigint.recv() => {
info!("Received SIGINT signal");
}
_ = sigterm.recv() => {
info!("Received SIGTERM signal");
}
}
info!("Shutting down gracefully...");
if let Err(e) = service.stop().await {
error!("Error during shutdown: {}", e);
}
Ok(())
}
/// Mask sensitive parts of connection strings for logging.
pub fn mask_connection_string(url: &str) -> String {
if let Some(at_pos) = url.find('@') {
if let Some(proto_end) = url.find("://") {
let protocol = &url[..proto_end + 3];
let host_and_path = &url[at_pos..];
return format!("{}***:***{}", protocol, host_and_path);
}
}
"***:***@***".to_string()
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_mask_connection_string() {
let url = "postgresql://user:password@localhost:5432/attune";
let masked = mask_connection_string(url);
assert!(!masked.contains("user"));
assert!(!masked.contains("password"));
assert!(masked.contains("@localhost"));
}
#[test]
fn test_mask_connection_string_no_credentials() {
let url = "postgresql://localhost:5432/attune";
let masked = mask_connection_string(url);
assert_eq!(masked, "***:***@***");
}
#[test]
fn test_mask_rabbitmq_connection() {
let url = "amqp://admin:secret@rabbitmq:5672/%2F";
let masked = mask_connection_string(url);
assert!(!masked.contains("admin"));
assert!(!masked.contains("secret"));
assert!(masked.contains("@rabbitmq"));
}
}

View File

@@ -10,6 +10,10 @@ repository.workspace = true
name = "attune-worker" name = "attune-worker"
path = "src/main.rs" path = "src/main.rs"
[[bin]]
name = "attune-agent"
path = "src/agent_main.rs"
[dependencies] [dependencies]
attune-common = { path = "../common" } attune-common = { path = "../common" }
tokio = { workspace = true } tokio = { workspace = true }

View File

@@ -0,0 +1,220 @@
//! Attune Universal Worker Agent
//!
//! This is the entrypoint for the universal worker agent binary (`attune-agent`).
//! Unlike the standard `attune-worker` binary which requires explicit runtime
//! configuration, the agent automatically detects available interpreters in the
//! container environment and configures itself accordingly.
//!
//! ## Usage
//!
//! The agent is designed to be injected into any container image. On startup it:
//!
//! 1. Probes the system for available interpreters (python3, node, bash, etc.)
//! 2. Sets `ATTUNE_WORKER_RUNTIMES` based on what it finds
//! 3. Loads configuration (env vars are the primary config source)
//! 4. Initializes and runs the standard `WorkerService`
//!
//! ## Configuration
//!
//! Environment variables (primary):
//! - `ATTUNE__DATABASE__URL` — PostgreSQL connection string
//! - `ATTUNE__MESSAGE_QUEUE__URL` — RabbitMQ connection string
//! - `ATTUNE_WORKER_RUNTIMES` — Override auto-detection with explicit runtime list
//! - `ATTUNE_CONFIG` — Path to optional config YAML file
//!
//! CLI arguments:
//! - `--config` / `-c` — Path to configuration file (optional)
//! - `--name` / `-n` — Worker name override
//! - `--detect-only` — Run runtime detection, print results, and exit
use anyhow::Result;
use attune_common::agent_bootstrap::{bootstrap_runtime_env, print_detect_only_report};
use attune_common::config::Config;
use clap::Parser;
use tokio::signal::unix::{signal, SignalKind};
use tracing::{info, warn};
use attune_worker::dynamic_runtime::auto_register_detected_runtimes;
use attune_worker::runtime_detect::DetectedRuntime;
use attune_worker::service::WorkerService;
#[derive(Parser, Debug)]
#[command(name = "attune-agent")]
#[command(
version,
about = "Attune Universal Worker Agent - Injected into any container to auto-detect and execute actions",
long_about = "The Attune Agent automatically discovers available runtime interpreters \
in the current environment and registers as a worker capable of executing \
actions for those runtimes. It is designed to be injected into arbitrary \
container images without requiring manual runtime configuration."
)]
struct Args {
/// Path to configuration file (optional — env vars are the primary config source)
#[arg(short, long)]
config: Option<String>,
/// Worker name (overrides config and auto-generated name)
#[arg(short, long)]
name: Option<String>,
/// Run runtime detection, print results, and exit without starting the worker
#[arg(long)]
detect_only: bool,
}
fn main() -> Result<()> {
// Install HMAC-only JWT crypto provider (must be before any token operations)
attune_common::auth::install_crypto_provider();
// Initialize tracing
tracing_subscriber::fmt()
.with_target(false)
.with_thread_ids(true)
.init();
let args = Args::parse();
info!("Starting Attune Universal Worker Agent");
info!("Agent binary: attune-agent {}", env!("CARGO_PKG_VERSION"));
// Safe: no async runtime or worker threads are running yet.
std::env::set_var("ATTUNE_AGENT_MODE", "true");
std::env::set_var("ATTUNE_AGENT_BINARY_NAME", "attune-agent");
std::env::set_var("ATTUNE_AGENT_BINARY_VERSION", env!("CARGO_PKG_VERSION"));
let bootstrap = bootstrap_runtime_env("ATTUNE_WORKER_RUNTIMES");
let agent_detected_runtimes = bootstrap.detected_runtimes.clone();
// --- Handle --detect-only (synchronous, no async runtime needed) ---
if args.detect_only {
print_detect_only_report("ATTUNE_WORKER_RUNTIMES", &bootstrap);
return Ok(());
}
// --- Set config path env var (synchronous, before tokio runtime) ---
if let Some(ref config_path) = args.config {
// Safe: no other threads are running yet (tokio runtime not started).
std::env::set_var("ATTUNE_CONFIG", config_path);
}
// --- Build the tokio runtime and run the async portion ---
let runtime = tokio::runtime::Runtime::new()?;
runtime.block_on(async_main(args, agent_detected_runtimes))
}
/// The async portion of the agent entrypoint. Called from `main()` via
/// `runtime.block_on()` after all environment variable mutations are complete.
async fn async_main(
args: Args,
agent_detected_runtimes: Option<Vec<DetectedRuntime>>,
) -> Result<()> {
// --- Phase 2: Load configuration ---
let mut config = Config::load()?;
config.validate()?;
// Override worker name if provided via CLI
if let Some(name) = args.name {
if let Some(ref mut worker_config) = config.worker {
worker_config.name = Some(name);
} else {
config.worker = Some(attune_common::config::WorkerConfig {
name: Some(name),
worker_type: None,
runtime_id: None,
host: None,
port: None,
capabilities: None,
max_concurrent_tasks: 10,
heartbeat_interval: 30,
task_timeout: 300,
max_stdout_bytes: 10 * 1024 * 1024,
max_stderr_bytes: 10 * 1024 * 1024,
shutdown_timeout: Some(30),
stream_logs: true,
});
}
}
info!("Configuration loaded successfully");
info!("Environment: {}", config.environment);
// --- Phase 2b: Dynamic runtime registration ---
//
// Before creating the WorkerService (which loads runtimes from the DB into
// its runtime registry), ensure that every detected runtime has a
// corresponding entry in the database. This handles the case where the
// agent detects a runtime (e.g., Ruby) that has a template in the core
// pack but hasn't been explicitly loaded by this agent before.
if let Some(ref detected) = agent_detected_runtimes {
info!(
"Ensuring {} detected runtime(s) are registered in the database...",
detected.len()
);
// We need a temporary DB connection for dynamic registration.
// WorkerService::new() will create its own connection, so this is
// a short-lived pool just for the registration step.
let db = attune_common::db::Database::new(&config.database).await?;
let pool = db.pool().clone();
match auto_register_detected_runtimes(&pool, detected).await {
Ok(count) => {
if count > 0 {
info!(
"Dynamic registration complete: {} new runtime(s) added to database",
count
);
} else {
info!("Dynamic registration: all detected runtimes already in database");
}
}
Err(e) => {
warn!(
"Dynamic runtime registration failed (non-fatal, continuing): {}",
e
);
}
}
}
// --- Phase 3: Initialize and run the worker service ---
let service = WorkerService::new(config).await?;
// If we auto-detected runtimes, pass them to the worker service so that
// registration includes the full `detected_interpreters` capability
// (binary paths + versions) and the `agent_mode` flag.
let mut service = if let Some(detected) = agent_detected_runtimes {
info!(
"Passing {} detected runtime(s) to worker registration",
detected.len()
);
service.with_detected_runtimes(detected)
} else {
service
};
info!("Attune Agent is ready");
service.start().await?;
// Setup signal handlers for graceful shutdown
let mut sigint = signal(SignalKind::interrupt())?;
let mut sigterm = signal(SignalKind::terminate())?;
tokio::select! {
_ = sigint.recv() => {
info!("Received SIGINT signal");
}
_ = sigterm.recv() => {
info!("Received SIGTERM signal");
}
}
info!("Shutting down gracefully...");
service.stop().await?;
info!("Attune Agent shutdown complete");
Ok(())
}

View File

@@ -0,0 +1,541 @@
//! Dynamic Runtime Registration Module
//!
//! When the agent detects an interpreter on the local system (e.g., Ruby, Go, Perl)
//! that does not yet have a corresponding runtime entry in the database, this module
//! handles auto-registering it so that the normal runtime-loading pipeline in
//! `WorkerService::new()` picks it up.
//!
//! ## Registration Strategy
//!
//! For each detected runtime the agent found:
//!
//! 1. **Look up by name** in the database using alias-aware matching.
//! 2. **If found** → already registered (either from a pack YAML or a previous
//! agent run). Nothing to do.
//! 3. **If not found** → search for a runtime *template* in loaded packs whose
//! normalized name matches. Templates are pack-registered runtimes (e.g.,
//! `core.ruby`) that provide the full `execution_config` needed to invoke
//! the interpreter, manage environments, and install dependencies.
//! 4. **If a template is found** → clone it as an auto-detected runtime with
//! `auto_detected = true` and populate `detection_config` with what the
//! agent discovered (binary path, version, etc.).
//! 5. **If no template exists** → create a minimal runtime with just the
//! detected interpreter binary path and file extension. This lets the agent
//! execute simple scripts immediately, even without a full template.
//! 6. Mark all auto-registered runtimes with `auto_detected = true`.
use attune_common::error::Result;
use attune_common::models::runtime::Runtime;
use attune_common::repositories::runtime::{CreateRuntimeInput, RuntimeRepository};
use attune_common::repositories::{Create, FindByRef, List};
use serde_json::json;
use sqlx::PgPool;
use tracing::{debug, info, warn};
use crate::runtime_detect::DetectedRuntime;
/// Canonical file extensions for runtimes that the auto-detection module knows
/// about. Used when creating minimal runtime entries without a template.
fn default_file_extension(runtime_name: &str) -> Option<&'static str> {
match runtime_name {
"shell" => Some(".sh"),
"python" => Some(".py"),
"node" => Some(".js"),
"ruby" => Some(".rb"),
"go" => Some(".go"),
"java" => Some(".java"),
"perl" => Some(".pl"),
"r" => Some(".R"),
_ => None,
}
}
/// Ensure that every detected runtime has a corresponding entry in the
/// `runtime` table. Runtimes that already exist (from pack loading or a
/// previous agent run) are left untouched. Missing runtimes are created
/// either from a matching pack template or as a minimal auto-detected entry.
///
/// This function should be called **before** `WorkerService::new()` so that
/// the normal runtime-loading pipeline finds all detected runtimes in the DB.
///
/// Returns the number of runtimes that were newly registered.
pub async fn auto_register_detected_runtimes(
pool: &PgPool,
detected: &[DetectedRuntime],
) -> Result<usize> {
if detected.is_empty() {
return Ok(0);
}
info!(
"Checking {} detected runtime(s) for dynamic registration...",
detected.len()
);
// Load all existing runtimes once to avoid repeated queries.
let existing_runtimes = RuntimeRepository::list(pool).await.unwrap_or_default();
let mut registered_count = 0;
for detected_rt in detected {
let canonical_name = detected_rt.name.to_ascii_lowercase();
// Check if a runtime with a matching name already exists in the DB.
// Primary: check if the detected name appears in any existing runtime's aliases.
// Secondary: check if the ref ends with the canonical name (e.g., "core.ruby").
let already_exists = existing_runtimes.iter().any(|r| {
// Primary: check if the detected name is in this runtime's aliases
r.aliases.iter().any(|a| a == &canonical_name)
// Secondary: check if the ref ends with the canonical name (e.g., "core.ruby")
|| r.r#ref.ends_with(&format!(".{}", canonical_name))
});
if already_exists {
debug!(
"Runtime '{}' (canonical: '{}') already exists in database, skipping",
detected_rt.name, canonical_name
);
continue;
}
// No existing runtime — try to find a template from loaded packs.
// Templates are pack-registered runtimes whose normalized name matches
// (e.g., `core.ruby` for detected runtime "ruby"). Since we already
// checked `existing_runtimes` above and found nothing, we look for
// runtimes by ref pattern. Common convention: `core.<name>`.
let template_ref = format!("core.{}", canonical_name);
let template = RuntimeRepository::find_by_ref(pool, &template_ref)
.await
.unwrap_or(None);
let detection_config = build_detection_config(detected_rt);
if let Some(tmpl) = template {
// Clone the template as an auto-detected runtime.
// The template already has the full execution_config, distributions, etc.
// We just re-create it with auto_detected = true.
info!(
"Found template '{}' for detected runtime '{}', registering auto-detected clone",
tmpl.r#ref, detected_rt.name
);
// Use a distinct ref so we don't collide with the template.
let auto_ref = format!("auto.{}", canonical_name);
// Check if the auto ref already exists (race condition / previous run)
if RuntimeRepository::find_by_ref(pool, &auto_ref)
.await
.unwrap_or(None)
.is_some()
{
debug!(
"Auto-detected runtime '{}' already registered from a previous run",
auto_ref
);
continue;
}
let input = CreateRuntimeInput {
r#ref: auto_ref.clone(),
pack: tmpl.pack,
pack_ref: tmpl.pack_ref.clone(),
description: Some(format!(
"Auto-detected {} runtime (from template {})",
detected_rt.name, tmpl.r#ref
)),
name: tmpl.name.clone(),
aliases: tmpl.aliases.clone(),
distributions: tmpl.distributions.clone(),
installation: tmpl.installation.clone(),
execution_config: build_execution_config_from_template(&tmpl, detected_rt),
auto_detected: true,
detection_config,
};
match RuntimeRepository::create(pool, input).await {
Ok(rt) => {
info!(
"Auto-registered runtime '{}' (ID: {}) from template '{}'",
auto_ref, rt.id, tmpl.r#ref
);
registered_count += 1;
}
Err(e) => {
// Unique constraint violation is fine (concurrent agent start)
warn!("Failed to auto-register runtime '{}': {}", auto_ref, e);
}
}
} else {
// No template found — create a minimal runtime entry.
info!(
"No template found for detected runtime '{}', creating minimal entry",
detected_rt.name
);
let auto_ref = format!("auto.{}", canonical_name);
if RuntimeRepository::find_by_ref(pool, &auto_ref)
.await
.unwrap_or(None)
.is_some()
{
debug!(
"Auto-detected runtime '{}' already registered from a previous run",
auto_ref
);
continue;
}
let execution_config = build_minimal_execution_config(detected_rt);
let input = CreateRuntimeInput {
r#ref: auto_ref.clone(),
pack: None,
pack_ref: None,
description: Some(format!(
"Auto-detected {} runtime at {}",
detected_rt.name, detected_rt.path
)),
name: capitalize_runtime_name(&canonical_name),
aliases: default_aliases(&canonical_name),
distributions: build_minimal_distributions(detected_rt),
installation: None,
execution_config,
auto_detected: true,
detection_config,
};
match RuntimeRepository::create(pool, input).await {
Ok(rt) => {
info!(
"Auto-registered minimal runtime '{}' (ID: {})",
auto_ref, rt.id
);
registered_count += 1;
}
Err(e) => {
warn!("Failed to auto-register runtime '{}': {}", auto_ref, e);
}
}
}
}
if registered_count > 0 {
info!(
"Dynamic runtime registration complete: {} new runtime(s) registered",
registered_count
);
} else {
info!("Dynamic runtime registration complete: all detected runtimes already in database");
}
Ok(registered_count)
}
/// Build the `detection_config` JSONB value from a detected runtime.
/// This metadata records how the agent discovered this runtime, enabling
/// re-verification and diagnostics.
fn build_detection_config(detected: &DetectedRuntime) -> serde_json::Value {
let mut config = json!({
"detected_path": detected.path,
"detected_name": detected.name,
});
if let Some(ref version) = detected.version {
config["detected_version"] = json!(version);
}
config
}
/// Build an execution config based on a template runtime, but with the
/// detected interpreter path substituted in. This ensures the auto-detected
/// runtime uses the actual binary path found on the system.
fn build_execution_config_from_template(
template: &Runtime,
detected: &DetectedRuntime,
) -> serde_json::Value {
let mut config = template.execution_config.clone();
// If the template has an interpreter config, update the binary path
// to the one we actually detected on this system.
if let Some(interpreter) = config.get_mut("interpreter") {
if let Some(obj) = interpreter.as_object_mut() {
obj.insert("binary".to_string(), json!(detected.path));
}
}
// If the template has an environment config with an interpreter_path
// that uses a template variable, leave it as-is (it will be resolved
// at execution time). But if it's a hardcoded absolute path, update it.
if let Some(env) = config.get_mut("environment") {
if let Some(obj) = env.as_object_mut() {
if let Some(interp_path) = obj.get("interpreter_path") {
if let Some(path_str) = interp_path.as_str() {
// Only leave template variables alone
if !path_str.contains('{') {
obj.insert("interpreter_path".to_string(), json!(detected.path));
}
}
}
}
}
config
}
/// Build a minimal execution config for a runtime with no template.
/// This provides enough information for `ProcessRuntime` to invoke the
/// interpreter directly, without environment or dependency management.
fn build_minimal_execution_config(detected: &DetectedRuntime) -> serde_json::Value {
let canonical = detected.name.to_ascii_lowercase();
let file_ext = default_file_extension(&canonical);
let mut config = json!({
"interpreter": {
"binary": detected.path,
"args": [],
}
});
if let Some(ext) = file_ext {
config["interpreter"]["file_extension"] = json!(ext);
}
config
}
/// Build minimal distributions metadata for a runtime with no template.
/// Includes a basic verification command using the detected binary path.
fn build_minimal_distributions(detected: &DetectedRuntime) -> serde_json::Value {
json!({
"verification": {
"commands": [
{
"binary": &detected.path,
"args": ["--version"],
"exit_code": 0,
"priority": 1
}
]
}
})
}
/// Default aliases for auto-detected runtimes that have no template.
/// These match what the core pack YAMLs declare but serve as fallback
/// when the template hasn't been loaded.
fn default_aliases(canonical_name: &str) -> Vec<String> {
match canonical_name {
"shell" => vec!["shell".into(), "bash".into(), "sh".into()],
"python" => vec!["python".into(), "python3".into()],
"node" => vec!["node".into(), "nodejs".into(), "node.js".into()],
"ruby" => vec!["ruby".into(), "rb".into()],
"go" => vec!["go".into(), "golang".into()],
"java" => vec!["java".into(), "jdk".into(), "openjdk".into()],
"perl" => vec!["perl".into(), "perl5".into()],
"r" => vec!["r".into(), "rscript".into()],
_ => vec![canonical_name.to_string()],
}
}
/// Capitalize a runtime name for display (e.g., "ruby" → "Ruby", "r" → "R").
fn capitalize_runtime_name(name: &str) -> String {
let mut chars = name.chars();
match chars.next() {
None => String::new(),
Some(first) => {
let upper: String = first.to_uppercase().collect();
format!("{}{}", upper, chars.as_str())
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_default_file_extension() {
assert_eq!(default_file_extension("shell"), Some(".sh"));
assert_eq!(default_file_extension("python"), Some(".py"));
assert_eq!(default_file_extension("node"), Some(".js"));
assert_eq!(default_file_extension("ruby"), Some(".rb"));
assert_eq!(default_file_extension("go"), Some(".go"));
assert_eq!(default_file_extension("java"), Some(".java"));
assert_eq!(default_file_extension("perl"), Some(".pl"));
assert_eq!(default_file_extension("r"), Some(".R"));
assert_eq!(default_file_extension("unknown"), None);
}
#[test]
fn test_capitalize_runtime_name() {
assert_eq!(capitalize_runtime_name("ruby"), "Ruby");
assert_eq!(capitalize_runtime_name("go"), "Go");
assert_eq!(capitalize_runtime_name("r"), "R");
assert_eq!(capitalize_runtime_name("perl"), "Perl");
assert_eq!(capitalize_runtime_name("java"), "Java");
assert_eq!(capitalize_runtime_name(""), "");
}
#[test]
fn test_build_detection_config_with_version() {
let detected = DetectedRuntime {
name: "ruby".to_string(),
path: "/usr/bin/ruby".to_string(),
version: Some("3.3.0".to_string()),
};
let config = build_detection_config(&detected);
assert_eq!(config["detected_path"], "/usr/bin/ruby");
assert_eq!(config["detected_name"], "ruby");
assert_eq!(config["detected_version"], "3.3.0");
}
#[test]
fn test_build_detection_config_without_version() {
let detected = DetectedRuntime {
name: "perl".to_string(),
path: "/usr/bin/perl".to_string(),
version: None,
};
let config = build_detection_config(&detected);
assert_eq!(config["detected_path"], "/usr/bin/perl");
assert_eq!(config["detected_name"], "perl");
assert!(config.get("detected_version").is_none());
}
#[test]
fn test_build_minimal_execution_config() {
let detected = DetectedRuntime {
name: "ruby".to_string(),
path: "/usr/bin/ruby".to_string(),
version: Some("3.3.0".to_string()),
};
let config = build_minimal_execution_config(&detected);
assert_eq!(config["interpreter"]["binary"], "/usr/bin/ruby");
assert_eq!(config["interpreter"]["file_extension"], ".rb");
assert_eq!(config["interpreter"]["args"], json!([]));
}
#[test]
fn test_build_minimal_execution_config_unknown_runtime() {
let detected = DetectedRuntime {
name: "custom".to_string(),
path: "/opt/custom/bin/custom".to_string(),
version: None,
};
let config = build_minimal_execution_config(&detected);
assert_eq!(config["interpreter"]["binary"], "/opt/custom/bin/custom");
// Unknown runtime has no file extension
assert!(config["interpreter"].get("file_extension").is_none());
}
#[test]
fn test_build_minimal_distributions() {
let detected = DetectedRuntime {
name: "ruby".to_string(),
path: "/usr/bin/ruby".to_string(),
version: Some("3.3.0".to_string()),
};
let distros = build_minimal_distributions(&detected);
let commands = distros["verification"]["commands"].as_array().unwrap();
assert_eq!(commands.len(), 1);
assert_eq!(commands[0]["binary"], "/usr/bin/ruby");
}
#[test]
fn test_build_execution_config_from_template_updates_binary() {
let template = Runtime {
id: 1,
r#ref: "core.ruby".to_string(),
pack: Some(1),
pack_ref: Some("core".to_string()),
description: Some("Ruby Runtime".to_string()),
name: "Ruby".to_string(),
aliases: vec!["ruby".to_string(), "rb".to_string()],
distributions: json!({}),
installation: None,
installers: json!({}),
execution_config: json!({
"interpreter": {
"binary": "ruby",
"args": [],
"file_extension": ".rb"
},
"env_vars": {
"GEM_HOME": "{env_dir}/gems"
}
}),
auto_detected: false,
detection_config: json!({}),
created: chrono::Utc::now(),
updated: chrono::Utc::now(),
};
let detected = DetectedRuntime {
name: "ruby".to_string(),
path: "/usr/local/bin/ruby3.3".to_string(),
version: Some("3.3.0".to_string()),
};
let config = build_execution_config_from_template(&template, &detected);
// Binary should be updated to the detected path
assert_eq!(config["interpreter"]["binary"], "/usr/local/bin/ruby3.3");
// Other fields should be preserved from the template
assert_eq!(config["interpreter"]["file_extension"], ".rb");
assert_eq!(config["env_vars"]["GEM_HOME"], "{env_dir}/gems");
}
#[test]
fn test_build_execution_config_from_template_preserves_template_vars() {
let template = Runtime {
id: 1,
r#ref: "core.python".to_string(),
pack: Some(1),
pack_ref: Some("core".to_string()),
description: None,
name: "Python".to_string(),
aliases: vec!["python".to_string(), "python3".to_string()],
distributions: json!({}),
installation: None,
installers: json!({}),
execution_config: json!({
"interpreter": {
"binary": "python3",
"file_extension": ".py"
},
"environment": {
"interpreter_path": "{env_dir}/bin/python3",
"create_command": ["python3", "-m", "venv", "{env_dir}"]
}
}),
auto_detected: false,
detection_config: json!({}),
created: chrono::Utc::now(),
updated: chrono::Utc::now(),
};
let detected = DetectedRuntime {
name: "python".to_string(),
path: "/usr/bin/python3.12".to_string(),
version: Some("3.12.1".to_string()),
};
let config = build_execution_config_from_template(&template, &detected);
// Binary should be updated
assert_eq!(config["interpreter"]["binary"], "/usr/bin/python3.12");
// Template-variable interpreter_path should be preserved (contains '{')
assert_eq!(
config["environment"]["interpreter_path"],
"{env_dir}/bin/python3"
);
}
}

View File

@@ -35,7 +35,7 @@ use attune_common::repositories::pack::PackRepository;
use attune_common::repositories::runtime::RuntimeRepository; use attune_common::repositories::runtime::RuntimeRepository;
use attune_common::repositories::runtime_version::RuntimeVersionRepository; use attune_common::repositories::runtime_version::RuntimeVersionRepository;
use attune_common::repositories::{FindById, List}; use attune_common::repositories::{FindById, List};
use attune_common::runtime_detection::runtime_in_filter; use attune_common::runtime_detection::runtime_aliases_match_filter;
// Re-export the utility that the API also uses so callers can reach it from // Re-export the utility that the API also uses so callers can reach it from
// either crate without adding a direct common dependency for this one function. // either crate without adding a direct common dependency for this one function.
@@ -207,7 +207,7 @@ pub async fn setup_environments_for_registered_pack(
.iter() .iter()
.filter(|name| { .filter(|name| {
if let Some(filter) = runtime_filter { if let Some(filter) = runtime_filter {
runtime_in_filter(name, filter) runtime_aliases_match_filter(&[name.to_string()], filter)
} else { } else {
true true
} }
@@ -463,12 +463,12 @@ async fn process_runtime_for_pack(
runtime_envs_dir: &Path, runtime_envs_dir: &Path,
pack_result: &mut PackEnvSetupResult, pack_result: &mut PackEnvSetupResult,
) { ) {
// Apply worker runtime filter (alias-aware matching) // Apply worker runtime filter (alias-aware matching via declared aliases)
if let Some(filter) = runtime_filter { if let Some(filter) = runtime_filter {
if !runtime_in_filter(rt_name, filter) { if !runtime_aliases_match_filter(&rt.aliases, filter) {
debug!( debug!(
"Runtime '{}' not in worker filter, skipping for pack '{}'", "Runtime '{}' not in worker filter (aliases: {:?}), skipping for pack '{}'",
rt_name, pack_ref, rt_name, rt.aliases, pack_ref,
); );
return; return;
} }

View File

@@ -19,6 +19,7 @@ use attune_common::models::runtime::RuntimeExecutionConfig;
use attune_common::models::{runtime::Runtime as RuntimeModel, Action, Execution, ExecutionStatus}; use attune_common::models::{runtime::Runtime as RuntimeModel, Action, Execution, ExecutionStatus};
use attune_common::repositories::artifact::{ArtifactRepository, ArtifactVersionRepository}; use attune_common::repositories::artifact::{ArtifactRepository, ArtifactVersionRepository};
use attune_common::repositories::execution::{ExecutionRepository, UpdateExecutionInput}; use attune_common::repositories::execution::{ExecutionRepository, UpdateExecutionInput};
use attune_common::repositories::runtime::SELECT_COLUMNS as RUNTIME_SELECT_COLUMNS;
use attune_common::repositories::runtime_version::RuntimeVersionRepository; use attune_common::repositories::runtime_version::RuntimeVersionRepository;
use attune_common::repositories::{FindById, Update}; use attune_common::repositories::{FindById, Update};
use attune_common::version_matching::select_best_version; use attune_common::version_matching::select_best_version;
@@ -410,15 +411,14 @@ impl ActionExecutor {
// Load runtime information if specified // Load runtime information if specified
let runtime_record = if let Some(runtime_id) = action.runtime { let runtime_record = if let Some(runtime_id) = action.runtime {
match sqlx::query_as::<_, RuntimeModel>( let query = format!(
r#"SELECT id, ref, pack, pack_ref, description, name, "SELECT {} FROM runtime WHERE id = $1",
distributions, installation, installers, execution_config, RUNTIME_SELECT_COLUMNS
created, updated );
FROM runtime WHERE id = $1"#, match sqlx::query_as::<_, RuntimeModel>(&query)
) .bind(runtime_id)
.bind(runtime_id) .fetch_optional(&self.pool)
.fetch_optional(&self.pool) .await
.await
{ {
Ok(Some(runtime)) => { Ok(Some(runtime)) => {
debug!( debug!(

View File

@@ -4,16 +4,19 @@
//! which executes actions in various runtime environments. //! which executes actions in various runtime environments.
pub mod artifacts; pub mod artifacts;
pub mod dynamic_runtime;
pub mod env_setup; pub mod env_setup;
pub mod executor; pub mod executor;
pub mod heartbeat; pub mod heartbeat;
pub mod registration; pub mod registration;
pub mod runtime; pub mod runtime;
pub mod runtime_detect;
pub mod secrets; pub mod secrets;
pub mod service; pub mod service;
pub mod version_verify; pub mod version_verify;
// Re-export commonly used types // Re-export commonly used types
pub use dynamic_runtime::auto_register_detected_runtimes;
pub use executor::ActionExecutor; pub use executor::ActionExecutor;
pub use heartbeat::HeartbeatManager; pub use heartbeat::HeartbeatManager;
pub use registration::WorkerRegistration; pub use registration::WorkerRegistration;
@@ -21,7 +24,8 @@ pub use runtime::{
ExecutionContext, ExecutionResult, LocalRuntime, NativeRuntime, ProcessRuntime, Runtime, ExecutionContext, ExecutionResult, LocalRuntime, NativeRuntime, ProcessRuntime, Runtime,
RuntimeError, RuntimeResult, RuntimeError, RuntimeResult,
}; };
pub use runtime_detect::DetectedRuntime;
pub use secrets::SecretManager; pub use secrets::SecretManager;
pub use service::WorkerService; pub use service::{StartupMode, WorkerService};
// Re-export test executor from common (shared business logic) // Re-export test executor from common (shared business logic)
pub use attune_common::test_executor::{TestConfig, TestExecutor}; pub use attune_common::test_executor::{TestConfig, TestExecutor};

View File

@@ -13,6 +13,12 @@ use sqlx::PgPool;
use std::collections::HashMap; use std::collections::HashMap;
use tracing::{info, warn}; use tracing::{info, warn};
use crate::runtime_detect::DetectedRuntime;
const ATTUNE_AGENT_MODE_ENV: &str = "ATTUNE_AGENT_MODE";
const ATTUNE_AGENT_BINARY_NAME_ENV: &str = "ATTUNE_AGENT_BINARY_NAME";
const ATTUNE_AGENT_BINARY_VERSION_ENV: &str = "ATTUNE_AGENT_BINARY_VERSION";
/// Worker registration manager /// Worker registration manager
pub struct WorkerRegistration { pub struct WorkerRegistration {
pool: PgPool, pool: PgPool,
@@ -27,12 +33,60 @@ pub struct WorkerRegistration {
} }
impl WorkerRegistration { impl WorkerRegistration {
fn env_truthy(name: &str) -> bool {
std::env::var(name)
.ok()
.map(|value| matches!(value.trim().to_ascii_lowercase().as_str(), "1" | "true"))
.unwrap_or(false)
}
fn inject_agent_capabilities(capabilities: &mut HashMap<String, serde_json::Value>) {
if Self::env_truthy(ATTUNE_AGENT_MODE_ENV) {
capabilities.insert("agent_mode".to_string(), json!(true));
}
if let Ok(binary_name) = std::env::var(ATTUNE_AGENT_BINARY_NAME_ENV) {
let binary_name = binary_name.trim();
if !binary_name.is_empty() {
capabilities.insert("agent_binary_name".to_string(), json!(binary_name));
}
}
if let Ok(binary_version) = std::env::var(ATTUNE_AGENT_BINARY_VERSION_ENV) {
let binary_version = binary_version.trim();
if !binary_version.is_empty() {
capabilities.insert("agent_binary_version".to_string(), json!(binary_version));
}
}
}
fn legacy_worker_name() -> Option<String> {
std::env::var("ATTUNE_WORKER_NAME")
.ok()
.map(|value| value.trim().to_string())
.filter(|value| !value.is_empty())
}
fn legacy_worker_type() -> Option<WorkerType> {
let value = std::env::var("ATTUNE_WORKER_TYPE").ok()?;
match value.trim().to_ascii_lowercase().as_str() {
"local" => Some(WorkerType::Local),
"remote" => Some(WorkerType::Remote),
"container" => Some(WorkerType::Container),
other => {
warn!("Ignoring unrecognized ATTUNE_WORKER_TYPE value: {}", other);
None
}
}
}
/// Create a new worker registration manager /// Create a new worker registration manager
pub fn new(pool: PgPool, config: &Config) -> Self { pub fn new(pool: PgPool, config: &Config) -> Self {
let worker_name = config let worker_name = config
.worker .worker
.as_ref() .as_ref()
.and_then(|w| w.name.clone()) .and_then(|w| w.name.clone())
.or_else(Self::legacy_worker_name)
.unwrap_or_else(|| { .unwrap_or_else(|| {
format!( format!(
"worker-{}", "worker-{}",
@@ -46,6 +100,7 @@ impl WorkerRegistration {
.worker .worker
.as_ref() .as_ref()
.and_then(|w| w.worker_type) .and_then(|w| w.worker_type)
.or_else(Self::legacy_worker_type)
.unwrap_or(WorkerType::Local); .unwrap_or(WorkerType::Local);
let worker_role = WorkerRole::Action; let worker_role = WorkerRole::Action;
@@ -84,6 +139,8 @@ impl WorkerRegistration {
json!(env!("CARGO_PKG_VERSION")), json!(env!("CARGO_PKG_VERSION")),
); );
Self::inject_agent_capabilities(&mut capabilities);
// Placeholder for runtimes (will be detected asynchronously) // Placeholder for runtimes (will be detected asynchronously)
capabilities.insert("runtimes".to_string(), json!(Vec::<String>::new())); capabilities.insert("runtimes".to_string(), json!(Vec::<String>::new()));
@@ -100,6 +157,51 @@ impl WorkerRegistration {
} }
} }
/// Store detected runtime interpreter metadata in capabilities.
///
/// This is used by the agent (`attune-agent`) to record the full details of
/// auto-detected interpreters — binary paths and versions — alongside the
/// simple `runtimes` string list used for backward compatibility.
///
/// The data is stored under the `detected_interpreters` capability key as a
/// JSON array of objects:
/// ```json
/// [
/// {"name": "python", "path": "/usr/bin/python3", "version": "3.12.1"},
/// {"name": "shell", "path": "/bin/bash", "version": "5.2.15"}
/// ]
/// ```
pub fn set_detected_runtimes(&mut self, runtimes: Vec<DetectedRuntime>) {
let interpreters: Vec<serde_json::Value> = runtimes
.iter()
.map(|rt| {
json!({
"name": rt.name,
"path": rt.path,
"version": rt.version,
})
})
.collect();
self.capabilities
.insert("detected_interpreters".to_string(), json!(interpreters));
info!(
"Stored {} detected interpreter(s) in capabilities",
runtimes.len()
);
}
/// Mark this worker as running in agent mode.
///
/// Agent-mode workers auto-detect their runtimes at startup (as opposed to
/// being configured via `ATTUNE_WORKER_RUNTIMES` or config files). Setting
/// this flag allows the system to distinguish agents from standard workers.
pub fn set_agent_mode(&mut self, is_agent: bool) {
self.capabilities
.insert("agent_mode".to_string(), json!(is_agent));
}
/// Detect available runtimes using the unified runtime detector /// Detect available runtimes using the unified runtime detector
pub async fn detect_capabilities(&mut self, config: &Config) -> Result<()> { pub async fn detect_capabilities(&mut self, config: &Config) -> Result<()> {
info!("Detecting worker capabilities..."); info!("Detecting worker capabilities...");
@@ -346,4 +448,96 @@ mod tests {
registration.deregister().await.unwrap(); registration.deregister().await.unwrap();
} }
#[test]
fn test_detected_runtimes_json_structure() {
// Test the JSON structure that set_detected_runtimes builds
let runtimes = vec![
DetectedRuntime {
name: "python".to_string(),
path: "/usr/bin/python3".to_string(),
version: Some("3.12.1".to_string()),
},
DetectedRuntime {
name: "shell".to_string(),
path: "/bin/bash".to_string(),
version: None,
},
];
let interpreters: Vec<serde_json::Value> = runtimes
.iter()
.map(|rt| {
json!({
"name": rt.name,
"path": rt.path,
"version": rt.version,
})
})
.collect();
let json_value = json!(interpreters);
// Verify structure
let arr = json_value.as_array().unwrap();
assert_eq!(arr.len(), 2);
assert_eq!(arr[0]["name"], "python");
assert_eq!(arr[0]["path"], "/usr/bin/python3");
assert_eq!(arr[0]["version"], "3.12.1");
assert_eq!(arr[1]["name"], "shell");
assert_eq!(arr[1]["path"], "/bin/bash");
assert!(arr[1]["version"].is_null());
}
#[test]
fn test_detected_runtimes_empty() {
let runtimes: Vec<DetectedRuntime> = vec![];
let interpreters: Vec<serde_json::Value> = runtimes
.iter()
.map(|rt| {
json!({
"name": rt.name,
"path": rt.path,
"version": rt.version,
})
})
.collect();
let json_value = json!(interpreters);
assert_eq!(json_value.as_array().unwrap().len(), 0);
}
#[test]
fn test_agent_mode_capability_value() {
// Verify the JSON value for agent_mode capability
let value = json!(true);
assert_eq!(value, true);
let value = json!(false);
assert_eq!(value, false);
}
#[test]
fn test_inject_agent_capabilities_from_env() {
std::env::set_var(ATTUNE_AGENT_MODE_ENV, "TRUE");
std::env::set_var(ATTUNE_AGENT_BINARY_NAME_ENV, "attune-agent");
std::env::set_var(ATTUNE_AGENT_BINARY_VERSION_ENV, "1.2.3");
let mut capabilities = HashMap::new();
WorkerRegistration::inject_agent_capabilities(&mut capabilities);
assert_eq!(capabilities.get("agent_mode"), Some(&json!(true)));
assert_eq!(
capabilities.get("agent_binary_name"),
Some(&json!("attune-agent"))
);
assert_eq!(
capabilities.get("agent_binary_version"),
Some(&json!("1.2.3"))
);
std::env::remove_var(ATTUNE_AGENT_MODE_ENV);
std::env::remove_var(ATTUNE_AGENT_BINARY_NAME_ENV);
std::env::remove_var(ATTUNE_AGENT_BINARY_VERSION_ENV);
}
} }

View File

@@ -24,9 +24,27 @@ use attune_common::models::runtime::{
}; };
use std::collections::HashMap; use std::collections::HashMap;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use std::sync::{Arc, Mutex as StdMutex, OnceLock};
use tokio::process::Command; use tokio::process::Command;
use tracing::{debug, error, info, warn}; use tracing::{debug, error, info, warn};
/// Per-directory locks for lazy environment setup to prevent concurrent
/// setup of the same environment from corrupting it. When two executions
/// for the same pack arrive concurrently (e.g. in agent mode), both may
/// see `!env_dir.exists()` and race to run `setup_pack_environment`.
/// This map provides a per-directory async mutex so that only one setup
/// runs at a time for each env_dir path.
static ENV_SETUP_LOCKS: OnceLock<StdMutex<HashMap<PathBuf, Arc<tokio::sync::Mutex<()>>>>> =
OnceLock::new();
fn get_env_setup_lock(env_dir: &Path) -> Arc<tokio::sync::Mutex<()>> {
let locks = ENV_SETUP_LOCKS.get_or_init(|| StdMutex::new(HashMap::new()));
let mut map = locks.lock().unwrap();
map.entry(env_dir.to_path_buf())
.or_insert_with(|| Arc::new(tokio::sync::Mutex::new(())))
.clone()
}
fn bash_single_quote_escape(s: &str) -> String { fn bash_single_quote_escape(s: &str) -> String {
s.replace('\'', "'\\''") s.replace('\'', "'\\''")
} }
@@ -615,88 +633,127 @@ impl Runtime for ProcessRuntime {
None None
}; };
// Runtime environments are set up proactively — either at worker startup // Lazy environment setup: if the environment directory doesn't exist but
// (scanning all registered packs) or via pack.registered MQ events when a // should (i.e., there's an environment config and the pack dir exists),
// new pack is installed. We only log a warning here if the expected // create it on-demand. This is the primary code path for agent mode where
// environment directory is missing so operators can investigate. // proactive startup setup is skipped, but it also serves as a safety net
if effective_config.environment.is_some() && pack_dir.exists() && !env_dir.exists() { // for standard workers if the environment was somehow missed.
warn!( // Acquire a per-directory async lock to serialize environment setup.
"Runtime environment for pack '{}' not found at {}. \ // This prevents concurrent executions for the same pack from racing
The environment should have been created at startup or on pack registration. \ // to create or repair the environment simultaneously.
Proceeding with system interpreter as fallback.", if effective_config.environment.is_some() && pack_dir.exists() {
context.action_ref, let env_lock = get_env_setup_lock(&env_dir);
env_dir.display(), let _guard = env_lock.lock().await;
);
}
// If the environment directory exists but contains a broken interpreter // --- Lazy environment creation (double-checked after lock) ---
// (e.g. broken symlinks from a venv created in a different container), if !env_dir.exists() {
// attempt to recreate it before resolving the interpreter. info!(
if effective_config.environment.is_some() && env_dir.exists() && pack_dir.exists() { "Runtime environment for pack '{}' not found at {}. \
if let Some(ref env_cfg) = effective_config.environment { Creating on first use (lazy setup).",
if let Some(ref interp_template) = env_cfg.interpreter_path { context.action_ref,
let mut vars = std::collections::HashMap::new(); env_dir.display(),
vars.insert("env_dir", env_dir.to_string_lossy().to_string()); );
vars.insert("pack_dir", pack_dir.to_string_lossy().to_string());
let resolved = RuntimeExecutionConfig::resolve_template(interp_template, &vars);
let resolved_path = std::path::PathBuf::from(&resolved);
// Check for a broken symlink: symlink_metadata succeeds for let setup_runtime = ProcessRuntime::new(
// the link itself even when its target is missing, while self.runtime_name.clone(),
// exists() (which follows symlinks) returns false. effective_config.clone(),
let is_broken_symlink = !resolved_path.exists() self.packs_base_dir.clone(),
&& std::fs::symlink_metadata(&resolved_path) self.runtime_envs_dir.clone(),
.map(|m| m.file_type().is_symlink()) );
.unwrap_or(false); match setup_runtime
.setup_pack_environment(&pack_dir, &env_dir)
if is_broken_symlink { .await
let target = std::fs::read_link(&resolved_path) {
.map(|t| t.display().to_string()) Ok(()) => {
.unwrap_or_else(|_| "<unreadable>".to_string()); info!(
warn!( "Successfully created environment for pack '{}' at {} (lazy setup)",
"Detected broken symlink at '{}' -> '{}' in venv for pack '{}'. \
Removing broken environment and recreating...",
resolved_path.display(),
target,
context.action_ref, context.action_ref,
env_dir.display(),
); );
}
Err(e) => {
warn!(
"Failed to create environment for pack '{}' at {}: {}. \
Proceeding with system interpreter as fallback.",
context.action_ref,
env_dir.display(),
e,
);
}
}
}
// Remove the broken environment directory // --- Broken-symlink repair (also under the per-directory lock) ---
if let Err(e) = std::fs::remove_dir_all(&env_dir) { // If the environment directory exists but contains a broken interpreter
// (e.g. broken symlinks from a venv created in a different container),
// attempt to recreate it before resolving the interpreter.
if env_dir.exists() {
if let Some(ref env_cfg) = effective_config.environment {
if let Some(ref interp_template) = env_cfg.interpreter_path {
let mut vars = std::collections::HashMap::new();
vars.insert("env_dir", env_dir.to_string_lossy().to_string());
vars.insert("pack_dir", pack_dir.to_string_lossy().to_string());
let resolved =
RuntimeExecutionConfig::resolve_template(interp_template, &vars);
let resolved_path = std::path::PathBuf::from(&resolved);
// Check for a broken symlink: symlink_metadata succeeds for
// the link itself even when its target is missing, while
// exists() (which follows symlinks) returns false.
let is_broken_symlink = !resolved_path.exists()
&& std::fs::symlink_metadata(&resolved_path)
.map(|m| m.file_type().is_symlink())
.unwrap_or(false);
if is_broken_symlink {
let target = std::fs::read_link(&resolved_path)
.map(|t| t.display().to_string())
.unwrap_or_else(|_| "<unreadable>".to_string());
warn!( warn!(
"Failed to remove broken environment at {}: {}. \ "Detected broken symlink at '{}' -> '{}' in venv for pack '{}'. \
Will proceed with system interpreter.", Removing broken environment and recreating...",
env_dir.display(), resolved_path.display(),
e, target,
context.action_ref,
); );
} else {
// Recreate the environment using a temporary ProcessRuntime // Remove the broken environment directory
// with the effective (possibly version-specific) config. if let Err(e) = std::fs::remove_dir_all(&env_dir) {
let setup_runtime = ProcessRuntime::new( warn!(
self.runtime_name.clone(), "Failed to remove broken environment at {}: {}. \
effective_config.clone(), Will proceed with system interpreter.",
self.packs_base_dir.clone(), env_dir.display(),
self.runtime_envs_dir.clone(), e,
); );
match setup_runtime } else {
.setup_pack_environment(&pack_dir, &env_dir) // Recreate the environment using a temporary ProcessRuntime
.await // with the effective (possibly version-specific) config.
{ let setup_runtime = ProcessRuntime::new(
Ok(()) => { self.runtime_name.clone(),
info!( effective_config.clone(),
"Successfully recreated environment for pack '{}' at {}", self.packs_base_dir.clone(),
context.action_ref, self.runtime_envs_dir.clone(),
env_dir.display(), );
); match setup_runtime
} .setup_pack_environment(&pack_dir, &env_dir)
Err(e) => { .await
warn!( {
"Failed to recreate environment for pack '{}' at {}: {}. \ Ok(()) => {
Will proceed with system interpreter.", info!(
context.action_ref, "Successfully recreated environment for pack '{}' at {}",
env_dir.display(), context.action_ref,
e, env_dir.display(),
); );
}
Err(e) => {
warn!(
"Failed to recreate environment for pack '{}' at {}: {}. \
Will proceed with system interpreter.",
context.action_ref,
env_dir.display(),
e,
);
}
} }
} }
} }

View File

@@ -0,0 +1,12 @@
//! Compatibility wrapper around the shared agent runtime detection module.
pub use attune_common::agent_runtime_detection::{
detect_runtimes, format_as_env_value, DetectedRuntime,
};
pub fn print_detection_report(runtimes: &[DetectedRuntime]) {
attune_common::agent_runtime_detection::print_detection_report_for_env(
"ATTUNE_WORKER_RUNTIMES",
runtimes,
);
}

View File

@@ -23,7 +23,7 @@ use attune_common::mq::{
MessageEnvelope, MessageType, PackRegisteredPayload, Publisher, PublisherConfig, MessageEnvelope, MessageType, PackRegisteredPayload, Publisher, PublisherConfig,
}; };
use attune_common::repositories::{execution::ExecutionRepository, FindById}; use attune_common::repositories::{execution::ExecutionRepository, FindById};
use attune_common::runtime_detection::runtime_in_filter; use attune_common::runtime_detection::runtime_aliases_match_filter;
use chrono::Utc; use chrono::Utc;
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use sqlx::PgPool; use sqlx::PgPool;
@@ -45,12 +45,32 @@ use crate::runtime::local::LocalRuntime;
use crate::runtime::native::NativeRuntime; use crate::runtime::native::NativeRuntime;
use crate::runtime::process::ProcessRuntime; use crate::runtime::process::ProcessRuntime;
use crate::runtime::RuntimeRegistry; use crate::runtime::RuntimeRegistry;
use crate::runtime_detect::DetectedRuntime;
use crate::secrets::SecretManager; use crate::secrets::SecretManager;
use crate::version_verify; use crate::version_verify;
use attune_common::repositories::runtime::RuntimeRepository; use attune_common::repositories::runtime::RuntimeRepository;
use attune_common::repositories::List; use attune_common::repositories::List;
/// Controls how the worker initializes its runtime environment.
///
/// The standard `attune-worker` binary uses `Worker` mode with proactive
/// setup at startup, while the `attune-agent` binary uses `Agent` mode
/// with lazy (on-demand) initialization.
#[derive(Debug, Clone)]
pub enum StartupMode {
/// Full worker mode: proactive environment setup, full version
/// verification sweep at startup. Used by `attune-worker`.
Worker,
/// Agent mode: lazy environment setup (on first use), on-demand
/// version verification, auto-detected runtimes. Used by `attune-agent`.
Agent {
/// Runtimes detected by the auto-detection module.
detected_runtimes: Vec<DetectedRuntime>,
},
}
/// Message payload for execution.scheduled events /// Message payload for execution.scheduled events
#[derive(Debug, Clone, Serialize, Deserialize)] #[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ExecutionScheduledPayload { pub struct ExecutionScheduledPayload {
@@ -93,6 +113,10 @@ pub struct WorkerService {
/// Tracks cancellation requests that arrived before the in-memory token /// Tracks cancellation requests that arrived before the in-memory token
/// for an execution had been registered. /// for an execution had been registered.
pending_cancellations: Arc<Mutex<HashSet<i64>>>, pending_cancellations: Arc<Mutex<HashSet<i64>>>,
/// Controls whether this worker runs in full `Worker` mode (proactive
/// environment setup, full version verification) or `Agent` mode (lazy
/// setup, auto-detected runtimes).
startup_mode: StartupMode,
} }
impl WorkerService { impl WorkerService {
@@ -229,10 +253,10 @@ impl WorkerService {
// Uses alias-aware matching so that e.g. filter "node" // Uses alias-aware matching so that e.g. filter "node"
// matches DB runtime name "Node.js" (lowercased to "node.js"). // matches DB runtime name "Node.js" (lowercased to "node.js").
if let Some(ref filter) = runtime_filter { if let Some(ref filter) = runtime_filter {
if !runtime_in_filter(&rt_name, filter) { if !runtime_aliases_match_filter(&rt.aliases, filter) {
debug!( debug!(
"Skipping runtime '{}' (not in ATTUNE_WORKER_RUNTIMES filter)", "Skipping runtime '{}' (aliases {:?} not in ATTUNE_WORKER_RUNTIMES filter)",
rt_name rt_name, rt.aliases
); );
continue; continue;
} }
@@ -402,9 +426,26 @@ impl WorkerService {
in_flight_tasks: Arc::new(Mutex::new(JoinSet::new())), in_flight_tasks: Arc::new(Mutex::new(JoinSet::new())),
cancel_tokens: Arc::new(Mutex::new(HashMap::new())), cancel_tokens: Arc::new(Mutex::new(HashMap::new())),
pending_cancellations: Arc::new(Mutex::new(HashSet::new())), pending_cancellations: Arc::new(Mutex::new(HashSet::new())),
startup_mode: StartupMode::Worker,
}) })
} }
/// Set agent-detected runtimes for inclusion in worker registration.
///
/// When the worker is started as `attune-agent`, the agent entrypoint
/// auto-detects available interpreters and passes them here. During
/// [`start()`](Self::start), the detection results are stored in the
/// worker's capabilities as `detected_interpreters` (structured JSON
/// with binary paths and versions) and the `agent_mode` flag is set.
///
/// This method is a no-op for the standard `attune-worker` binary.
pub fn with_detected_runtimes(mut self, runtimes: Vec<DetectedRuntime>) -> Self {
self.startup_mode = StartupMode::Agent {
detected_runtimes: runtimes,
};
self
}
/// Start the worker service /// Start the worker service
pub async fn start(&mut self) -> Result<()> { pub async fn start(&mut self) -> Result<()> {
info!("Starting Worker Service"); info!("Starting Worker Service");
@@ -413,6 +454,21 @@ impl WorkerService {
let worker_id = { let worker_id = {
let mut reg = self.registration.write().await; let mut reg = self.registration.write().await;
reg.detect_capabilities(&self.config).await?; reg.detect_capabilities(&self.config).await?;
// If running as an agent, store the detected interpreter metadata
// and set the agent_mode flag before registering.
if let StartupMode::Agent {
ref detected_runtimes,
} = self.startup_mode
{
reg.set_detected_runtimes(detected_runtimes.clone());
reg.set_agent_mode(true);
info!(
"Agent mode: {} detected interpreter(s) will be stored in capabilities",
detected_runtimes.len()
);
}
reg.register().await? reg.register().await?
}; };
self.worker_id = Some(worker_id); self.worker_id = Some(worker_id);
@@ -430,16 +486,26 @@ impl WorkerService {
})?; })?;
info!("Worker-specific message queue infrastructure setup completed"); info!("Worker-specific message queue infrastructure setup completed");
// Verify which runtime versions are available on this system. match &self.startup_mode {
// This updates the `available` flag in the database so that StartupMode::Worker => {
// `select_best_version()` only considers genuinely present versions. // Verify which runtime versions are available on this system.
self.verify_runtime_versions().await; // This updates the `available` flag in the database so that
// `select_best_version()` only considers genuinely present versions.
self.verify_runtime_versions().await;
// Proactively set up runtime environments for all registered packs. // Proactively set up runtime environments for all registered packs.
// This runs before we start consuming execution messages so that // This runs before we start consuming execution messages so that
// environments are ready by the time the first execution arrives. // environments are ready by the time the first execution arrives.
// Now version-aware: creates per-version environments where needed. // Now version-aware: creates per-version environments where needed.
self.scan_and_setup_environments().await; self.scan_and_setup_environments().await;
}
StartupMode::Agent { .. } => {
// Skip proactive setup — will happen lazily on first execution
info!(
"Agent mode: deferring environment setup and version verification to first use"
);
}
}
// Start heartbeat // Start heartbeat
self.heartbeat.start().await?; self.heartbeat.start().await?;

View File

@@ -17,7 +17,7 @@ use tracing::{debug, info, warn};
use attune_common::models::RuntimeVersion; use attune_common::models::RuntimeVersion;
use attune_common::repositories::runtime_version::RuntimeVersionRepository; use attune_common::repositories::runtime_version::RuntimeVersionRepository;
use attune_common::runtime_detection::runtime_in_filter; use attune_common::runtime_detection::runtime_aliases_match_filter;
/// Result of verifying all runtime versions at startup. /// Result of verifying all runtime versions at startup.
#[derive(Debug)] #[derive(Debug)]
@@ -95,7 +95,7 @@ pub async fn verify_all_runtime_versions(
.to_lowercase(); .to_lowercase();
if let Some(filter) = runtime_filter { if let Some(filter) = runtime_filter {
if !runtime_in_filter(&rt_base_name, filter) { if !runtime_aliases_match_filter(&[rt_base_name.to_string()], filter) {
debug!( debug!(
"Skipping version '{}' of runtime '{}' (not in worker runtime filter)", "Skipping version '{}' of runtime '{}' (not in worker runtime filter)",
version.version, version.runtime_ref, version.version, version.runtime_ref,

216
docker-compose.agent.yaml Normal file
View File

@@ -0,0 +1,216 @@
# Agent-Based Worker Services
#
# This override file demonstrates how to add custom runtime workers to Attune
# by injecting the universal worker agent into any container image.
#
# Usage:
# docker compose -f docker-compose.yaml -f docker-compose.agent.yaml up -d
#
# Prerequisites:
# The init-agent service (defined in docker-compose.yaml) must be present.
# It builds the statically-linked worker and sensor agent binaries and populates the agent_bin volume.
#
# How it works:
# 1. init-agent builds musl-static injected agent binaries and copies them to the agent_bin volume
# 2. Each worker service mounts agent_bin read-only and uses the agent as its entrypoint
# 3. The agent auto-detects available runtimes in the container (python, ruby, node, etc.)
# 4. No Dockerfile needed — just point at any container image with your desired runtime
#
# Adding your own worker:
# Copy one of the examples below and change:
# - service name (e.g., worker-my-runtime)
# - image (any Docker image with your runtime installed)
# - ATTUNE_WORKER_NAME (unique name for this worker)
# - Optionally set ATTUNE_WORKER_RUNTIMES to override auto-detection
services:
# ---------------------------------------------------------------------------
# Ruby Worker — Official Ruby image with auto-detected runtime
# ---------------------------------------------------------------------------
worker-ruby:
image: ruby:3.3-slim
container_name: attune-worker-ruby
depends_on:
init-agent:
condition: service_completed_successfully
init-packs:
condition: service_completed_successfully
migrations:
condition: service_completed_successfully
postgres:
condition: service_healthy
rabbitmq:
condition: service_healthy
entrypoint: ["/opt/attune/agent/attune-agent"]
stop_grace_period: 45s
environment:
RUST_LOG: info
ATTUNE_CONFIG: /opt/attune/config/config.yaml
ATTUNE_WORKER_NAME: worker-ruby-01
ATTUNE_WORKER_TYPE: container
ATTUNE__SECURITY__JWT_SECRET: ${JWT_SECRET:-docker-dev-secret-change-in-production}
ATTUNE__SECURITY__ENCRYPTION_KEY: ${ENCRYPTION_KEY:-docker-dev-encryption-key-please-change-in-production-32plus}
ATTUNE__DATABASE__URL: postgresql://attune:attune@postgres:5432/attune
ATTUNE__MESSAGE_QUEUE__URL: amqp://attune:attune@rabbitmq:5672
ATTUNE_API_URL: http://attune-api:8080
# ATTUNE_WORKER_RUNTIMES omitted — auto-detected as ruby,shell
volumes:
- agent_bin:/opt/attune/agent:ro
- ${ATTUNE_DOCKER_CONFIG_PATH:-./config.docker.yaml}:/opt/attune/config/config.yaml:ro
- packs_data:/opt/attune/packs:ro
- runtime_envs:/opt/attune/runtime_envs
- artifacts_data:/opt/attune/artifacts
healthcheck:
test: ["CMD-SHELL", "pgrep -f attune-agent || exit 1"]
interval: 30s
timeout: 10s
retries: 3
start_period: 20s
networks:
- attune-network
restart: unless-stopped
# ---------------------------------------------------------------------------
# Python 3.12 Worker — Use a specific Python version via the agent
# ---------------------------------------------------------------------------
# This demonstrates using the agent with a specific Python version instead of
# the built-in worker-python service (which uses debian:bookworm-slim + apt).
#
# worker-python312:
# image: python:3.12-slim
# container_name: attune-worker-python312
# depends_on:
# init-agent:
# condition: service_completed_successfully
# init-packs:
# condition: service_completed_successfully
# migrations:
# condition: service_completed_successfully
# postgres:
# condition: service_healthy
# rabbitmq:
# condition: service_healthy
# entrypoint: ["/opt/attune/agent/attune-agent"]
# stop_grace_period: 45s
# environment:
# RUST_LOG: info
# ATTUNE_CONFIG: /opt/attune/config/config.yaml
# ATTUNE_WORKER_NAME: worker-python312-01
# ATTUNE_WORKER_TYPE: container
# ATTUNE__SECURITY__JWT_SECRET: ${JWT_SECRET:-docker-dev-secret-change-in-production}
# ATTUNE__SECURITY__ENCRYPTION_KEY: ${ENCRYPTION_KEY:-docker-dev-encryption-key-please-change-in-production-32plus}
# ATTUNE__DATABASE__URL: postgresql://attune:attune@postgres:5432/attune
# ATTUNE__MESSAGE_QUEUE__URL: amqp://attune:attune@rabbitmq:5672
# ATTUNE_API_URL: http://attune-api:8080
# volumes:
# - agent_bin:/opt/attune/agent:ro
# - ${ATTUNE_DOCKER_CONFIG_PATH:-./config.docker.yaml}:/opt/attune/config/config.yaml:ro
# - packs_data:/opt/attune/packs:ro
# - runtime_envs:/opt/attune/runtime_envs
# - artifacts_data:/opt/attune/artifacts
# healthcheck:
# test: ["CMD-SHELL", "pgrep -f attune-agent || exit 1"]
# interval: 30s
# timeout: 10s
# retries: 3
# start_period: 20s
# networks:
# - attune-network
# restart: unless-stopped
# ---------------------------------------------------------------------------
# GPU Worker — NVIDIA CUDA image for GPU-accelerated workloads
# ---------------------------------------------------------------------------
# Requires: NVIDIA Container Toolkit installed on the Docker host
# See: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/
#
# worker-gpu:
# image: nvidia/cuda:12.3.1-runtime-ubuntu22.04
# container_name: attune-worker-gpu
# depends_on:
# init-agent:
# condition: service_completed_successfully
# init-packs:
# condition: service_completed_successfully
# migrations:
# condition: service_completed_successfully
# postgres:
# condition: service_healthy
# rabbitmq:
# condition: service_healthy
# entrypoint: ["/opt/attune/agent/attune-agent"]
# runtime: nvidia
# stop_grace_period: 45s
# environment:
# RUST_LOG: info
# ATTUNE_CONFIG: /opt/attune/config/config.yaml
# ATTUNE_WORKER_NAME: worker-gpu-01
# ATTUNE_WORKER_TYPE: container
# ATTUNE_WORKER_RUNTIMES: python,shell # Manual override — CUDA image has python pre-installed
# ATTUNE__SECURITY__JWT_SECRET: ${JWT_SECRET:-docker-dev-secret-change-in-production}
# ATTUNE__SECURITY__ENCRYPTION_KEY: ${ENCRYPTION_KEY:-docker-dev-encryption-key-please-change-in-production-32plus}
# ATTUNE__DATABASE__URL: postgresql://attune:attune@postgres:5432/attune
# ATTUNE__MESSAGE_QUEUE__URL: amqp://attune:attune@rabbitmq:5672
# ATTUNE_API_URL: http://attune-api:8080
# volumes:
# - agent_bin:/opt/attune/agent:ro
# - ${ATTUNE_DOCKER_CONFIG_PATH:-./config.docker.yaml}:/opt/attune/config/config.yaml:ro
# - packs_data:/opt/attune/packs:ro
# - runtime_envs:/opt/attune/runtime_envs
# - artifacts_data:/opt/attune/artifacts
# healthcheck:
# test: ["CMD-SHELL", "pgrep -f attune-agent || exit 1"]
# interval: 30s
# timeout: 10s
# retries: 3
# start_period: 20s
# networks:
# - attune-network
# restart: unless-stopped
# ---------------------------------------------------------------------------
# Custom Image Worker — Template for any container image
# ---------------------------------------------------------------------------
# Copy this template and customize for your runtime:
#
# worker-custom:
# image: my-org/my-custom-image:latest
# container_name: attune-worker-custom
# depends_on:
# init-agent:
# condition: service_completed_successfully
# init-packs:
# condition: service_completed_successfully
# migrations:
# condition: service_completed_successfully
# postgres:
# condition: service_healthy
# rabbitmq:
# condition: service_healthy
# entrypoint: ["/opt/attune/agent/attune-agent"]
# stop_grace_period: 45s
# environment:
# RUST_LOG: info
# ATTUNE_CONFIG: /opt/attune/config/config.yaml
# ATTUNE_WORKER_NAME: worker-custom-01
# ATTUNE_WORKER_TYPE: container
# ATTUNE__SECURITY__JWT_SECRET: ${JWT_SECRET:-docker-dev-secret-change-in-production}
# ATTUNE__SECURITY__ENCRYPTION_KEY: ${ENCRYPTION_KEY:-docker-dev-encryption-key-please-change-in-production-32plus}
# ATTUNE__DATABASE__URL: postgresql://attune:attune@postgres:5432/attune
# ATTUNE__MESSAGE_QUEUE__URL: amqp://attune:attune@rabbitmq:5672
# ATTUNE_API_URL: http://attune-api:8080
# volumes:
# - agent_bin:/opt/attune/agent:ro
# - ${ATTUNE_DOCKER_CONFIG_PATH:-./config.docker.yaml}:/opt/attune/config/config.yaml:ro
# - packs_data:/opt/attune/packs:ro
# - runtime_envs:/opt/attune/runtime_envs
# - artifacts_data:/opt/attune/artifacts
# healthcheck:
# test: ["CMD-SHELL", "pgrep -f attune-agent || exit 1"]
# interval: 30s
# timeout: 10s
# retries: 3
# start_period: 20s
# networks:
# - attune-network
# restart: unless-stopped

View File

@@ -126,6 +126,29 @@ services:
restart: on-failure restart: on-failure
entrypoint: "" # Override Python image entrypoint entrypoint: "" # Override Python image entrypoint
# Agent binary volume population (builds the statically-linked worker and sensor agents)
# Other containers can use these binaries by mounting agent_bin and running
# /opt/attune/agent/attune-agent or /opt/attune/agent/attune-sensor-agent.
init-agent:
build:
context: .
dockerfile: docker/Dockerfile.agent
target: agent-init
args:
BUILDKIT_INLINE_CACHE: 1
container_name: attune-init-agent
volumes:
- agent_bin:/opt/attune/agent
entrypoint:
[
"/bin/sh",
"-c",
"cp /usr/local/bin/attune-agent /opt/attune/agent/attune-agent && cp /usr/local/bin/attune-sensor-agent /opt/attune/agent/attune-sensor-agent && chmod +x /opt/attune/agent/attune-agent /opt/attune/agent/attune-sensor-agent && /usr/local/bin/attune-agent --version > /opt/attune/agent/attune-agent.version && /usr/local/bin/attune-sensor-agent --version > /opt/attune/agent/attune-sensor-agent.version && echo 'Agent binaries copied successfully'",
]
restart: "no"
networks:
- attune-network
rabbitmq: rabbitmq:
image: rabbitmq:3.13-management-alpine image: rabbitmq:3.13-management-alpine
container_name: attune-rabbitmq container_name: attune-rabbitmq
@@ -199,7 +222,10 @@ services:
- runtime_envs:/opt/attune/runtime_envs - runtime_envs:/opt/attune/runtime_envs
- artifacts_data:/opt/attune/artifacts - artifacts_data:/opt/attune/artifacts
- api_logs:/opt/attune/logs - api_logs:/opt/attune/logs
- agent_bin:/opt/attune/agent:ro
depends_on: depends_on:
init-agent:
condition: service_completed_successfully
init-packs: init-packs:
condition: service_completed_successfully condition: service_completed_successfully
init-user: init-user:
@@ -271,19 +297,17 @@ services:
# ============================================================================ # ============================================================================
# Workers # Workers
# ============================================================================ # ============================================================================
# Default agent-based workers
# These use stock runtime images and inject the statically-linked attune-agent
# from the shared agent_bin volume instead of baking attune-worker into each image.
worker-shell: worker-shell:
build: image: debian:bookworm-slim
context: .
dockerfile: docker/Dockerfile.worker.optimized
target: worker-base
args:
BUILDKIT_INLINE_CACHE: 1
container_name: attune-worker-shell container_name: attune-worker-shell
entrypoint: ["/opt/attune/agent/attune-agent"]
stop_grace_period: 45s stop_grace_period: 45s
environment: environment:
RUST_LOG: info RUST_LOG: info
ATTUNE_CONFIG: /opt/attune/config/config.yaml ATTUNE_CONFIG: /opt/attune/config/config.yaml
ATTUNE_WORKER_RUNTIMES: shell
ATTUNE_WORKER_TYPE: container ATTUNE_WORKER_TYPE: container
ATTUNE_WORKER_NAME: worker-shell-01 ATTUNE_WORKER_NAME: worker-shell-01
ATTUNE__SECURITY__JWT_SECRET: ${JWT_SECRET:-docker-dev-secret-change-in-production} ATTUNE__SECURITY__JWT_SECRET: ${JWT_SECRET:-docker-dev-secret-change-in-production}
@@ -292,6 +316,7 @@ services:
ATTUNE__MESSAGE_QUEUE__URL: amqp://attune:attune@rabbitmq:5672 ATTUNE__MESSAGE_QUEUE__URL: amqp://attune:attune@rabbitmq:5672
ATTUNE_API_URL: http://attune-api:8080 ATTUNE_API_URL: http://attune-api:8080
volumes: volumes:
- agent_bin:/opt/attune/agent:ro
- ${ATTUNE_DOCKER_CONFIG_PATH:-./config.docker.yaml}:/opt/attune/config/config.yaml:ro - ${ATTUNE_DOCKER_CONFIG_PATH:-./config.docker.yaml}:/opt/attune/config/config.yaml:ro
- packs_data:/opt/attune/packs:ro - packs_data:/opt/attune/packs:ro
- ./packs.dev:/opt/attune/packs.dev:rw - ./packs.dev:/opt/attune/packs.dev:rw
@@ -299,6 +324,8 @@ services:
- artifacts_data:/opt/attune/artifacts - artifacts_data:/opt/attune/artifacts
- worker_shell_logs:/opt/attune/logs - worker_shell_logs:/opt/attune/logs
depends_on: depends_on:
init-agent:
condition: service_completed_successfully
init-packs: init-packs:
condition: service_completed_successfully condition: service_completed_successfully
init-user: init-user:
@@ -310,7 +337,7 @@ services:
rabbitmq: rabbitmq:
condition: service_healthy condition: service_healthy
healthcheck: healthcheck:
test: ["CMD-SHELL", "pgrep -f attune-worker || exit 1"] test: ["CMD-SHELL", "pgrep -f attune-agent || exit 1"]
interval: 30s interval: 30s
timeout: 10s timeout: 10s
retries: 3 retries: 3
@@ -319,20 +346,15 @@ services:
- attune-network - attune-network
restart: unless-stopped restart: unless-stopped
# Python worker - Shell + Python runtime # Python worker - official Python image with agent auto-detection
worker-python: worker-python:
build: image: python:3.12-slim
context: .
dockerfile: docker/Dockerfile.worker.optimized
target: worker-python
args:
BUILDKIT_INLINE_CACHE: 1
container_name: attune-worker-python container_name: attune-worker-python
entrypoint: ["/opt/attune/agent/attune-agent"]
stop_grace_period: 45s stop_grace_period: 45s
environment: environment:
RUST_LOG: info RUST_LOG: info
ATTUNE_CONFIG: /opt/attune/config/config.yaml ATTUNE_CONFIG: /opt/attune/config/config.yaml
ATTUNE_WORKER_RUNTIMES: shell,python
ATTUNE_WORKER_TYPE: container ATTUNE_WORKER_TYPE: container
ATTUNE_WORKER_NAME: worker-python-01 ATTUNE_WORKER_NAME: worker-python-01
ATTUNE__SECURITY__JWT_SECRET: ${JWT_SECRET:-docker-dev-secret-change-in-production} ATTUNE__SECURITY__JWT_SECRET: ${JWT_SECRET:-docker-dev-secret-change-in-production}
@@ -341,6 +363,7 @@ services:
ATTUNE__MESSAGE_QUEUE__URL: amqp://attune:attune@rabbitmq:5672 ATTUNE__MESSAGE_QUEUE__URL: amqp://attune:attune@rabbitmq:5672
ATTUNE_API_URL: http://attune-api:8080 ATTUNE_API_URL: http://attune-api:8080
volumes: volumes:
- agent_bin:/opt/attune/agent:ro
- ${ATTUNE_DOCKER_CONFIG_PATH:-./config.docker.yaml}:/opt/attune/config/config.yaml:ro - ${ATTUNE_DOCKER_CONFIG_PATH:-./config.docker.yaml}:/opt/attune/config/config.yaml:ro
- packs_data:/opt/attune/packs:ro - packs_data:/opt/attune/packs:ro
- ./packs.dev:/opt/attune/packs.dev:rw - ./packs.dev:/opt/attune/packs.dev:rw
@@ -348,6 +371,8 @@ services:
- artifacts_data:/opt/attune/artifacts - artifacts_data:/opt/attune/artifacts
- worker_python_logs:/opt/attune/logs - worker_python_logs:/opt/attune/logs
depends_on: depends_on:
init-agent:
condition: service_completed_successfully
init-packs: init-packs:
condition: service_completed_successfully condition: service_completed_successfully
init-user: init-user:
@@ -359,7 +384,7 @@ services:
rabbitmq: rabbitmq:
condition: service_healthy condition: service_healthy
healthcheck: healthcheck:
test: ["CMD-SHELL", "pgrep -f attune-worker || exit 1"] test: ["CMD-SHELL", "pgrep -f attune-agent || exit 1"]
interval: 30s interval: 30s
timeout: 10s timeout: 10s
retries: 3 retries: 3
@@ -368,20 +393,15 @@ services:
- attune-network - attune-network
restart: unless-stopped restart: unless-stopped
# Node worker - Shell + Node.js runtime # Node worker - official Node.js image with agent auto-detection
worker-node: worker-node:
build: image: node:22-slim
context: .
dockerfile: docker/Dockerfile.worker.optimized
target: worker-node
args:
BUILDKIT_INLINE_CACHE: 1
container_name: attune-worker-node container_name: attune-worker-node
entrypoint: ["/opt/attune/agent/attune-agent"]
stop_grace_period: 45s stop_grace_period: 45s
environment: environment:
RUST_LOG: info RUST_LOG: info
ATTUNE_CONFIG: /opt/attune/config/config.yaml ATTUNE_CONFIG: /opt/attune/config/config.yaml
ATTUNE_WORKER_RUNTIMES: shell,node
ATTUNE_WORKER_TYPE: container ATTUNE_WORKER_TYPE: container
ATTUNE_WORKER_NAME: worker-node-01 ATTUNE_WORKER_NAME: worker-node-01
ATTUNE__SECURITY__JWT_SECRET: ${JWT_SECRET:-docker-dev-secret-change-in-production} ATTUNE__SECURITY__JWT_SECRET: ${JWT_SECRET:-docker-dev-secret-change-in-production}
@@ -390,6 +410,7 @@ services:
ATTUNE__MESSAGE_QUEUE__URL: amqp://attune:attune@rabbitmq:5672 ATTUNE__MESSAGE_QUEUE__URL: amqp://attune:attune@rabbitmq:5672
ATTUNE_API_URL: http://attune-api:8080 ATTUNE_API_URL: http://attune-api:8080
volumes: volumes:
- agent_bin:/opt/attune/agent:ro
- ${ATTUNE_DOCKER_CONFIG_PATH:-./config.docker.yaml}:/opt/attune/config/config.yaml:ro - ${ATTUNE_DOCKER_CONFIG_PATH:-./config.docker.yaml}:/opt/attune/config/config.yaml:ro
- packs_data:/opt/attune/packs:ro - packs_data:/opt/attune/packs:ro
- ./packs.dev:/opt/attune/packs.dev:rw - ./packs.dev:/opt/attune/packs.dev:rw
@@ -397,6 +418,8 @@ services:
- artifacts_data:/opt/attune/artifacts - artifacts_data:/opt/attune/artifacts
- worker_node_logs:/opt/attune/logs - worker_node_logs:/opt/attune/logs
depends_on: depends_on:
init-agent:
condition: service_completed_successfully
init-packs: init-packs:
condition: service_completed_successfully condition: service_completed_successfully
init-user: init-user:
@@ -408,7 +431,7 @@ services:
rabbitmq: rabbitmq:
condition: service_healthy condition: service_healthy
healthcheck: healthcheck:
test: ["CMD-SHELL", "pgrep -f attune-worker || exit 1"] test: ["CMD-SHELL", "pgrep -f attune-agent || exit 1"]
interval: 30s interval: 30s
timeout: 10s timeout: 10s
retries: 3 retries: 3
@@ -417,19 +440,17 @@ services:
- attune-network - attune-network
restart: unless-stopped restart: unless-stopped
# Full worker - All runtimes (shell, python, node, native) # Full worker - Python + Node image with manual native capability override
worker-full: worker-full:
build: image: nikolaik/python-nodejs:python3.12-nodejs22-slim
context: .
dockerfile: docker/Dockerfile.worker.optimized
target: worker-full
args:
BUILDKIT_INLINE_CACHE: 1
container_name: attune-worker-full container_name: attune-worker-full
entrypoint: ["/opt/attune/agent/attune-agent"]
stop_grace_period: 45s stop_grace_period: 45s
environment: environment:
RUST_LOG: info RUST_LOG: info
ATTUNE_CONFIG: /opt/attune/config/config.yaml ATTUNE_CONFIG: /opt/attune/config/config.yaml
# Keep native support enabled explicitly; the agent auto-detects interpreters
# but "native" is a capability flag rather than a discoverable interpreter.
ATTUNE_WORKER_RUNTIMES: shell,python,node,native ATTUNE_WORKER_RUNTIMES: shell,python,node,native
ATTUNE_WORKER_TYPE: container ATTUNE_WORKER_TYPE: container
ATTUNE_WORKER_NAME: worker-full-01 ATTUNE_WORKER_NAME: worker-full-01
@@ -439,6 +460,7 @@ services:
ATTUNE__MESSAGE_QUEUE__URL: amqp://attune:attune@rabbitmq:5672 ATTUNE__MESSAGE_QUEUE__URL: amqp://attune:attune@rabbitmq:5672
ATTUNE_API_URL: http://attune-api:8080 ATTUNE_API_URL: http://attune-api:8080
volumes: volumes:
- agent_bin:/opt/attune/agent:ro
- ${ATTUNE_DOCKER_CONFIG_PATH:-./config.docker.yaml}:/opt/attune/config/config.yaml:ro - ${ATTUNE_DOCKER_CONFIG_PATH:-./config.docker.yaml}:/opt/attune/config/config.yaml:ro
- packs_data:/opt/attune/packs:ro - packs_data:/opt/attune/packs:ro
- ./packs.dev:/opt/attune/packs.dev:rw - ./packs.dev:/opt/attune/packs.dev:rw
@@ -446,6 +468,8 @@ services:
- artifacts_data:/opt/attune/artifacts - artifacts_data:/opt/attune/artifacts
- worker_full_logs:/opt/attune/logs - worker_full_logs:/opt/attune/logs
depends_on: depends_on:
init-agent:
condition: service_completed_successfully
init-packs: init-packs:
condition: service_completed_successfully condition: service_completed_successfully
init-user: init-user:
@@ -457,7 +481,7 @@ services:
rabbitmq: rabbitmq:
condition: service_healthy condition: service_healthy
healthcheck: healthcheck:
test: ["CMD-SHELL", "pgrep -f attune-worker || exit 1"] test: ["CMD-SHELL", "pgrep -f attune-agent || exit 1"]
interval: 30s interval: 30s
timeout: 10s timeout: 10s
retries: 3 retries: 3
@@ -466,18 +490,18 @@ services:
- attune-network - attune-network
restart: unless-stopped restart: unless-stopped
# Default sensor service now uses the injected sensor agent inside a stock runtime image.
sensor: sensor:
build: image: nikolaik/python-nodejs:python3.12-nodejs22-slim
context: .
dockerfile: docker/Dockerfile.sensor.optimized
target: sensor-full
args:
BUILDKIT_INLINE_CACHE: 1
container_name: attune-sensor container_name: attune-sensor
entrypoint: ["/opt/attune/agent/attune-sensor-agent"]
stop_grace_period: 45s stop_grace_period: 45s
environment: environment:
RUST_LOG: debug RUST_LOG: debug
ATTUNE_CONFIG: /opt/attune/config/config.yaml ATTUNE_CONFIG: /opt/attune/config/config.yaml
# Keep native support enabled explicitly; interpreter auto-detection does
# not infer the synthetic "native" capability.
ATTUNE_SENSOR_RUNTIMES: shell,python,node,native
ATTUNE__SECURITY__JWT_SECRET: ${JWT_SECRET:-docker-dev-secret-change-in-production} ATTUNE__SECURITY__JWT_SECRET: ${JWT_SECRET:-docker-dev-secret-change-in-production}
ATTUNE__SECURITY__ENCRYPTION_KEY: ${ENCRYPTION_KEY:-docker-dev-encryption-key-please-change-in-production-32plus} ATTUNE__SECURITY__ENCRYPTION_KEY: ${ENCRYPTION_KEY:-docker-dev-encryption-key-please-change-in-production-32plus}
ATTUNE__DATABASE__URL: postgresql://attune:attune@postgres:5432/attune ATTUNE__DATABASE__URL: postgresql://attune:attune@postgres:5432/attune
@@ -488,12 +512,15 @@ services:
ATTUNE_MQ_URL: amqp://attune:attune@rabbitmq:5672 ATTUNE_MQ_URL: amqp://attune:attune@rabbitmq:5672
ATTUNE_PACKS_BASE_DIR: /opt/attune/packs ATTUNE_PACKS_BASE_DIR: /opt/attune/packs
volumes: volumes:
- agent_bin:/opt/attune/agent:ro
- ${ATTUNE_DOCKER_CONFIG_PATH:-./config.docker.yaml}:/opt/attune/config/config.yaml:ro - ${ATTUNE_DOCKER_CONFIG_PATH:-./config.docker.yaml}:/opt/attune/config/config.yaml:ro
- packs_data:/opt/attune/packs:rw - packs_data:/opt/attune/packs:rw
- ./packs.dev:/opt/attune/packs.dev:rw - ./packs.dev:/opt/attune/packs.dev:rw
- runtime_envs:/opt/attune/runtime_envs - runtime_envs:/opt/attune/runtime_envs
- sensor_logs:/opt/attune/logs - sensor_logs:/opt/attune/logs
depends_on: depends_on:
init-agent:
condition: service_completed_successfully
init-packs: init-packs:
condition: service_completed_successfully condition: service_completed_successfully
init-user: init-user:
@@ -621,6 +648,8 @@ volumes:
driver: local driver: local
artifacts_data: artifacts_data:
driver: local driver: local
agent_bin:
driver: local
# ============================================================================ # ============================================================================
# Networks # Networks

159
docker/Dockerfile.agent Normal file
View File

@@ -0,0 +1,159 @@
# Multi-stage Dockerfile for the Attune injected agent binaries
#
# Builds statically-linked `attune-agent` and `attune-sensor-agent` binaries
# using musl, suitable for injection into arbitrary runtime containers.
#
# Stages:
# builder - Cross-compile with musl for a fully static binary
# agent-binary - Minimal scratch image containing just the binary
# agent-init - BusyBox-based image for use as a Kubernetes init container
# or Docker Compose volume-populating service (has `cp`)
#
# Usage:
# # Build the minimal binary-only image:
# DOCKER_BUILDKIT=1 docker buildx build --target agent-binary -f docker/Dockerfile.agent -t attune-agent:binary .
#
# # Build the init container image (for volume population via `cp`):
# DOCKER_BUILDKIT=1 docker buildx build --target agent-init -f docker/Dockerfile.agent -t attune-agent:latest .
#
# # Use in docker-compose.yaml to populate a shared volume:
# # agent-init:
# # image: attune-agent:latest
# # command: ["cp", "/usr/local/bin/attune-agent", "/shared/attune-agent"]
# # volumes:
# # - agent_binary:/shared
#
# Note: `attune-agent` lives in the worker crate and `attune-sensor-agent`
# lives in the sensor crate.
ARG RUST_VERSION=1.92
ARG DEBIAN_VERSION=bookworm
# ============================================================================
# Stage 1: Builder - Cross-compile a statically-linked binary with musl
# ============================================================================
FROM rust:${RUST_VERSION}-${DEBIAN_VERSION} AS builder
# Install musl toolchain for static linking
RUN apt-get update && apt-get install -y \
musl-tools \
pkg-config \
libssl-dev \
ca-certificates \
&& rm -rf /var/lib/apt/lists/*
# Add the musl target for fully static binaries
RUN rustup target add x86_64-unknown-linux-musl
WORKDIR /build
# Increase rustc stack size to prevent SIGSEGV during release builds
ENV RUST_MIN_STACK=67108864
# Enable SQLx offline mode — compile-time query checking without a live database
ENV SQLX_OFFLINE=true
# ---------------------------------------------------------------------------
# Dependency caching layer
# Copy only Cargo metadata first so `cargo fetch` is cached when only source
# code changes. This follows the same selective-copy optimization pattern as
# the other active Dockerfiles in this directory.
# ---------------------------------------------------------------------------
COPY Cargo.toml Cargo.lock ./
COPY crates/common/Cargo.toml ./crates/common/Cargo.toml
COPY crates/api/Cargo.toml ./crates/api/Cargo.toml
COPY crates/executor/Cargo.toml ./crates/executor/Cargo.toml
COPY crates/sensor/Cargo.toml ./crates/sensor/Cargo.toml
COPY crates/core-timer-sensor/Cargo.toml ./crates/core-timer-sensor/Cargo.toml
COPY crates/worker/Cargo.toml ./crates/worker/Cargo.toml
COPY crates/notifier/Cargo.toml ./crates/notifier/Cargo.toml
COPY crates/cli/Cargo.toml ./crates/cli/Cargo.toml
# Create minimal stub sources so cargo can resolve the workspace and fetch deps.
# These are ONLY used for `cargo fetch` — never compiled.
# NOTE: The worker crate has TWO binary targets and the sensor crate now has
# two binary targets as well, so we create stubs for all of them.
RUN mkdir -p crates/common/src && echo "" > crates/common/src/lib.rs && \
mkdir -p crates/api/src && echo "fn main(){}" > crates/api/src/main.rs && \
mkdir -p crates/executor/src && echo "fn main(){}" > crates/executor/src/main.rs && \
mkdir -p crates/executor/benches && echo "fn main(){}" > crates/executor/benches/context_clone.rs && \
mkdir -p crates/sensor/src && echo "fn main(){}" > crates/sensor/src/main.rs && \
echo "fn main(){}" > crates/sensor/src/agent_main.rs && \
mkdir -p crates/core-timer-sensor/src && echo "fn main(){}" > crates/core-timer-sensor/src/main.rs && \
mkdir -p crates/worker/src && echo "fn main(){}" > crates/worker/src/main.rs && \
echo "fn main(){}" > crates/worker/src/agent_main.rs && \
mkdir -p crates/notifier/src && echo "fn main(){}" > crates/notifier/src/main.rs && \
mkdir -p crates/cli/src && echo "fn main(){}" > crates/cli/src/main.rs
# Download all dependencies (cached unless Cargo.toml/Cargo.lock change)
# registry/git use sharing=shared — cargo handles concurrent reads safely
RUN --mount=type=cache,target=/usr/local/cargo/registry,sharing=shared \
--mount=type=cache,target=/usr/local/cargo/git,sharing=shared \
cargo fetch
# ---------------------------------------------------------------------------
# Build layer
# Copy real source code and compile only the agent binary with musl
# ---------------------------------------------------------------------------
COPY migrations/ ./migrations/
COPY crates/ ./crates/
# Build the injected agent binaries, statically linked with musl.
# Uses a dedicated cache ID (agent-target) so the musl target directory
# doesn't collide with the glibc target cache used by other Dockerfiles.
RUN --mount=type=cache,target=/usr/local/cargo/registry,sharing=shared \
--mount=type=cache,target=/usr/local/cargo/git,sharing=shared \
--mount=type=cache,id=agent-target,target=/build/target,sharing=locked \
cargo build --release --target x86_64-unknown-linux-musl --bin attune-agent --bin attune-sensor-agent && \
cp /build/target/x86_64-unknown-linux-musl/release/attune-agent /build/attune-agent && \
cp /build/target/x86_64-unknown-linux-musl/release/attune-sensor-agent /build/attune-sensor-agent
# Strip the binaries to minimize size
RUN strip /build/attune-agent && strip /build/attune-sensor-agent
# Verify the binaries are statically linked and functional
RUN ls -lh /build/attune-agent /build/attune-sensor-agent && \
file /build/attune-agent && \
file /build/attune-sensor-agent && \
ldd /build/attune-agent 2>&1 || true && \
ldd /build/attune-sensor-agent 2>&1 || true
# ============================================================================
# Stage 2: agent-binary - Minimal image with just the static binary
# ============================================================================
# This is the smallest possible image — a single static binary on scratch.
# Useful when you only need to extract the binary (e.g., via COPY --from).
FROM scratch AS agent-binary
COPY --from=builder /build/attune-agent /usr/local/bin/attune-agent
COPY --from=builder /build/attune-sensor-agent /usr/local/bin/attune-sensor-agent
ENTRYPOINT ["/usr/local/bin/attune-agent"]
# ============================================================================
# Stage 3: agent-init - Init container for volume population
# ============================================================================
# Uses busybox so we have `cp`, `sh`, etc. for use as a Docker Compose or
# Kubernetes init container that copies the agent binary into a shared volume.
#
# Example docker-compose.yaml usage:
# agent-init:
# image: attune-agent:latest
# command: ["cp", "/usr/local/bin/attune-agent", "/shared/attune-agent"]
# volumes:
# - agent_binary:/shared
#
# my-worker-container:
# image: python:3.12
# command: ["/agent/attune-agent"]
# volumes:
# - agent_binary:/agent:ro
# depends_on:
# agent-init:
# condition: service_completed_successfully
FROM busybox:1.36 AS agent-init
COPY --from=builder /build/attune-agent /usr/local/bin/attune-agent
COPY --from=builder /build/attune-sensor-agent /usr/local/bin/attune-sensor-agent
ENTRYPOINT ["/usr/local/bin/attune-agent"]

View File

@@ -1,10 +0,0 @@
FROM python:3.11-slim
COPY packs /source/packs
COPY scripts/load_core_pack.py /scripts/load_core_pack.py
COPY docker/init-packs.sh /init-packs.sh
RUN pip install --no-cache-dir psycopg2-binary pyyaml && \
chmod +x /init-packs.sh
CMD ["/bin/sh", "/init-packs.sh"]

View File

@@ -1,7 +0,0 @@
FROM postgres:16-alpine
COPY docker/init-user.sh /init-user.sh
RUN chmod +x /init-user.sh
CMD ["/bin/sh", "/init-user.sh"]

View File

@@ -1,9 +0,0 @@
FROM postgres:16-alpine
COPY migrations /migrations
COPY docker/run-migrations.sh /run-migrations.sh
COPY docker/init-roles.sql /docker/init-roles.sql
RUN chmod +x /run-migrations.sh
CMD ["/bin/sh", "/run-migrations.sh"]

View File

@@ -51,6 +51,7 @@ RUN mkdir -p crates/common/src && echo "" > crates/common/src/lib.rs && \
mkdir -p crates/sensor/src && echo "fn main(){}" > crates/sensor/src/main.rs && \ mkdir -p crates/sensor/src && echo "fn main(){}" > crates/sensor/src/main.rs && \
mkdir -p crates/core-timer-sensor/src && echo "fn main(){}" > crates/core-timer-sensor/src/main.rs && \ mkdir -p crates/core-timer-sensor/src && echo "fn main(){}" > crates/core-timer-sensor/src/main.rs && \
mkdir -p crates/worker/src && echo "fn main(){}" > crates/worker/src/main.rs && \ mkdir -p crates/worker/src && echo "fn main(){}" > crates/worker/src/main.rs && \
echo "fn main(){}" > crates/worker/src/agent_main.rs && \
mkdir -p crates/notifier/src && echo "fn main(){}" > crates/notifier/src/main.rs && \ mkdir -p crates/notifier/src && echo "fn main(){}" > crates/notifier/src/main.rs && \
mkdir -p crates/cli/src && echo "fn main(){}" > crates/cli/src/main.rs mkdir -p crates/cli/src && echo "fn main(){}" > crates/cli/src/main.rs

View File

@@ -51,6 +51,7 @@ RUN mkdir -p crates/executor/src && echo "fn main() {}" > crates/executor/src/ma
RUN mkdir -p crates/executor/benches && echo "fn main() {}" > crates/executor/benches/context_clone.rs RUN mkdir -p crates/executor/benches && echo "fn main() {}" > crates/executor/benches/context_clone.rs
RUN mkdir -p crates/sensor/src && echo "fn main() {}" > crates/sensor/src/main.rs RUN mkdir -p crates/sensor/src && echo "fn main() {}" > crates/sensor/src/main.rs
RUN mkdir -p crates/worker/src && echo "fn main() {}" > crates/worker/src/main.rs RUN mkdir -p crates/worker/src && echo "fn main() {}" > crates/worker/src/main.rs
RUN echo "fn main() {}" > crates/worker/src/agent_main.rs
RUN mkdir -p crates/notifier/src && echo "fn main() {}" > crates/notifier/src/main.rs RUN mkdir -p crates/notifier/src && echo "fn main() {}" > crates/notifier/src/main.rs
RUN mkdir -p crates/cli/src && echo "fn main() {}" > crates/cli/src/main.rs RUN mkdir -p crates/cli/src && echo "fn main() {}" > crates/cli/src/main.rs

View File

@@ -1,176 +0,0 @@
# Multi-stage Dockerfile for Attune sensor service
#
# Simple and robust: build the entire workspace, then copy the sensor binary
# into different runtime base images depending on language support needed.
#
# Targets:
# sensor-base - Native sensors only (lightweight)
# sensor-full - Native + Python + Node.js sensors
#
# Usage:
# DOCKER_BUILDKIT=1 docker build --target sensor-base -t attune-sensor:base -f docker/Dockerfile.sensor.optimized .
# DOCKER_BUILDKIT=1 docker build --target sensor-full -t attune-sensor:full -f docker/Dockerfile.sensor.optimized .
#
# Note: Packs are NOT copied into the image — they are mounted as volumes at runtime.
ARG RUST_VERSION=1.92
ARG DEBIAN_VERSION=bookworm
ARG NODE_VERSION=20
# ============================================================================
# Stage 1: Builder - Compile the entire workspace
# ============================================================================
FROM rust:${RUST_VERSION}-${DEBIAN_VERSION} AS builder
RUN apt-get update && apt-get install -y \
pkg-config \
libssl-dev \
ca-certificates \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /build
# Increase rustc stack size to prevent SIGSEGV during release builds
ENV RUST_MIN_STACK=67108864
# Copy dependency metadata first so `cargo fetch` layer is cached
# when only source code changes (Cargo.toml/Cargo.lock stay the same)
COPY Cargo.toml Cargo.lock ./
COPY crates/common/Cargo.toml ./crates/common/Cargo.toml
COPY crates/api/Cargo.toml ./crates/api/Cargo.toml
COPY crates/executor/Cargo.toml ./crates/executor/Cargo.toml
COPY crates/sensor/Cargo.toml ./crates/sensor/Cargo.toml
COPY crates/core-timer-sensor/Cargo.toml ./crates/core-timer-sensor/Cargo.toml
COPY crates/worker/Cargo.toml ./crates/worker/Cargo.toml
COPY crates/notifier/Cargo.toml ./crates/notifier/Cargo.toml
COPY crates/cli/Cargo.toml ./crates/cli/Cargo.toml
# Create minimal stub sources so cargo can resolve the workspace and fetch deps.
# These are ONLY used for `cargo fetch` — never compiled.
RUN mkdir -p crates/common/src && echo "" > crates/common/src/lib.rs && \
mkdir -p crates/api/src && echo "fn main(){}" > crates/api/src/main.rs && \
mkdir -p crates/executor/src && echo "fn main(){}" > crates/executor/src/main.rs && \
mkdir -p crates/executor/benches && echo "fn main(){}" > crates/executor/benches/context_clone.rs && \
mkdir -p crates/sensor/src && echo "fn main(){}" > crates/sensor/src/main.rs && \
mkdir -p crates/core-timer-sensor/src && echo "fn main(){}" > crates/core-timer-sensor/src/main.rs && \
mkdir -p crates/worker/src && echo "fn main(){}" > crates/worker/src/main.rs && \
mkdir -p crates/notifier/src && echo "fn main(){}" > crates/notifier/src/main.rs && \
mkdir -p crates/cli/src && echo "fn main(){}" > crates/cli/src/main.rs
# Download all dependencies (cached unless Cargo.toml/Cargo.lock change)
RUN --mount=type=cache,target=/usr/local/cargo/registry,sharing=shared \
--mount=type=cache,target=/usr/local/cargo/git,sharing=shared \
cargo fetch
# Now copy the real source code and migrations
COPY migrations/ ./migrations/
COPY crates/ ./crates/
# Build the entire workspace in release mode.
# All binaries are compiled together, sharing dependency compilation.
# target cache uses sharing=locked so concurrent service builds serialize
# writes to the shared compilation cache instead of corrupting it.
RUN --mount=type=cache,target=/usr/local/cargo/registry,sharing=shared \
--mount=type=cache,target=/usr/local/cargo/git,sharing=shared \
--mount=type=cache,target=/build/target,sharing=locked \
cargo build --release --workspace --bins -j 4 && \
cp /build/target/release/attune-sensor /build/attune-sensor
# Verify the binary was built
RUN ls -lh /build/attune-sensor && \
file /build/attune-sensor
# ============================================================================
# Stage 2a: Base Sensor (Native sensors only)
# Runtime capabilities: native binary sensors
# ============================================================================
FROM debian:${DEBIAN_VERSION}-slim AS sensor-base
RUN apt-get update && apt-get install -y \
ca-certificates \
libssl3 \
curl \
bash \
procps \
&& rm -rf /var/lib/apt/lists/*
RUN useradd -m -u 1000 attune && \
mkdir -p /opt/attune/packs /opt/attune/logs /opt/attune/runtime_envs /opt/attune/config && \
chown -R attune:attune /opt/attune
WORKDIR /opt/attune
COPY --from=builder /build/attune-sensor /usr/local/bin/attune-sensor
COPY migrations/ ./migrations/
USER attune
ENV RUST_LOG=info
ENV ATTUNE_CONFIG=/opt/attune/config/config.yaml
HEALTHCHECK --interval=30s --timeout=3s --start-period=20s --retries=3 \
CMD kill -0 1 || exit 1
CMD ["/usr/local/bin/attune-sensor"]
# ============================================================================
# Stage 2b: Full Sensor (Native + Python + Node.js sensors)
# Runtime capabilities: native, python, node
#
# Uses debian-slim + apt python3 + NodeSource node so that interpreter
# paths (/usr/bin/python3, /usr/bin/node) are identical to the worker
# containers. This avoids broken symlinks and path mismatches when
# sensors and workers share the runtime_envs volume.
# ============================================================================
FROM debian:${DEBIAN_VERSION}-slim AS sensor-full
# Re-declare global ARG so it's available in RUN commands within this stage
# (global ARGs are only automatically available in FROM instructions)
ARG NODE_VERSION=20
RUN apt-get update && apt-get install -y \
ca-certificates \
libssl3 \
curl \
bash \
build-essential \
python3 \
python3-pip \
python3-venv \
procps \
&& rm -rf /var/lib/apt/lists/*
# Install Node.js from NodeSource (same method and version as workers)
RUN curl -fsSL https://deb.nodesource.com/setup_${NODE_VERSION}.x | bash - && \
apt-get install -y nodejs && \
rm -rf /var/lib/apt/lists/*
# Create python symlink for convenience
RUN ln -sf /usr/bin/python3 /usr/bin/python
# Install common Python packages used by sensor scripts
# Use --break-system-packages for Debian 12+ pip-in-system-python restrictions
RUN pip3 install --no-cache-dir --break-system-packages \
requests>=2.31.0 \
pyyaml>=6.0 \
jinja2>=3.1.0 \
python-dateutil>=2.8.0
RUN useradd -m -u 1000 attune && \
mkdir -p /opt/attune/packs /opt/attune/logs /opt/attune/runtime_envs /opt/attune/config && \
chown -R attune:attune /opt/attune
WORKDIR /opt/attune
COPY --from=builder /build/attune-sensor /usr/local/bin/attune-sensor
COPY migrations/ ./migrations/
USER attune
ENV RUST_LOG=info
ENV ATTUNE_CONFIG=/opt/attune/config/config.yaml
HEALTHCHECK --interval=30s --timeout=3s --start-period=20s --retries=3 \
CMD kill -0 1 || exit 1
CMD ["/usr/local/bin/attune-sensor"]

View File

@@ -1,269 +0,0 @@
# Multi-stage Dockerfile for Attune worker service
#
# Simple and robust: build the entire workspace, then copy the worker binary
# into different runtime base images depending on language support needed.
# No dummy source compilation, no selective crate copying, no fragile hacks.
#
# Targets:
# worker-base - Shell only (lightweight)
# worker-python - Shell + Python
# worker-node - Shell + Node.js
# worker-full - Shell + Python + Node.js + Native
#
# Usage:
# DOCKER_BUILDKIT=1 docker build --target worker-base -t attune-worker:base -f docker/Dockerfile.worker.optimized .
# DOCKER_BUILDKIT=1 docker build --target worker-python -t attune-worker:python -f docker/Dockerfile.worker.optimized .
# DOCKER_BUILDKIT=1 docker build --target worker-node -t attune-worker:node -f docker/Dockerfile.worker.optimized .
# DOCKER_BUILDKIT=1 docker build --target worker-full -t attune-worker:full -f docker/Dockerfile.worker.optimized .
#
# Note: Packs are NOT copied into the image — they are mounted as volumes at runtime.
ARG RUST_VERSION=1.92
ARG DEBIAN_VERSION=bookworm
ARG NODE_VERSION=20
# ============================================================================
# Stage 1: Builder - Compile the entire workspace
# ============================================================================
FROM rust:${RUST_VERSION}-${DEBIAN_VERSION} AS builder
RUN apt-get update && apt-get install -y \
pkg-config \
libssl-dev \
ca-certificates \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /build
# Increase rustc stack size to prevent SIGSEGV during release builds
ENV RUST_MIN_STACK=67108864
# Copy dependency metadata first so `cargo fetch` layer is cached
# when only source code changes (Cargo.toml/Cargo.lock stay the same)
COPY Cargo.toml Cargo.lock ./
COPY crates/common/Cargo.toml ./crates/common/Cargo.toml
COPY crates/api/Cargo.toml ./crates/api/Cargo.toml
COPY crates/executor/Cargo.toml ./crates/executor/Cargo.toml
COPY crates/sensor/Cargo.toml ./crates/sensor/Cargo.toml
COPY crates/core-timer-sensor/Cargo.toml ./crates/core-timer-sensor/Cargo.toml
COPY crates/worker/Cargo.toml ./crates/worker/Cargo.toml
COPY crates/notifier/Cargo.toml ./crates/notifier/Cargo.toml
COPY crates/cli/Cargo.toml ./crates/cli/Cargo.toml
# Create minimal stub sources so cargo can resolve the workspace and fetch deps.
# Unlike the old approach, these are ONLY used for `cargo fetch` — never compiled.
RUN mkdir -p crates/common/src && echo "" > crates/common/src/lib.rs && \
mkdir -p crates/api/src && echo "fn main(){}" > crates/api/src/main.rs && \
mkdir -p crates/executor/src && echo "fn main(){}" > crates/executor/src/main.rs && \
mkdir -p crates/executor/benches && echo "fn main(){}" > crates/executor/benches/context_clone.rs && \
mkdir -p crates/sensor/src && echo "fn main(){}" > crates/sensor/src/main.rs && \
mkdir -p crates/core-timer-sensor/src && echo "fn main(){}" > crates/core-timer-sensor/src/main.rs && \
mkdir -p crates/worker/src && echo "fn main(){}" > crates/worker/src/main.rs && \
mkdir -p crates/notifier/src && echo "fn main(){}" > crates/notifier/src/main.rs && \
mkdir -p crates/cli/src && echo "fn main(){}" > crates/cli/src/main.rs
# Download all dependencies (cached unless Cargo.toml/Cargo.lock change)
RUN --mount=type=cache,target=/usr/local/cargo/registry,sharing=shared \
--mount=type=cache,target=/usr/local/cargo/git,sharing=shared \
cargo fetch
# Now copy the real source code and migrations
COPY migrations/ ./migrations/
COPY crates/ ./crates/
# Build the entire workspace in release mode.
# All binaries are compiled together, sharing dependency compilation.
# target cache uses sharing=locked so concurrent service builds serialize
# writes to the shared compilation cache instead of corrupting it.
RUN --mount=type=cache,target=/usr/local/cargo/registry,sharing=shared \
--mount=type=cache,target=/usr/local/cargo/git,sharing=shared \
--mount=type=cache,target=/build/target,sharing=locked \
cargo build --release --workspace --bins -j 4 && \
cp /build/target/release/attune-worker /build/attune-worker
# Verify the binary was built
RUN ls -lh /build/attune-worker && \
file /build/attune-worker
# ============================================================================
# Stage 2a: Base Worker (Shell only)
# Runtime capabilities: shell
# ============================================================================
FROM debian:${DEBIAN_VERSION}-slim AS worker-base
RUN apt-get update && apt-get install -y \
ca-certificates \
libssl3 \
curl \
bash \
procps \
&& rm -rf /var/lib/apt/lists/*
RUN useradd -m -u 1000 attune && \
mkdir -p /opt/attune/packs /opt/attune/logs /opt/attune/runtime_envs /opt/attune/config && \
chown -R attune:attune /opt/attune
WORKDIR /opt/attune
COPY --from=builder /build/attune-worker /usr/local/bin/attune-worker
USER attune
ENV ATTUNE_WORKER_RUNTIMES="shell"
ENV ATTUNE_WORKER_TYPE="container"
ENV RUST_LOG=info
ENV ATTUNE_CONFIG=/opt/attune/config/config.yaml
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
CMD pgrep -f attune-worker || exit 1
CMD ["/usr/local/bin/attune-worker"]
# ============================================================================
# Stage 2b: Python Worker (Shell + Python)
# Runtime capabilities: shell, python
#
# Uses debian-slim + apt python3 (NOT the python: Docker image) so that
# python3 lives at /usr/bin/python3 — the same path as worker-full.
# This avoids broken venv symlinks when multiple workers share the
# runtime_envs volume.
# ============================================================================
FROM debian:${DEBIAN_VERSION}-slim AS worker-python
RUN apt-get update && apt-get install -y \
ca-certificates \
libssl3 \
curl \
build-essential \
python3 \
python3-pip \
python3-venv \
procps \
&& rm -rf /var/lib/apt/lists/*
# Create python symlink for convenience
RUN ln -sf /usr/bin/python3 /usr/bin/python
# Use --break-system-packages for Debian 12+ pip-in-system-python restrictions
RUN pip3 install --no-cache-dir --break-system-packages \
requests>=2.31.0 \
pyyaml>=6.0 \
jinja2>=3.1.0 \
python-dateutil>=2.8.0
RUN useradd -m -u 1000 attune && \
mkdir -p /opt/attune/packs /opt/attune/logs /opt/attune/runtime_envs /opt/attune/config && \
chown -R attune:attune /opt/attune
WORKDIR /opt/attune
COPY --from=builder /build/attune-worker /usr/local/bin/attune-worker
USER attune
ENV ATTUNE_WORKER_RUNTIMES="shell,python"
ENV ATTUNE_WORKER_TYPE="container"
ENV RUST_LOG=info
ENV ATTUNE_CONFIG=/opt/attune/config/config.yaml
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
CMD pgrep -f attune-worker || exit 1
CMD ["/usr/local/bin/attune-worker"]
# ============================================================================
# Stage 2c: Node Worker (Shell + Node.js)
# Runtime capabilities: shell, node
#
# Uses debian-slim + NodeSource apt repo (NOT the node: Docker image) so that
# node lives at /usr/bin/node — the same path as worker-full.
# This avoids path mismatches when multiple workers share volumes.
# ============================================================================
FROM debian:${DEBIAN_VERSION}-slim AS worker-node
ARG NODE_VERSION=20
RUN apt-get update && apt-get install -y \
ca-certificates \
libssl3 \
curl \
procps \
&& rm -rf /var/lib/apt/lists/*
# Install Node.js from NodeSource (same method as worker-full)
RUN curl -fsSL https://deb.nodesource.com/setup_${NODE_VERSION}.x | bash - && \
apt-get install -y nodejs && \
rm -rf /var/lib/apt/lists/*
RUN useradd -m -u 1000 attune && \
mkdir -p /opt/attune/packs /opt/attune/logs /opt/attune/runtime_envs /opt/attune/config && \
chown -R attune:attune /opt/attune
WORKDIR /opt/attune
COPY --from=builder /build/attune-worker /usr/local/bin/attune-worker
USER attune
ENV ATTUNE_WORKER_RUNTIMES="shell,node"
ENV ATTUNE_WORKER_TYPE="container"
ENV RUST_LOG=info
ENV ATTUNE_CONFIG=/opt/attune/config/config.yaml
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
CMD pgrep -f attune-worker || exit 1
CMD ["/usr/local/bin/attune-worker"]
# ============================================================================
# Stage 2d: Full Worker (All runtimes)
# Runtime capabilities: shell, python, node, native
# ============================================================================
FROM debian:${DEBIAN_VERSION} AS worker-full
ARG NODE_VERSION=20
RUN apt-get update && apt-get install -y \
ca-certificates \
libssl3 \
curl \
build-essential \
python3 \
python3-pip \
python3-venv \
procps \
&& rm -rf /var/lib/apt/lists/*
# Install Node.js from NodeSource (same method and version as worker-node)
RUN curl -fsSL https://deb.nodesource.com/setup_${NODE_VERSION}.x | bash - && \
apt-get install -y nodejs && \
rm -rf /var/lib/apt/lists/*
RUN ln -sf /usr/bin/python3 /usr/bin/python
# Use --break-system-packages for Debian 12+ pip-in-system-python restrictions
RUN pip3 install --no-cache-dir --break-system-packages \
requests>=2.31.0 \
pyyaml>=6.0 \
jinja2>=3.1.0 \
python-dateutil>=2.8.0
RUN useradd -m -u 1000 attune && \
mkdir -p /opt/attune/packs /opt/attune/logs /opt/attune/runtime_envs /opt/attune/config && \
chown -R attune:attune /opt/attune
WORKDIR /opt/attune
COPY --from=builder /build/attune-worker /usr/local/bin/attune-worker
USER attune
ENV ATTUNE_WORKER_RUNTIMES="shell,python,node,native"
ENV ATTUNE_WORKER_TYPE="container"
ENV RUST_LOG=info
ENV ATTUNE_CONFIG=/opt/attune/config/config.yaml
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
CMD pgrep -f attune-worker || exit 1
CMD ["/usr/local/bin/attune-worker"]

View File

@@ -33,13 +33,11 @@ curl -X POST http://localhost:8080/auth/login \
- Uses build argument `SERVICE` to specify which service to build - Uses build argument `SERVICE` to specify which service to build
- Example: `docker build --build-arg SERVICE=api -f docker/Dockerfile.optimized -t attune-api .` - Example: `docker build --build-arg SERVICE=api -f docker/Dockerfile.optimized -t attune-api .`
- **`Dockerfile.worker.optimized`** - Multi-stage Dockerfile for containerized workers with different runtime capabilities - **`Dockerfile.agent`** - Multi-stage Dockerfile for the statically-linked agent image
- Supports 4 variants: `worker-base`, `worker-python`, `worker-node`, `worker-full` - Builds the `agent-init` image used to populate the shared agent binary volume
- See [README.worker.md](./README.worker.md) for details
- **`Dockerfile.pack-binaries`** - Pack binary builder used by `scripts/build-pack-binaries.sh`
- **`Dockerfile.sensor.optimized`** - Multi-stage Dockerfile for the sensor service
- Supports `sensor-base` and `sensor-full`
- **`Dockerfile.web`** - Multi-stage Dockerfile for React Web UI - **`Dockerfile.web`** - Multi-stage Dockerfile for React Web UI
- Builds with Node.js and serves with Nginx - Builds with Node.js and serves with Nginx
- Includes runtime environment variable injection - Includes runtime environment variable injection
@@ -122,8 +120,8 @@ docker compose build api
# Web UI # Web UI
docker compose build web docker compose build web
# Worker service # Notifier service
docker compose build worker docker compose build notifier
``` ```
### Build with Custom Args ### Build with Custom Args

View File

@@ -1,364 +0,0 @@
# Attune Worker Containers
This directory contains Docker configurations for building Attune worker containers with different runtime capabilities.
## Overview
Attune workers can run in containers with specialized runtime environments. Workers automatically declare their capabilities when they register with the system, enabling intelligent action scheduling based on runtime requirements.
## Worker Variants
### Base Worker (`worker-base`)
- **Runtimes**: `shell`
- **Base Image**: Debian Bookworm Slim
- **Size**: ~580 MB
- **Use Case**: Lightweight workers for shell scripts and basic automation
- **Build**: `make docker-build-worker-base`
### Python Worker (`worker-python`)
- **Runtimes**: `shell`, `python`
- **Base Image**: Python 3.11 Slim
- **Size**: ~1.2 GB
- **Includes**: pip, virtualenv, common Python libraries (requests, pyyaml, jinja2, python-dateutil)
- **Use Case**: Python actions and scripts with dependencies
- **Build**: `make docker-build-worker-python`
### Node.js Worker (`worker-node`)
- **Runtimes**: `shell`, `node`
- **Base Image**: Node 20 Slim
- **Size**: ~760 MB
- **Includes**: npm, yarn
- **Use Case**: JavaScript/TypeScript actions and npm packages
- **Build**: `make docker-build-worker-node`
### Full Worker (`worker-full`)
- **Runtimes**: `shell`, `python`, `node`, `native`
- **Base Image**: Debian Bookworm
- **Size**: ~1.6 GB
- **Includes**: Python 3.x, Node.js 20, build tools
- **Use Case**: General-purpose automation requiring multiple runtimes
- **Build**: `make docker-build-worker-full`
## Building Worker Images
### Build All Variants
```bash
make docker-build-workers
```
### Build Individual Variants
```bash
# Base worker (shell only)
make docker-build-worker-base
# Python worker
make docker-build-worker-python
# Node.js worker
make docker-build-worker-node
# Full worker (all runtimes)
make docker-build-worker-full
```
### Direct Docker Build
```bash
# Using Docker directly with BuildKit
DOCKER_BUILDKIT=1 docker build \
--target worker-python \
-t attune-worker:python \
-f docker/Dockerfile.worker.optimized \
.
```
## Running Workers
### Using Docker Compose
```bash
# Start specific worker type
docker-compose up -d worker-python
# Start all workers
docker-compose up -d worker-shell worker-python worker-node worker-full
# Scale workers
docker-compose up -d --scale worker-python=3
```
### Using Docker Run
```bash
docker run -d \
--name worker-python-01 \
--network attune_attune-network \
-e ATTUNE_WORKER_NAME=worker-python-01 \
-e ATTUNE_WORKER_RUNTIMES=shell,python \
-e ATTUNE__DATABASE__URL=postgresql://attune:attune@postgres:5432/attune \
-e ATTUNE__MESSAGE_QUEUE__URL=amqp://attune:attune@rabbitmq:5672 \
-v $(pwd)/packs:/opt/attune/packs:ro \
attune-worker:python
```
## Runtime Capability Declaration
Workers declare their capabilities in three ways (in order of precedence):
### 1. Environment Variable (Highest Priority)
```bash
ATTUNE_WORKER_RUNTIMES="shell,python,custom"
```
### 2. Configuration File
```yaml
worker:
capabilities:
runtimes: ["shell", "python"]
```
### 3. Auto-Detection (Fallback)
Workers automatically detect available runtimes by checking for binaries:
- `python3` or `python` → adds `python`
- `node` → adds `node`
- Always includes `shell` and `native`
## Configuration
### Key Environment Variables
| Variable | Description | Example |
|----------|-------------|---------|
| `ATTUNE_WORKER_NAME` | Unique worker identifier | `worker-python-01` |
| `ATTUNE_WORKER_RUNTIMES` | Comma-separated runtime list | `shell,python` |
| `ATTUNE_WORKER_TYPE` | Worker type | `container` |
| `ATTUNE__DATABASE__URL` | PostgreSQL connection | `postgresql://...` |
| `ATTUNE__MESSAGE_QUEUE__URL` | RabbitMQ connection | `amqp://...` |
| `RUST_LOG` | Log level | `info`, `debug`, `trace` |
### Resource Limits
Set CPU and memory limits in `docker-compose.override.yml`:
```yaml
services:
worker-python:
deploy:
resources:
limits:
cpus: '2.0'
memory: 2G
reservations:
cpus: '0.5'
memory: 512M
```
## Custom Worker Images
### Extend Python Worker
Create a custom worker with additional packages:
```dockerfile
# Dockerfile.worker.ml
FROM attune-worker:python
USER root
# Install ML packages
RUN pip install --no-cache-dir \
pandas \
numpy \
scikit-learn \
torch
USER attune
ENV ATTUNE_WORKER_RUNTIMES="shell,python,ml"
```
Build and run:
```bash
docker build -t attune-worker:ml -f Dockerfile.worker.ml .
docker run -d --name worker-ml-01 ... attune-worker:ml
```
### Add New Runtime
Example: Adding Ruby support
```dockerfile
FROM attune-worker:base
USER root
RUN apt-get update && apt-get install -y \
ruby-full \
&& rm -rf /var/lib/apt/lists/*
USER attune
ENV ATTUNE_WORKER_RUNTIMES="shell,ruby"
```
## Architecture
### Multi-stage Build
The `Dockerfile.worker.optimized` uses a multi-stage build pattern:
1. **Builder Stage**: Compiles the Rust worker binary
- Uses BuildKit cache mounts for fast incremental builds
- Shared across all worker variants
2. **Runtime Stages**: Creates specialized worker images
- `worker-base`: Minimal shell runtime
- `worker-python`: Python runtime
- `worker-node`: Node.js runtime
- `worker-full`: All runtimes
### Build Cache
BuildKit cache mounts dramatically speed up builds:
- First build: ~5-6 minutes
- Incremental builds: ~30-60 seconds
Cache is shared across builds using `sharing=locked` to prevent race conditions.
## Security
### Non-root Execution
All workers run as user `attune` (UID 1000)
### Read-only Packs
Pack files are mounted read-only to prevent modification:
```yaml
volumes:
- ./packs:/opt/attune/packs:ro # :ro = read-only
```
### Network Isolation
Workers run in isolated Docker network with only necessary service access
### Secret Management
Use environment variables for sensitive data; never hardcode in images
## Monitoring
### Check Worker Registration
```bash
docker-compose exec postgres psql -U attune -d attune -c \
"SELECT name, worker_type, status, capabilities->>'runtimes' as runtimes FROM worker;"
```
### View Logs
```bash
docker-compose logs -f worker-python
```
### Check Resource Usage
```bash
docker stats attune-worker-python
```
### Verify Health
```bash
docker-compose ps | grep worker
```
## Troubleshooting
### Worker Not Registering
**Check database connectivity:**
```bash
docker-compose logs worker-python | grep -i database
```
**Verify environment:**
```bash
docker-compose exec worker-python env | grep ATTUNE
```
### Runtime Not Detected
**Check runtime availability:**
```bash
docker-compose exec worker-python python3 --version
docker-compose exec worker-python node --version
```
**Force runtime declaration:**
```bash
ATTUNE_WORKER_RUNTIMES=shell,python
```
### Actions Not Scheduled
**Verify runtime match:**
```sql
-- Check action runtime requirement
SELECT a.ref, r.name as runtime
FROM action a
JOIN runtime r ON a.runtime = r.id
WHERE a.ref = 'core.my_action';
-- Check worker capabilities
SELECT name, capabilities->>'runtimes'
FROM worker
WHERE status = 'active';
```
## Performance
### Image Sizes
| Image | Size | Build Time (Cold) | Build Time (Cached) |
|-------|------|-------------------|---------------------|
| worker-base | ~580 MB | ~5 min | ~30 sec |
| worker-python | ~1.2 GB | ~6 min | ~45 sec |
| worker-node | ~760 MB | ~6 min | ~45 sec |
| worker-full | ~1.6 GB | ~7 min | ~60 sec |
### Optimization Tips
1. **Use specific variants**: Don't use `worker-full` if you only need Python
2. **Enable BuildKit**: Dramatically speeds up builds
3. **Layer caching**: Order Dockerfile commands from least to most frequently changed
4. **Multi-stage builds**: Keeps runtime images small
## Files
- `Dockerfile.worker.optimized` - Multi-stage worker Dockerfile with all variants
- `README.worker.md` - This file
- `../docker-compose.yaml` - Service definitions for all workers
## References
- [Worker Containerization Design](../docs/worker-containerization.md)
- [Quick Start Guide](../docs/worker-containers-quickstart.md)
- [Worker Service Architecture](../docs/architecture/worker-service.md)
- [Production Deployment](../docs/production-deployment.md)
## Quick Commands
```bash
# Build all workers
make docker-build-workers
# Start all workers
docker-compose up -d worker-shell worker-python worker-node worker-full
# Check worker status
docker-compose exec postgres psql -U attune -d attune -c \
"SELECT name, status, capabilities FROM worker;"
# View Python worker logs
docker-compose logs -f worker-python
# Restart worker
docker-compose restart worker-python
# Scale Python workers
docker-compose up -d --scale worker-python=3
# Stop all workers
docker-compose stop worker-shell worker-python worker-node worker-full
```

View File

@@ -0,0 +1,219 @@
# Quick Reference: Agent-Based Workers
> **TL;DR**: Inject the `attune-agent` binary into _any_ container image to turn it into an Attune worker. No Dockerfiles. No Rust compilation. ~12 lines of YAML.
## How It Works
1. The `init-agent` service (in `docker-compose.yaml`) builds the statically-linked `attune-agent` binary and copies it into the `agent_bin` volume
2. Your worker service mounts `agent_bin` read-only and uses the agent as its entrypoint
3. On startup, the agent auto-detects available runtimes (Python, Ruby, Node.js, Shell, etc.)
4. The worker registers with Attune and starts processing executions
## Quick Start
### Option A: Use the override file
```bash
# Start all services including the example Ruby agent worker
docker compose -f docker-compose.yaml -f docker-compose.agent.yaml up -d
```
The `docker-compose.agent.yaml` file includes a ready-to-use Ruby worker and commented-out templates for Python 3.12, GPU, and custom images.
### Option B: Add to docker-compose.override.yaml
Create a `docker-compose.override.yaml` in the project root:
```yaml
services:
worker-my-runtime:
image: my-org/my-custom-image:latest
container_name: attune-worker-my-runtime
depends_on:
init-agent:
condition: service_completed_successfully
init-packs:
condition: service_completed_successfully
migrations:
condition: service_completed_successfully
postgres:
condition: service_healthy
rabbitmq:
condition: service_healthy
entrypoint: ["/opt/attune/agent/attune-agent"]
stop_grace_period: 45s
environment:
RUST_LOG: info
ATTUNE_CONFIG: /opt/attune/config/config.yaml
ATTUNE_WORKER_NAME: worker-my-runtime-01
ATTUNE_WORKER_TYPE: container
ATTUNE__SECURITY__JWT_SECRET: ${JWT_SECRET:-docker-dev-secret-change-in-production}
ATTUNE__SECURITY__ENCRYPTION_KEY: ${ENCRYPTION_KEY:-docker-dev-encryption-key-please-change-in-production-32plus}
ATTUNE__DATABASE__URL: postgresql://attune:attune@postgres:5432/attune
ATTUNE__MESSAGE_QUEUE__URL: amqp://attune:attune@rabbitmq:5672
ATTUNE_API_URL: http://attune-api:8080
volumes:
- agent_bin:/opt/attune/agent:ro
- ${ATTUNE_DOCKER_CONFIG_PATH:-./config.docker.yaml}:/opt/attune/config/config.yaml:ro
- packs_data:/opt/attune/packs:ro
- runtime_envs:/opt/attune/runtime_envs
- artifacts_data:/opt/attune/artifacts
healthcheck:
test: ["CMD-SHELL", "pgrep -f attune-agent || exit 1"]
interval: 30s
timeout: 10s
retries: 3
start_period: 20s
networks:
- attune-network
restart: unless-stopped
```
Then run:
```bash
docker compose up -d
```
Docker Compose automatically merges `docker-compose.override.yaml`.
## Required Volumes
Every agent worker needs these volumes:
| Volume | Mount Path | Mode | Purpose |
|--------|-----------|------|---------|
| `agent_bin` | `/opt/attune/agent` | `ro` | The statically-linked agent binary |
| `packs_data` | `/opt/attune/packs` | `ro` | Pack files (actions, workflows, etc.) |
| `runtime_envs` | `/opt/attune/runtime_envs` | `rw` | Isolated runtime environments (venvs, node_modules) |
| `artifacts_data` | `/opt/attune/artifacts` | `rw` | File-backed artifact storage |
| Config YAML | `/opt/attune/config/config.yaml` | `ro` | Attune configuration |
## Required Environment Variables
| Variable | Description | Example |
|----------|-------------|---------|
| `ATTUNE_CONFIG` | Path to config file inside container | `/opt/attune/config/config.yaml` |
| `ATTUNE_WORKER_NAME` | Unique worker name | `worker-ruby-01` |
| `ATTUNE_WORKER_TYPE` | Worker type | `container` |
| `ATTUNE__DATABASE__URL` | PostgreSQL connection string | `postgresql://attune:attune@postgres:5432/attune` |
| `ATTUNE__MESSAGE_QUEUE__URL` | RabbitMQ connection string | `amqp://attune:attune@rabbitmq:5672` |
| `ATTUNE__SECURITY__JWT_SECRET` | JWT signing secret | (use env var) |
| `ATTUNE__SECURITY__ENCRYPTION_KEY` | Encryption key for secrets | (use env var) |
### Optional Environment Variables
| Variable | Description | Default |
|----------|-------------|---------|
| `ATTUNE_WORKER_RUNTIMES` | Override auto-detection | Auto-detected |
| `ATTUNE_API_URL` | API URL for token generation | `http://attune-api:8080` |
| `RUST_LOG` | Log level | `info` |
## Runtime Auto-Detection
The agent probes for these runtimes automatically:
| Runtime | Probed Binaries |
|---------|----------------|
| Shell | `bash`, `sh` |
| Python | `python3`, `python` |
| Node.js | `node`, `nodejs` |
| Ruby | `ruby` |
| Go | `go` |
| Java | `java` |
| R | `Rscript` |
| Perl | `perl` |
To override, set `ATTUNE_WORKER_RUNTIMES`:
```yaml
environment:
ATTUNE_WORKER_RUNTIMES: python,shell # Only advertise Python and Shell
```
## Testing Detection
Run the agent in detect-only mode to see what it finds:
```bash
# In a running container
docker exec <container> /opt/attune/agent/attune-agent --detect-only
# Or start a throwaway container
docker run --rm -v agent_bin:/opt/attune/agent:ro ruby:3.3-slim /opt/attune/agent/attune-agent --detect-only
```
## Examples
### Ruby Worker
```yaml
worker-ruby:
image: ruby:3.3-slim
entrypoint: ["/opt/attune/agent/attune-agent"]
# ... (standard depends_on, volumes, env, networks)
```
### Node.js 22 Worker
```yaml
worker-node22:
image: node:22-slim
entrypoint: ["/opt/attune/agent/attune-agent"]
# ...
```
### GPU Worker (NVIDIA CUDA)
```yaml
worker-gpu:
image: nvidia/cuda:12.3.1-runtime-ubuntu22.04
runtime: nvidia
entrypoint: ["/opt/attune/agent/attune-agent"]
environment:
ATTUNE_WORKER_RUNTIMES: python,shell # Override — CUDA image has python
# ...
```
### Multi-Runtime Custom Image
```yaml
worker-data-science:
image: my-org/data-science:latest # Has Python, R, and Julia
entrypoint: ["/opt/attune/agent/attune-agent"]
# Agent auto-detects all available runtimes
# ...
```
## Comparison: Traditional vs Agent Workers
| Aspect | Traditional Worker | Agent Worker |
|--------|-------------------|--------------|
| Docker build | Required (5+ min) | None |
| Dockerfile | Custom per runtime | Not needed |
| Base image | `debian:bookworm-slim` | Any image |
| Runtime install | Via apt/NodeSource | Pre-installed in image |
| Configuration | Manual `ATTUNE_WORKER_RUNTIMES` | Auto-detected |
| Binary | Compiled into image | Injected via volume |
| Update cycle | Rebuild image | Restart `init-agent` |
## Troubleshooting
### Agent binary not found
```
exec /opt/attune/agent/attune-agent: no such file or directory
```
The `init-agent` service hasn't completed. Check:
```bash
docker compose logs init-agent
```
### "No runtimes detected"
The container image doesn't have any recognized interpreters in `$PATH`. Either:
- Use an image that includes your runtime (e.g., `ruby:3.3-slim`)
- Set `ATTUNE_WORKER_RUNTIMES` manually
### Connection refused to PostgreSQL/RabbitMQ
Ensure your `depends_on` conditions include `postgres` and `rabbitmq` health checks, and that the container is on the `attune-network`.
## See Also
- [Universal Worker Agent Plan](plans/universal-worker-agent.md) — Full architecture document
- [Docker Deployment](docker-deployment.md) — General Docker setup
- [Worker Service](architecture/worker-service.md) — Worker architecture details

View File

@@ -0,0 +1,146 @@
# Quick Reference: Kubernetes Agent Workers
Agent-based workers let you run Attune actions inside **any container image** by injecting a statically-linked `attune-agent` binary via a Kubernetes init container. No custom Dockerfile required — just point at an image that has your runtime installed.
## How It Works
1. An **init container** (`agent-loader`) copies the `attune-agent` binary from the `attune-agent` image into an `emptyDir` volume
2. The **worker container** uses your chosen image (e.g., `ruby:3.3`) and runs the agent binary as its entrypoint
3. The agent **auto-detects** available runtimes (python, ruby, node, shell, etc.) and registers with Attune
4. Actions targeting those runtimes are routed to the agent worker via RabbitMQ
## Helm Values
Add entries to `agentWorkers` in your `values.yaml`:
```yaml
agentWorkers:
- name: ruby
image: ruby:3.3
replicas: 2
- name: python-gpu
image: nvidia/cuda:12.3.1-runtime-ubuntu22.04
replicas: 1
runtimes: [python, shell]
runtimeClassName: nvidia
nodeSelector:
gpu: "true"
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
resources:
limits:
nvidia.com/gpu: 1
- name: custom
image: my-org/my-custom-image:latest
replicas: 1
env:
- name: MY_CUSTOM_VAR
value: my-value
```
### Supported Fields
| Field | Required | Default | Description |
|-------|----------|---------|-------------|
| `name` | Yes | — | Unique name (used in Deployment and worker names) |
| `image` | Yes | — | Container image with your desired runtime(s) |
| `replicas` | No | `1` | Number of pod replicas |
| `runtimes` | No | `[]` (auto-detect) | List of runtimes to expose (e.g., `[python, shell]`) |
| `resources` | No | `{}` | Kubernetes resource requests/limits |
| `env` | No | — | Extra environment variables (`[{name, value}]`) |
| `imagePullPolicy` | No | — | Pull policy for the worker image |
| `logLevel` | No | `info` | `RUST_LOG` level |
| `runtimeClassName` | No | — | Kubernetes RuntimeClass (e.g., `nvidia`) |
| `nodeSelector` | No | — | Node selector for pod scheduling |
| `tolerations` | No | — | Tolerations for pod scheduling |
| `stopGracePeriod` | No | `45` | Termination grace period (seconds) |
## Install / Upgrade
```bash
helm upgrade --install attune oci://registry.example.com/namespace/helm/attune \
--version 0.3.0 \
--set global.imageRegistry=registry.example.com \
--set global.imageNamespace=namespace \
--set global.imageTag=0.3.0 \
-f my-values.yaml
```
## What Gets Created
For each `agentWorkers` entry, the chart creates a Deployment named `<release>-attune-agent-worker-<name>` with:
- **Init containers**:
- `agent-loader` — copies the agent binary from the `attune-agent` image to an `emptyDir` volume
- `wait-for-schema` — polls PostgreSQL until the Attune schema is ready
- `wait-for-packs` — waits for the core pack to be available on the shared PVC
- **Worker container** — runs `attune-agent` as the entrypoint inside your chosen image
- **Volumes**: `agent-bin` (emptyDir), `config` (ConfigMap), `packs` (PVC, read-only), `runtime-envs` (PVC), `artifacts` (PVC)
## Runtime Auto-Detection
When `runtimes` is empty (the default), the agent probes the container for interpreters:
| Runtime | Probed Binaries |
|---------|----------------|
| Shell | `bash`, `sh` |
| Python | `python3`, `python` |
| Node.js | `node`, `nodejs` |
| Ruby | `ruby` |
| Go | `go` |
| Java | `java` |
| R | `Rscript` |
| Perl | `perl` |
Set `runtimes` explicitly to skip auto-detection and only register the listed runtimes.
## Prerequisites
- The `attune-agent` image must be available in your registry (built from `docker/Dockerfile.agent`, target `agent-init`)
- Shared PVCs (`packs`, `runtime-envs`, `artifacts`) must support `ReadWriteMany` if agent workers run on different nodes than the standard worker
- The Attune database and RabbitMQ must be reachable from agent worker pods
## Differences from the Standard Worker
| Aspect | Standard Worker (`worker`) | Agent Worker (`agentWorkers`) |
|--------|---------------------------|-------------------------------|
| Image | Built from `Dockerfile.worker.optimized` | Any image (ruby, python, cuda, etc.) |
| Binary | Baked into the image | Injected via init container |
| Runtimes | Configured at build time | Auto-detected or explicitly listed |
| Use case | Known, pre-built runtime combos | Custom images, exotic runtimes, GPU |
Both worker types coexist — actions are routed to whichever worker has the matching runtime registered.
## Troubleshooting
**Agent binary not found**: Check that the `agent-loader` init container completed. View its logs:
```bash
kubectl logs <pod> -c agent-loader
```
**Runtime not detected**: Run the agent with `--detect-only` to see what it finds:
```bash
kubectl exec <pod> -c worker -- /opt/attune/agent/attune-agent --detect-only
```
**Worker not registering**: Check the worker container logs for database/MQ connectivity:
```bash
kubectl logs <pod> -c worker
```
**Packs not available**: Ensure the `init-packs` job has completed and the PVC is mounted:
```bash
kubectl get jobs | grep init-packs
kubectl exec <pod> -c worker -- ls /opt/attune/packs/core/
```
## See Also
- [Agent Workers (Docker Compose)](QUICKREF-agent-workers.md)
- [Universal Worker Agent Plan](plans/universal-worker-agent.md)
- [Gitea Registry and Helm](deployment/gitea-registry-and-helm.md)
- [Production Deployment](deployment/production-deployment.md)

View File

@@ -19,6 +19,7 @@ The workflow publishes these images to the Gitea OCI registry:
- `attune-migrations` - `attune-migrations`
- `attune-init-user` - `attune-init-user`
- `attune-init-packs` - `attune-init-packs`
- `attune-agent`
The Helm chart is pushed as an OCI chart to: The Helm chart is pushed as an OCI chart to:

View File

@@ -0,0 +1,266 @@
# Sensor Agent Injection Plan
## Overview
The sensor service is positioned similarly to the worker service: it is a long-running process that dispatches sensor commands into underlying runtimes rather than containing runtime-specific logic in the service binary itself. The worker side now supports injected, statically-linked agent binaries that run inside arbitrary container images. This plan extends the same model to sensors.
Goal:
- Replace the pre-built `attune-sensor` container image in default deployments with an injected sensor agent binary running inside stock runtime images
- Reuse the existing runtime auto-detection and capability reporting model
- Preserve current sensor behavior, including runtime-based execution, registration, heartbeat, and graceful shutdown
Non-goals:
- Converging worker and sensor into a single binary
- Redesigning sensor scheduling or runtime execution semantics
- Removing existing `ATTUNE_SENSOR_RUNTIMES` overrides
## Current State
Relevant implementation points:
- Sensor startup entrypoint: [crates/sensor/src/main.rs](/home/david/Codebase/attune/crates/sensor/src/main.rs)
- Sensor service orchestration: [crates/sensor/src/service.rs](/home/david/Codebase/attune/crates/sensor/src/service.rs)
- Sensor capability registration: [crates/sensor/src/sensor_worker_registration.rs](/home/david/Codebase/attune/crates/sensor/src/sensor_worker_registration.rs)
- Shared runtime detection: [crates/common/src/runtime_detection.rs](/home/david/Codebase/attune/crates/common/src/runtime_detection.rs)
- Current sensor container build: [docker/Dockerfile.sensor.optimized](/home/david/Codebase/attune/docker/Dockerfile.sensor.optimized)
- Existing worker-agent design reference: [docs/plans/universal-worker-agent.md](/home/david/Codebase/attune/docs/plans/universal-worker-agent.md)
Observations:
- Sensors already use the same three-tier capability detection model as workers:
- `ATTUNE_SENSOR_RUNTIMES`
- config file capabilities
- database-driven verification
- The main missing piece is packaging and entrypoint behavior, not capability modeling
- The current sensor Compose service still depends on a pre-built Rust binary baked into the container image
- The sensor manager relies on shared runtime environment assumptions such as interpreter paths and `runtime_envs` compatibility
## Proposed Architecture
Introduce a dedicated injected binary, `attune-sensor-agent`, analogous to the existing `attune-agent` for workers.
Responsibilities of `attune-sensor-agent`:
- Probe the container for available interpreters before the Tokio runtime starts
- Respect `ATTUNE_SENSOR_RUNTIMES` as a hard override
- Populate `ATTUNE_SENSOR_RUNTIMES` automatically when unset
- Support `--detect-only` for diagnostics
- Load config and start `SensorService`
This should remain a separate binary from `attune-agent`.
Reasoning:
- `attune-agent` is worker-specific today and boots `WorkerService`
- Sensor startup and runtime semantics are related but not identical
- A shared bootstrap library is useful; a single polymorphic agent binary is not necessary
## Implementation Phases
### Phase 1: Add Sensor Agent Binary
Add a new binary target under the sensor crate, likely:
- `name = "attune-sensor-agent"`
- `path = "src/agent_main.rs"`
The new binary should mirror the startup shape of [crates/worker/src/agent_main.rs](/home/david/Codebase/attune/crates/worker/src/agent_main.rs), but target sensors instead of workers.
Expected behavior:
1. Install the crypto provider
2. Initialize tracing
3. Parse CLI flags:
- `--config`
- `--name`
- `--detect-only`
4. Detect runtimes synchronously before Tokio starts
5. Set `ATTUNE_SENSOR_RUNTIMES` when auto-detection is used
6. Load config
7. Apply sensor name override if provided
8. Start `SensorService`
9. Handle SIGINT/SIGTERM and call `stop()`
### Phase 2: Reuse and Extract Shared Bootstrap Logic
Avoid duplicating the worker-agent detection/bootstrap code blindly.
Extract shared pieces into a reusable location, likely one of:
- `attune-common`
- a small shared helper module in `crates/common`
- a narrow internal library module used by both worker and sensor crates
Candidate shared logic:
- pre-Tokio runtime detection flow
- override handling
- `--detect-only` reporting
- environment mutation rules
Keep service-specific startup separate:
- worker agent starts `WorkerService`
- sensor agent starts `SensorService`
### Phase 3: Docker Build Support for Injected Sensor Agent
Extend the current agent binary build pipeline so the statically-linked sensor agent can be published into the same shared volume model used for workers.
Options:
- Extend [docker/Dockerfile.agent](/home/david/Codebase/attune/docker/Dockerfile.agent) to build and copy both `attune-agent` and `attune-sensor-agent`
- Or add a sibling Dockerfile if the combined build becomes unclear
Preferred outcome:
- `init-agent` populates `/opt/attune/agent/attune-agent`
- `init-agent` also populates `/opt/attune/agent/attune-sensor-agent`
Constraints:
- Keep the binaries statically linked
- Preserve the existing API binary-serving flow from the `agent_bin` volume
- Do not break current worker agent consumers
### Phase 4: Compose Integration for Sensor Agent Injection
Replace the current `sensor` service in [docker-compose.yaml](/home/david/Codebase/attune/docker-compose.yaml) with an agent-injected service.
Target shape:
- stock runtime image instead of `docker/Dockerfile.sensor.optimized`
- `entrypoint: ["/opt/attune/agent/attune-sensor-agent"]`
- `depends_on.init-agent`
- same config, packs, runtime env, and log/artifact mounts as required
Required environment variables must be preserved, especially:
- `ATTUNE_CONFIG`
- `ATTUNE__DATABASE__URL`
- `ATTUNE__MESSAGE_QUEUE__URL`
- `ATTUNE_API_URL`
- `ATTUNE_MQ_URL`
- `ATTUNE_PACKS_BASE_DIR`
Recommended default image strategy:
- Use a stock image that includes the default runtimes the sensor service should expose
- Be conservative about path compatibility with worker-created runtime environments
### Phase 5: Native Capability Handling
Sensors have the same edge case as workers: `native` is a capability but not a discoverable interpreter.
Implication:
- Pure auto-detection can discover Python, Node, Shell, Ruby, etc.
- It cannot infer `native` safely from interpreter probing alone
Plan:
- Keep explicit `ATTUNE_SENSOR_RUNTIMES=...,native` for any default full-capability sensor image
- Revisit later only if native becomes a first-class explicit capability outside interpreter discovery
### Phase 6: Runtime Environment Compatibility
The current sensor image documents an important invariant: sensors and workers share `runtime_envs`, so interpreter paths must remain compatible.
This must remain true after the migration.
Validation criteria:
- Python virtual environments created by workers remain usable by sensors
- Node runtime assumptions remain compatible across images
- No new symlink breakage due to mismatched interpreter installation paths
If necessary, prefer stock images whose paths align with the worker fleet, or explicitly document where sensor and worker images are allowed to diverge.
### Phase 7: Documentation and Examples
After implementation:
- Update [docs/plans/universal-worker-agent.md](/home/david/Codebase/attune/docs/plans/universal-worker-agent.md) with a sensor extension or cross-reference
- Update [docker-compose.yaml](/home/david/Codebase/attune/docker-compose.yaml)
- Update [docker-compose.agent.yaml](/home/david/Codebase/attune/docker-compose.agent.yaml) if it should also include sensor examples
- Add or update quick references for sensor agent injection
The message should be clear:
- Workers and sensors both support injected static agent binaries
- Runtime images are now decoupled from Rust service image builds
## Recommended Implementation Order
1. Add `attune-sensor-agent` binary and make it boot `SensorService`
2. Extract shared bootstrap logic from the worker-agent path
3. Extend the agent Docker build/init path to include the sensor agent binary
4. Replace the Compose `sensor` service with an injected sensor-agent container
5. Validate runtime detection and one end-to-end Python, Node, and native sensor path
6. Update docs and examples
## Risks
### Worker-Agent Coupling
Risk:
- Trying to reuse `attune-agent` directly for sensors will conflate worker and sensor startup semantics
Mitigation:
- Keep separate binaries with shared helper code only where it is truly generic
### Native Capability Loss
Risk:
- Auto-detection does not capture `native`
Mitigation:
- Preserve explicit `ATTUNE_SENSOR_RUNTIMES` where native support is required
### Runtime Path Mismatch
Risk:
- Switching to a stock image may reintroduce broken venv or interpreter path issues
Mitigation:
- Validate image interpreter paths against shared `runtime_envs`
- Prefer images that align with worker path conventions when possible
### Missing Environment Contract
Risk:
- The sensor manager currently depends on env vars such as `ATTUNE_API_URL`, `ATTUNE_MQ_URL`, and `ATTUNE_PACKS_BASE_DIR`
Mitigation:
- Preserve these in the injected sensor container definition
- Avoid relying solely on config fields unless the code is updated accordingly
## Validation Checklist
- `attune-sensor-agent --detect-only` reports detected runtimes correctly
- `ATTUNE_SENSOR_RUNTIMES` override still takes precedence
- Sensor registration records expected runtime capabilities in the `worker` table
- Sensor heartbeat and deregistration still work
- Python-based sensors execute successfully
- Node-based sensors execute successfully
- Native sensors execute successfully when `native` is explicitly enabled
- Shared `runtime_envs` remain usable between workers and sensors
- `docker compose config` validates cleanly after Compose changes
## Deliverables
- New `attune-sensor-agent` binary target
- Shared bootstrap/runtime-detection helpers as needed
- Updated agent build/init pipeline producing a sensor agent binary
- Updated Compose deployment using injected sensor agent containers
- Documentation updates covering the sensor agent model

File diff suppressed because it is too large Load Diff

View File

@@ -96,6 +96,7 @@ CREATE TABLE runtime (
pack_ref TEXT, pack_ref TEXT,
description TEXT, description TEXT,
name TEXT NOT NULL, name TEXT NOT NULL,
aliases TEXT[] NOT NULL DEFAULT '{}'::text[],
distributions JSONB NOT NULL, distributions JSONB NOT NULL,
installation JSONB, installation JSONB,
@@ -131,6 +132,17 @@ CREATE TABLE runtime (
-- {manifest_path} - absolute path to the dependency manifest file -- {manifest_path} - absolute path to the dependency manifest file
execution_config JSONB NOT NULL DEFAULT '{}'::jsonb, execution_config JSONB NOT NULL DEFAULT '{}'::jsonb,
-- Whether this runtime was auto-registered by an agent
-- (vs. loaded from a pack's YAML file during pack registration)
auto_detected BOOLEAN NOT NULL DEFAULT FALSE,
-- Detection metadata for auto-discovered runtimes.
-- Stores how the agent discovered this runtime (binary path, version, etc.)
-- enables re-verification on restart.
-- Example: { "detected_path": "/usr/bin/ruby", "detected_name": "ruby",
-- "detected_version": "3.3.0" }
detection_config JSONB NOT NULL DEFAULT '{}'::jsonb,
created TIMESTAMPTZ NOT NULL DEFAULT NOW(), created TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated TIMESTAMPTZ NOT NULL DEFAULT NOW(), updated TIMESTAMPTZ NOT NULL DEFAULT NOW(),
@@ -145,6 +157,9 @@ CREATE INDEX idx_runtime_created ON runtime(created DESC);
CREATE INDEX idx_runtime_name ON runtime(name); CREATE INDEX idx_runtime_name ON runtime(name);
CREATE INDEX idx_runtime_verification ON runtime USING GIN ((distributions->'verification')); CREATE INDEX idx_runtime_verification ON runtime USING GIN ((distributions->'verification'));
CREATE INDEX idx_runtime_execution_config ON runtime USING GIN (execution_config); CREATE INDEX idx_runtime_execution_config ON runtime USING GIN (execution_config);
CREATE INDEX idx_runtime_auto_detected ON runtime(auto_detected);
CREATE INDEX idx_runtime_detection_config ON runtime USING GIN (detection_config);
CREATE INDEX idx_runtime_aliases ON runtime USING GIN (aliases);
-- Trigger -- Trigger
CREATE TRIGGER update_runtime_updated CREATE TRIGGER update_runtime_updated
@@ -156,10 +171,13 @@ CREATE TRIGGER update_runtime_updated
COMMENT ON TABLE runtime IS 'Runtime environments for executing actions and sensors (unified)'; COMMENT ON TABLE runtime IS 'Runtime environments for executing actions and sensors (unified)';
COMMENT ON COLUMN runtime.ref IS 'Unique runtime reference (format: pack.name, e.g., core.python)'; COMMENT ON COLUMN runtime.ref IS 'Unique runtime reference (format: pack.name, e.g., core.python)';
COMMENT ON COLUMN runtime.name IS 'Runtime name (e.g., "Python", "Node.js", "Shell")'; COMMENT ON COLUMN runtime.name IS 'Runtime name (e.g., "Python", "Node.js", "Shell")';
COMMENT ON COLUMN runtime.aliases IS 'Lowercase alias names for this runtime (e.g., ["ruby", "rb"] for the Ruby runtime). Used for alias-aware matching during auto-detection and scheduling.';
COMMENT ON COLUMN runtime.distributions IS 'Runtime distribution metadata including verification commands, version requirements, and capabilities'; COMMENT ON COLUMN runtime.distributions IS 'Runtime distribution metadata including verification commands, version requirements, and capabilities';
COMMENT ON COLUMN runtime.installation IS 'Installation requirements and instructions including package managers and setup steps'; COMMENT ON COLUMN runtime.installation IS 'Installation requirements and instructions including package managers and setup steps';
COMMENT ON COLUMN runtime.installers IS 'Array of installer actions to create pack-specific runtime environments. Each installer defines commands to set up isolated environments (e.g., Python venv, npm install).'; COMMENT ON COLUMN runtime.installers IS 'Array of installer actions to create pack-specific runtime environments. Each installer defines commands to set up isolated environments (e.g., Python venv, npm install).';
COMMENT ON COLUMN runtime.execution_config IS 'Execution configuration: interpreter, environment setup, and dependency management. Drives how the worker executes actions and how pack install sets up environments.'; COMMENT ON COLUMN runtime.execution_config IS 'Execution configuration: interpreter, environment setup, and dependency management. Drives how the worker executes actions and how pack install sets up environments.';
COMMENT ON COLUMN runtime.auto_detected IS 'Whether this runtime was auto-registered by an agent (true) vs. loaded from a pack YAML (false)';
COMMENT ON COLUMN runtime.detection_config IS 'Detection metadata for auto-discovered runtimes: binaries probed, version regex, detected path/version';
-- ============================================================================ -- ============================================================================
-- RUNTIME VERSION TABLE -- RUNTIME VERSION TABLE

View File

@@ -0,0 +1,48 @@
ref: core.go
pack_ref: core
name: Go
aliases: [go, golang]
description: Go runtime for compiling and running Go scripts and programs
distributions:
verification:
commands:
- binary: go
args:
- "version"
exit_code: 0
pattern: "go\\d+\\."
priority: 1
min_version: "1.18"
recommended_version: "1.22"
installation:
package_managers:
- apt
- snap
- brew
module_support: true
execution_config:
interpreter:
binary: go
args:
- "run"
file_extension: ".go"
environment:
env_type: gopath
dir_name: gopath
create_command:
- sh
- "-c"
- "mkdir -p {env_dir}"
interpreter_path: null
dependencies:
manifest_file: go.mod
install_command:
- sh
- "-c"
- "cd {pack_dir} && GOPATH={env_dir} go mod download 2>/dev/null || true"
env_vars:
GOPATH: "{env_dir}"
GOMODCACHE: "{env_dir}/pkg/mod"

View File

@@ -0,0 +1,31 @@
ref: core.java
pack_ref: core
name: Java
aliases: [java, jdk, openjdk]
description: Java runtime for executing Java programs and scripts
distributions:
verification:
commands:
- binary: java
args:
- "-version"
exit_code: 0
pattern: "version \"\\d+"
priority: 1
min_version: "11"
recommended_version: "21"
installation:
interpreters:
- java
- javac
package_managers:
- maven
- gradle
execution_config:
interpreter:
binary: java
args: []
file_extension: ".java"

View File

@@ -1,6 +1,7 @@
ref: core.native ref: core.native
pack_ref: core pack_ref: core
name: Native name: Native
aliases: [native, builtin, standalone]
description: Native compiled runtime (Rust, Go, C, etc.) - executes binaries directly without an interpreter description: Native compiled runtime (Rust, Go, C, etc.) - executes binaries directly without an interpreter
distributions: distributions:

View File

@@ -1,6 +1,7 @@
ref: core.nodejs ref: core.nodejs
pack_ref: core pack_ref: core
name: Node.js name: Node.js
aliases: [node, nodejs, "node.js"]
description: Node.js runtime for JavaScript-based actions and sensors description: Node.js runtime for JavaScript-based actions and sensors
distributions: distributions:

View File

@@ -0,0 +1,47 @@
ref: core.perl
pack_ref: core
name: Perl
aliases: [perl, perl5]
description: Perl runtime for script execution with optional CPAN dependency management
distributions:
verification:
commands:
- binary: perl
args:
- "--version"
exit_code: 0
pattern: "perl.*v\\d+\\."
priority: 1
min_version: "5.20"
recommended_version: "5.38"
installation:
package_managers:
- cpanm
- cpan
interpreters:
- perl
execution_config:
interpreter:
binary: perl
args: []
file_extension: ".pl"
environment:
env_type: local_lib
dir_name: perl5
create_command:
- sh
- "-c"
- "mkdir -p {env_dir}/lib/perl5"
interpreter_path: null
dependencies:
manifest_file: cpanfile
install_command:
- sh
- "-c"
- "cd {pack_dir} && PERL5LIB={env_dir}/lib/perl5 PERL_LOCAL_LIB_ROOT={env_dir} cpanm --local-lib {env_dir} --installdeps --quiet . 2>/dev/null || true"
env_vars:
PERL5LIB: "{env_dir}/lib/perl5"
PERL_LOCAL_LIB_ROOT: "{env_dir}"

View File

@@ -1,6 +1,7 @@
ref: core.python ref: core.python
pack_ref: core pack_ref: core
name: Python name: Python
aliases: [python, python3]
description: Python 3 runtime for actions and sensors with automatic environment management description: Python 3 runtime for actions and sensors with automatic environment management
distributions: distributions:

View File

@@ -0,0 +1,48 @@
ref: core.r
pack_ref: core
name: R
aliases: [r, rscript]
description: R runtime for statistical computing and data analysis scripts
distributions:
verification:
commands:
- binary: Rscript
args:
- "--version"
exit_code: 0
pattern: "\\d+\\.\\d+\\.\\d+"
priority: 1
min_version: "4.0.0"
recommended_version: "4.4.0"
installation:
package_managers:
- install.packages
- renv
interpreters:
- Rscript
- R
execution_config:
interpreter:
binary: Rscript
args:
- "--vanilla"
file_extension: ".R"
environment:
env_type: renv
dir_name: renv
create_command:
- sh
- "-c"
- "mkdir -p {env_dir}/library"
interpreter_path: null
dependencies:
manifest_file: renv.lock
install_command:
- sh
- "-c"
- 'cd {pack_dir} && R_LIBS_USER={env_dir}/library Rscript -e "if (file.exists(''renv.lock'')) renv::restore(library=''{env_dir}/library'', prompt=FALSE)" 2>/dev/null || true'
env_vars:
R_LIBS_USER: "{env_dir}/library"

View File

@@ -0,0 +1,49 @@
ref: core.ruby
pack_ref: core
name: Ruby
aliases: [ruby, rb]
description: Ruby runtime for script execution with automatic gem environment management
distributions:
verification:
commands:
- binary: ruby
args:
- "--version"
exit_code: 0
pattern: "ruby \\d+\\."
priority: 1
min_version: "2.7"
recommended_version: "3.2"
installation:
package_managers:
- gem
- bundler
interpreters:
- ruby
portable: false
execution_config:
interpreter:
binary: ruby
args: []
file_extension: ".rb"
environment:
env_type: gem_home
dir_name: gems
create_command:
- sh
- "-c"
- "mkdir -p {env_dir}/gems"
interpreter_path: null
dependencies:
manifest_file: Gemfile
install_command:
- sh
- "-c"
- "cd {pack_dir} && GEM_HOME={env_dir}/gems GEM_PATH={env_dir}/gems bundle install --quiet 2>/dev/null || true"
env_vars:
GEM_HOME: "{env_dir}/gems"
GEM_PATH: "{env_dir}/gems"
BUNDLE_PATH: "{env_dir}/gems"

View File

@@ -1,6 +1,7 @@
ref: core.shell ref: core.shell
pack_ref: core pack_ref: core
name: Shell name: Shell
aliases: [shell, bash, sh]
description: Shell (bash/sh) runtime for script execution - always available description: Shell (bash/sh) runtime for script execution - always available
distributions: distributions:

122
scripts/attune-agent-wrapper.sh Executable file
View File

@@ -0,0 +1,122 @@
#!/bin/sh
# attune-agent-wrapper.sh — Bootstrap the Attune agent in any container
#
# This script provides a simple way to start the Attune universal worker agent
# in containers where the agent binary isn't available via a shared volume.
# It first checks for a volume-mounted binary, then falls back to downloading
# it from the Attune API.
#
# Usage:
# As a container entrypoint:
# entrypoint: ["/opt/attune/scripts/attune-agent-wrapper.sh"]
#
# With Docker Compose volume mount:
# volumes:
# - ./scripts/attune-agent-wrapper.sh:/opt/attune/scripts/attune-agent-wrapper.sh:ro
#
# Environment Variables:
# ATTUNE_AGENT_DIR - Directory for the agent binary (default: /opt/attune/agent)
# ATTUNE_AGENT_URL - URL to download the agent binary from
# (default: http://attune-api:8080/api/v1/agent/binary)
# ATTUNE_AGENT_TOKEN - Optional bootstrap token for authenticated downloads
# ATTUNE_AGENT_ARCH - Target architecture (default: auto-detected via uname -m)
#
set -e
AGENT_DIR="${ATTUNE_AGENT_DIR:-/opt/attune/agent}"
AGENT_BIN="$AGENT_DIR/attune-agent"
AGENT_URL="${ATTUNE_AGENT_URL:-http://attune-api:8080/api/v1/agent/binary}"
# SECURITY: The default URL uses plain HTTP, which is fine for internal Docker
# networking. For cross-network or production deployments, set ATTUNE_AGENT_URL
# to an HTTPS endpoint and consider setting ATTUNE_AGENT_TOKEN to authenticate.
AGENT_TOKEN="${ATTUNE_AGENT_TOKEN:-}"
# Auto-detect architecture if not specified
if [ -z "$ATTUNE_AGENT_ARCH" ]; then
MACHINE=$(uname -m)
case "$MACHINE" in
x86_64|amd64) ATTUNE_AGENT_ARCH="x86_64" ;;
aarch64|arm64) ATTUNE_AGENT_ARCH="aarch64" ;;
*)
echo "[attune] WARNING: Unknown architecture '$MACHINE', defaulting to x86_64" >&2
ATTUNE_AGENT_ARCH="x86_64"
;;
esac
fi
# Use volume-mounted binary if available
if [ -x "$AGENT_BIN" ]; then
echo "[attune] Agent binary found at $AGENT_BIN"
exec "$AGENT_BIN" "$@"
fi
# Download the agent binary
echo "[attune] Agent binary not found at $AGENT_BIN, downloading..."
echo "[attune] URL: $AGENT_URL"
echo "[attune] Architecture: $ATTUNE_AGENT_ARCH"
DOWNLOAD_URL="${AGENT_URL}?arch=${ATTUNE_AGENT_ARCH}"
mkdir -p "$AGENT_DIR"
# Build auth header if token is provided
AUTH_HEADER=""
if [ -n "$AGENT_TOKEN" ]; then
AUTH_HEADER="X-Agent-Token: $AGENT_TOKEN"
fi
# Download with retries (agent might start before API is ready)
MAX_RETRIES=10
RETRY_DELAY=5
ATTEMPT=0
while [ $ATTEMPT -lt $MAX_RETRIES ]; do
ATTEMPT=$((ATTEMPT + 1))
if command -v curl >/dev/null 2>&1; then
if [ -n "$AUTH_HEADER" ]; then
if curl -fsSL --retry 3 --retry-delay 2 -o "$AGENT_BIN" -H "$AUTH_HEADER" "$DOWNLOAD_URL" 2>/dev/null; then
break
fi
else
if curl -fsSL --retry 3 --retry-delay 2 -o "$AGENT_BIN" "$DOWNLOAD_URL" 2>/dev/null; then
break
fi
fi
elif command -v wget >/dev/null 2>&1; then
if [ -n "$AUTH_HEADER" ]; then
if wget -q -O "$AGENT_BIN" --header="$AUTH_HEADER" "$DOWNLOAD_URL" 2>/dev/null; then
break
fi
else
if wget -q -O "$AGENT_BIN" "$DOWNLOAD_URL" 2>/dev/null; then
break
fi
fi
else
echo "[attune] ERROR: Neither curl nor wget available. Cannot download agent." >&2
echo "[attune] Install curl or wget, or mount the agent binary via volume." >&2
exit 1
fi
if [ $ATTEMPT -lt $MAX_RETRIES ]; then
echo "[attune] Download attempt $ATTEMPT/$MAX_RETRIES failed, retrying in ${RETRY_DELAY}s..."
sleep $RETRY_DELAY
else
echo "[attune] ERROR: Failed to download agent binary after $MAX_RETRIES attempts." >&2
echo "[attune] Check that the API is running and ATTUNE_AGENT_URL is correct." >&2
exit 1
fi
done
chmod +x "$AGENT_BIN"
# Verify the binary works
if ! "$AGENT_BIN" --version >/dev/null 2>&1; then
echo "[attune] WARNING: Downloaded binary may not be compatible with this system." >&2
echo "[attune] Architecture: $ATTUNE_AGENT_ARCH ($(uname -m))" >&2
echo "[attune] File type: $(file "$AGENT_BIN" 2>/dev/null || echo 'unknown')" >&2
fi
echo "[attune] Agent binary downloaded successfully ($(wc -c < "$AGENT_BIN") bytes)."
echo "[attune] Starting agent..."
exec "$AGENT_BIN" "$@"

View File

@@ -302,6 +302,7 @@ class PackLoader:
name = runtime_data.get("name", ref.split(".")[-1]) name = runtime_data.get("name", ref.split(".")[-1])
description = runtime_data.get("description", "") description = runtime_data.get("description", "")
aliases = [alias.lower() for alias in runtime_data.get("aliases", [])]
distributions = json.dumps(runtime_data.get("distributions", {})) distributions = json.dumps(runtime_data.get("distributions", {}))
installation = json.dumps(runtime_data.get("installation", {})) installation = json.dumps(runtime_data.get("installation", {}))
execution_config = json.dumps(runtime_data.get("execution_config", {})) execution_config = json.dumps(runtime_data.get("execution_config", {}))
@@ -310,12 +311,13 @@ class PackLoader:
""" """
INSERT INTO runtime ( INSERT INTO runtime (
ref, pack, pack_ref, name, description, ref, pack, pack_ref, name, description,
distributions, installation, execution_config aliases, distributions, installation, execution_config
) )
VALUES (%s, %s, %s, %s, %s, %s, %s, %s) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)
ON CONFLICT (ref) DO UPDATE SET ON CONFLICT (ref) DO UPDATE SET
name = EXCLUDED.name, name = EXCLUDED.name,
description = EXCLUDED.description, description = EXCLUDED.description,
aliases = EXCLUDED.aliases,
distributions = EXCLUDED.distributions, distributions = EXCLUDED.distributions,
installation = EXCLUDED.installation, installation = EXCLUDED.installation,
execution_config = EXCLUDED.execution_config, execution_config = EXCLUDED.execution_config,
@@ -328,6 +330,7 @@ class PackLoader:
self.pack_ref, self.pack_ref,
name, name,
description, description,
aliases,
distributions, distributions,
installation, installation,
execution_config, execution_config,
@@ -338,6 +341,8 @@ class PackLoader:
runtime_ids[ref] = runtime_id runtime_ids[ref] = runtime_id
# Also index by lowercase name for easy lookup by runner_type # Also index by lowercase name for easy lookup by runner_type
runtime_ids[name.lower()] = runtime_id runtime_ids[name.lower()] = runtime_id
for alias in aliases:
runtime_ids[alias] = runtime_id
print(f" ✓ Runtime '{ref}' (ID: {runtime_id})") print(f" ✓ Runtime '{ref}' (ID: {runtime_id})")
cursor.close() cursor.close()

View File

@@ -0,0 +1,65 @@
# Universal Worker Agent Phase 2: Runtime Detection ↔ Worker Registration Integration
**Date**: 2026-02-05
## Summary
Integrated the Phase 1 runtime auto-detection module with the worker registration system so that `attune-agent` workers register with rich interpreter metadata (binary paths, versions) in their capabilities, enabling the system to distinguish agents from standard workers and know exactly which interpreters are available and where.
## Changes
### 1. `crates/worker/src/runtime_detect.rs`
- Added `Serialize` and `Deserialize` derives to `DetectedRuntime` so instances can be stored as structured JSON in worker capabilities.
### 2. `crates/worker/src/registration.rs`
- Added `use crate::runtime_detect::DetectedRuntime` import.
- Added `set_detected_runtimes(&mut self, runtimes: Vec<DetectedRuntime>)` method that stores detected interpreter metadata under the `detected_interpreters` capability key as a JSON array of `{name, path, version}` objects.
- Added `set_agent_mode(&mut self, is_agent: bool)` method that sets an `agent_mode` boolean capability to distinguish agent workers from standard workers.
- Both methods are additive — the existing `runtimes` string list capability remains for backward compatibility.
### 3. `crates/worker/src/service.rs`
- Added `detected_runtimes: Option<Vec<DetectedRuntime>>` field to `WorkerService` (initialized to `None` in `new()`).
- Added `pub fn with_detected_runtimes(mut self, runtimes: Vec<DetectedRuntime>) -> Self` builder method that stores agent detection results for use during `start()`.
- Updated `start()` to call `registration.set_detected_runtimes()` and `registration.set_agent_mode(true)` before `register()` when detected runtimes are present.
- Standard `attune-worker` binary is completely unaffected — the field stays `None` and no agent-specific code runs.
### 4. `crates/worker/src/agent_main.rs`
- Added `agent_detected_runtimes: Option<Vec<DetectedRuntime>>` variable to stash detection results.
- After auto-detection runs and sets `ATTUNE_WORKER_RUNTIMES`, the detected `Vec` is saved into `agent_detected_runtimes`.
- After `WorkerService::new()`, calls `.with_detected_runtimes(detected)` if auto-detection ran, so the registration includes full interpreter metadata.
### 5. `crates/worker/src/lib.rs`
- Added `pub use runtime_detect::DetectedRuntime` re-export for convenient access.
## Capability Format
After Phase 2, an agent worker's `capabilities` JSON in the `worker` table looks like:
```json
{
"runtimes": ["shell", "python", "node"],
"max_concurrent_executions": 10,
"worker_version": "0.1.0",
"agent_mode": true,
"detected_interpreters": [
{"name": "shell", "path": "/bin/bash", "version": "5.2.15"},
{"name": "python", "path": "/usr/bin/python3", "version": "3.12.1"},
{"name": "node", "path": "/usr/bin/node", "version": "20.11.0"}
]
}
```
Standard `attune-worker` instances do NOT have `agent_mode` or `detected_interpreters` keys.
## Design Decisions
- **Builder pattern** (`with_detected_runtimes`) rather than a separate constructor — keeps the API surface minimal and avoids duplicating `new()` logic.
- **Explicit `set_agent_mode`** separate from `set_detected_runtimes` — allows independent control, though in practice they're always called together for agents.
- **JSON serialization via `serde_json::json!()` macro** rather than `serde_json::to_value(&runtimes)` — gives explicit control over the capability shape and avoids coupling the DB format to the Rust struct layout.
- **No changes to the `Worker` model or database schema** — `detected_interpreters` and `agent_mode` are stored inside the existing `capabilities` JSONB column.
## Verification
- `cargo check --workspace` — zero errors, zero warnings
- `cargo test -p attune-worker` — 139 tests pass (105 unit + 17 dependency isolation + 8 log truncation + 7 security + 2 doc-tests)
- Standard `attune-worker` binary path is completely unchanged

View File

@@ -0,0 +1,69 @@
# Universal Worker Agent — Phase 2: Runtime Auto-Detection Integration
**Date**: 2026-02-05
## Overview
Implemented Phase 2 of the Universal Worker Agent plan (`docs/plans/universal-worker-agent.md`), which integrates the runtime auto-detection module (built in Phase 1) with the worker registration system. Agent workers now register with rich interpreter metadata — binary paths and versions — alongside the simple runtime name list used for backward compatibility.
## Changes
### 1. `crates/worker/src/runtime_detect.rs`
- Added `Serialize` and `Deserialize` derives to `DetectedRuntime` so detection results can be stored as JSON in worker capabilities.
### 2. `crates/worker/src/registration.rs`
- Added `use crate::runtime_detect::DetectedRuntime` import.
- **`set_detected_runtimes(runtimes: Vec<DetectedRuntime>)`** — Stores interpreter metadata under the `detected_interpreters` capability key as a structured JSON array (each entry has `name`, `path`, `version`). This supplements the existing `runtimes` string list for backward compatibility.
- **`set_agent_mode(is_agent: bool)`** — Sets an `agent_mode` boolean capability so the system can distinguish agent-mode workers from standard workers.
### 3. `crates/worker/src/service.rs`
- Added `detected_runtimes: Option<Vec<DetectedRuntime>>` field to `WorkerService` (defaults to `None`).
- **`with_detected_runtimes(self, runtimes) -> Self`** — Builder method to pass agent-detected runtimes into the service. No-op for standard `attune-worker`.
- Updated `start()` to call `set_detected_runtimes()` + `set_agent_mode(true)` on the registration before `register()` when detected runtimes are present.
### 4. `crates/worker/src/agent_main.rs`
- Added `agent_detected_runtimes: Option<Vec<DetectedRuntime>>` variable to stash detection results.
- After auto-detection runs, the detected runtimes are saved (previously they were consumed by the env var setup and discarded).
- After `WorkerService::new()`, chains `.with_detected_runtimes(detected)` to pass the metadata through.
### 5. `crates/worker/src/lib.rs`
- Re-exported `DetectedRuntime` from `runtime_detect` module for external use.
## Worker Capabilities (Agent Mode)
When an `attune-agent` registers, its capabilities JSON now includes:
```json
{
"runtimes": ["shell", "python", "node"],
"detected_interpreters": [
{"name": "shell", "path": "/bin/bash", "version": "5.2.15"},
{"name": "python", "path": "/usr/bin/python3", "version": "3.12.1"},
{"name": "node", "path": "/usr/bin/node", "version": "20.11.0"}
],
"agent_mode": true,
"max_concurrent_executions": 10,
"worker_version": "0.1.0"
}
```
Standard `attune-worker` registrations are unchanged — no `detected_interpreters` or `agent_mode` keys.
## Phase 2 Sub-tasks
| Sub-task | Status | Notes |
|----------|--------|-------|
| 2.1 Interpreter Discovery Module | ✅ Done (Phase 1) | `runtime_detect.rs` already existed |
| 2.2 Integration with Worker Registration | ✅ Done | Rich capabilities + agent_mode flag |
| 2.3 Runtime Hints File | ⏭️ Deferred | Optional enhancement, not needed yet |
## Verification
- `cargo check --workspace` — zero errors, zero warnings
- `cargo test -p attune-worker` — all tests pass (unit, integration, doc-tests)
- No breaking changes to the standard `attune-worker` binary

View File

@@ -0,0 +1,72 @@
# Universal Worker Agent — Phase 3: WorkerService Dual-Mode Refactor
**Date**: 2026-02-05
## Overview
Implemented Phase 3 of the Universal Worker Agent plan (`docs/plans/universal-worker-agent.md`): refactoring `WorkerService` for clean code reuse between the full `attune-worker` and the `attune-agent` binary, without code duplication.
## Changes
### 1. `StartupMode` Enum (`crates/worker/src/service.rs`)
Added a `StartupMode` enum that controls how the worker initializes its runtime environment:
- **`Worker`** — Full worker mode with proactive environment setup and full version verification sweep at startup. This is the existing behavior used by `attune-worker`.
- **`Agent { detected_runtimes }`** — Agent mode with lazy (on-demand) environment setup and deferred version verification. Used by `attune-agent`. Carries the auto-detected runtimes from Phase 2.
### 2. `WorkerService` Struct Refactoring (`crates/worker/src/service.rs`)
- Replaced the `detected_runtimes: Option<Vec<DetectedRuntime>>` field with `startup_mode: StartupMode`
- `new()` defaults to `StartupMode::Worker`
- `with_detected_runtimes()` now sets `StartupMode::Agent { detected_runtimes }` — the method signature is unchanged, so `agent_main.rs` requires no modifications
### 3. Conditional Startup in `start()` (`crates/worker/src/service.rs`)
The `start()` method now branches on `self.startup_mode`:
- **Worker mode**: Runs `verify_runtime_versions()` and `scan_and_setup_environments()` proactively (existing behavior, unchanged)
- **Agent mode**: Skips both with an info log — environments will be created lazily on first execution
Agent capability registration (`set_detected_runtimes()`, `set_agent_mode()`) also uses the `StartupMode` match instead of the old `Option` check.
### 4. Lazy Environment Setup (`crates/worker/src/runtime/process.rs`)
Updated `ProcessRuntime::execute()` to perform on-demand environment creation when the env directory is missing. Previously, a missing env dir produced a warning and fell back to the system interpreter. Now it:
1. Logs an info message about lazy setup
2. Creates a temporary `ProcessRuntime` with the effective config
3. Calls `setup_pack_environment()` to create the environment (venv, node_modules, etc.)
4. Falls back to the system interpreter only if creation fails
This is the primary code path for agent mode (where proactive startup setup is skipped) but also serves as a safety net for standard workers.
### 5. Re-export (`crates/worker/src/lib.rs`)
`StartupMode` is re-exported from the `attune_worker` crate root for external use.
## Files Modified
| File | Change |
|------|--------|
| `crates/worker/src/service.rs` | Added `StartupMode` enum, replaced `detected_runtimes` field, conditional startup logic |
| `crates/worker/src/runtime/process.rs` | Lazy on-demand environment creation in `execute()` |
| `crates/worker/src/lib.rs` | Re-export `StartupMode` |
| `AGENTS.md` | Updated development status (Phase 3 complete, Phases 47 in progress) |
## Test Results
All 139 tests pass:
- 105 unit tests
- 17 dependency isolation tests
- 8 log truncation tests
- 7 security tests
- 2 doc-tests
Zero compiler warnings across the workspace.
## Design Decisions
- **No code duplication**: The `StartupMode` enum parameterizes `WorkerService` rather than creating a separate `AgentService`. All execution machinery (runtimes, consumers, heartbeat, cancellation) is shared.
- **Lazy setup as safety net**: The on-demand environment creation in `ProcessRuntime::execute()` benefits both modes — agents rely on it as the primary path, while standard workers get it as a fallback if proactive setup missed something.
- **Backward compatible API**: `with_detected_runtimes()` keeps its signature, so `agent_main.rs` needed no changes.

View File

@@ -0,0 +1,77 @@
# Universal Worker Agent: Phase 4 — Docker Compose Integration
**Date**: 2026-02-05
**Phase**: 4 of 7
**Scope**: Docker Compose integration for agent-based workers
## Summary
Added Docker Compose infrastructure to make it trivial to add agent-based workers to an Attune deployment. Users can now inject the statically-linked `attune-agent` binary into any container image via a shared volume, turning it into a fully functional Attune worker with auto-detected runtimes — no Dockerfiles, no Rust compilation.
## Changes
### docker-compose.yaml
- Added `init-agent` service between `init-packs` and `rabbitmq`
- Builds from `docker/Dockerfile.agent` (target: `agent-init`)
- Copies the statically-linked binary to the `agent_bin` volume at `/opt/attune/agent/attune-agent`
- Runs once (`restart: "no"`) and completes immediately
- Added `agent_bin` named volume to the volumes section
### docker-compose.agent.yaml (new)
- Override file with example agent-based worker services
- **Active (uncommented)**: `worker-ruby` using `ruby:3.3-slim`
- **Commented templates**: Python 3.12, NVIDIA CUDA GPU, and custom image workers
- All workers follow the same pattern: mount `agent_bin` read-only, use `attune-agent` as entrypoint, share standard volumes
### Makefile
- Added `docker-up-agent` target: `docker compose -f docker-compose.yaml -f docker-compose.agent.yaml up -d`
- Added `docker-down-agent` target: corresponding `down` command
- Updated `.PHONY` and help text
### docs/QUICKREF-agent-workers.md (new)
- Quick-reference guide for adding agent-based workers
- Covers: how it works, quick start (override file vs docker-compose.override.yaml), required volumes, required environment variables, runtime auto-detection, testing detection, examples (Ruby, Node.js, GPU, multi-runtime), comparison table (traditional vs agent workers), troubleshooting
## Usage
```bash
# Start everything including the Ruby agent worker
make docker-up-agent
# Or manually
docker compose -f docker-compose.yaml -f docker-compose.agent.yaml up -d
# Stop
make docker-down-agent
```
Adding a new runtime worker is ~12 lines of YAML in `docker-compose.override.yaml`:
```yaml
services:
worker-my-runtime:
image: my-org/my-image:latest
depends_on:
init-agent:
condition: service_completed_successfully
# ... standard health checks
entrypoint: ["/opt/attune/agent/attune-agent"]
volumes:
- agent_bin:/opt/attune/agent:ro
- packs_data:/opt/attune/packs:ro
- runtime_envs:/opt/attune/runtime_envs
- artifacts_data:/opt/attune/artifacts
- ${ATTUNE_DOCKER_CONFIG_PATH:-./config.docker.yaml}:/opt/attune/config/config.yaml:ro
networks:
- attune-network
```
## Dependencies
- **Requires**: Phase 1 (agent binary build infrastructure) — `docker/Dockerfile.agent` must exist
- **Requires**: Phase 3 (WorkerService dual-mode refactor) — agent auto-detection and lazy env setup
## Next Steps
- **Phase 5**: API binary download endpoint (`GET /api/v1/agent/binary`)
- **Phase 6**: Database runtime registry extensions (runtime template packs)
- **Phase 7**: Kubernetes support (InitContainer pattern, Helm chart)

View File

@@ -0,0 +1,100 @@
# Universal Worker Agent — Phase 6: Database & Runtime Registry Extensions
**Date**: 2026-02
**Phase**: 6 of 7 (Universal Worker Agent)
**Plan**: `docs/plans/universal-worker-agent.md`
## Overview
Phase 6 extends the runtime registry so that the universal worker agent (`attune-agent`) can work with arbitrary runtimes — including languages like Ruby, Go, Java, Perl, and R — without requiring every possible runtime to be pre-registered in the database by an administrator.
## Changes Made
### 6.1 Extended Runtime Detection Metadata
**Migration** (`migrations/20250101000012_agent_runtime_detection.sql`):
- Added `auto_detected BOOLEAN NOT NULL DEFAULT FALSE` column to `runtime` table — distinguishes agent-created runtimes from pack-loaded ones
- Added `detection_config JSONB NOT NULL DEFAULT '{}'` column — stores detection metadata (detected binary path, version, runtime name)
- Added index `idx_runtime_auto_detected` for efficient filtering
**Rust Model** (`crates/common/src/models.rs`):
- Added `auto_detected: bool` and `detection_config: JsonDict` fields to the `Runtime` struct
**Repository** (`crates/common/src/repositories/runtime.rs`):
- Added `SELECT_COLUMNS` constant centralising the column list for all runtime queries
- Added `auto_detected` and `detection_config` to `CreateRuntimeInput` and `UpdateRuntimeInput`
- Updated ALL 7 SELECT queries, 2 RETURNING clauses, and the INSERT statement to include the new columns
- Updated the `update` method to support setting `auto_detected` and `detection_config`
**External query sites updated**:
- `crates/common/src/runtime_detection.rs``detect_from_database()`
- `crates/common/src/pack_environment.rs``get_runtime()`
- `crates/worker/src/executor.rs``prepare_execution_context()`
**All `CreateRuntimeInput` construction sites updated** (7 files):
- `crates/api/src/routes/runtimes.rs`
- `crates/common/src/pack_registry/loader.rs`
- `crates/common/tests/helpers.rs`
- `crates/common/tests/repository_runtime_tests.rs`
- `crates/common/tests/repository_worker_tests.rs`
- `crates/executor/tests/fifo_ordering_integration_test.rs`
- `crates/executor/tests/policy_enforcer_tests.rs`
### 6.2 Runtime Template Packs
Added 5 new runtime YAML definitions in `packs/core/runtimes/`:
| File | Ref | Interpreter | Environment | Dependencies |
|------|-----|-------------|-------------|--------------|
| `ruby.yaml` | `core.ruby` | `ruby` (.rb) | GEM_HOME isolation | Gemfile → bundle install |
| `go.yaml` | `core.go` | `go run` (.go) | GOPATH isolation | go.mod → go mod download |
| `java.yaml` | `core.java` | `java` (.java) | None (simple) | None |
| `perl.yaml` | `core.perl` | `perl` (.pl) | local::lib isolation | cpanfile → cpanm |
| `r.yaml` | `core.r` | `Rscript --vanilla` (.R) | renv isolation | renv.lock → renv::restore() |
Each includes verification commands matching the auto-detection module's probe strategy.
### 6.3 Dynamic Runtime Registration
**New module** (`crates/worker/src/dynamic_runtime.rs`):
- `auto_register_detected_runtimes(pool, detected)` — main entry point called from `agent_main.rs` BEFORE `WorkerService::new()`
- For each detected runtime:
1. Alias-aware lookup in existing DB runtimes (via `normalize_runtime_name`)
2. If not found, looks for a template runtime by ref pattern `core.<name>`
3. If template found, clones it with `auto_detected = true` and substitutes the detected binary path
4. If no template, creates a minimal runtime with just the interpreter binary and file extension
5. Auto-registered runtimes use ref format `auto.<name>` (e.g., `auto.ruby`)
- Helper functions: `build_detection_config()`, `build_execution_config_from_template()`, `build_minimal_execution_config()`, `build_minimal_distributions()`, `capitalize_runtime_name()`
- 8 unit tests covering all helpers
**Agent entrypoint** (`crates/worker/src/agent_main.rs`):
- Added Phase 2b between config loading and `WorkerService::new()`
- Creates a temporary DB connection and calls `auto_register_detected_runtimes()` for all detected runtimes
- Non-fatal: registration failures are logged as warnings, agent continues
**Runtime name normalization** (`crates/common/src/runtime_detection.rs`):
- Extended `normalize_runtime_name()` with 5 new alias groups:
- `ruby`/`rb``ruby`
- `go`/`golang``go`
- `java`/`jdk`/`openjdk``java`
- `perl`/`perl5``perl`
- `r`/`rscript``r`
- Added 5 new unit tests + 6 new assertions in existing filter tests
## Architecture Decisions
1. **Dynamic registration before WorkerService::new()**: The `WorkerService` constructor loads runtimes from the DB into an immutable `RuntimeRegistry` wrapped in `Arc`. Rather than restructuring this, dynamic registration runs beforehand so the normal loading pipeline picks up the new entries.
2. **Template-based cloning**: Auto-detected runtimes clone their execution config from pack templates (e.g., `core.ruby`) when available, inheriting environment management, dependency installation, and env_vars configuration. Only the interpreter binary path is substituted with the actual detected path.
3. **Minimal fallback**: When no template exists, a bare-minimum runtime entry is created with just the interpreter binary. This enables immediate script execution without environment/dependency management.
4. **`auto.` ref prefix**: Auto-detected runtimes use `auto.<name>` refs to avoid collisions with pack-registered templates (which use `core.<name>` or `<pack>.<name>`).
## Test Results
- **Worker crate**: 114 passed, 0 failed, 3 ignored
- **Common crate**: 321 passed, 0 failed
- **API crate**: 110 passed, 0 failed, 1 ignored
- **Executor crate**: 115 passed, 0 failed, 1 ignored
- **Workspace check**: Zero errors, zero warnings

View File

@@ -0,0 +1,62 @@
# Universal Worker Agent Phase 5: API Binary Download Endpoint
**Date**: 2026-03-21
**Phase**: Universal Worker Agent Phase 5
**Status**: Complete
## Overview
Implemented the API binary download endpoint for the Attune universal worker agent. This enables deployments where shared Docker volumes are impractical (Kubernetes, ECS, remote Docker hosts) by allowing containers to download the agent binary directly from the Attune API at startup.
## Changes
### New Files
- **`crates/api/src/routes/agent.rs`** — Two new unauthenticated API endpoints:
- `GET /api/v1/agent/binary` — Streams the statically-linked `attune-agent` binary as `application/octet-stream`. Supports `?arch=x86_64|aarch64|arm64` query parameter (defaults to `x86_64`). Tries arch-specific binary (`attune-agent-{arch}`) first, falls back to generic (`attune-agent`). Uses `ReaderStream` for memory-efficient streaming. Optional bootstrap token authentication via `X-Agent-Token` header or `token` query parameter.
- `GET /api/v1/agent/info` — Returns JSON metadata about available agent binaries (architectures, sizes, availability status, version).
- **`scripts/attune-agent-wrapper.sh`** — Bootstrap entrypoint script for containers without volume-mounted agent binary. Features:
- Auto-detects host architecture via `uname -m`
- Checks for volume-mounted binary first (zero-overhead fast path)
- Downloads from API with retry logic (10 attempts, 5s delay) using `curl` or `wget`
- Supports bootstrap token via `ATTUNE_AGENT_TOKEN` env var
- Verifies downloaded binary compatibility
- Configurable via `ATTUNE_AGENT_DIR`, `ATTUNE_AGENT_URL`, `ATTUNE_AGENT_ARCH` env vars
### Modified Files
- **`crates/common/src/config.rs`** — Added `AgentConfig` struct with `binary_dir` (path to agent binaries) and `bootstrap_token` (optional auth). Added `agent: Option<AgentConfig>` field to `Config`.
- **`crates/api/src/routes/mod.rs`** — Added `pub mod agent` and `pub use agent::routes as agent_routes`.
- **`crates/api/src/server.rs`** — Added `.merge(routes::agent_routes())` to the API v1 router.
- **`crates/api/src/openapi.rs`** — Registered both endpoints in OpenAPI paths, added `AgentBinaryInfo` and `AgentArchInfo` schemas, added `"agent"` tag. Updated endpoint count test assertions (+2 paths, +2 operations).
- **`config.docker.yaml`** — Added `agent.binary_dir: /opt/attune/agent` configuration.
- **`config.development.yaml`** — Added commented-out agent config pointing to local musl build output.
- **`docker-compose.yaml`** — API service now mounts `agent_bin` volume read-only at `/opt/attune/agent` and depends on `init-agent` service completing successfully.
- **`AGENTS.md`** — Updated development status (Phase 5 complete), updated agent_bin volume description, added agent config to Key Settings.
## Architecture Decisions
1. **Unauthenticated endpoint** — The agent needs to download its binary before it can authenticate with JWT. An optional lightweight bootstrap token (`agent.bootstrap_token`) provides security when needed.
2. **Streaming response** — Uses `tokio_util::io::ReaderStream` to stream the ~20MB binary without loading it entirely into memory.
3. **Architecture whitelist** — Only `x86_64`, `aarch64`, and `arm64` (alias) are accepted, preventing path traversal attacks.
4. **Graceful fallback** — Arch-specific binary (`attune-agent-x86_64`) → generic binary (`attune-agent`) → 404. This supports both multi-arch and single-arch deployments.
5. **Volume-first strategy** — The wrapper script checks for a volume-mounted binary before attempting download, so Docker Compose deployments with the `agent_bin` volume pay zero network overhead.
## Testing
- All 4 OpenAPI tests pass (including updated endpoint count: 59 paths, 83 operations)
- All 21 config tests pass (including `AgentConfig` integration)
- API crate compiles with zero warnings
- Common crate compiles with zero warnings

View File

@@ -0,0 +1,72 @@
# Universal Worker Agent — Phase 1: Static Binary Build Infrastructure
**Date**: 2026-03-21
## Summary
Implemented Phase 1 of the Universal Worker Agent plan (`docs/plans/universal-worker-agent.md`), establishing the build infrastructure for a statically-linked `attune-agent` binary that can be injected into any container to turn it into an Attune worker.
## Problem
Adding support for new runtime environments (Ruby, Go, Java, R, etc.) required building custom Docker images for each combination. This meant modifying `Dockerfile.worker.optimized`, installing interpreters via apt, managing a combinatorial explosion of worker variants, and rebuilding images (~5 min) for every change.
## Solution
Phase 1 lays the groundwork for flipping the model: instead of baking the worker into custom images, a single static binary is injected into **any** container at startup. This phase delivers:
1. **TLS backend audit** — confirmed the worker crate has zero `native-tls` or `openssl` dependencies, making musl static linking viable without any TLS backend changes
2. **New binary target**`attune-agent` alongside `attune-worker` in the same crate
3. **Runtime auto-detection module** — probes container environments for interpreters
4. **Dockerfile for static builds** — multi-stage musl cross-compilation
5. **Makefile targets** — local and Docker build commands
## Changes
### New Files
- **`crates/worker/src/agent_main.rs`** — Agent entrypoint with three-phase startup: (1) auto-detect runtimes or respect `ATTUNE_WORKER_RUNTIMES` override, (2) load config, (3) run `WorkerService`. Includes `--detect-only` flag for diagnostic probing.
- **`crates/worker/src/runtime_detect.rs`** — Database-free runtime detection module. Probes 8 interpreter families (shell, python, node, ruby, go, java, r, perl) via `which`-style PATH lookup with fallbacks. Captures version strings. 18 unit tests covering version parsing, display formatting, binary lookup, and detection pipeline.
- **`docker/Dockerfile.agent`** — Multi-stage Dockerfile:
- `builder` stage: cross-compiles with `x86_64-unknown-linux-musl` target, BuildKit cache mounts
- `agent-binary` stage: `FROM scratch` with just the static binary
- `agent-init` stage: busybox-based for Docker Compose/K8s init container volume population
### Modified Files
- **`crates/worker/Cargo.toml`** — Added second `[[bin]]` target for `attune-agent`
- **`crates/worker/src/lib.rs`** — Added `pub mod runtime_detect`
- **`Makefile`** — Added targets: `build-agent` (local musl build), `docker-build-agent`, `run-agent`, `run-agent-release`
- **`docker/Dockerfile.worker.optimized`** — Added `agent_main.rs` stub for second binary target
- **`docker/Dockerfile.optimized`** — Added `agent_main.rs` stub
- **`docker/Dockerfile.sensor.optimized`** — Added `agent_main.rs` stub
- **`docker/Dockerfile.pack-binaries`** — Added `agent_main.rs` stub
- **`AGENTS.md`** — Documented agent service, runtime auto-detection, Docker build, Makefile targets
## Key Design Decisions
1. **Same crate, new binary** — The agent lives as a second `[[bin]]` target in `crates/worker` rather than a separate crate. This gives zero code duplication and the same test suite covers both binaries. Can be split into a separate crate later if binary size becomes a concern.
2. **No TLS changes needed** — The plan anticipated needing to switch from `native-tls` to `rustls` workspace-wide. Audit revealed the worker crate already uses `rustls` exclusively (`native-tls` only enters via `tokio-tungstenite` in CLI and `ldap3` in API, neither of which the worker depends on).
3. **Database-free detection** — The `runtime_detect` module is deliberately separate from `attune_common::runtime_detection` (which queries the database). The agent must discover runtimes before any DB connectivity, using pure filesystem probing.
4. **All Dockerfiles updated** — Since the worker crate now has two binary targets, all Dockerfiles that create workspace stubs for `cargo fetch` need a stub for `agent_main.rs`. Missing this would break Docker builds.
## Verification
- `cargo check --all-targets --workspace` — zero warnings ✅
- `cargo test -p attune-worker` — all 37 tests pass (18 new runtime_detect tests + 19 existing) ✅
- `cargo run --bin attune-agent -- --detect-only` — successfully detected 6 runtimes on dev machine ✅
- `cargo run --bin attune-agent -- --help` — correct CLI documentation ✅
## Next Steps (Phases 27)
See `docs/plans/universal-worker-agent.md` for the remaining phases:
- **Phase 2**: Integration with worker registration (auto-detected runtimes → DB)
- **Phase 3**: Refactor `WorkerService` for dual modes (lazy env setup)
- **Phase 4**: Docker Compose init service for agent volume
- **Phase 5**: API binary download endpoint
- **Phase 6**: Database runtime registry extensions
- **Phase 7**: Kubernetes support (init containers, Helm chart)

View File

@@ -0,0 +1,46 @@
# Universal Worker Agent Phase 7: Kubernetes Support
**Date**: 2026-02-05
## Summary
Implemented Kubernetes support for agent-based workers in the Attune Helm chart, completing Phase 7 of the Universal Worker Agent plan. Users can now deploy the `attune-agent` binary into any container image on Kubernetes using the InitContainer pattern — the same approach used by Tekton and Argo.
## Changes
### Helm Chart (`charts/attune/`)
- **`templates/agent-workers.yaml`** (new): Helm template that iterates over `agentWorkers[]` values and creates a Deployment per entry. Each Deployment includes:
- `agent-loader` init container — copies the statically-linked `attune-agent` binary from the `attune-agent` image into an `emptyDir` volume
- `wait-for-schema` init container — polls PostgreSQL until the Attune schema is ready
- `wait-for-packs` init container — waits for the core pack on the shared PVC
- Worker container — runs the user's chosen image with the agent binary as entrypoint
- Volumes: `agent-bin` (emptyDir), `config` (ConfigMap), `packs` (PVC, read-only), `runtime-envs` (PVC), `artifacts` (PVC)
- **`values.yaml`**: Added `images.agent` (repository, tag, pullPolicy) and `agentWorkers: []` with full documentation of supported fields: `name`, `image`, `replicas`, `runtimes`, `resources`, `env`, `imagePullPolicy`, `logLevel`, `runtimeClassName`, `nodeSelector`, `tolerations`, `stopGracePeriod`
- **`templates/NOTES.txt`**: Updated to list enabled agent workers on install/upgrade
### CI/CD (`.gitea/workflows/publish.yml`)
- Added `attune-agent` to the image build matrix (target: `agent-init`, dockerfile: `docker/Dockerfile.agent`) so the agent image is published alongside all other Attune images
### Documentation
- **`docs/QUICKREF-kubernetes-agent-workers.md`** (new): Quick-reference guide covering how agent workers work on Kubernetes, all supported Helm values fields, runtime auto-detection table, differences from the standard worker, and troubleshooting steps
- **`docs/deployment/gitea-registry-and-helm.md`**: Added `attune-agent` to the published images list
- **`docs/plans/universal-worker-agent.md`**: Marked Phase 7 as complete with implementation details
### AGENTS.md
- Moved Phase 7 from "In Progress" to "Complete" with a summary of what was implemented
## Design Decisions
1. **emptyDir volume** (not PVC) for the agent binary — each pod gets its own copy via the init container. This avoids needing a shared RWX volume just for a single static binary and follows the standard Kubernetes sidecar injection pattern used by Tekton, Argo, and Istio.
2. **Pod-level scheduling fields**`runtimeClassName`, `nodeSelector`, and `tolerations` are exposed at the pod spec level (not container level) to support GPU scheduling via NVIDIA RuntimeClass and node affinity for specialized hardware.
3. **Runtime auto-detect by default** — when `runtimes` is empty (the default), the agent probes the container for interpreters. Users can override with an explicit list to skip detection and limit which runtimes are registered.
4. **Consistent patterns** — the template reuses the same `wait-for-schema` and `wait-for-packs` init containers, `envFrom` secret injection, and volume mount structure as the existing worker Deployment in `applications.yaml`.