attune-system/attune
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: attune-system:0782675a2b7816abf9e06e43d77caf19de0b74b1
Branches Tags
attune-system:main attune-system:perms-mgmt attune-system:autoload-workers
...
compare: attune-system:d4c6240485683ef5863788f3c2813a9447113540
Branches Tags
attune-system:main attune-system:perms-mgmt attune-system:autoload-workers

3 Commits
0782675a2b ... d4c6240485

Author SHA1 Message Date
David Culbreth
d4c6240485 agent workers 2026-03-21 10:05:02 -05:00
David Culbreth
4d5a3b1bf5 agent-style workers 2026-03-21 08:27:20 -05:00
David Culbreth
8ba7e3bb84 [wip] universal workers 2026-03-21 07:32:11 -05:00
62 changed files with 5603 additions and 230 deletions
Expand all files Collapse all files

6
.gitea/workflows/publish.yml
View File

@@ -163,6 +163,12 @@ jobs:
context: .
target: ""
build_args: ""
- name: agent
repository: attune-agent
dockerfile: docker/Dockerfile.agent
context: .
target: agent-init
build_args: ""
steps:
- name: Checkout
uses: actions/checkout@v4

28
AGENTS.md
View File

File diff suppressed because one or more lines are too long

1
Cargo.lock generated
View File

@@ -490,6 +490,7 @@ dependencies = [
"sha1",
"sha2",
"sqlx",
"subtle",
"tar",
"tempfile",
"thiserror 2.0.18",

2
Cargo.toml
View File

@@ -21,7 +21,7 @@ repository = "https://git.rdrx.app/attune-system/attune"
[workspace.dependencies]
# Async runtime
tokio = { version = "1.50", features = ["full"] }
tokio-util = "0.7"
tokio-util = { version = "0.7", features = ["io"] }
tokio-stream = { version = "0.1", features = ["sync"] }
# Web framework

41
Makefile
View File

@@ -4,7 +4,9 @@
docker-build-workers docker-build-worker-base docker-build-worker-python \
docker-build-worker-node docker-build-worker-full deny ci-rust ci-web-blocking ci-web-advisory \
ci-security-blocking ci-security-advisory ci-blocking ci-advisory \
fmt-check pre-commit install-git-hooks
fmt-check pre-commit install-git-hooks \
build-agent docker-build-agent run-agent run-agent-release \
docker-up-agent docker-down-agent
# Default target
help:
@@ -60,6 +62,14 @@ help:
@echo " make docker-up - Start services with docker compose"
@echo " make docker-down - Stop services"
@echo ""
@echo "Agent (Universal Worker):"
@echo " make build-agent - Build statically-linked agent binary (musl)"
@echo " make docker-build-agent - Build agent Docker image"
@echo " make run-agent - Run agent in development mode"
@echo " make run-agent-release - Run agent in release mode"
@echo " make docker-up-agent - Start all services + agent workers (ruby, etc.)"
@echo " make docker-down-agent - Stop agent stack"
@echo ""
@echo "Development:"
@echo " make watch - Watch and rebuild on changes"
@echo " make install-tools - Install development tools"
@@ -251,14 +261,43 @@ docker-build-worker-full:
DOCKER_BUILDKIT=1 docker build --target worker-full -t attune-worker:full -f docker/Dockerfile.worker.optimized .
@echo "✅ Full worker image built: attune-worker:full"
# Agent binary (statically-linked for injection into any container)
build-agent:
@echo "Installing musl target (if not already installed)..."
rustup target add x86_64-unknown-linux-musl 2>/dev/null || true
@echo "Building statically-linked agent binary..."
SQLX_OFFLINE=true cargo build --release --target x86_64-unknown-linux-musl --bin attune-agent
strip target/x86_64-unknown-linux-musl/release/attune-agent
@echo "✅ Agent binary built: target/x86_64-unknown-linux-musl/release/attune-agent"
@ls -lh target/x86_64-unknown-linux-musl/release/attune-agent
docker-build-agent:
@echo "Building agent Docker image (statically-linked binary)..."
DOCKER_BUILDKIT=1 docker buildx build --target agent-init -f docker/Dockerfile.agent -t attune-agent:latest .
@echo "✅ Agent image built: attune-agent:latest"
run-agent:
cargo run --bin attune-agent
run-agent-release:
cargo run --bin attune-agent --release
docker-up:
@echo "Starting all services with Docker Compose..."
docker compose up -d
docker-up-agent:
@echo "Starting all services + agent-based workers..."
docker compose -f docker-compose.yaml -f docker-compose.agent.yaml up -d
docker-down:
@echo "Stopping all services..."
docker compose down
docker-down-agent:
@echo "Stopping all services (including agent workers)..."
docker compose -f docker-compose.yaml -f docker-compose.agent.yaml down
docker-down-volumes:
@echo "Stopping all services and removing volumes (WARNING: deletes data)..."
docker compose down -v

20
charts/attune/templates/NOTES.txt
View File

@@ -1,3 +1,23 @@
1. Set `global.imageRegistry`, `global.imageNamespace`, and `global.imageTag` so the chart pulls the images published by the Gitea workflow.
2. Set `web.config.apiUrl` and `web.config.wsUrl` to browser-reachable endpoints before exposing the web UI.
3. The shared `packs`, `runtime_envs`, and `artifacts` PVCs default to `ReadWriteMany`; your cluster storage class must support RWX or you need to override those claims.
{{- if .Values.agentWorkers }}
Agent-based workers enabled:
{{- range .Values.agentWorkers }}
- {{ .name }}: image={{ .image }}, replicas={{ .replicas | default 1 }}
{{- if .runtimes }} runtimes={{ join "," .runtimes }}{{ else }} runtimes=auto-detect{{ end }}
{{- end }}
Each agent worker uses an init container to copy the statically-linked
attune-agent binary into the worker pod via an emptyDir volume. The agent
auto-detects available runtimes in the container and registers with Attune.
To add more agent workers, append entries to `agentWorkers` in your values:
agentWorkers:
- name: my-runtime
image: my-org/my-image:latest
replicas: 1
runtimes: [] # auto-detect
{{- end }}

137
charts/attune/templates/agent-workers.yaml Normal file
View File

@@ -0,0 +1,137 @@
{{- range .Values.agentWorkers }}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "attune.fullname" $ }}-agent-worker-{{ .name }}
labels:
{{- include "attune.labels" $ | nindent 4 }}
app.kubernetes.io/component: agent-worker-{{ .name }}
spec:
replicas: {{ .replicas | default 1 }}
selector:
matchLabels:
{{- include "attune.selectorLabels" $ | nindent 6 }}
app.kubernetes.io/component: agent-worker-{{ .name }}
template:
metadata:
labels:
{{- include "attune.selectorLabels" $ | nindent 8 }}
app.kubernetes.io/component: agent-worker-{{ .name }}
spec:
{{- if $.Values.global.imagePullSecrets }}
imagePullSecrets:
{{- toYaml $.Values.global.imagePullSecrets | nindent 8 }}
{{- end }}
{{- if .runtimeClassName }}
runtimeClassName: {{ .runtimeClassName }}
{{- end }}
{{- if .nodeSelector }}
nodeSelector:
{{- toYaml .nodeSelector | nindent 8 }}
{{- end }}
{{- if .tolerations }}
tolerations:
{{- toYaml .tolerations | nindent 8 }}
{{- end }}
{{- if .stopGracePeriod }}
terminationGracePeriodSeconds: {{ .stopGracePeriod }}
{{- else }}
terminationGracePeriodSeconds: 45
{{- end }}
initContainers:
- name: agent-loader
image: {{ include "attune.image" (dict "root" $ "image" $.Values.images.agent) }}
imagePullPolicy: {{ $.Values.images.agent.pullPolicy }}
command: ["cp", "/usr/local/bin/attune-agent", "/opt/attune/agent/attune-agent"]
volumeMounts:
- name: agent-bin
mountPath: /opt/attune/agent
- name: wait-for-schema
image: postgres:16-alpine
command: ["/bin/sh", "-ec"]
args:
- |
until PGPASSWORD="$DB_PASSWORD" psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" -tAc "SELECT to_regclass('${DB_SCHEMA}.identity')" | grep -q identity; do
echo "waiting for schema";
sleep 2;
done
envFrom:
- secretRef:
name: {{ include "attune.secretName" $ }}
- name: wait-for-packs
image: busybox:1.36
command: ["/bin/sh", "-ec"]
args:
- |
until [ -f /opt/attune/packs/core/pack.yaml ]; do
echo "waiting for packs";
sleep 2;
done
volumeMounts:
- name: packs
mountPath: /opt/attune/packs
containers:
- name: worker
image: {{ .image }}
{{- if .imagePullPolicy }}
imagePullPolicy: {{ .imagePullPolicy }}
{{- end }}
command: ["/opt/attune/agent/attune-agent"]
envFrom:
- secretRef:
name: {{ include "attune.secretName" $ }}
env:
- name: ATTUNE_CONFIG
value: /opt/attune/config.yaml
- name: ATTUNE__DATABASE__SCHEMA
value: {{ $.Values.database.schema | quote }}
- name: ATTUNE_WORKER_TYPE
value: container
- name: ATTUNE_WORKER_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: ATTUNE_API_URL
value: http://{{ include "attune.apiServiceName" $ }}:{{ $.Values.api.service.port }}
- name: RUST_LOG
value: {{ .logLevel | default "info" }}
{{- if .runtimes }}
- name: ATTUNE_WORKER_RUNTIMES
value: {{ join "," .runtimes | quote }}
{{- end }}
{{- if .env }}
{{- toYaml .env | nindent 12 }}
{{- end }}
resources:
{{- toYaml (.resources | default dict) | nindent 12 }}
volumeMounts:
- name: agent-bin
mountPath: /opt/attune/agent
readOnly: true
- name: config
mountPath: /opt/attune/config.yaml
subPath: config.yaml
- name: packs
mountPath: /opt/attune/packs
readOnly: true
- name: runtime-envs
mountPath: /opt/attune/runtime_envs
- name: artifacts
mountPath: /opt/attune/artifacts
volumes:
- name: agent-bin
emptyDir: {}
- name: config
configMap:
name: {{ include "attune.fullname" $ }}-config
- name: packs
persistentVolumeClaim:
claimName: {{ include "attune.fullname" $ }}-packs
- name: runtime-envs
persistentVolumeClaim:
claimName: {{ include "attune.fullname" $ }}-runtime-envs
- name: artifacts
persistentVolumeClaim:
claimName: {{ include "attune.fullname" $ }}-artifacts
{{- end }}

58
charts/attune/values.yaml
View File

@@ -131,6 +131,10 @@ images:
repository: attune-init-packs
tag: ""
pullPolicy: IfNotPresent
agent:
repository: attune-agent
tag: ""
pullPolicy: IfNotPresent
jobs:
migrations:
@@ -191,3 +195,57 @@ web:
- path: /
pathType: Prefix
tls: []
# Agent-based workers
# These deploy the universal worker agent into any container image.
# The agent auto-detects available runtimes (python, ruby, node, etc.)
# and registers with the Attune platform.
#
# Each entry creates a separate Deployment with an init container that
# copies the statically-linked agent binary into the worker container.
#
# Supported fields per worker:
# name (required) - Unique name for this worker (used in resource names)
# image (required) - Container image with your desired runtime(s)
# replicas (optional) - Number of pod replicas (default: 1)
# runtimes (optional) - List of runtimes to expose; [] = auto-detect
# resources (optional) - Kubernetes resource requests/limits
# env (optional) - Extra environment variables (list of {name, value})
# imagePullPolicy (optional) - Pull policy for the worker image
# logLevel (optional) - RUST_LOG level (default: "info")
# runtimeClassName (optional) - Kubernetes RuntimeClass (e.g., "nvidia" for GPU)
# nodeSelector (optional) - Node selector map for pod scheduling
# tolerations (optional) - Tolerations list for pod scheduling
# stopGracePeriod (optional) - Termination grace period in seconds (default: 45)
#
# Examples:
# agentWorkers:
# - name: ruby
# image: ruby:3.3
# replicas: 2
# runtimes: [] # auto-detect
# resources: {}
#
# - name: python-gpu
# image: nvidia/cuda:12.3.1-runtime-ubuntu22.04
# replicas: 1
# runtimes: [python, shell]
# runtimeClassName: nvidia
# nodeSelector:
# gpu: "true"
# tolerations:
# - key: nvidia.com/gpu
# operator: Exists
# effect: NoSchedule
# resources:
# limits:
# nvidia.com/gpu: 1
#
# - name: custom
# image: my-org/my-custom-image:latest
# replicas: 1
# runtimes: []
# env:
# - name: MY_CUSTOM_VAR
# value: my-value
agentWorkers: []

5
config.development.yaml
View File

@@ -125,3 +125,8 @@ executor:
scheduled_timeout: 120 # 2 minutes (faster feedback in dev)
timeout_check_interval: 30 # Check every 30 seconds
enable_timeout_monitor: true
# Agent binary distribution (optional - for local development)
# Binary is built via: make build-agent
# agent:
# binary_dir: ./target/x86_64-unknown-linux-musl/release

1
crates/api/Cargo.toml
View File

@@ -89,6 +89,7 @@ hmac = "0.12"
sha1 = "0.10"
sha2 = { workspace = true }
hex = "0.4"
subtle = "2.6"
# OpenAPI/Swagger
utoipa = { workspace = true, features = ["axum_extras"] }

17
crates/api/src/openapi.rs
View File

@@ -199,6 +199,10 @@ use crate::dto::{
crate::routes::webhooks::disable_webhook,
crate::routes::webhooks::regenerate_webhook_key,
crate::routes::webhooks::receive_webhook,
// Agent
crate::routes::agent::download_agent_binary,
crate::routes::agent::agent_info,
),
components(
schemas(
@@ -341,6 +345,10 @@ use crate::dto::{
WebhookReceiverRequest,
WebhookReceiverResponse,
ApiResponse<WebhookReceiverResponse>,
// Agent DTOs
crate::routes::agent::AgentBinaryInfo,
crate::routes::agent::AgentArchInfo,
)
),
modifiers(&SecurityAddon),
@@ -359,6 +367,7 @@ use crate::dto::{
(name = "secrets", description = "Secret management endpoints"),
(name = "workflows", description = "Workflow management endpoints"),
(name = "webhooks", description = "Webhook management and receiver endpoints"),
(name = "agent", description = "Agent binary distribution endpoints"),
)
)]
pub struct ApiDoc;
@@ -441,14 +450,14 @@ mod tests {
// We have 57 unique paths with 81 total operations (HTTP methods)
// This test ensures we don't accidentally remove endpoints
assert!(
path_count >= 57,
"Expected at least 57 unique API paths, found {}",
path_count >= 59,
"Expected at least 59 unique API paths, found {}",
path_count
);
assert!(
operation_count >= 81,
"Expected at least 81 API operations, found {}",
operation_count >= 83,
"Expected at least 83 API operations, found {}",
operation_count
);

482
crates/api/src/routes/agent.rs Normal file
View File

@@ -0,0 +1,482 @@
//! Agent binary download endpoints
//!
//! Provides endpoints for downloading the attune-agent binary for injection
//! into arbitrary containers. This supports deployments where shared Docker
//! volumes are impractical (Kubernetes, ECS, remote Docker hosts).
use axum::{
body::Body,
extract::{Query, State},
http::{header, HeaderMap, StatusCode},
response::IntoResponse,
routing::get,
Json, Router,
};
use serde::{Deserialize, Serialize};
use std::sync::Arc;
use subtle::ConstantTimeEq;
use tokio::fs;
use tokio_util::io::ReaderStream;
use utoipa::{IntoParams, ToSchema};
use crate::state::AppState;
/// Query parameters for the binary download endpoint
#[derive(Debug, Deserialize, IntoParams)]
pub struct BinaryDownloadParams {
/// Target architecture (x86_64, aarch64). Defaults to x86_64.
#[param(example = "x86_64")]
pub arch: Option<String>,
/// Optional bootstrap token for authentication
pub token: Option<String>,
}
/// Agent binary metadata
#[derive(Debug, Serialize, ToSchema)]
pub struct AgentBinaryInfo {
/// Available architectures
pub architectures: Vec<AgentArchInfo>,
/// Agent version (from build)
pub version: String,
}
/// Per-architecture binary info
#[derive(Debug, Serialize, ToSchema)]
pub struct AgentArchInfo {
/// Architecture name
pub arch: String,
/// Binary size in bytes
pub size_bytes: u64,
/// Whether this binary is available
pub available: bool,
}
/// Validate that the architecture name is safe (no path traversal) and normalize it.
fn validate_arch(arch: &str) -> Result<&str, (StatusCode, Json<serde_json::Value>)> {
match arch {
"x86_64" | "aarch64" => Ok(arch),
// Accept arm64 as an alias for aarch64
"arm64" => Ok("aarch64"),
_ => Err((
StatusCode::BAD_REQUEST,
Json(serde_json::json!({
"error": "Invalid architecture",
"message": format!("Unsupported architecture '{}'. Supported: x86_64, aarch64", arch),
})),
)),
}
}
/// Validate bootstrap token if configured.
///
/// If the agent config has a `bootstrap_token` set, the request must provide it
/// via the `X-Agent-Token` header or the `token` query parameter. If no token
/// is configured, access is unrestricted.
fn validate_token(
config: &attune_common::config::Config,
headers: &HeaderMap,
query_token: &Option<String>,
) -> Result<(), (StatusCode, Json<serde_json::Value>)> {
let expected_token = config
.agent
.as_ref()
.and_then(|ac| ac.bootstrap_token.as_ref());
let expected_token = match expected_token {
Some(t) => t,
None => {
use std::sync::Once;
static WARN_ONCE: Once = Once::new();
WARN_ONCE.call_once(|| {
tracing::warn!(
"Agent binary download endpoint has no bootstrap_token configured. \
Anyone with network access to the API can download the agent binary. \
Set agent.bootstrap_token in config to restrict access."
);
});
return Ok(());
}
};
// Check X-Agent-Token header first, then query param
let provided_token = headers
.get("x-agent-token")
.and_then(|v| v.to_str().ok())
.map(|s| s.to_string())
.or_else(|| query_token.clone());
match provided_token {
Some(ref t) if bool::from(t.as_bytes().ct_eq(expected_token.as_bytes())) => Ok(()),
Some(_) => Err((
StatusCode::UNAUTHORIZED,
Json(serde_json::json!({
"error": "Invalid token",
"message": "The provided bootstrap token is invalid",
})),
)),
None => Err((
StatusCode::UNAUTHORIZED,
Json(serde_json::json!({
"error": "Token required",
"message": "A bootstrap token is required. Provide via X-Agent-Token header or token query parameter.",
})),
)),
}
}
/// Download the agent binary
///
/// Returns the statically-linked attune-agent binary for the requested architecture.
/// The binary can be injected into any container to turn it into an Attune worker.
#[utoipa::path(
get,
path = "/api/v1/agent/binary",
params(BinaryDownloadParams),
responses(
(status = 200, description = "Agent binary", content_type = "application/octet-stream"),
(status = 400, description = "Invalid architecture"),
(status = 401, description = "Invalid or missing bootstrap token"),
(status = 404, description = "Agent binary not found"),
(status = 503, description = "Agent binary distribution not configured"),
),
tag = "agent"
)]
pub async fn download_agent_binary(
State(state): State<Arc<AppState>>,
headers: HeaderMap,
Query(params): Query<BinaryDownloadParams>,
) -> Result<impl IntoResponse, (StatusCode, Json<serde_json::Value>)> {
// Validate bootstrap token if configured
validate_token(&state.config, &headers, &params.token)?;
let agent_config = state.config.agent.as_ref().ok_or_else(|| {
(
StatusCode::SERVICE_UNAVAILABLE,
Json(serde_json::json!({
"error": "Not configured",
"message": "Agent binary distribution is not configured. Set agent.binary_dir in config.",
})),
)
})?;
let arch = params.arch.as_deref().unwrap_or("x86_64");
let arch = validate_arch(arch)?;
let binary_dir = std::path::Path::new(&agent_config.binary_dir);
// Try arch-specific binary first, then fall back to generic name.
// IMPORTANT: The generic `attune-agent` binary is only safe to serve for
// x86_64 requests, because the current build pipeline produces an
// x86_64-unknown-linux-musl binary. Serving it for aarch64/arm64 would
// give the caller an incompatible executable (exec format error).
let arch_specific = binary_dir.join(format!("attune-agent-{}", arch));
let generic = binary_dir.join("attune-agent");
let binary_path = if arch_specific.exists() {
arch_specific
} else if arch == "x86_64" && generic.exists() {
tracing::debug!(
"Arch-specific binary not found at {:?}, falling back to generic {:?} (safe for x86_64)",
arch_specific,
generic
);
generic
} else {
tracing::warn!(
"Agent binary not found. Checked: {:?} and {:?}",
arch_specific,
generic
);
return Err((
StatusCode::NOT_FOUND,
Json(serde_json::json!({
"error": "Not found",
"message": format!(
"Agent binary not found for architecture '{}'. Ensure the agent binary is built and placed in '{}'.",
arch,
agent_config.binary_dir
),
})),
));
};
// Get file metadata for Content-Length
let metadata = fs::metadata(&binary_path).await.map_err(|e| {
tracing::error!(
"Failed to read agent binary metadata at {:?}: {}",
binary_path,
e
);
(
StatusCode::INTERNAL_SERVER_ERROR,
Json(serde_json::json!({
"error": "Internal error",
"message": "Failed to read agent binary",
})),
)
})?;
// Open file for streaming
let file = fs::File::open(&binary_path).await.map_err(|e| {
tracing::error!("Failed to open agent binary at {:?}: {}", binary_path, e);
(
StatusCode::INTERNAL_SERVER_ERROR,
Json(serde_json::json!({
"error": "Internal error",
"message": "Failed to open agent binary",
})),
)
})?;
let stream = ReaderStream::new(file);
let body = Body::from_stream(stream);
let headers_response = [
(header::CONTENT_TYPE, "application/octet-stream".to_string()),
(
header::CONTENT_DISPOSITION,
"attachment; filename=\"attune-agent\"".to_string(),
),
(header::CONTENT_LENGTH, metadata.len().to_string()),
(header::CACHE_CONTROL, "public, max-age=3600".to_string()),
];
tracing::info!(
arch = arch,
size_bytes = metadata.len(),
path = ?binary_path,
"Serving agent binary download"
);
Ok((headers_response, body))
}
/// Get agent binary metadata
///
/// Returns information about available agent binaries, including
/// supported architectures and binary sizes.
#[utoipa::path(
get,
path = "/api/v1/agent/info",
responses(
(status = 200, description = "Agent binary info", body = AgentBinaryInfo),
(status = 503, description = "Agent binary distribution not configured"),
),
tag = "agent"
)]
pub async fn agent_info(
State(state): State<Arc<AppState>>,
) -> Result<impl IntoResponse, (StatusCode, Json<serde_json::Value>)> {
let agent_config = state.config.agent.as_ref().ok_or_else(|| {
(
StatusCode::SERVICE_UNAVAILABLE,
Json(serde_json::json!({
"error": "Not configured",
"message": "Agent binary distribution is not configured.",
})),
)
})?;
let binary_dir = std::path::Path::new(&agent_config.binary_dir);
let architectures = ["x86_64", "aarch64"];
let mut arch_infos = Vec::new();
for arch in &architectures {
let arch_specific = binary_dir.join(format!("attune-agent-{}", arch));
let generic = binary_dir.join("attune-agent");
// Only fall back to the generic binary for x86_64, since the build
// pipeline currently produces x86_64-only generic binaries.
let (available, size_bytes) = if arch_specific.exists() {
match fs::metadata(&arch_specific).await {
Ok(m) => (true, m.len()),
Err(_) => (false, 0),
}
} else if *arch == "x86_64" && generic.exists() {
match fs::metadata(&generic).await {
Ok(m) => (true, m.len()),
Err(_) => (false, 0),
}
} else {
(false, 0)
};
arch_infos.push(AgentArchInfo {
arch: arch.to_string(),
size_bytes,
available,
});
}
Ok(Json(AgentBinaryInfo {
architectures: arch_infos,
version: env!("CARGO_PKG_VERSION").to_string(),
}))
}
/// Create agent routes
pub fn routes() -> Router<Arc<AppState>> {
Router::new()
.route("/agent/binary", get(download_agent_binary))
.route("/agent/info", get(agent_info))
}
#[cfg(test)]
mod tests {
use super::*;
use attune_common::config::AgentConfig;
use axum::http::{HeaderMap, HeaderValue};
// ── validate_arch tests ─────────────────────────────────────────
#[test]
fn test_validate_arch_valid_x86_64() {
let result = validate_arch("x86_64");
assert!(result.is_ok());
assert_eq!(result.unwrap(), "x86_64");
}
#[test]
fn test_validate_arch_valid_aarch64() {
let result = validate_arch("aarch64");
assert!(result.is_ok());
assert_eq!(result.unwrap(), "aarch64");
}
#[test]
fn test_validate_arch_arm64_alias() {
// "arm64" is an alias for "aarch64"
let result = validate_arch("arm64");
assert!(result.is_ok());
assert_eq!(result.unwrap(), "aarch64");
}
#[test]
fn test_validate_arch_invalid() {
let result = validate_arch("mips");
assert!(result.is_err());
let (status, body) = result.unwrap_err();
assert_eq!(status, StatusCode::BAD_REQUEST);
assert_eq!(body.0["error"], "Invalid architecture");
}
// ── validate_token tests ────────────────────────────────────────
/// Helper: build a minimal Config with the given agent config.
/// Only the `agent` field is relevant for `validate_token`.
fn test_config(agent: Option<AgentConfig>) -> attune_common::config::Config {
let manifest_dir = std::env::var("CARGO_MANIFEST_DIR").unwrap_or_else(|_| ".".to_string());
let config_path = format!("{}/../../config.test.yaml", manifest_dir);
let mut config = attune_common::config::Config::load_from_file(&config_path)
.expect("Failed to load test config");
config.agent = agent;
config
}
#[test]
fn test_validate_token_no_config() {
// When no agent config is set at all, no token is required.
let config = test_config(None);
let headers = HeaderMap::new();
let query_token = None;
let result = validate_token(&config, &headers, &query_token);
assert!(result.is_ok());
}
#[test]
fn test_validate_token_no_bootstrap_token_configured() {
// Agent config exists but bootstrap_token is None → no token required.
let config = test_config(Some(AgentConfig {
binary_dir: "/tmp/test".to_string(),
bootstrap_token: None,
}));
let headers = HeaderMap::new();
let query_token = None;
let result = validate_token(&config, &headers, &query_token);
assert!(result.is_ok());
}
#[test]
fn test_validate_token_valid_from_header() {
let config = test_config(Some(AgentConfig {
binary_dir: "/tmp/test".to_string(),
bootstrap_token: Some("s3cret-bootstrap".to_string()),
}));
let mut headers = HeaderMap::new();
headers.insert(
"x-agent-token",
HeaderValue::from_static("s3cret-bootstrap"),
);
let query_token = None;
let result = validate_token(&config, &headers, &query_token);
assert!(result.is_ok());
}
#[test]
fn test_validate_token_valid_from_query() {
let config = test_config(Some(AgentConfig {
binary_dir: "/tmp/test".to_string(),
bootstrap_token: Some("s3cret-bootstrap".to_string()),
}));
let headers = HeaderMap::new();
let query_token = Some("s3cret-bootstrap".to_string());
let result = validate_token(&config, &headers, &query_token);
assert!(result.is_ok());
}
#[test]
fn test_validate_token_invalid() {
let config = test_config(Some(AgentConfig {
binary_dir: "/tmp/test".to_string(),
bootstrap_token: Some("correct-token".to_string()),
}));
let mut headers = HeaderMap::new();
headers.insert("x-agent-token", HeaderValue::from_static("wrong-token"));
let query_token = None;
let result = validate_token(&config, &headers, &query_token);
assert!(result.is_err());
let (status, body) = result.unwrap_err();
assert_eq!(status, StatusCode::UNAUTHORIZED);
assert_eq!(body.0["error"], "Invalid token");
}
#[test]
fn test_validate_token_missing_when_required() {
// bootstrap_token is configured but caller provides nothing.
let config = test_config(Some(AgentConfig {
binary_dir: "/tmp/test".to_string(),
bootstrap_token: Some("required-token".to_string()),
}));
let headers = HeaderMap::new();
let query_token = None;
let result = validate_token(&config, &headers, &query_token);
assert!(result.is_err());
let (status, body) = result.unwrap_err();
assert_eq!(status, StatusCode::UNAUTHORIZED);
assert_eq!(body.0["error"], "Token required");
}
#[test]
fn test_validate_token_header_takes_precedence_over_query() {
// When both header and query provide a token, the header value is
// checked first (it appears first in the or_else chain). Provide a
// valid token in the header and an invalid one in the query — should
// succeed because the header matches.
let config = test_config(Some(AgentConfig {
binary_dir: "/tmp/test".to_string(),
bootstrap_token: Some("the-real-token".to_string()),
}));
let mut headers = HeaderMap::new();
headers.insert("x-agent-token", HeaderValue::from_static("the-real-token"));
let query_token = Some("wrong-token".to_string());
let result = validate_token(&config, &headers, &query_token);
assert!(result.is_ok());
}
}

2
crates/api/src/routes/mod.rs
View File

@@ -1,6 +1,7 @@
//! API route modules
pub mod actions;
pub mod agent;
pub mod analytics;
pub mod artifacts;
pub mod auth;
@@ -19,6 +20,7 @@ pub mod webhooks;
pub mod workflows;
pub use actions::routes as action_routes;
pub use agent::routes as agent_routes;
pub use analytics::routes as analytics_routes;
pub use artifacts::routes as artifact_routes;
pub use auth::routes as auth_routes;

3
crates/api/src/routes/runtimes.rs
View File

@@ -179,6 +179,8 @@ pub async fn create_runtime(
distributions: request.distributions,
installation: request.installation,
execution_config: request.execution_config,
auto_detected: false,
detection_config: serde_json::json!({}),
},
)
.await?;
@@ -232,6 +234,7 @@ pub async fn update_runtime(
NullableJsonPatch::Clear => Patch::Clear,
}),
execution_config: request.execution_config,
..Default::default()
},
)
.await?;

1
crates/api/src/server.rs
View File

@@ -60,6 +60,7 @@ impl Server {
.merge(routes::history_routes())
.merge(routes::analytics_routes())
.merge(routes::artifact_routes())
.merge(routes::agent_routes())
.with_state(self.state.clone());
// Auth routes at root level (not versioned for frontend compatibility)

138
crates/api/tests/agent_tests.rs Normal file
View File

@@ -0,0 +1,138 @@
//! Integration tests for agent binary distribution endpoints
//!
//! The agent endpoints (`/api/v1/agent/binary` and `/api/v1/agent/info`) are
//! intentionally unauthenticated — the agent needs to download its binary
//! before it has JWT credentials. An optional `bootstrap_token` can restrict
//! access, but that is validated inside the handler, not via RequireAuth
//! middleware.
//!
//! The test configuration (`config.test.yaml`) does NOT include an `agent`
//! section, so both endpoints return 503 Service Unavailable. This is the
//! correct behaviour: the endpoints are reachable (no 401/404 from middleware)
//! but the feature is not configured.
use axum::http::StatusCode;
#[allow(dead_code)]
mod helpers;
use helpers::TestContext;
// ── /api/v1/agent/info ──────────────────────────────────────────────
#[tokio::test]
#[ignore = "integration test — requires database"]
async fn test_agent_info_not_configured() {
let ctx = TestContext::new()
.await
.expect("Failed to create test context");
let response = ctx
.get("/api/v1/agent/info", None)
.await
.expect("Failed to make request");
// Agent config is not set in config.test.yaml, so the handler returns 503.
assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE);
let body: serde_json::Value = response.json().await.expect("Failed to parse JSON");
assert_eq!(body["error"], "Not configured");
}
#[tokio::test]
#[ignore = "integration test — requires database"]
async fn test_agent_info_no_auth_required() {
// Verify that the endpoint is reachable WITHOUT any JWT token.
// If RequireAuth middleware were applied, this would return 401.
// Instead we expect 503 (not configured) — proving the endpoint
// is publicly accessible.
let ctx = TestContext::new()
.await
.expect("Failed to create test context");
let response = ctx
.get("/api/v1/agent/info", None)
.await
.expect("Failed to make request");
// Must NOT be 401 Unauthorized — the endpoint has no auth middleware.
assert_ne!(
response.status(),
StatusCode::UNAUTHORIZED,
"agent/info should not require authentication"
);
// Should be 503 because agent config is absent.
assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE);
}
// ── /api/v1/agent/binary ────────────────────────────────────────────
#[tokio::test]
#[ignore = "integration test — requires database"]
async fn test_agent_binary_not_configured() {
let ctx = TestContext::new()
.await
.expect("Failed to create test context");
let response = ctx
.get("/api/v1/agent/binary", None)
.await
.expect("Failed to make request");
// Agent config is not set in config.test.yaml, so the handler returns 503.
assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE);
let body: serde_json::Value = response.json().await.expect("Failed to parse JSON");
assert_eq!(body["error"], "Not configured");
}
#[tokio::test]
#[ignore = "integration test — requires database"]
async fn test_agent_binary_no_auth_required() {
// Same reasoning as test_agent_info_no_auth_required: the binary
// download endpoint must be publicly accessible (no RequireAuth).
// When no bootstrap_token is configured, any caller can reach the
// handler. We still get 503 because the agent feature itself is
// not configured in the test environment.
let ctx = TestContext::new()
.await
.expect("Failed to create test context");
let response = ctx
.get("/api/v1/agent/binary", None)
.await
.expect("Failed to make request");
// Must NOT be 401 Unauthorized — the endpoint has no auth middleware.
assert_ne!(
response.status(),
StatusCode::UNAUTHORIZED,
"agent/binary should not require authentication when no bootstrap_token is configured"
);
// Should be 503 because agent config is absent.
assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE);
}
#[tokio::test]
#[ignore = "integration test — requires database"]
async fn test_agent_binary_invalid_arch() {
// Architecture validation (`validate_arch`) rejects unsupported values
// with 400 Bad Request. However, in the handler the execution order is:
// 1. validate_token (passes — no bootstrap_token configured)
// 2. check agent config (fails with 503 — not configured)
// 3. validate_arch (never reached)
//
// So even with an invalid arch like "mips", we get 503 from the config
// check before the arch is ever validated. The arch validation is covered
// by unit tests in routes/agent.rs instead.
let ctx = TestContext::new()
.await
.expect("Failed to create test context");
let response = ctx
.get("/api/v1/agent/binary?arch=mips", None)
.await
.expect("Failed to make request");
// 503 from the agent-config-not-set check, NOT 400 from arch validation.
assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE);
}

14
crates/common/src/config.rs
View File

@@ -677,6 +677,15 @@ impl Default for PackRegistryConfig {
}
}
/// Agent binary distribution configuration
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AgentConfig {
/// Directory containing agent binary files
pub binary_dir: String,
/// Optional bootstrap token for authenticating agent binary downloads
pub bootstrap_token: Option<String>,
}
/// Executor service configuration
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ExecutorConfig {
@@ -770,6 +779,9 @@ pub struct Config {
/// Executor configuration (optional, for executor service)
pub executor: Option<ExecutorConfig>,
/// Agent configuration (optional, for agent binary distribution)
pub agent: Option<AgentConfig>,
}
fn default_service_name() -> String {
@@ -1066,6 +1078,7 @@ mod tests {
notifier: None,
pack_registry: PackRegistryConfig::default(),
executor: None,
agent: None,
};
assert_eq!(config.service_name, "attune");
@@ -1144,6 +1157,7 @@ mod tests {
notifier: None,
pack_registry: PackRegistryConfig::default(),
executor: None,
agent: None,
};
assert!(config.validate().is_ok());

2
crates/common/src/models.rs
View File

@@ -780,6 +780,8 @@ pub mod runtime {
pub installation: Option<JsonDict>,
pub installers: JsonDict,
pub execution_config: JsonDict,
pub auto_detected: bool,
pub detection_config: JsonDict,
pub created: DateTime<Utc>,
pub updated: DateTime<Utc>,
}

16
crates/common/src/pack_environment.rs
View File

@@ -10,7 +10,7 @@ use crate::config::Config;
use crate::error::{Error, Result};
use crate::models::Runtime;
use crate::repositories::action::ActionRepository;
use crate::repositories::runtime::RuntimeRepository;
use crate::repositories::runtime::{self, RuntimeRepository};
use crate::repositories::FindById as _;
use serde_json::Value as JsonValue;
use sqlx::{PgPool, Row};
@@ -370,15 +370,11 @@ impl PackEnvironmentManager {
// ========================================================================
async fn get_runtime(&self, runtime_id: i64) -> Result<Runtime> {
sqlx::query_as::<_, Runtime>(
r#"
SELECT id, ref, pack, pack_ref, description, name,
distributions, installation, installers, execution_config,
created, updated
FROM runtime
WHERE id = $1
"#,
)
let query = format!(
"SELECT {} FROM runtime WHERE id = $1",
runtime::SELECT_COLUMNS
);
sqlx::query_as::<_, Runtime>(&query)
.bind(runtime_id)
.fetch_one(&self.pool)
.await

3
crates/common/src/pack_registry/loader.rs
View File

@@ -418,6 +418,7 @@ impl<'a> PackComponentLoader<'a> {
None => Patch::Clear,
}),
execution_config: Some(execution_config),
..Default::default()
};
match RuntimeRepository::update(self.pool, existing.id, update_input).await {
@@ -448,6 +449,8 @@ impl<'a> PackComponentLoader<'a> {
distributions,
installation,
execution_config,
auto_detected: false,
detection_config: serde_json::json!({}),
};
match RuntimeRepository::create(self.pool, input).await {

118
crates/common/src/repositories/runtime.rs
View File

@@ -23,6 +23,13 @@ impl Repository for RuntimeRepository {
}
}
/// Columns selected for all Runtime queries. Centralised here so that
/// schema changes only need one update.
pub const SELECT_COLUMNS: &str = "id, ref, pack, pack_ref, description, name, \
distributions, installation, installers, execution_config, \
auto_detected, detection_config, \
created, updated";
/// Input for creating a new runtime
#[derive(Debug, Clone)]
pub struct CreateRuntimeInput {
@@ -34,6 +41,8 @@ pub struct CreateRuntimeInput {
pub distributions: JsonDict,
pub installation: Option<JsonDict>,
pub execution_config: JsonDict,
pub auto_detected: bool,
pub detection_config: JsonDict,
}
/// Input for updating a runtime
@@ -44,6 +53,8 @@ pub struct UpdateRuntimeInput {
pub distributions: Option<JsonDict>,
pub installation: Option<Patch<JsonDict>>,
pub execution_config: Option<JsonDict>,
pub auto_detected: Option<bool>,
pub detection_config: Option<JsonDict>,
}
#[async_trait::async_trait]
@@ -52,15 +63,8 @@ impl FindById for RuntimeRepository {
where
E: Executor<'e, Database = Postgres> + 'e,
{
let runtime = sqlx::query_as::<_, Runtime>(
r#"
SELECT id, ref, pack, pack_ref, description, name,
distributions, installation, installers, execution_config,
created, updated
FROM runtime
WHERE id = $1
"#,
)
let query = format!("SELECT {} FROM runtime WHERE id = $1", SELECT_COLUMNS);
let runtime = sqlx::query_as::<_, Runtime>(&query)
.bind(id)
.fetch_optional(executor)
.await?;
@@ -75,15 +79,8 @@ impl FindByRef for RuntimeRepository {
where
E: Executor<'e, Database = Postgres> + 'e,
{
let runtime = sqlx::query_as::<_, Runtime>(
r#"
SELECT id, ref, pack, pack_ref, description, name,
distributions, installation, installers, execution_config,
created, updated
FROM runtime
WHERE ref = $1
"#,
)
let query = format!("SELECT {} FROM runtime WHERE ref = $1", SELECT_COLUMNS);
let runtime = sqlx::query_as::<_, Runtime>(&query)
.bind(ref_str)
.fetch_optional(executor)
.await?;
@@ -98,15 +95,8 @@ impl List for RuntimeRepository {
where
E: Executor<'e, Database = Postgres> + 'e,
{
let runtimes = sqlx::query_as::<_, Runtime>(
r#"
SELECT id, ref, pack, pack_ref, description, name,
distributions, installation, installers, execution_config,
created, updated
FROM runtime
ORDER BY ref ASC
"#,
)
let query = format!("SELECT {} FROM runtime ORDER BY ref ASC", SELECT_COLUMNS);
let runtimes = sqlx::query_as::<_, Runtime>(&query)
.fetch_all(executor)
.await?;
@@ -122,16 +112,15 @@ impl Create for RuntimeRepository {
where
E: Executor<'e, Database = Postgres> + 'e,
{
let runtime = sqlx::query_as::<_, Runtime>(
r#"
INSERT INTO runtime (ref, pack, pack_ref, description, name,
distributions, installation, installers, execution_config)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)
RETURNING id, ref, pack, pack_ref, description, name,
distributions, installation, installers, execution_config,
created, updated
"#,
)
let query = format!(
"INSERT INTO runtime (ref, pack, pack_ref, description, name, \
distributions, installation, installers, execution_config, \
auto_detected, detection_config) \
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11) \
RETURNING {}",
SELECT_COLUMNS
);
let runtime = sqlx::query_as::<_, Runtime>(&query)
.bind(&input.r#ref)
.bind(input.pack)
.bind(&input.pack_ref)
@@ -141,6 +130,8 @@ impl Create for RuntimeRepository {
.bind(&input.installation)
.bind(serde_json::json!({}))
.bind(&input.execution_config)
.bind(input.auto_detected)
.bind(&input.detection_config)
.fetch_one(executor)
.await?;
@@ -209,6 +200,24 @@ impl Update for RuntimeRepository {
has_updates = true;
}
if let Some(auto_detected) = input.auto_detected {
if has_updates {
query.push(", ");
}
query.push("auto_detected = ");
query.push_bind(auto_detected);
has_updates = true;
}
if let Some(detection_config) = &input.detection_config {
if has_updates {
query.push(", ");
}
query.push("detection_config = ");
query.push_bind(detection_config);
has_updates = true;
}
if !has_updates {
// No updates requested, fetch and return existing entity
return Self::get_by_id(executor, id).await;
@@ -216,10 +225,7 @@ impl Update for RuntimeRepository {
query.push(", updated = NOW() WHERE id = ");
query.push_bind(id);
query.push(
" RETURNING id, ref, pack, pack_ref, description, name, \
distributions, installation, installers, execution_config, created, updated",
);
query.push(&format!(" RETURNING {}", SELECT_COLUMNS));
let runtime = query
.build_query_as::<Runtime>()
@@ -251,16 +257,11 @@ impl RuntimeRepository {
where
E: Executor<'e, Database = Postgres> + 'e,
{
let runtimes = sqlx::query_as::<_, Runtime>(
r#"
SELECT id, ref, pack, pack_ref, description, name,
distributions, installation, installers, execution_config,
created, updated
FROM runtime
WHERE pack = $1
ORDER BY ref ASC
"#,
)
let query = format!(
"SELECT {} FROM runtime WHERE pack = $1 ORDER BY ref ASC",
SELECT_COLUMNS
);
let runtimes = sqlx::query_as::<_, Runtime>(&query)
.bind(pack_id)
.fetch_all(executor)
.await?;
@@ -273,16 +274,11 @@ impl RuntimeRepository {
where
E: Executor<'e, Database = Postgres> + 'e,
{
let runtime = sqlx::query_as::<_, Runtime>(
r#"
SELECT id, ref, pack, pack_ref, description, name,
distributions, installation, installers, execution_config,
created, updated
FROM runtime
WHERE LOWER(name) = LOWER($1)
LIMIT 1
"#,
)
let query = format!(
"SELECT {} FROM runtime WHERE LOWER(name) = LOWER($1) LIMIT 1",
SELECT_COLUMNS
);
let runtime = sqlx::query_as::<_, Runtime>(&query)
.bind(name)
.fetch_optional(executor)
.await?;

96
crates/common/src/runtime_detection.rs
View File

@@ -12,6 +12,7 @@
use crate::config::Config;
use crate::error::Result;
use crate::models::Runtime;
use crate::repositories::runtime::SELECT_COLUMNS;
use serde_json::json;
use sqlx::PgPool;
use std::collections::HashMap;
@@ -33,16 +34,22 @@ use tracing::{debug, info, warn};
/// use attune_common::runtime_detection::normalize_runtime_name;
/// assert_eq!(normalize_runtime_name("node.js"), "node");
/// assert_eq!(normalize_runtime_name("nodejs"), "node");
/// assert_eq!(normalize_runtime_name("python3"), "python");
/// assert_eq!(normalize_runtime_name("shell"), "shell");
/// assert_eq!(normalize_runtime_name("Python3"), "python");
/// assert_eq!(normalize_runtime_name("Shell"), "shell");
/// ```
pub fn normalize_runtime_name(name: &str) -> &str {
match name {
"node" | "nodejs" | "node.js" => "node",
"python" | "python3" => "python",
"bash" | "sh" | "shell" => "shell",
"native" | "builtin" | "standalone" => "native",
other => other,
pub fn normalize_runtime_name(name: &str) -> String {
let lower = name.to_ascii_lowercase();
match lower.as_str() {
"node" | "nodejs" | "node.js" => "node".to_string(),
"python" | "python3" => "python".to_string(),
"bash" | "sh" | "shell" => "shell".to_string(),
"native" | "builtin" | "standalone" => "native".to_string(),
"ruby" | "rb" => "ruby".to_string(),
"go" | "golang" => "go".to_string(),
"java" | "jdk" | "openjdk" => "java".to_string(),
"perl" | "perl5" => "perl".to_string(),
"r" | "rscript" => "r".to_string(),
_ => lower,
}
}
@@ -51,9 +58,7 @@ pub fn normalize_runtime_name(name: &str) -> &str {
/// Both sides are lowercased and then normalized before comparison so that,
/// e.g., a filter value of `"node"` matches a database runtime name `"Node.js"`.
pub fn runtime_matches_filter(rt_name: &str, filter_entry: &str) -> bool {
let rt_lower = rt_name.to_ascii_lowercase();
let filter_lower = filter_entry.to_ascii_lowercase();
normalize_runtime_name(&rt_lower) == normalize_runtime_name(&filter_lower)
normalize_runtime_name(rt_name) == normalize_runtime_name(filter_entry)
}
/// Check if a runtime name matches any entry in a filter list.
@@ -156,15 +161,8 @@ impl RuntimeDetector {
info!("Querying database for runtime definitions...");
// Query all runtimes from database
let runtimes = sqlx::query_as::<_, Runtime>(
r#"
SELECT id, ref, pack, pack_ref, description, name,
distributions, installation, installers, execution_config,
created, updated
FROM runtime
ORDER BY ref
"#,
)
let query = format!("SELECT {} FROM runtime ORDER BY ref", SELECT_COLUMNS);
let runtimes = sqlx::query_as::<_, Runtime>(&query)
.fetch_all(&self.pool)
.await?;
@@ -363,11 +361,61 @@ mod tests {
assert_eq!(normalize_runtime_name("standalone"), "native");
}
#[test]
fn test_normalize_runtime_name_ruby_variants() {
assert_eq!(normalize_runtime_name("ruby"), "ruby");
assert_eq!(normalize_runtime_name("rb"), "ruby");
}
#[test]
fn test_normalize_runtime_name_go_variants() {
assert_eq!(normalize_runtime_name("go"), "go");
assert_eq!(normalize_runtime_name("golang"), "go");
}
#[test]
fn test_normalize_runtime_name_java_variants() {
assert_eq!(normalize_runtime_name("java"), "java");
assert_eq!(normalize_runtime_name("jdk"), "java");
assert_eq!(normalize_runtime_name("openjdk"), "java");
}
#[test]
fn test_normalize_runtime_name_perl_variants() {
assert_eq!(normalize_runtime_name("perl"), "perl");
assert_eq!(normalize_runtime_name("perl5"), "perl");
}
#[test]
fn test_normalize_runtime_name_r_variants() {
assert_eq!(normalize_runtime_name("r"), "r");
assert_eq!(normalize_runtime_name("rscript"), "r");
}
#[test]
fn test_normalize_runtime_name_passthrough() {
assert_eq!(normalize_runtime_name("custom_runtime"), "custom_runtime");
}
#[test]
fn test_normalize_runtime_name_case_insensitive() {
assert_eq!(normalize_runtime_name("Node"), "node");
assert_eq!(normalize_runtime_name("NodeJS"), "node");
assert_eq!(normalize_runtime_name("Node.js"), "node");
assert_eq!(normalize_runtime_name("Python"), "python");
assert_eq!(normalize_runtime_name("Python3"), "python");
assert_eq!(normalize_runtime_name("Shell"), "shell");
assert_eq!(normalize_runtime_name("BASH"), "shell");
assert_eq!(normalize_runtime_name("Ruby"), "ruby");
assert_eq!(normalize_runtime_name("Go"), "go");
assert_eq!(normalize_runtime_name("GoLang"), "go");
assert_eq!(normalize_runtime_name("Java"), "java");
assert_eq!(normalize_runtime_name("JDK"), "java");
assert_eq!(normalize_runtime_name("Perl"), "perl");
assert_eq!(normalize_runtime_name("R"), "r");
assert_eq!(normalize_runtime_name("Custom_Runtime"), "custom_runtime");
}
#[test]
fn test_runtime_matches_filter() {
// Node.js DB name lowercased vs worker filter "node"
@@ -390,6 +438,12 @@ mod tests {
assert!(runtime_matches_filter("python", "Python"));
assert!(runtime_matches_filter("Shell", "shell"));
assert!(runtime_matches_filter("NODEJS", "node"));
assert!(runtime_matches_filter("Ruby", "ruby"));
assert!(runtime_matches_filter("ruby", "rb"));
assert!(runtime_matches_filter("Go", "golang"));
assert!(runtime_matches_filter("R", "rscript"));
assert!(runtime_matches_filter("Java", "jdk"));
assert!(runtime_matches_filter("Perl", "perl5"));
assert!(!runtime_matches_filter("Python", "node"));
}

2
crates/common/tests/helpers.rs
View File

@@ -964,6 +964,8 @@ impl RuntimeFixture {
distributions: self.distributions,
installation: self.installation,
execution_config: self.execution_config,
auto_detected: false,
detection_config: serde_json::json!({}),
};
RuntimeRepository::create(pool, input).await

6
crates/common/tests/repository_runtime_tests.rs
View File

@@ -79,6 +79,8 @@ impl RuntimeFixture {
"file_extension": ".py"
}
}),
auto_detected: false,
detection_config: json!({}),
}
}
@@ -102,6 +104,8 @@ impl RuntimeFixture {
"file_extension": ".sh"
}
}),
auto_detected: false,
detection_config: json!({}),
}
}
}
@@ -268,6 +272,7 @@ async fn test_update_runtime() {
"method": "npm"
}))),
execution_config: None,
..Default::default()
};
let updated = RuntimeRepository::update(&pool, created.id, update_input.clone())
@@ -299,6 +304,7 @@ async fn test_update_runtime_partial() {
distributions: None,
installation: None,
execution_config: None,
..Default::default()
};
let updated = RuntimeRepository::update(&pool, created.id, update_input.clone())

2
crates/common/tests/repository_worker_tests.rs
View File

@@ -583,6 +583,8 @@ async fn test_worker_with_runtime() {
"file_extension": ".sh"
}
}),
auto_detected: false,
detection_config: json!({}),
};
let runtime = RuntimeRepository::create(&pool, runtime_input)

2
crates/executor/tests/fifo_ordering_integration_test.rs
View File

@@ -81,6 +81,8 @@ async fn _create_test_runtime(pool: &PgPool, suffix: &str) -> i64 {
"file_extension": ".py"
}
}),
auto_detected: false,
detection_config: json!({}),
};
RuntimeRepository::create(pool, runtime_input)

2
crates/executor/tests/policy_enforcer_tests.rs
View File

@@ -76,6 +76,8 @@ async fn create_test_runtime(pool: &PgPool, suffix: &str) -> i64 {
"file_extension": ".py"
}
}),
auto_detected: false,
detection_config: json!({}),
};
let runtime = RuntimeRepository::create(pool, runtime_input)

4
crates/worker/Cargo.toml
View File

@@ -10,6 +10,10 @@ repository.workspace = true
name = "attune-worker"
path = "src/main.rs"
[[bin]]
name = "attune-agent"
path = "src/agent_main.rs"
[dependencies]
attune-common = { path = "../common" }
tokio = { workspace = true }

318
crates/worker/src/agent_main.rs Normal file
View File

@@ -0,0 +1,318 @@
//! Attune Universal Worker Agent
//!
//! This is the entrypoint for the universal worker agent binary (`attune-agent`).
//! Unlike the standard `attune-worker` binary which requires explicit runtime
//! configuration, the agent automatically detects available interpreters in the
//! container environment and configures itself accordingly.
//!
//! ## Usage
//!
//! The agent is designed to be injected into any container image. On startup it:
//!
//! 1. Probes the system for available interpreters (python3, node, bash, etc.)
//! 2. Sets `ATTUNE_WORKER_RUNTIMES` based on what it finds
//! 3. Loads configuration (env vars are the primary config source)
//! 4. Initializes and runs the standard `WorkerService`
//!
//! ## Configuration
//!
//! Environment variables (primary):
//! - `ATTUNE__DATABASE__URL` — PostgreSQL connection string
//! - `ATTUNE__MESSAGE_QUEUE__URL` — RabbitMQ connection string
//! - `ATTUNE_WORKER_RUNTIMES` — Override auto-detection with explicit runtime list
//! - `ATTUNE_CONFIG` — Path to optional config YAML file
//!
//! CLI arguments:
//! - `--config` / `-c` — Path to configuration file (optional)
//! - `--name` / `-n` — Worker name override
//! - `--detect-only` — Run runtime detection, print results, and exit
use anyhow::Result;
use attune_common::config::Config;
use clap::Parser;
use tokio::signal::unix::{signal, SignalKind};
use tracing::{info, warn};
use attune_worker::dynamic_runtime::auto_register_detected_runtimes;
use attune_worker::runtime_detect::{detect_runtimes, print_detection_report};
use attune_worker::service::WorkerService;
#[derive(Parser, Debug)]
#[command(name = "attune-agent")]
#[command(
about = "Attune Universal Worker Agent - Injected into any container to auto-detect and execute actions",
long_about = "The Attune Agent automatically discovers available runtime interpreters \
in the current environment and registers as a worker capable of executing \
actions for those runtimes. It is designed to be injected into arbitrary \
container images without requiring manual runtime configuration."
)]
struct Args {
/// Path to configuration file (optional — env vars are the primary config source)
#[arg(short, long)]
config: Option<String>,
/// Worker name (overrides config and auto-generated name)
#[arg(short, long)]
name: Option<String>,
/// Run runtime detection, print results, and exit without starting the worker
#[arg(long)]
detect_only: bool,
}
fn main() -> Result<()> {
// Install HMAC-only JWT crypto provider (must be before any token operations)
attune_common::auth::install_crypto_provider();
// Initialize tracing
tracing_subscriber::fmt()
.with_target(false)
.with_thread_ids(true)
.init();
let args = Args::parse();
info!("Starting Attune Universal Worker Agent");
// --- Phase 1: Runtime auto-detection (synchronous, before tokio runtime) ---
//
// All std::env::set_var calls MUST happen here, before we create the tokio
// runtime, to avoid undefined behavior from mutating the process environment
// while other threads are running.
//
// Check if the user has explicitly set ATTUNE_WORKER_RUNTIMES. If so, skip
// auto-detection and respect their override. Otherwise, probe the system for
// available interpreters.
let runtimes_override = std::env::var("ATTUNE_WORKER_RUNTIMES").ok();
// Holds the detected runtimes so we can pass them to WorkerService later.
// Populated in both branches: auto-detection and override (filtered to
// match the override list).
let mut agent_detected_runtimes: Option<Vec<attune_worker::runtime_detect::DetectedRuntime>> =
None;
if let Some(ref override_value) = runtimes_override {
info!(
"ATTUNE_WORKER_RUNTIMES already set (override): {}",
override_value
);
// Even with an explicit override, run detection so we can register
// the overridden runtimes in the database and advertise accurate
// capability metadata (binary paths, versions). Without this, the
// worker would accept work for runtimes that were never registered
// locally — e.g. ruby/go on a fresh deployment.
info!("Running auto-detection for override-specified runtimes...");
let detected = detect_runtimes();
// Filter detected runtimes to only those matching the override list,
// so we don't register runtimes the user explicitly excluded.
let override_names: Vec<&str> = override_value.split(',').map(|s| s.trim()).collect();
let filtered: Vec<_> = detected
.into_iter()
.filter(|rt| {
let normalized = attune_common::runtime_detection::normalize_runtime_name(&rt.name);
override_names.iter().any(|ov| {
attune_common::runtime_detection::normalize_runtime_name(ov) == normalized
})
})
.collect();
if filtered.is_empty() {
warn!(
"None of the override runtimes ({}) were found on this system! \
The agent may not be able to execute any actions.",
override_value
);
} else {
info!(
"Matched {} override runtime(s) to detected interpreters:",
filtered.len()
);
for rt in &filtered {
match &rt.version {
Some(ver) => info!(" ✓ {} — {} ({})", rt.name, rt.path, ver),
None => info!(" ✓ {} — {}", rt.name, rt.path),
}
}
agent_detected_runtimes = Some(filtered);
}
} else {
info!("No ATTUNE_WORKER_RUNTIMES override — running auto-detection...");
let detected = detect_runtimes();
if detected.is_empty() {
warn!("No runtimes detected! The agent may not be able to execute any actions.");
} else {
info!("Detected {} runtime(s):", detected.len());
for rt in &detected {
match &rt.version {
Some(ver) => info!(" ✓ {} — {} ({})", rt.name, rt.path, ver),
None => info!(" ✓ {} — {}", rt.name, rt.path),
}
}
// Build comma-separated runtime list and set the env var so that
// Config::load() and WorkerService pick it up downstream.
let runtime_list: Vec<&str> = detected.iter().map(|r| r.name.as_str()).collect();
let runtime_csv = runtime_list.join(",");
info!("Setting ATTUNE_WORKER_RUNTIMES={}", runtime_csv);
// Safe: no other threads are running yet (tokio runtime not started).
std::env::set_var("ATTUNE_WORKER_RUNTIMES", &runtime_csv);
// Stash for Phase 2: pass to WorkerService for rich capability registration
agent_detected_runtimes = Some(detected);
}
}
// --- Handle --detect-only (synchronous, no async runtime needed) ---
if args.detect_only {
if runtimes_override.is_some() {
// User set an override, but --detect-only should show what's actually
// on this system regardless, so re-run detection.
info!(
"--detect-only: re-running detection to show what is available on this system..."
);
println!("NOTE: ATTUNE_WORKER_RUNTIMES is set — auto-detection was skipped during normal startup.");
println!(" Showing what auto-detection would find on this system:");
println!();
let detected = detect_runtimes();
print_detection_report(&detected);
} else if let Some(ref detected) = agent_detected_runtimes {
print_detection_report(detected);
} else {
// No detection ran (empty results), run it fresh
let detected = detect_runtimes();
print_detection_report(&detected);
}
return Ok(());
}
// --- Set config path env var (synchronous, before tokio runtime) ---
if let Some(ref config_path) = args.config {
// Safe: no other threads are running yet (tokio runtime not started).
std::env::set_var("ATTUNE_CONFIG", config_path);
}
// --- Build the tokio runtime and run the async portion ---
let runtime = tokio::runtime::Runtime::new()?;
runtime.block_on(async_main(args, agent_detected_runtimes))
}
/// The async portion of the agent entrypoint. Called from `main()` via
/// `runtime.block_on()` after all environment variable mutations are complete.
async fn async_main(
args: Args,
agent_detected_runtimes: Option<Vec<attune_worker::runtime_detect::DetectedRuntime>>,
) -> Result<()> {
// --- Phase 2: Load configuration ---
let mut config = Config::load()?;
config.validate()?;
// Override worker name if provided via CLI
if let Some(name) = args.name {
if let Some(ref mut worker_config) = config.worker {
worker_config.name = Some(name);
} else {
config.worker = Some(attune_common::config::WorkerConfig {
name: Some(name),
worker_type: None,
runtime_id: None,
host: None,
port: None,
capabilities: None,
max_concurrent_tasks: 10,
heartbeat_interval: 30,
task_timeout: 300,
max_stdout_bytes: 10 * 1024 * 1024,
max_stderr_bytes: 10 * 1024 * 1024,
shutdown_timeout: Some(30),
stream_logs: true,
});
}
}
info!("Configuration loaded successfully");
info!("Environment: {}", config.environment);
// --- Phase 2b: Dynamic runtime registration ---
//
// Before creating the WorkerService (which loads runtimes from the DB into
// its runtime registry), ensure that every detected runtime has a
// corresponding entry in the database. This handles the case where the
// agent detects a runtime (e.g., Ruby) that has a template in the core
// pack but hasn't been explicitly loaded by this agent before.
if let Some(ref detected) = agent_detected_runtimes {
info!(
"Ensuring {} detected runtime(s) are registered in the database...",
detected.len()
);
// We need a temporary DB connection for dynamic registration.
// WorkerService::new() will create its own connection, so this is
// a short-lived pool just for the registration step.
let db = attune_common::db::Database::new(&config.database).await?;
let pool = db.pool().clone();
match auto_register_detected_runtimes(&pool, detected).await {
Ok(count) => {
if count > 0 {
info!(
"Dynamic registration complete: {} new runtime(s) added to database",
count
);
} else {
info!("Dynamic registration: all detected runtimes already in database");
}
}
Err(e) => {
warn!(
"Dynamic runtime registration failed (non-fatal, continuing): {}",
e
);
}
}
}
// --- Phase 3: Initialize and run the worker service ---
let service = WorkerService::new(config).await?;
// If we auto-detected runtimes, pass them to the worker service so that
// registration includes the full `detected_interpreters` capability
// (binary paths + versions) and the `agent_mode` flag.
let mut service = if let Some(detected) = agent_detected_runtimes {
info!(
"Passing {} detected runtime(s) to worker registration",
detected.len()
);
service.with_detected_runtimes(detected)
} else {
service
};
info!("Attune Agent is ready");
service.start().await?;
// Setup signal handlers for graceful shutdown
let mut sigint = signal(SignalKind::interrupt())?;
let mut sigterm = signal(SignalKind::terminate())?;
tokio::select! {
_ = sigint.recv() => {
info!("Received SIGINT signal");
}
_ = sigterm.recv() => {
info!("Received SIGTERM signal");
}
}
info!("Shutting down gracefully...");
service.stop().await?;
info!("Attune Agent shutdown complete");
Ok(())
}

518
crates/worker/src/dynamic_runtime.rs Normal file
View File

@@ -0,0 +1,518 @@
//! Dynamic Runtime Registration Module
//!
//! When the agent detects an interpreter on the local system (e.g., Ruby, Go, Perl)
//! that does not yet have a corresponding runtime entry in the database, this module
//! handles auto-registering it so that the normal runtime-loading pipeline in
//! `WorkerService::new()` picks it up.
//!
//! ## Registration Strategy
//!
//! For each detected runtime the agent found:
//!
//! 1. **Look up by name** in the database using alias-aware matching
//! (via [`normalize_runtime_name`]).
//! 2. **If found** → already registered (either from a pack YAML or a previous
//! agent run). Nothing to do.
//! 3. **If not found** → search for a runtime *template* in loaded packs whose
//! normalized name matches. Templates are pack-registered runtimes (e.g.,
//! `core.ruby`) that provide the full `execution_config` needed to invoke
//! the interpreter, manage environments, and install dependencies.
//! 4. **If a template is found** → clone it as an auto-detected runtime with
//! `auto_detected = true` and populate `detection_config` with what the
//! agent discovered (binary path, version, etc.).
//! 5. **If no template exists** → create a minimal runtime with just the
//! detected interpreter binary path and file extension. This lets the agent
//! execute simple scripts immediately, even without a full template.
//! 6. Mark all auto-registered runtimes with `auto_detected = true`.
use attune_common::error::Result;
use attune_common::models::runtime::Runtime;
use attune_common::repositories::runtime::{CreateRuntimeInput, RuntimeRepository};
use attune_common::repositories::{Create, FindByRef, List};
use attune_common::runtime_detection::normalize_runtime_name;
use serde_json::json;
use sqlx::PgPool;
use tracing::{debug, info, warn};
use crate::runtime_detect::DetectedRuntime;
/// Canonical file extensions for runtimes that the auto-detection module knows
/// about. Used when creating minimal runtime entries without a template.
fn default_file_extension(runtime_name: &str) -> Option<&'static str> {
match runtime_name {
"shell" => Some(".sh"),
"python" => Some(".py"),
"node" => Some(".js"),
"ruby" => Some(".rb"),
"go" => Some(".go"),
"java" => Some(".java"),
"perl" => Some(".pl"),
"r" => Some(".R"),
_ => None,
}
}
/// Ensure that every detected runtime has a corresponding entry in the
/// `runtime` table. Runtimes that already exist (from pack loading or a
/// previous agent run) are left untouched. Missing runtimes are created
/// either from a matching pack template or as a minimal auto-detected entry.
///
/// This function should be called **before** `WorkerService::new()` so that
/// the normal runtime-loading pipeline finds all detected runtimes in the DB.
///
/// Returns the number of runtimes that were newly registered.
pub async fn auto_register_detected_runtimes(
pool: &PgPool,
detected: &[DetectedRuntime],
) -> Result<usize> {
if detected.is_empty() {
return Ok(0);
}
info!(
"Checking {} detected runtime(s) for dynamic registration...",
detected.len()
);
// Load all existing runtimes once to avoid repeated queries.
let existing_runtimes = RuntimeRepository::list(pool).await.unwrap_or_default();
let mut registered_count = 0;
for detected_rt in detected {
let canonical_name = normalize_runtime_name(&detected_rt.name);
// Check if a runtime with a matching name already exists in the DB.
// We normalize both sides for alias-aware comparison.
// normalize_runtime_name lowercases internally, so no need to pre-lowercase.
let already_exists = existing_runtimes
.iter()
.any(|r| normalize_runtime_name(&r.name) == canonical_name);
if already_exists {
debug!(
"Runtime '{}' (canonical: '{}') already exists in database, skipping",
detected_rt.name, canonical_name
);
continue;
}
// No existing runtime — try to find a template from loaded packs.
// Templates are pack-registered runtimes whose normalized name matches
// (e.g., `core.ruby` for detected runtime "ruby"). Since we already
// checked `existing_runtimes` above and found nothing, we look for
// runtimes by ref pattern. Common convention: `core.<name>`.
let template_ref = format!("core.{}", canonical_name);
let template = RuntimeRepository::find_by_ref(pool, &template_ref)
.await
.unwrap_or(None);
let detection_config = build_detection_config(detected_rt);
if let Some(tmpl) = template {
// Clone the template as an auto-detected runtime.
// The template already has the full execution_config, distributions, etc.
// We just re-create it with auto_detected = true.
info!(
"Found template '{}' for detected runtime '{}', registering auto-detected clone",
tmpl.r#ref, detected_rt.name
);
// Use a distinct ref so we don't collide with the template.
let auto_ref = format!("auto.{}", canonical_name);
// Check if the auto ref already exists (race condition / previous run)
if RuntimeRepository::find_by_ref(pool, &auto_ref)
.await
.unwrap_or(None)
.is_some()
{
debug!(
"Auto-detected runtime '{}' already registered from a previous run",
auto_ref
);
continue;
}
let input = CreateRuntimeInput {
r#ref: auto_ref.clone(),
pack: tmpl.pack,
pack_ref: tmpl.pack_ref.clone(),
description: Some(format!(
"Auto-detected {} runtime (from template {})",
detected_rt.name, tmpl.r#ref
)),
name: tmpl.name.clone(),
distributions: tmpl.distributions.clone(),
installation: tmpl.installation.clone(),
execution_config: build_execution_config_from_template(&tmpl, detected_rt),
auto_detected: true,
detection_config,
};
match RuntimeRepository::create(pool, input).await {
Ok(rt) => {
info!(
"Auto-registered runtime '{}' (ID: {}) from template '{}'",
auto_ref, rt.id, tmpl.r#ref
);
registered_count += 1;
}
Err(e) => {
// Unique constraint violation is fine (concurrent agent start)
warn!("Failed to auto-register runtime '{}': {}", auto_ref, e);
}
}
} else {
// No template found — create a minimal runtime entry.
info!(
"No template found for detected runtime '{}', creating minimal entry",
detected_rt.name
);
let auto_ref = format!("auto.{}", canonical_name);
if RuntimeRepository::find_by_ref(pool, &auto_ref)
.await
.unwrap_or(None)
.is_some()
{
debug!(
"Auto-detected runtime '{}' already registered from a previous run",
auto_ref
);
continue;
}
let execution_config = build_minimal_execution_config(detected_rt);
let input = CreateRuntimeInput {
r#ref: auto_ref.clone(),
pack: None,
pack_ref: None,
description: Some(format!(
"Auto-detected {} runtime at {}",
detected_rt.name, detected_rt.path
)),
name: capitalize_runtime_name(&canonical_name),
distributions: build_minimal_distributions(detected_rt),
installation: None,
execution_config,
auto_detected: true,
detection_config,
};
match RuntimeRepository::create(pool, input).await {
Ok(rt) => {
info!(
"Auto-registered minimal runtime '{}' (ID: {})",
auto_ref, rt.id
);
registered_count += 1;
}
Err(e) => {
warn!("Failed to auto-register runtime '{}': {}", auto_ref, e);
}
}
}
}
if registered_count > 0 {
info!(
"Dynamic runtime registration complete: {} new runtime(s) registered",
registered_count
);
} else {
info!("Dynamic runtime registration complete: all detected runtimes already in database");
}
Ok(registered_count)
}
/// Build the `detection_config` JSONB value from a detected runtime.
/// This metadata records how the agent discovered this runtime, enabling
/// re-verification and diagnostics.
fn build_detection_config(detected: &DetectedRuntime) -> serde_json::Value {
let mut config = json!({
"detected_path": detected.path,
"detected_name": detected.name,
});
if let Some(ref version) = detected.version {
config["detected_version"] = json!(version);
}
config
}
/// Build an execution config based on a template runtime, but with the
/// detected interpreter path substituted in. This ensures the auto-detected
/// runtime uses the actual binary path found on the system.
fn build_execution_config_from_template(
template: &Runtime,
detected: &DetectedRuntime,
) -> serde_json::Value {
let mut config = template.execution_config.clone();
// If the template has an interpreter config, update the binary path
// to the one we actually detected on this system.
if let Some(interpreter) = config.get_mut("interpreter") {
if let Some(obj) = interpreter.as_object_mut() {
obj.insert("binary".to_string(), json!(detected.path));
}
}
// If the template has an environment config with an interpreter_path
// that uses a template variable, leave it as-is (it will be resolved
// at execution time). But if it's a hardcoded absolute path, update it.
if let Some(env) = config.get_mut("environment") {
if let Some(obj) = env.as_object_mut() {
if let Some(interp_path) = obj.get("interpreter_path") {
if let Some(path_str) = interp_path.as_str() {
// Only leave template variables alone
if !path_str.contains('{') {
obj.insert("interpreter_path".to_string(), json!(detected.path));
}
}
}
}
}
config
}
/// Build a minimal execution config for a runtime with no template.
/// This provides enough information for `ProcessRuntime` to invoke the
/// interpreter directly, without environment or dependency management.
fn build_minimal_execution_config(detected: &DetectedRuntime) -> serde_json::Value {
let canonical = normalize_runtime_name(&detected.name);
let file_ext = default_file_extension(&canonical);
let mut config = json!({
"interpreter": {
"binary": detected.path,
"args": [],
}
});
if let Some(ext) = file_ext {
config["interpreter"]["file_extension"] = json!(ext);
}
config
}
/// Build minimal distributions metadata for a runtime with no template.
/// Includes a basic verification command using the detected binary path.
fn build_minimal_distributions(detected: &DetectedRuntime) -> serde_json::Value {
json!({
"verification": {
"commands": [
{
"binary": &detected.path,
"args": ["--version"],
"exit_code": 0,
"priority": 1
}
]
}
})
}
/// Capitalize a runtime name for display (e.g., "ruby" → "Ruby", "r" → "R").
fn capitalize_runtime_name(name: &str) -> String {
let mut chars = name.chars();
match chars.next() {
None => String::new(),
Some(first) => {
let upper: String = first.to_uppercase().collect();
format!("{}{}", upper, chars.as_str())
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_default_file_extension() {
assert_eq!(default_file_extension("shell"), Some(".sh"));
assert_eq!(default_file_extension("python"), Some(".py"));
assert_eq!(default_file_extension("node"), Some(".js"));
assert_eq!(default_file_extension("ruby"), Some(".rb"));
assert_eq!(default_file_extension("go"), Some(".go"));
assert_eq!(default_file_extension("java"), Some(".java"));
assert_eq!(default_file_extension("perl"), Some(".pl"));
assert_eq!(default_file_extension("r"), Some(".R"));
assert_eq!(default_file_extension("unknown"), None);
}
#[test]
fn test_capitalize_runtime_name() {
assert_eq!(capitalize_runtime_name("ruby"), "Ruby");
assert_eq!(capitalize_runtime_name("go"), "Go");
assert_eq!(capitalize_runtime_name("r"), "R");
assert_eq!(capitalize_runtime_name("perl"), "Perl");
assert_eq!(capitalize_runtime_name("java"), "Java");
assert_eq!(capitalize_runtime_name(""), "");
}
#[test]
fn test_build_detection_config_with_version() {
let detected = DetectedRuntime {
name: "ruby".to_string(),
path: "/usr/bin/ruby".to_string(),
version: Some("3.3.0".to_string()),
};
let config = build_detection_config(&detected);
assert_eq!(config["detected_path"], "/usr/bin/ruby");
assert_eq!(config["detected_name"], "ruby");
assert_eq!(config["detected_version"], "3.3.0");
}
#[test]
fn test_build_detection_config_without_version() {
let detected = DetectedRuntime {
name: "perl".to_string(),
path: "/usr/bin/perl".to_string(),
version: None,
};
let config = build_detection_config(&detected);
assert_eq!(config["detected_path"], "/usr/bin/perl");
assert_eq!(config["detected_name"], "perl");
assert!(config.get("detected_version").is_none());
}
#[test]
fn test_build_minimal_execution_config() {
let detected = DetectedRuntime {
name: "ruby".to_string(),
path: "/usr/bin/ruby".to_string(),
version: Some("3.3.0".to_string()),
};
let config = build_minimal_execution_config(&detected);
assert_eq!(config["interpreter"]["binary"], "/usr/bin/ruby");
assert_eq!(config["interpreter"]["file_extension"], ".rb");
assert_eq!(config["interpreter"]["args"], json!([]));
}
#[test]
fn test_build_minimal_execution_config_unknown_runtime() {
let detected = DetectedRuntime {
name: "custom".to_string(),
path: "/opt/custom/bin/custom".to_string(),
version: None,
};
let config = build_minimal_execution_config(&detected);
assert_eq!(config["interpreter"]["binary"], "/opt/custom/bin/custom");
// Unknown runtime has no file extension
assert!(config["interpreter"].get("file_extension").is_none());
}
#[test]
fn test_build_minimal_distributions() {
let detected = DetectedRuntime {
name: "ruby".to_string(),
path: "/usr/bin/ruby".to_string(),
version: Some("3.3.0".to_string()),
};
let distros = build_minimal_distributions(&detected);
let commands = distros["verification"]["commands"].as_array().unwrap();
assert_eq!(commands.len(), 1);
assert_eq!(commands[0]["binary"], "/usr/bin/ruby");
}
#[test]
fn test_build_execution_config_from_template_updates_binary() {
let template = Runtime {
id: 1,
r#ref: "core.ruby".to_string(),
pack: Some(1),
pack_ref: Some("core".to_string()),
description: Some("Ruby Runtime".to_string()),
name: "Ruby".to_string(),
distributions: json!({}),
installation: None,
installers: json!({}),
execution_config: json!({
"interpreter": {
"binary": "ruby",
"args": [],
"file_extension": ".rb"
},
"env_vars": {
"GEM_HOME": "{env_dir}/gems"
}
}),
auto_detected: false,
detection_config: json!({}),
created: chrono::Utc::now(),
updated: chrono::Utc::now(),
};
let detected = DetectedRuntime {
name: "ruby".to_string(),
path: "/usr/local/bin/ruby3.3".to_string(),
version: Some("3.3.0".to_string()),
};
let config = build_execution_config_from_template(&template, &detected);
// Binary should be updated to the detected path
assert_eq!(config["interpreter"]["binary"], "/usr/local/bin/ruby3.3");
// Other fields should be preserved from the template
assert_eq!(config["interpreter"]["file_extension"], ".rb");
assert_eq!(config["env_vars"]["GEM_HOME"], "{env_dir}/gems");
}
#[test]
fn test_build_execution_config_from_template_preserves_template_vars() {
let template = Runtime {
id: 1,
r#ref: "core.python".to_string(),
pack: Some(1),
pack_ref: Some("core".to_string()),
description: None,
name: "Python".to_string(),
distributions: json!({}),
installation: None,
installers: json!({}),
execution_config: json!({
"interpreter": {
"binary": "python3",
"file_extension": ".py"
},
"environment": {
"interpreter_path": "{env_dir}/bin/python3",
"create_command": ["python3", "-m", "venv", "{env_dir}"]
}
}),
auto_detected: false,
detection_config: json!({}),
created: chrono::Utc::now(),
updated: chrono::Utc::now(),
};
let detected = DetectedRuntime {
name: "python".to_string(),
path: "/usr/bin/python3.12".to_string(),
version: Some("3.12.1".to_string()),
};
let config = build_execution_config_from_template(&template, &detected);
// Binary should be updated
assert_eq!(config["interpreter"]["binary"], "/usr/bin/python3.12");
// Template-variable interpreter_path should be preserved (contains '{')
assert_eq!(
config["environment"]["interpreter_path"],
"{env_dir}/bin/python3"
);
}
}

12
crates/worker/src/executor.rs
View File

@@ -19,6 +19,7 @@ use attune_common::models::runtime::RuntimeExecutionConfig;
use attune_common::models::{runtime::Runtime as RuntimeModel, Action, Execution, ExecutionStatus};
use attune_common::repositories::artifact::{ArtifactRepository, ArtifactVersionRepository};
use attune_common::repositories::execution::{ExecutionRepository, UpdateExecutionInput};
use attune_common::repositories::runtime::SELECT_COLUMNS as RUNTIME_SELECT_COLUMNS;
use attune_common::repositories::runtime_version::RuntimeVersionRepository;
use attune_common::repositories::{FindById, Update};
use attune_common::version_matching::select_best_version;
@@ -410,12 +411,11 @@ impl ActionExecutor {
// Load runtime information if specified
let runtime_record = if let Some(runtime_id) = action.runtime {
match sqlx::query_as::<_, RuntimeModel>(
r#"SELECT id, ref, pack, pack_ref, description, name,
distributions, installation, installers, execution_config,
created, updated
FROM runtime WHERE id = $1"#,
)
let query = format!(
"SELECT {} FROM runtime WHERE id = $1",
RUNTIME_SELECT_COLUMNS
);
match sqlx::query_as::<_, RuntimeModel>(&query)
.bind(runtime_id)
.fetch_optional(&self.pool)
.await

6
crates/worker/src/lib.rs
View File

@@ -4,16 +4,19 @@
//! which executes actions in various runtime environments.
pub mod artifacts;
pub mod dynamic_runtime;
pub mod env_setup;
pub mod executor;
pub mod heartbeat;
pub mod registration;
pub mod runtime;
pub mod runtime_detect;
pub mod secrets;
pub mod service;
pub mod version_verify;
// Re-export commonly used types
pub use dynamic_runtime::auto_register_detected_runtimes;
pub use executor::ActionExecutor;
pub use heartbeat::HeartbeatManager;
pub use registration::WorkerRegistration;
@@ -21,7 +24,8 @@ pub use runtime::{
ExecutionContext, ExecutionResult, LocalRuntime, NativeRuntime, ProcessRuntime, Runtime,
RuntimeError, RuntimeResult,
};
pub use runtime_detect::DetectedRuntime;
pub use secrets::SecretManager;
pub use service::WorkerService;
pub use service::{StartupMode, WorkerService};
// Re-export test executor from common (shared business logic)
pub use attune_common::test_executor::{TestConfig, TestExecutor};

115
crates/worker/src/registration.rs
View File

@@ -13,6 +13,8 @@ use sqlx::PgPool;
use std::collections::HashMap;
use tracing::{info, warn};
use crate::runtime_detect::DetectedRuntime;
/// Worker registration manager
pub struct WorkerRegistration {
pool: PgPool,
@@ -100,6 +102,51 @@ impl WorkerRegistration {
}
}
/// Store detected runtime interpreter metadata in capabilities.
///
/// This is used by the agent (`attune-agent`) to record the full details of
/// auto-detected interpreters — binary paths and versions — alongside the
/// simple `runtimes` string list used for backward compatibility.
///
/// The data is stored under the `detected_interpreters` capability key as a
/// JSON array of objects:
/// ```json
/// [
/// {"name": "python", "path": "/usr/bin/python3", "version": "3.12.1"},
/// {"name": "shell", "path": "/bin/bash", "version": "5.2.15"}
/// ]
/// ```
pub fn set_detected_runtimes(&mut self, runtimes: Vec<DetectedRuntime>) {
let interpreters: Vec<serde_json::Value> = runtimes
.iter()
.map(|rt| {
json!({
"name": rt.name,
"path": rt.path,
"version": rt.version,
})
})
.collect();
self.capabilities
.insert("detected_interpreters".to_string(), json!(interpreters));
info!(
"Stored {} detected interpreter(s) in capabilities",
runtimes.len()
);
}
/// Mark this worker as running in agent mode.
///
/// Agent-mode workers auto-detect their runtimes at startup (as opposed to
/// being configured via `ATTUNE_WORKER_RUNTIMES` or config files). Setting
/// this flag allows the system to distinguish agents from standard workers.
pub fn set_agent_mode(&mut self, is_agent: bool) {
self.capabilities
.insert("agent_mode".to_string(), json!(is_agent));
}
/// Detect available runtimes using the unified runtime detector
pub async fn detect_capabilities(&mut self, config: &Config) -> Result<()> {
info!("Detecting worker capabilities...");
@@ -346,4 +393,72 @@ mod tests {
registration.deregister().await.unwrap();
}
#[test]
fn test_detected_runtimes_json_structure() {
// Test the JSON structure that set_detected_runtimes builds
let runtimes = vec![
DetectedRuntime {
name: "python".to_string(),
path: "/usr/bin/python3".to_string(),
version: Some("3.12.1".to_string()),
},
DetectedRuntime {
name: "shell".to_string(),
path: "/bin/bash".to_string(),
version: None,
},
];
let interpreters: Vec<serde_json::Value> = runtimes
.iter()
.map(|rt| {
json!({
"name": rt.name,
"path": rt.path,
"version": rt.version,
})
})
.collect();
let json_value = json!(interpreters);
// Verify structure
let arr = json_value.as_array().unwrap();
assert_eq!(arr.len(), 2);
assert_eq!(arr[0]["name"], "python");
assert_eq!(arr[0]["path"], "/usr/bin/python3");
assert_eq!(arr[0]["version"], "3.12.1");
assert_eq!(arr[1]["name"], "shell");
assert_eq!(arr[1]["path"], "/bin/bash");
assert!(arr[1]["version"].is_null());
}
#[test]
fn test_detected_runtimes_empty() {
let runtimes: Vec<DetectedRuntime> = vec![];
let interpreters: Vec<serde_json::Value> = runtimes
.iter()
.map(|rt| {
json!({
"name": rt.name,
"path": rt.path,
"version": rt.version,
})
})
.collect();
let json_value = json!(interpreters);
assert_eq!(json_value.as_array().unwrap().len(), 0);
}
#[test]
fn test_agent_mode_capability_value() {
// Verify the JSON value for agent_mode capability
let value = json!(true);
assert_eq!(value, true);
let value = json!(false);
assert_eq!(value, false);
}
}

77
crates/worker/src/runtime/process.rs
View File

@@ -24,9 +24,27 @@ use attune_common::models::runtime::{
};
use std::collections::HashMap;
use std::path::{Path, PathBuf};
use std::sync::{Arc, Mutex as StdMutex, OnceLock};
use tokio::process::Command;
use tracing::{debug, error, info, warn};
/// Per-directory locks for lazy environment setup to prevent concurrent
/// setup of the same environment from corrupting it. When two executions
/// for the same pack arrive concurrently (e.g. in agent mode), both may
/// see `!env_dir.exists()` and race to run `setup_pack_environment`.
/// This map provides a per-directory async mutex so that only one setup
/// runs at a time for each env_dir path.
static ENV_SETUP_LOCKS: OnceLock<StdMutex<HashMap<PathBuf, Arc<tokio::sync::Mutex<()>>>>> =
OnceLock::new();
fn get_env_setup_lock(env_dir: &Path) -> Arc<tokio::sync::Mutex<()>> {
let locks = ENV_SETUP_LOCKS.get_or_init(|| StdMutex::new(HashMap::new()));
let mut map = locks.lock().unwrap();
map.entry(env_dir.to_path_buf())
.or_insert_with(|| Arc::new(tokio::sync::Mutex::new(())))
.clone()
}
fn bash_single_quote_escape(s: &str) -> String {
s.replace('\'', "'\\''")
}
@@ -615,30 +633,68 @@ impl Runtime for ProcessRuntime {
None
};
// Runtime environments are set up proactively — either at worker startup
// (scanning all registered packs) or via pack.registered MQ events when a
// new pack is installed. We only log a warning here if the expected
// environment directory is missing so operators can investigate.
if effective_config.environment.is_some() && pack_dir.exists() && !env_dir.exists() {
warn!(
// Lazy environment setup: if the environment directory doesn't exist but
// should (i.e., there's an environment config and the pack dir exists),
// create it on-demand. This is the primary code path for agent mode where
// proactive startup setup is skipped, but it also serves as a safety net
// for standard workers if the environment was somehow missed.
// Acquire a per-directory async lock to serialize environment setup.
// This prevents concurrent executions for the same pack from racing
// to create or repair the environment simultaneously.
if effective_config.environment.is_some() && pack_dir.exists() {
let env_lock = get_env_setup_lock(&env_dir);
let _guard = env_lock.lock().await;
// --- Lazy environment creation (double-checked after lock) ---
if !env_dir.exists() {
info!(
"Runtime environment for pack '{}' not found at {}. \
The environment should have been created at startup or on pack registration. \
Proceeding with system interpreter as fallback.",
Creating on first use (lazy setup).",
context.action_ref,
env_dir.display(),
);
let setup_runtime = ProcessRuntime::new(
self.runtime_name.clone(),
effective_config.clone(),
self.packs_base_dir.clone(),
self.runtime_envs_dir.clone(),
);
match setup_runtime
.setup_pack_environment(&pack_dir, &env_dir)
.await
{
Ok(()) => {
info!(
"Successfully created environment for pack '{}' at {} (lazy setup)",
context.action_ref,
env_dir.display(),
);
}
Err(e) => {
warn!(
"Failed to create environment for pack '{}' at {}: {}. \
Proceeding with system interpreter as fallback.",
context.action_ref,
env_dir.display(),
e,
);
}
}
}
// --- Broken-symlink repair (also under the per-directory lock) ---
// If the environment directory exists but contains a broken interpreter
// (e.g. broken symlinks from a venv created in a different container),
// attempt to recreate it before resolving the interpreter.
if effective_config.environment.is_some() && env_dir.exists() && pack_dir.exists() {
if env_dir.exists() {
if let Some(ref env_cfg) = effective_config.environment {
if let Some(ref interp_template) = env_cfg.interpreter_path {
let mut vars = std::collections::HashMap::new();
vars.insert("env_dir", env_dir.to_string_lossy().to_string());
vars.insert("pack_dir", pack_dir.to_string_lossy().to_string());
let resolved = RuntimeExecutionConfig::resolve_template(interp_template, &vars);
let resolved =
RuntimeExecutionConfig::resolve_template(interp_template, &vars);
let resolved_path = std::path::PathBuf::from(&resolved);
// Check for a broken symlink: symlink_metadata succeeds for
@@ -704,6 +760,7 @@ impl Runtime for ProcessRuntime {
}
}
}
}
let interpreter = effective_config.resolve_interpreter_with_env(&pack_dir, env_dir_opt);

544
crates/worker/src/runtime_detect.rs Normal file
View File

@@ -0,0 +1,544 @@
//! Runtime Auto-Detection Module
//!
//! Provides lightweight, database-free runtime detection for the Universal Worker Agent.
//! Unlike [`attune_common::runtime_detection::RuntimeDetector`] which queries the database
//! for runtime definitions and verification metadata, this module probes the local system
//! directly by checking for well-known interpreter binaries on PATH.
//!
//! This is designed for the agent entrypoint (`attune-agent`) which is injected into
//! arbitrary containers and must discover what runtimes are available without any
//! database connectivity at detection time.
//!
//! # Detection Strategy
//!
//! For each candidate runtime, the detector:
//! 1. Checks if a binary exists and is executable using `which`-style PATH lookup
//! 2. Optionally runs a version command (e.g., `python3 --version`) to capture the version
//! 3. Returns a list of [`DetectedRuntime`] structs with name, path, and version info
//!
//! # Supported Runtimes
//!
//! | Runtime | Binaries checked (in order) | Version command |
//! |----------|-------------------------------|-------------------------|
//! | shell | `bash`, `sh` | `bash --version` |
//! | python | `python3`, `python` | `python3 --version` |
//! | node | `node`, `nodejs` | `node --version` |
//! | ruby | `ruby` | `ruby --version` |
//! | go | `go` | `go version` |
//! | java | `java` | `java -version` |
//! | r | `Rscript` | `Rscript --version` |
//! | perl | `perl` | `perl --version` |
use serde::{Deserialize, Serialize};
use std::fmt;
use std::process::Command;
use tracing::{debug, info};
/// A runtime interpreter discovered on the local system.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct DetectedRuntime {
/// Canonical runtime name (e.g., "shell", "python", "node").
/// These names align with the normalized names from
/// [`attune_common::runtime_detection::normalize_runtime_name`].
pub name: String,
/// Absolute path to the interpreter binary (as resolved by `which`).
pub path: String,
/// Version string if a version check command succeeded (e.g., "3.12.1").
pub version: Option<String>,
}
impl fmt::Display for DetectedRuntime {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match &self.version {
Some(v) => write!(f, "{} ({}, v{})", self.name, self.path, v),
None => write!(f, "{} ({})", self.name, self.path),
}
}
}
/// A candidate runtime to probe for during detection.
struct RuntimeCandidate {
/// Canonical name for this runtime (used in ATTUNE_WORKER_RUNTIMES).
name: &'static str,
/// Binary names to try, in priority order. The first one found wins.
binaries: &'static [&'static str],
/// Arguments to pass to the binary to get a version string.
version_args: &'static [&'static str],
/// How to extract the version from command output.
version_parser: VersionParser,
}
/// Strategy for parsing version output from a command.
enum VersionParser {
/// Extract a version pattern like "X.Y.Z" from the combined stdout+stderr output.
/// This handles the common case where the version appears somewhere in the output
/// (e.g., "Python 3.12.1", "node v20.11.0", "go1.22.0").
SemverLike,
/// Java uses `-version` which writes to stderr, and the format is
/// `openjdk version "21.0.1"` or `java version "1.8.0_392"`.
JavaStyle,
}
/// All candidate runtimes to probe, in detection order.
fn candidates() -> Vec<RuntimeCandidate> {
vec![
RuntimeCandidate {
name: "shell",
binaries: &["bash", "sh"],
version_args: &["--version"],
version_parser: VersionParser::SemverLike,
},
RuntimeCandidate {
name: "python",
binaries: &["python3", "python"],
version_args: &["--version"],
version_parser: VersionParser::SemverLike,
},
RuntimeCandidate {
name: "node",
binaries: &["node", "nodejs"],
version_args: &["--version"],
version_parser: VersionParser::SemverLike,
},
RuntimeCandidate {
name: "ruby",
binaries: &["ruby"],
version_args: &["--version"],
version_parser: VersionParser::SemverLike,
},
RuntimeCandidate {
name: "go",
binaries: &["go"],
version_args: &["version"],
version_parser: VersionParser::SemverLike,
},
RuntimeCandidate {
name: "java",
binaries: &["java"],
version_args: &["-version"],
version_parser: VersionParser::JavaStyle,
},
RuntimeCandidate {
name: "r",
binaries: &["Rscript"],
version_args: &["--version"],
version_parser: VersionParser::SemverLike,
},
RuntimeCandidate {
name: "perl",
binaries: &["perl"],
version_args: &["--version"],
version_parser: VersionParser::SemverLike,
},
]
}
/// Detect available runtimes by probing the local system for known interpreter binaries.
///
/// This function performs synchronous subprocess calls (`std::process::Command`) since
/// it is a one-time startup operation. It checks each candidate runtime's binaries
/// in priority order using `which`-style PATH lookup, and optionally captures the
/// interpreter version.
///
/// # Returns
///
/// A vector of [`DetectedRuntime`] for each runtime that was found on the system.
/// The order matches the detection order (shell first, then python, node, etc.).
///
/// # Example
///
/// ```no_run
/// use attune_worker::runtime_detect::detect_runtimes;
///
/// let runtimes = detect_runtimes();
/// for rt in &runtimes {
/// println!("Found: {}", rt);
/// }
/// // Convert to ATTUNE_WORKER_RUNTIMES format
/// let names: Vec<&str> = runtimes.iter().map(|r| r.name.as_str()).collect();
/// println!("ATTUNE_WORKER_RUNTIMES={}", names.join(","));
/// ```
pub fn detect_runtimes() -> Vec<DetectedRuntime> {
info!("Starting runtime auto-detection...");
let mut detected = Vec::new();
for candidate in candidates() {
match detect_single_runtime(&candidate) {
Some(runtime) => {
info!(" ✓ Detected: {}", runtime);
detected.push(runtime);
}
None => {
debug!(" ✗ Not found: {}", candidate.name);
}
}
}
info!(
"Runtime auto-detection complete: found {} runtime(s): [{}]",
detected.len(),
detected
.iter()
.map(|r| r.name.as_str())
.collect::<Vec<_>>()
.join(", ")
);
detected
}
/// Attempt to detect a single runtime by checking its candidate binaries.
fn detect_single_runtime(candidate: &RuntimeCandidate) -> Option<DetectedRuntime> {
for binary in candidate.binaries {
if let Some(path) = which_binary(binary) {
debug!(
"Found {} at {} (for runtime '{}')",
binary, path, candidate.name
);
// Attempt to get version info (non-fatal if it fails)
let version = get_version(&path, candidate.version_args, &candidate.version_parser);
return Some(DetectedRuntime {
name: candidate.name.to_string(),
path,
version,
});
}
}
None
}
/// Look up a binary on PATH, similar to the `which` command.
///
/// Uses `which <binary>` on the system to resolve the full path.
/// Returns `None` if the binary is not found or `which` fails.
fn which_binary(binary: &str) -> Option<String> {
// First check well-known absolute paths for shell interpreters
// (these may not be on PATH in minimal containers)
if binary == "bash" || binary == "sh" {
let absolute_path = format!("/bin/{}", binary);
if std::path::Path::new(&absolute_path).exists() {
return Some(absolute_path);
}
}
// Fall back to PATH lookup via `which`
match Command::new("which").arg(binary).output() {
Ok(output) => {
if output.status.success() {
let path = String::from_utf8_lossy(&output.stdout).trim().to_string();
if !path.is_empty() {
Some(path)
} else {
None
}
} else {
None
}
}
Err(e) => {
// `which` itself not found — try `command -v` as fallback
debug!("'which' command failed ({}), trying 'command -v'", e);
match Command::new("sh")
.args(["-c", &format!("command -v {}", binary)])
.output()
{
Ok(output) if output.status.success() => {
let path = String::from_utf8_lossy(&output.stdout).trim().to_string();
if !path.is_empty() {
Some(path)
} else {
None
}
}
_ => None,
}
}
}
}
/// Run a version command and parse the version string from the output.
fn get_version(binary_path: &str, version_args: &[&str], parser: &VersionParser) -> Option<String> {
let output = match Command::new(binary_path).args(version_args).output() {
Ok(output) => output,
Err(e) => {
debug!("Failed to run version command for {}: {}", binary_path, e);
return None;
}
};
let stdout = String::from_utf8_lossy(&output.stdout);
let stderr = String::from_utf8_lossy(&output.stderr);
let combined = format!("{}{}", stdout, stderr);
match parser {
VersionParser::SemverLike => parse_semver_like(&combined),
VersionParser::JavaStyle => parse_java_version(&combined),
}
}
/// Extract a semver-like version (X.Y.Z or X.Y) from output text.
///
/// Handles common patterns:
/// - "Python 3.12.1"
/// - "node v20.11.0"
/// - "go version go1.22.0 linux/amd64"
/// - "GNU bash, version 5.2.15(1)-release"
/// - "ruby 3.2.2 (2023-03-30 revision e51014f9c0)"
/// - "perl 5, version 36, subversion 0 (v5.36.0)"
fn parse_semver_like(output: &str) -> Option<String> {
// Try to find a pattern like X.Y.Z or X.Y (with optional leading 'v')
// Also handle go's "go1.22.0" format
let re = regex::Regex::new(r"(?:v|go)?(\d+\.\d+(?:\.\d+)?)").ok()?;
if let Some(captures) = re.captures(output) {
captures.get(1).map(|m| m.as_str().to_string())
} else {
None
}
}
/// Parse Java's peculiar version output format.
///
/// Java writes to stderr and uses formats like:
/// - `openjdk version "21.0.1" 2023-10-17`
/// - `java version "1.8.0_392"`
fn parse_java_version(output: &str) -> Option<String> {
// Look for version inside quotes first
let quoted_re = regex::Regex::new(r#"version\s+"([^"]+)""#).ok()?;
if let Some(captures) = quoted_re.captures(output) {
return captures.get(1).map(|m| m.as_str().to_string());
}
// Fall back to semver-like parsing
parse_semver_like(output)
}
/// Format detected runtimes as a comma-separated string suitable for
/// the `ATTUNE_WORKER_RUNTIMES` environment variable.
///
/// # Example
///
/// ```no_run
/// use attune_worker::runtime_detect::{detect_runtimes, format_as_env_value};
///
/// let runtimes = detect_runtimes();
/// let env_val = format_as_env_value(&runtimes);
/// // e.g., "shell,python,node"
/// ```
pub fn format_as_env_value(runtimes: &[DetectedRuntime]) -> String {
runtimes
.iter()
.map(|r| r.name.as_str())
.collect::<Vec<_>>()
.join(",")
}
/// Print a human-readable detection report to stdout.
///
/// Used by the `--detect-only` flag to show detection results and exit.
pub fn print_detection_report(runtimes: &[DetectedRuntime]) {
println!("=== Attune Agent Runtime Detection Report ===");
println!();
if runtimes.is_empty() {
println!("No runtimes detected!");
println!();
println!("The agent could not find any supported interpreter binaries.");
println!("Ensure at least one of the following is installed and on PATH:");
println!(" - bash / sh (shell scripts)");
println!(" - python3 / python (Python scripts)");
println!(" - node / nodejs (Node.js scripts)");
println!(" - ruby (Ruby scripts)");
println!(" - go (Go programs)");
println!(" - java (Java programs)");
println!(" - Rscript (R scripts)");
println!(" - perl (Perl scripts)");
} else {
println!("Detected {} runtime(s):", runtimes.len());
println!();
for rt in runtimes {
let version_str = rt.version.as_deref().unwrap_or("unknown version");
println!(" ✓ {:<10} {} ({})", rt.name, rt.path, version_str);
}
}
println!();
println!("ATTUNE_WORKER_RUNTIMES={}", format_as_env_value(runtimes));
println!();
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_parse_semver_like_python() {
assert_eq!(
parse_semver_like("Python 3.12.1"),
Some("3.12.1".to_string())
);
}
#[test]
fn test_parse_semver_like_node() {
assert_eq!(parse_semver_like("v20.11.0"), Some("20.11.0".to_string()));
}
#[test]
fn test_parse_semver_like_go() {
assert_eq!(
parse_semver_like("go version go1.22.0 linux/amd64"),
Some("1.22.0".to_string())
);
}
#[test]
fn test_parse_semver_like_bash() {
assert_eq!(
parse_semver_like("GNU bash, version 5.2.15(1)-release (x86_64-pc-linux-gnu)"),
Some("5.2.15".to_string())
);
}
#[test]
fn test_parse_semver_like_ruby() {
assert_eq!(
parse_semver_like("ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]"),
Some("3.2.2".to_string())
);
}
#[test]
fn test_parse_semver_like_two_part() {
assert_eq!(
parse_semver_like("SomeRuntime 1.5"),
Some("1.5".to_string())
);
}
#[test]
fn test_parse_semver_like_no_match() {
assert_eq!(parse_semver_like("no version here"), None);
}
#[test]
fn test_parse_java_version_openjdk() {
assert_eq!(
parse_java_version(r#"openjdk version "21.0.1" 2023-10-17"#),
Some("21.0.1".to_string())
);
}
#[test]
fn test_parse_java_version_legacy() {
assert_eq!(
parse_java_version(r#"java version "1.8.0_392""#),
Some("1.8.0_392".to_string())
);
}
#[test]
fn test_format_as_env_value_empty() {
let runtimes: Vec<DetectedRuntime> = vec![];
assert_eq!(format_as_env_value(&runtimes), "");
}
#[test]
fn test_format_as_env_value_multiple() {
let runtimes = vec![
DetectedRuntime {
name: "shell".to_string(),
path: "/bin/bash".to_string(),
version: Some("5.2.15".to_string()),
},
DetectedRuntime {
name: "python".to_string(),
path: "/usr/bin/python3".to_string(),
version: Some("3.12.1".to_string()),
},
DetectedRuntime {
name: "node".to_string(),
path: "/usr/bin/node".to_string(),
version: None,
},
];
assert_eq!(format_as_env_value(&runtimes), "shell,python,node");
}
#[test]
fn test_detected_runtime_display_with_version() {
let rt = DetectedRuntime {
name: "python".to_string(),
path: "/usr/bin/python3".to_string(),
version: Some("3.12.1".to_string()),
};
assert_eq!(format!("{}", rt), "python (/usr/bin/python3, v3.12.1)");
}
#[test]
fn test_detected_runtime_display_without_version() {
let rt = DetectedRuntime {
name: "shell".to_string(),
path: "/bin/bash".to_string(),
version: None,
};
assert_eq!(format!("{}", rt), "shell (/bin/bash)");
}
#[test]
fn test_detect_runtimes_runs_without_panic() {
// This test verifies the detection logic doesn't panic,
// regardless of what's actually installed on the system.
let runtimes = detect_runtimes();
// We should at least find a shell on any Unix system
// but we don't assert that since test environments vary.
let _ = runtimes;
}
#[test]
fn test_which_binary_sh() {
// /bin/sh should exist on virtually all Unix systems
let result = which_binary("sh");
assert!(result.is_some(), "Expected to find 'sh' on this system");
}
#[test]
fn test_which_binary_nonexistent() {
let result = which_binary("definitely_not_a_real_binary_xyz123");
assert!(result.is_none());
}
#[test]
fn test_candidates_order() {
let c = candidates();
assert_eq!(c[0].name, "shell");
assert_eq!(c[1].name, "python");
assert_eq!(c[2].name, "node");
assert_eq!(c[3].name, "ruby");
assert_eq!(c[4].name, "go");
assert_eq!(c[5].name, "java");
assert_eq!(c[6].name, "r");
assert_eq!(c[7].name, "perl");
}
#[test]
fn test_candidates_binaries_priority() {
let c = candidates();
// shell prefers bash over sh
assert_eq!(c[0].binaries, &["bash", "sh"]);
// python prefers python3 over python
assert_eq!(c[1].binaries, &["python3", "python"]);
// node prefers node over nodejs
assert_eq!(c[2].binaries, &["node", "nodejs"]);
}
}

66
crates/worker/src/service.rs
View File

@@ -45,12 +45,32 @@ use crate::runtime::local::LocalRuntime;
use crate::runtime::native::NativeRuntime;
use crate::runtime::process::ProcessRuntime;
use crate::runtime::RuntimeRegistry;
use crate::runtime_detect::DetectedRuntime;
use crate::secrets::SecretManager;
use crate::version_verify;
use attune_common::repositories::runtime::RuntimeRepository;
use attune_common::repositories::List;
/// Controls how the worker initializes its runtime environment.
///
/// The standard `attune-worker` binary uses `Worker` mode with proactive
/// setup at startup, while the `attune-agent` binary uses `Agent` mode
/// with lazy (on-demand) initialization.
#[derive(Debug, Clone)]
pub enum StartupMode {
/// Full worker mode: proactive environment setup, full version
/// verification sweep at startup. Used by `attune-worker`.
Worker,
/// Agent mode: lazy environment setup (on first use), on-demand
/// version verification, auto-detected runtimes. Used by `attune-agent`.
Agent {
/// Runtimes detected by the auto-detection module.
detected_runtimes: Vec<DetectedRuntime>,
},
}
/// Message payload for execution.scheduled events
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ExecutionScheduledPayload {
@@ -93,6 +113,10 @@ pub struct WorkerService {
/// Tracks cancellation requests that arrived before the in-memory token
/// for an execution had been registered.
pending_cancellations: Arc<Mutex<HashSet<i64>>>,
/// Controls whether this worker runs in full `Worker` mode (proactive
/// environment setup, full version verification) or `Agent` mode (lazy
/// setup, auto-detected runtimes).
startup_mode: StartupMode,
}
impl WorkerService {
@@ -402,9 +426,26 @@ impl WorkerService {
in_flight_tasks: Arc::new(Mutex::new(JoinSet::new())),
cancel_tokens: Arc::new(Mutex::new(HashMap::new())),
pending_cancellations: Arc::new(Mutex::new(HashSet::new())),
startup_mode: StartupMode::Worker,
})
}
/// Set agent-detected runtimes for inclusion in worker registration.
///
/// When the worker is started as `attune-agent`, the agent entrypoint
/// auto-detects available interpreters and passes them here. During
/// [`start()`](Self::start), the detection results are stored in the
/// worker's capabilities as `detected_interpreters` (structured JSON
/// with binary paths and versions) and the `agent_mode` flag is set.
///
/// This method is a no-op for the standard `attune-worker` binary.
pub fn with_detected_runtimes(mut self, runtimes: Vec<DetectedRuntime>) -> Self {
self.startup_mode = StartupMode::Agent {
detected_runtimes: runtimes,
};
self
}
/// Start the worker service
pub async fn start(&mut self) -> Result<()> {
info!("Starting Worker Service");
@@ -413,6 +454,21 @@ impl WorkerService {
let worker_id = {
let mut reg = self.registration.write().await;
reg.detect_capabilities(&self.config).await?;
// If running as an agent, store the detected interpreter metadata
// and set the agent_mode flag before registering.
if let StartupMode::Agent {
ref detected_runtimes,
} = self.startup_mode
{
reg.set_detected_runtimes(detected_runtimes.clone());
reg.set_agent_mode(true);
info!(
"Agent mode: {} detected interpreter(s) will be stored in capabilities",
detected_runtimes.len()
);
}
reg.register().await?
};
self.worker_id = Some(worker_id);
@@ -430,6 +486,8 @@ impl WorkerService {
})?;
info!("Worker-specific message queue infrastructure setup completed");
match &self.startup_mode {
StartupMode::Worker => {
// Verify which runtime versions are available on this system.
// This updates the `available` flag in the database so that
// `select_best_version()` only considers genuinely present versions.
@@ -440,6 +498,14 @@ impl WorkerService {
// environments are ready by the time the first execution arrives.
// Now version-aware: creates per-version environments where needed.
self.scan_and_setup_environments().await;
}
StartupMode::Agent { .. } => {
// Skip proactive setup — will happen lazily on first execution
info!(
"Agent mode: deferring environment setup and version verification to first use"
);
}
}
// Start heartbeat
self.heartbeat.start().await?;

216
docker-compose.agent.yaml Normal file
View File

@@ -0,0 +1,216 @@
# Agent-Based Worker Services
#
# This override file demonstrates how to add custom runtime workers to Attune
# by injecting the universal worker agent into any container image.
#
# Usage:
# docker compose -f docker-compose.yaml -f docker-compose.agent.yaml up -d
#
# Prerequisites:
# The init-agent service (defined in docker-compose.yaml) must be present.
# It builds the statically-linked agent binary and populates the agent_bin volume.
#
# How it works:
# 1. init-agent builds a musl-static attune-agent binary and copies it to the agent_bin volume
# 2. Each worker service mounts agent_bin read-only and uses the agent as its entrypoint
# 3. The agent auto-detects available runtimes in the container (python, ruby, node, etc.)
# 4. No Dockerfile needed — just point at any container image with your desired runtime
#
# Adding your own worker:
# Copy one of the examples below and change:
# - service name (e.g., worker-my-runtime)
# - image (any Docker image with your runtime installed)
# - ATTUNE_WORKER_NAME (unique name for this worker)
# - Optionally set ATTUNE_WORKER_RUNTIMES to override auto-detection
services:
# ---------------------------------------------------------------------------
# Ruby Worker — Official Ruby image with auto-detected runtime
# ---------------------------------------------------------------------------
worker-ruby:
image: ruby:3.3-slim
container_name: attune-worker-ruby
depends_on:
init-agent:
condition: service_completed_successfully
init-packs:
condition: service_completed_successfully
migrations:
condition: service_completed_successfully
postgres:
condition: service_healthy
rabbitmq:
condition: service_healthy
entrypoint: ["/opt/attune/agent/attune-agent"]
stop_grace_period: 45s
environment:
RUST_LOG: info
ATTUNE_CONFIG: /opt/attune/config/config.yaml
ATTUNE_WORKER_NAME: worker-ruby-01
ATTUNE_WORKER_TYPE: container
ATTUNE__SECURITY__JWT_SECRET: ${JWT_SECRET:-docker-dev-secret-change-in-production}
ATTUNE__SECURITY__ENCRYPTION_KEY: ${ENCRYPTION_KEY:-docker-dev-encryption-key-please-change-in-production-32plus}
ATTUNE__DATABASE__URL: postgresql://attune:attune@postgres:5432/attune
ATTUNE__MESSAGE_QUEUE__URL: amqp://attune:attune@rabbitmq:5672
ATTUNE_API_URL: http://attune-api:8080
# ATTUNE_WORKER_RUNTIMES omitted — auto-detected as ruby,shell
volumes:
- agent_bin:/opt/attune/agent:ro
- ${ATTUNE_DOCKER_CONFIG_PATH:-./config.docker.yaml}:/opt/attune/config/config.yaml:ro
- packs_data:/opt/attune/packs:ro
- runtime_envs:/opt/attune/runtime_envs
- artifacts_data:/opt/attune/artifacts
healthcheck:
test: ["CMD-SHELL", "pgrep -f attune-agent || exit 1"]
interval: 30s
timeout: 10s
retries: 3
start_period: 20s
networks:
- attune-network
restart: unless-stopped
# ---------------------------------------------------------------------------
# Python 3.12 Worker — Use a specific Python version via the agent
# ---------------------------------------------------------------------------
# This demonstrates using the agent with a specific Python version instead of
# the built-in worker-python service (which uses debian:bookworm-slim + apt).
#
# worker-python312:
# image: python:3.12-slim
# container_name: attune-worker-python312
# depends_on:
# init-agent:
# condition: service_completed_successfully
# init-packs:
# condition: service_completed_successfully
# migrations:
# condition: service_completed_successfully
# postgres:
# condition: service_healthy
# rabbitmq:
# condition: service_healthy
# entrypoint: ["/opt/attune/agent/attune-agent"]
# stop_grace_period: 45s
# environment:
# RUST_LOG: info
# ATTUNE_CONFIG: /opt/attune/config/config.yaml
# ATTUNE_WORKER_NAME: worker-python312-01
# ATTUNE_WORKER_TYPE: container
# ATTUNE__SECURITY__JWT_SECRET: ${JWT_SECRET:-docker-dev-secret-change-in-production}
# ATTUNE__SECURITY__ENCRYPTION_KEY: ${ENCRYPTION_KEY:-docker-dev-encryption-key-please-change-in-production-32plus}
# ATTUNE__DATABASE__URL: postgresql://attune:attune@postgres:5432/attune
# ATTUNE__MESSAGE_QUEUE__URL: amqp://attune:attune@rabbitmq:5672
# ATTUNE_API_URL: http://attune-api:8080
# volumes:
# - agent_bin:/opt/attune/agent:ro
# - ${ATTUNE_DOCKER_CONFIG_PATH:-./config.docker.yaml}:/opt/attune/config/config.yaml:ro
# - packs_data:/opt/attune/packs:ro
# - runtime_envs:/opt/attune/runtime_envs
# - artifacts_data:/opt/attune/artifacts
# healthcheck:
# test: ["CMD-SHELL", "pgrep -f attune-agent || exit 1"]
# interval: 30s
# timeout: 10s
# retries: 3
# start_period: 20s
# networks:
# - attune-network
# restart: unless-stopped
# ---------------------------------------------------------------------------
# GPU Worker — NVIDIA CUDA image for GPU-accelerated workloads
# ---------------------------------------------------------------------------
# Requires: NVIDIA Container Toolkit installed on the Docker host
# See: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/
#
# worker-gpu:
# image: nvidia/cuda:12.3.1-runtime-ubuntu22.04
# container_name: attune-worker-gpu
# depends_on:
# init-agent:
# condition: service_completed_successfully
# init-packs:
# condition: service_completed_successfully
# migrations:
# condition: service_completed_successfully
# postgres:
# condition: service_healthy
# rabbitmq:
# condition: service_healthy
# entrypoint: ["/opt/attune/agent/attune-agent"]
# runtime: nvidia
# stop_grace_period: 45s
# environment:
# RUST_LOG: info
# ATTUNE_CONFIG: /opt/attune/config/config.yaml
# ATTUNE_WORKER_NAME: worker-gpu-01
# ATTUNE_WORKER_TYPE: container
# ATTUNE_WORKER_RUNTIMES: python,shell # Manual override — CUDA image has python pre-installed
# ATTUNE__SECURITY__JWT_SECRET: ${JWT_SECRET:-docker-dev-secret-change-in-production}
# ATTUNE__SECURITY__ENCRYPTION_KEY: ${ENCRYPTION_KEY:-docker-dev-encryption-key-please-change-in-production-32plus}
# ATTUNE__DATABASE__URL: postgresql://attune:attune@postgres:5432/attune
# ATTUNE__MESSAGE_QUEUE__URL: amqp://attune:attune@rabbitmq:5672
# ATTUNE_API_URL: http://attune-api:8080
# volumes:
# - agent_bin:/opt/attune/agent:ro
# - ${ATTUNE_DOCKER_CONFIG_PATH:-./config.docker.yaml}:/opt/attune/config/config.yaml:ro
# - packs_data:/opt/attune/packs:ro
# - runtime_envs:/opt/attune/runtime_envs
# - artifacts_data:/opt/attune/artifacts
# healthcheck:
# test: ["CMD-SHELL", "pgrep -f attune-agent || exit 1"]
# interval: 30s
# timeout: 10s
# retries: 3
# start_period: 20s
# networks:
# - attune-network
# restart: unless-stopped
# ---------------------------------------------------------------------------
# Custom Image Worker — Template for any container image
# ---------------------------------------------------------------------------
# Copy this template and customize for your runtime:
#
# worker-custom:
# image: my-org/my-custom-image:latest
# container_name: attune-worker-custom
# depends_on:
# init-agent:
# condition: service_completed_successfully
# init-packs:
# condition: service_completed_successfully
# migrations:
# condition: service_completed_successfully
# postgres:
# condition: service_healthy
# rabbitmq:
# condition: service_healthy
# entrypoint: ["/opt/attune/agent/attune-agent"]
# stop_grace_period: 45s
# environment:
# RUST_LOG: info
# ATTUNE_CONFIG: /opt/attune/config/config.yaml
# ATTUNE_WORKER_NAME: worker-custom-01
# ATTUNE_WORKER_TYPE: container
# ATTUNE__SECURITY__JWT_SECRET: ${JWT_SECRET:-docker-dev-secret-change-in-production}
# ATTUNE__SECURITY__ENCRYPTION_KEY: ${ENCRYPTION_KEY:-docker-dev-encryption-key-please-change-in-production-32plus}
# ATTUNE__DATABASE__URL: postgresql://attune:attune@postgres:5432/attune
# ATTUNE__MESSAGE_QUEUE__URL: amqp://attune:attune@rabbitmq:5672
# ATTUNE_API_URL: http://attune-api:8080
# volumes:
# - agent_bin:/opt/attune/agent:ro
# - ${ATTUNE_DOCKER_CONFIG_PATH:-./config.docker.yaml}:/opt/attune/config/config.yaml:ro
# - packs_data:/opt/attune/packs:ro
# - runtime_envs:/opt/attune/runtime_envs
# - artifacts_data:/opt/attune/artifacts
# healthcheck:
# test: ["CMD-SHELL", "pgrep -f attune-agent || exit 1"]
# interval: 30s
# timeout: 10s
# retries: 3
# start_period: 20s
# networks:
# - attune-network
# restart: unless-stopped

25
docker-compose.yaml
View File

@@ -126,6 +126,28 @@ services:
restart: on-failure
entrypoint: "" # Override Python image entrypoint
# Agent binary volume population (builds the statically-linked agent and copies it to a shared volume)
# Other containers can use the agent binary by mounting agent_bin volume and running /opt/attune/agent/attune-agent
init-agent:
build:
context: .
dockerfile: docker/Dockerfile.agent
target: agent-init
args:
BUILDKIT_INLINE_CACHE: 1
container_name: attune-init-agent
volumes:
- agent_bin:/opt/attune/agent
entrypoint:
[
"/bin/sh",
"-c",
"cp /usr/local/bin/attune-agent /opt/attune/agent/attune-agent && chmod +x /opt/attune/agent/attune-agent && echo 'Agent binary copied successfully'",
]
restart: "no"
networks:
- attune-network
rabbitmq:
image: rabbitmq:3.13-management-alpine
container_name: attune-rabbitmq
@@ -199,6 +221,7 @@ services:
- runtime_envs:/opt/attune/runtime_envs
- artifacts_data:/opt/attune/artifacts
- api_logs:/opt/attune/logs
- agent_bin:/opt/attune/agent:ro
depends_on:
init-packs:
condition: service_completed_successfully
@@ -621,6 +644,8 @@ volumes:
driver: local
artifacts_data:
driver: local
agent_bin:
driver: local
# ============================================================================
# Networks

153
docker/Dockerfile.agent Normal file
View File

@@ -0,0 +1,153 @@
# Multi-stage Dockerfile for the Attune universal worker agent
#
# Builds a statically-linked attune-agent binary using musl, suitable for
# injection into ANY container as a sidecar or init container. The binary
# has zero runtime dependencies — no glibc, no libssl, no shared libraries.
#
# Stages:
# builder - Cross-compile with musl for a fully static binary
# agent-binary - Minimal scratch image containing just the binary
# agent-init - BusyBox-based image for use as a Kubernetes init container
# or Docker Compose volume-populating service (has `cp`)
#
# Usage:
# # Build the minimal binary-only image:
# DOCKER_BUILDKIT=1 docker buildx build --target agent-binary -f docker/Dockerfile.agent -t attune-agent:binary .
#
# # Build the init container image (for volume population via `cp`):
# DOCKER_BUILDKIT=1 docker buildx build --target agent-init -f docker/Dockerfile.agent -t attune-agent:latest .
#
# # Use in docker-compose.yaml to populate a shared volume:
# # agent-init:
# # image: attune-agent:latest
# # command: ["cp", "/usr/local/bin/attune-agent", "/shared/attune-agent"]
# # volumes:
# # - agent_binary:/shared
#
# Note: The agent binary is part of the worker crate (--bin attune-agent).
# It connects to the Attune API and executes actions inside the target container.
ARG RUST_VERSION=1.92
ARG DEBIAN_VERSION=bookworm
# ============================================================================
# Stage 1: Builder - Cross-compile a statically-linked binary with musl
# ============================================================================
FROM rust:${RUST_VERSION}-${DEBIAN_VERSION} AS builder
# Install musl toolchain for static linking
RUN apt-get update && apt-get install -y \
musl-tools \
pkg-config \
libssl-dev \
ca-certificates \
&& rm -rf /var/lib/apt/lists/*
# Add the musl target for fully static binaries
RUN rustup target add x86_64-unknown-linux-musl
WORKDIR /build
# Increase rustc stack size to prevent SIGSEGV during release builds
ENV RUST_MIN_STACK=67108864
# Enable SQLx offline mode — compile-time query checking without a live database
ENV SQLX_OFFLINE=true
# ---------------------------------------------------------------------------
# Dependency caching layer
# Copy only Cargo metadata first so `cargo fetch` is cached when only source
# code changes. This follows the same pattern as Dockerfile.worker.optimized.
# ---------------------------------------------------------------------------
COPY Cargo.toml Cargo.lock ./
COPY crates/common/Cargo.toml ./crates/common/Cargo.toml
COPY crates/api/Cargo.toml ./crates/api/Cargo.toml
COPY crates/executor/Cargo.toml ./crates/executor/Cargo.toml
COPY crates/sensor/Cargo.toml ./crates/sensor/Cargo.toml
COPY crates/core-timer-sensor/Cargo.toml ./crates/core-timer-sensor/Cargo.toml
COPY crates/worker/Cargo.toml ./crates/worker/Cargo.toml
COPY crates/notifier/Cargo.toml ./crates/notifier/Cargo.toml
COPY crates/cli/Cargo.toml ./crates/cli/Cargo.toml
# Create minimal stub sources so cargo can resolve the workspace and fetch deps.
# These are ONLY used for `cargo fetch` — never compiled.
# NOTE: The worker crate has TWO binary targets (attune-worker and attune-agent),
# so we create stubs for both to satisfy the workspace resolver.
RUN mkdir -p crates/common/src && echo "" > crates/common/src/lib.rs && \
mkdir -p crates/api/src && echo "fn main(){}" > crates/api/src/main.rs && \
mkdir -p crates/executor/src && echo "fn main(){}" > crates/executor/src/main.rs && \
mkdir -p crates/executor/benches && echo "fn main(){}" > crates/executor/benches/context_clone.rs && \
mkdir -p crates/sensor/src && echo "fn main(){}" > crates/sensor/src/main.rs && \
mkdir -p crates/core-timer-sensor/src && echo "fn main(){}" > crates/core-timer-sensor/src/main.rs && \
mkdir -p crates/worker/src && echo "fn main(){}" > crates/worker/src/main.rs && \
echo "fn main(){}" > crates/worker/src/agent_main.rs && \
mkdir -p crates/notifier/src && echo "fn main(){}" > crates/notifier/src/main.rs && \
mkdir -p crates/cli/src && echo "fn main(){}" > crates/cli/src/main.rs
# Download all dependencies (cached unless Cargo.toml/Cargo.lock change)
# registry/git use sharing=shared — cargo handles concurrent reads safely
RUN --mount=type=cache,target=/usr/local/cargo/registry,sharing=shared \
--mount=type=cache,target=/usr/local/cargo/git,sharing=shared \
cargo fetch
# ---------------------------------------------------------------------------
# Build layer
# Copy real source code and compile only the agent binary with musl
# ---------------------------------------------------------------------------
COPY migrations/ ./migrations/
COPY crates/ ./crates/
# Build ONLY the attune-agent binary, statically linked with musl.
# Uses a dedicated cache ID (agent-target) so the musl target directory
# doesn't collide with the glibc target cache used by other Dockerfiles.
RUN --mount=type=cache,target=/usr/local/cargo/registry,sharing=shared \
--mount=type=cache,target=/usr/local/cargo/git,sharing=shared \
--mount=type=cache,id=agent-target,target=/build/target,sharing=locked \
cargo build --release --target x86_64-unknown-linux-musl --bin attune-agent && \
cp /build/target/x86_64-unknown-linux-musl/release/attune-agent /build/attune-agent
# Strip the binary to minimize size
RUN strip /build/attune-agent
# Verify the binary is statically linked and functional
RUN ls -lh /build/attune-agent && \
file /build/attune-agent && \
ldd /build/attune-agent 2>&1 || true
# ============================================================================
# Stage 2: agent-binary - Minimal image with just the static binary
# ============================================================================
# This is the smallest possible image — a single static binary on scratch.
# Useful when you only need to extract the binary (e.g., via COPY --from).
FROM scratch AS agent-binary
COPY --from=builder /build/attune-agent /usr/local/bin/attune-agent
ENTRYPOINT ["/usr/local/bin/attune-agent"]
# ============================================================================
# Stage 3: agent-init - Init container for volume population
# ============================================================================
# Uses busybox so we have `cp`, `sh`, etc. for use as a Docker Compose or
# Kubernetes init container that copies the agent binary into a shared volume.
#
# Example docker-compose.yaml usage:
# agent-init:
# image: attune-agent:latest
# command: ["cp", "/usr/local/bin/attune-agent", "/shared/attune-agent"]
# volumes:
# - agent_binary:/shared
#
# my-worker-container:
# image: python:3.12
# command: ["/agent/attune-agent"]
# volumes:
# - agent_binary:/agent:ro
# depends_on:
# agent-init:
# condition: service_completed_successfully
FROM busybox:1.36 AS agent-init
COPY --from=builder /build/attune-agent /usr/local/bin/attune-agent
ENTRYPOINT ["/usr/local/bin/attune-agent"]

1
docker/Dockerfile.optimized
View File

@@ -51,6 +51,7 @@ RUN mkdir -p crates/common/src && echo "" > crates/common/src/lib.rs && \
mkdir -p crates/sensor/src && echo "fn main(){}" > crates/sensor/src/main.rs && \
mkdir -p crates/core-timer-sensor/src && echo "fn main(){}" > crates/core-timer-sensor/src/main.rs && \
mkdir -p crates/worker/src && echo "fn main(){}" > crates/worker/src/main.rs && \
echo "fn main(){}" > crates/worker/src/agent_main.rs && \
mkdir -p crates/notifier/src && echo "fn main(){}" > crates/notifier/src/main.rs && \
mkdir -p crates/cli/src && echo "fn main(){}" > crates/cli/src/main.rs

1
docker/Dockerfile.pack-binaries
View File

@@ -51,6 +51,7 @@ RUN mkdir -p crates/executor/src && echo "fn main() {}" > crates/executor/src/ma
RUN mkdir -p crates/executor/benches && echo "fn main() {}" > crates/executor/benches/context_clone.rs
RUN mkdir -p crates/sensor/src && echo "fn main() {}" > crates/sensor/src/main.rs
RUN mkdir -p crates/worker/src && echo "fn main() {}" > crates/worker/src/main.rs
RUN echo "fn main() {}" > crates/worker/src/agent_main.rs
RUN mkdir -p crates/notifier/src && echo "fn main() {}" > crates/notifier/src/main.rs
RUN mkdir -p crates/cli/src && echo "fn main() {}" > crates/cli/src/main.rs

1
docker/Dockerfile.sensor.optimized
View File

@@ -54,6 +54,7 @@ RUN mkdir -p crates/common/src && echo "" > crates/common/src/lib.rs && \
mkdir -p crates/sensor/src && echo "fn main(){}" > crates/sensor/src/main.rs && \
mkdir -p crates/core-timer-sensor/src && echo "fn main(){}" > crates/core-timer-sensor/src/main.rs && \
mkdir -p crates/worker/src && echo "fn main(){}" > crates/worker/src/main.rs && \
echo "fn main(){}" > crates/worker/src/agent_main.rs && \
mkdir -p crates/notifier/src && echo "fn main(){}" > crates/notifier/src/main.rs && \
mkdir -p crates/cli/src && echo "fn main(){}" > crates/cli/src/main.rs

1
docker/Dockerfile.worker.optimized
View File

@@ -59,6 +59,7 @@ RUN mkdir -p crates/common/src && echo "" > crates/common/src/lib.rs && \
mkdir -p crates/sensor/src && echo "fn main(){}" > crates/sensor/src/main.rs && \
mkdir -p crates/core-timer-sensor/src && echo "fn main(){}" > crates/core-timer-sensor/src/main.rs && \
mkdir -p crates/worker/src && echo "fn main(){}" > crates/worker/src/main.rs && \
echo "fn main(){}" > crates/worker/src/agent_main.rs && \
mkdir -p crates/notifier/src && echo "fn main(){}" > crates/notifier/src/main.rs && \
mkdir -p crates/cli/src && echo "fn main(){}" > crates/cli/src/main.rs

219
docs/QUICKREF-agent-workers.md Normal file
View File

@@ -0,0 +1,219 @@
# Quick Reference: Agent-Based Workers
> **TL;DR**: Inject the `attune-agent` binary into _any_ container image to turn it into an Attune worker. No Dockerfiles. No Rust compilation. ~12 lines of YAML.
## How It Works
1. The `init-agent` service (in `docker-compose.yaml`) builds the statically-linked `attune-agent` binary and copies it into the `agent_bin` volume
2. Your worker service mounts `agent_bin` read-only and uses the agent as its entrypoint
3. On startup, the agent auto-detects available runtimes (Python, Ruby, Node.js, Shell, etc.)
4. The worker registers with Attune and starts processing executions
## Quick Start
### Option A: Use the override file
```bash
# Start all services including the example Ruby agent worker
docker compose -f docker-compose.yaml -f docker-compose.agent.yaml up -d
```
The `docker-compose.agent.yaml` file includes a ready-to-use Ruby worker and commented-out templates for Python 3.12, GPU, and custom images.
### Option B: Add to docker-compose.override.yaml
Create a `docker-compose.override.yaml` in the project root:
```yaml
services:
worker-my-runtime:
image: my-org/my-custom-image:latest
container_name: attune-worker-my-runtime
depends_on:
init-agent:
condition: service_completed_successfully
init-packs:
condition: service_completed_successfully
migrations:
condition: service_completed_successfully
postgres:
condition: service_healthy
rabbitmq:
condition: service_healthy
entrypoint: ["/opt/attune/agent/attune-agent"]
stop_grace_period: 45s
environment:
RUST_LOG: info
ATTUNE_CONFIG: /opt/attune/config/config.yaml
ATTUNE_WORKER_NAME: worker-my-runtime-01
ATTUNE_WORKER_TYPE: container
ATTUNE__SECURITY__JWT_SECRET: ${JWT_SECRET:-docker-dev-secret-change-in-production}
ATTUNE__SECURITY__ENCRYPTION_KEY: ${ENCRYPTION_KEY:-docker-dev-encryption-key-please-change-in-production-32plus}
ATTUNE__DATABASE__URL: postgresql://attune:attune@postgres:5432/attune
ATTUNE__MESSAGE_QUEUE__URL: amqp://attune:attune@rabbitmq:5672
ATTUNE_API_URL: http://attune-api:8080
volumes:
- agent_bin:/opt/attune/agent:ro
- ${ATTUNE_DOCKER_CONFIG_PATH:-./config.docker.yaml}:/opt/attune/config/config.yaml:ro
- packs_data:/opt/attune/packs:ro
- runtime_envs:/opt/attune/runtime_envs
- artifacts_data:/opt/attune/artifacts
healthcheck:
test: ["CMD-SHELL", "pgrep -f attune-agent || exit 1"]
interval: 30s
timeout: 10s
retries: 3
start_period: 20s
networks:
- attune-network
restart: unless-stopped
```
Then run:
```bash
docker compose up -d
```
Docker Compose automatically merges `docker-compose.override.yaml`.
## Required Volumes
Every agent worker needs these volumes:
| Volume | Mount Path | Mode | Purpose |
|--------|-----------|------|---------|
| `agent_bin` | `/opt/attune/agent` | `ro` | The statically-linked agent binary |
| `packs_data` | `/opt/attune/packs` | `ro` | Pack files (actions, workflows, etc.) |
| `runtime_envs` | `/opt/attune/runtime_envs` | `rw` | Isolated runtime environments (venvs, node_modules) |
| `artifacts_data` | `/opt/attune/artifacts` | `rw` | File-backed artifact storage |
| Config YAML | `/opt/attune/config/config.yaml` | `ro` | Attune configuration |
## Required Environment Variables
| Variable | Description | Example |
|----------|-------------|---------|
| `ATTUNE_CONFIG` | Path to config file inside container | `/opt/attune/config/config.yaml` |
| `ATTUNE_WORKER_NAME` | Unique worker name | `worker-ruby-01` |
| `ATTUNE_WORKER_TYPE` | Worker type | `container` |
| `ATTUNE__DATABASE__URL` | PostgreSQL connection string | `postgresql://attune:attune@postgres:5432/attune` |
| `ATTUNE__MESSAGE_QUEUE__URL` | RabbitMQ connection string | `amqp://attune:attune@rabbitmq:5672` |
| `ATTUNE__SECURITY__JWT_SECRET` | JWT signing secret | (use env var) |
| `ATTUNE__SECURITY__ENCRYPTION_KEY` | Encryption key for secrets | (use env var) |
### Optional Environment Variables
| Variable | Description | Default |
|----------|-------------|---------|
| `ATTUNE_WORKER_RUNTIMES` | Override auto-detection | Auto-detected |
| `ATTUNE_API_URL` | API URL for token generation | `http://attune-api:8080` |
| `RUST_LOG` | Log level | `info` |
## Runtime Auto-Detection
The agent probes for these runtimes automatically:
| Runtime | Probed Binaries |
|---------|----------------|
| Shell | `bash`, `sh` |
| Python | `python3`, `python` |
| Node.js | `node`, `nodejs` |
| Ruby | `ruby` |
| Go | `go` |
| Java | `java` |
| R | `Rscript` |
| Perl | `perl` |
To override, set `ATTUNE_WORKER_RUNTIMES`:
```yaml
environment:
ATTUNE_WORKER_RUNTIMES: python,shell # Only advertise Python and Shell
```
## Testing Detection
Run the agent in detect-only mode to see what it finds:
```bash
# In a running container
docker exec <container> /opt/attune/agent/attune-agent --detect-only
# Or start a throwaway container
docker run --rm -v agent_bin:/opt/attune/agent:ro ruby:3.3-slim /opt/attune/agent/attune-agent --detect-only
```
## Examples
### Ruby Worker
```yaml
worker-ruby:
image: ruby:3.3-slim
entrypoint: ["/opt/attune/agent/attune-agent"]
# ... (standard depends_on, volumes, env, networks)
```
### Node.js 22 Worker
```yaml
worker-node22:
image: node:22-slim
entrypoint: ["/opt/attune/agent/attune-agent"]
# ...
```
### GPU Worker (NVIDIA CUDA)
```yaml
worker-gpu:
image: nvidia/cuda:12.3.1-runtime-ubuntu22.04
runtime: nvidia
entrypoint: ["/opt/attune/agent/attune-agent"]
environment:
ATTUNE_WORKER_RUNTIMES: python,shell # Override — CUDA image has python
# ...
```
### Multi-Runtime Custom Image
```yaml
worker-data-science:
image: my-org/data-science:latest # Has Python, R, and Julia
entrypoint: ["/opt/attune/agent/attune-agent"]
# Agent auto-detects all available runtimes
# ...
```
## Comparison: Traditional vs Agent Workers
| Aspect | Traditional Worker | Agent Worker |
|--------|-------------------|--------------|
| Docker build | Required (5+ min) | None |
| Dockerfile | Custom per runtime | Not needed |
| Base image | `debian:bookworm-slim` | Any image |
| Runtime install | Via apt/NodeSource | Pre-installed in image |
| Configuration | Manual `ATTUNE_WORKER_RUNTIMES` | Auto-detected |
| Binary | Compiled into image | Injected via volume |
| Update cycle | Rebuild image | Restart `init-agent` |
## Troubleshooting
### Agent binary not found
```
exec /opt/attune/agent/attune-agent: no such file or directory
```
The `init-agent` service hasn't completed. Check:
```bash
docker compose logs init-agent
```
### "No runtimes detected"
The container image doesn't have any recognized interpreters in `$PATH`. Either:
- Use an image that includes your runtime (e.g., `ruby:3.3-slim`)
- Set `ATTUNE_WORKER_RUNTIMES` manually
### Connection refused to PostgreSQL/RabbitMQ
Ensure your `depends_on` conditions include `postgres` and `rabbitmq` health checks, and that the container is on the `attune-network`.
## See Also
- [Universal Worker Agent Plan](plans/universal-worker-agent.md) — Full architecture document
- [Docker Deployment](docker-deployment.md) — General Docker setup
- [Worker Service](architecture/worker-service.md) — Worker architecture details

146
docs/QUICKREF-kubernetes-agent-workers.md Normal file
View File

@@ -0,0 +1,146 @@
# Quick Reference: Kubernetes Agent Workers
Agent-based workers let you run Attune actions inside **any container image** by injecting a statically-linked `attune-agent` binary via a Kubernetes init container. No custom Dockerfile required — just point at an image that has your runtime installed.
## How It Works
1. An **init container** (`agent-loader`) copies the `attune-agent` binary from the `attune-agent` image into an `emptyDir` volume
2. The **worker container** uses your chosen image (e.g., `ruby:3.3`) and runs the agent binary as its entrypoint
3. The agent **auto-detects** available runtimes (python, ruby, node, shell, etc.) and registers with Attune
4. Actions targeting those runtimes are routed to the agent worker via RabbitMQ
## Helm Values
Add entries to `agentWorkers` in your `values.yaml`:
```yaml
agentWorkers:
- name: ruby
image: ruby:3.3
replicas: 2
- name: python-gpu
image: nvidia/cuda:12.3.1-runtime-ubuntu22.04
replicas: 1
runtimes: [python, shell]
runtimeClassName: nvidia
nodeSelector:
gpu: "true"
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
resources:
limits:
nvidia.com/gpu: 1
- name: custom
image: my-org/my-custom-image:latest
replicas: 1
env:
- name: MY_CUSTOM_VAR
value: my-value
```
### Supported Fields
| Field | Required | Default | Description |
|-------|----------|---------|-------------|
| `name` | Yes | — | Unique name (used in Deployment and worker names) |
| `image` | Yes | — | Container image with your desired runtime(s) |
| `replicas` | No | `1` | Number of pod replicas |
| `runtimes` | No | `[]` (auto-detect) | List of runtimes to expose (e.g., `[python, shell]`) |
| `resources` | No | `{}` | Kubernetes resource requests/limits |
| `env` | No | — | Extra environment variables (`[{name, value}]`) |
| `imagePullPolicy` | No | — | Pull policy for the worker image |
| `logLevel` | No | `info` | `RUST_LOG` level |
| `runtimeClassName` | No | — | Kubernetes RuntimeClass (e.g., `nvidia`) |
| `nodeSelector` | No | — | Node selector for pod scheduling |
| `tolerations` | No | — | Tolerations for pod scheduling |
| `stopGracePeriod` | No | `45` | Termination grace period (seconds) |
## Install / Upgrade
```bash
helm upgrade --install attune oci://registry.example.com/namespace/helm/attune \
--version 0.3.0 \
--set global.imageRegistry=registry.example.com \
--set global.imageNamespace=namespace \
--set global.imageTag=0.3.0 \
-f my-values.yaml
```
## What Gets Created
For each `agentWorkers` entry, the chart creates a Deployment named `<release>-attune-agent-worker-<name>` with:
- **Init containers**:
- `agent-loader` — copies the agent binary from the `attune-agent` image to an `emptyDir` volume
- `wait-for-schema` — polls PostgreSQL until the Attune schema is ready
- `wait-for-packs` — waits for the core pack to be available on the shared PVC
- **Worker container** — runs `attune-agent` as the entrypoint inside your chosen image
- **Volumes**: `agent-bin` (emptyDir), `config` (ConfigMap), `packs` (PVC, read-only), `runtime-envs` (PVC), `artifacts` (PVC)
## Runtime Auto-Detection
When `runtimes` is empty (the default), the agent probes the container for interpreters:
| Runtime | Probed Binaries |
|---------|----------------|
| Shell | `bash`, `sh` |
| Python | `python3`, `python` |
| Node.js | `node`, `nodejs` |
| Ruby | `ruby` |
| Go | `go` |
| Java | `java` |
| R | `Rscript` |
| Perl | `perl` |
Set `runtimes` explicitly to skip auto-detection and only register the listed runtimes.
## Prerequisites
- The `attune-agent` image must be available in your registry (built from `docker/Dockerfile.agent`, target `agent-init`)
- Shared PVCs (`packs`, `runtime-envs`, `artifacts`) must support `ReadWriteMany` if agent workers run on different nodes than the standard worker
- The Attune database and RabbitMQ must be reachable from agent worker pods
## Differences from the Standard Worker
| Aspect | Standard Worker (`worker`) | Agent Worker (`agentWorkers`) |
|--------|---------------------------|-------------------------------|
| Image | Built from `Dockerfile.worker.optimized` | Any image (ruby, python, cuda, etc.) |
| Binary | Baked into the image | Injected via init container |
| Runtimes | Configured at build time | Auto-detected or explicitly listed |
| Use case | Known, pre-built runtime combos | Custom images, exotic runtimes, GPU |
Both worker types coexist — actions are routed to whichever worker has the matching runtime registered.
## Troubleshooting
**Agent binary not found**: Check that the `agent-loader` init container completed. View its logs:
```bash
kubectl logs <pod> -c agent-loader
```
**Runtime not detected**: Run the agent with `--detect-only` to see what it finds:
```bash
kubectl exec <pod> -c worker -- /opt/attune/agent/attune-agent --detect-only
```
**Worker not registering**: Check the worker container logs for database/MQ connectivity:
```bash
kubectl logs <pod> -c worker
```
**Packs not available**: Ensure the `init-packs` job has completed and the PVC is mounted:
```bash
kubectl get jobs | grep init-packs
kubectl exec <pod> -c worker -- ls /opt/attune/packs/core/
```
## See Also
- [Agent Workers (Docker Compose)](QUICKREF-agent-workers.md)
- [Universal Worker Agent Plan](plans/universal-worker-agent.md)
- [Gitea Registry and Helm](deployment/gitea-registry-and-helm.md)
- [Production Deployment](deployment/production-deployment.md)

1
docs/deployment/gitea-registry-and-helm.md
View File

@@ -19,6 +19,7 @@ The workflow publishes these images to the Gitea OCI registry:
- `attune-migrations`
- `attune-init-user`
- `attune-init-packs`
- `attune-agent`
The Helm chart is pushed as an OCI chart to:

1070
docs/plans/universal-worker-agent.md Normal file
View File

File diff suppressed because it is too large Load Diff

15
migrations/20250101000002_pack_system.sql
View File

@@ -131,6 +131,17 @@ CREATE TABLE runtime (
-- {manifest_path} - absolute path to the dependency manifest file
execution_config JSONB NOT NULL DEFAULT '{}'::jsonb,
-- Whether this runtime was auto-registered by an agent
-- (vs. loaded from a pack's YAML file during pack registration)
auto_detected BOOLEAN NOT NULL DEFAULT FALSE,
-- Detection metadata for auto-discovered runtimes.
-- Stores how the agent discovered this runtime (binary path, version, etc.)
-- enables re-verification on restart.
-- Example: { "detected_path": "/usr/bin/ruby", "detected_name": "ruby",
-- "detected_version": "3.3.0" }
detection_config JSONB NOT NULL DEFAULT '{}'::jsonb,
created TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated TIMESTAMPTZ NOT NULL DEFAULT NOW(),
@@ -145,6 +156,8 @@ CREATE INDEX idx_runtime_created ON runtime(created DESC);
CREATE INDEX idx_runtime_name ON runtime(name);
CREATE INDEX idx_runtime_verification ON runtime USING GIN ((distributions->'verification'));
CREATE INDEX idx_runtime_execution_config ON runtime USING GIN (execution_config);
CREATE INDEX idx_runtime_auto_detected ON runtime(auto_detected);
CREATE INDEX idx_runtime_detection_config ON runtime USING GIN (detection_config);
-- Trigger
CREATE TRIGGER update_runtime_updated
@@ -160,6 +173,8 @@ COMMENT ON COLUMN runtime.distributions IS 'Runtime distribution metadata includ
COMMENT ON COLUMN runtime.installation IS 'Installation requirements and instructions including package managers and setup steps';
COMMENT ON COLUMN runtime.installers IS 'Array of installer actions to create pack-specific runtime environments. Each installer defines commands to set up isolated environments (e.g., Python venv, npm install).';
COMMENT ON COLUMN runtime.execution_config IS 'Execution configuration: interpreter, environment setup, and dependency management. Drives how the worker executes actions and how pack install sets up environments.';
COMMENT ON COLUMN runtime.auto_detected IS 'Whether this runtime was auto-registered by an agent (true) vs. loaded from a pack YAML (false)';
COMMENT ON COLUMN runtime.detection_config IS 'Detection metadata for auto-discovered runtimes: binaries probed, version regex, detected path/version';
-- ============================================================================
-- RUNTIME VERSION TABLE

47
packs/core/runtimes/go.yaml Normal file
View File

@@ -0,0 +1,47 @@
ref: core.go
pack_ref: core
name: Go
description: Go runtime for compiling and running Go scripts and programs
distributions:
verification:
commands:
- binary: go
args:
- "version"
exit_code: 0
pattern: "go\\d+\\."
priority: 1
min_version: "1.18"
recommended_version: "1.22"
installation:
package_managers:
- apt
- snap
- brew
module_support: true
execution_config:
interpreter:
binary: go
args:
- "run"
file_extension: ".go"
environment:
env_type: gopath
dir_name: gopath
create_command:
- sh
- "-c"
- "mkdir -p {env_dir}"
interpreter_path: null
dependencies:
manifest_file: go.mod
install_command:
- sh
- "-c"
- "cd {pack_dir} && GOPATH={env_dir} go mod download 2>/dev/null || true"
env_vars:
GOPATH: "{env_dir}"
GOMODCACHE: "{env_dir}/pkg/mod"

30
packs/core/runtimes/java.yaml Normal file
View File

@@ -0,0 +1,30 @@
ref: core.java
pack_ref: core
name: Java
description: Java runtime for executing Java programs and scripts
distributions:
verification:
commands:
- binary: java
args:
- "-version"
exit_code: 0
pattern: "version \"\\d+"
priority: 1
min_version: "11"
recommended_version: "21"
installation:
interpreters:
- java
- javac
package_managers:
- maven
- gradle
execution_config:
interpreter:
binary: java
args: []
file_extension: ".java"

46
packs/core/runtimes/perl.yaml Normal file
View File

@@ -0,0 +1,46 @@
ref: core.perl
pack_ref: core
name: Perl
description: Perl runtime for script execution with optional CPAN dependency management
distributions:
verification:
commands:
- binary: perl
args:
- "--version"
exit_code: 0
pattern: "perl.*v\\d+\\."
priority: 1
min_version: "5.20"
recommended_version: "5.38"
installation:
package_managers:
- cpanm
- cpan
interpreters:
- perl
execution_config:
interpreter:
binary: perl
args: []
file_extension: ".pl"
environment:
env_type: local_lib
dir_name: perl5
create_command:
- sh
- "-c"
- "mkdir -p {env_dir}/lib/perl5"
interpreter_path: null
dependencies:
manifest_file: cpanfile
install_command:
- sh
- "-c"
- "cd {pack_dir} && PERL5LIB={env_dir}/lib/perl5 PERL_LOCAL_LIB_ROOT={env_dir} cpanm --local-lib {env_dir} --installdeps --quiet . 2>/dev/null || true"
env_vars:
PERL5LIB: "{env_dir}/lib/perl5"
PERL_LOCAL_LIB_ROOT: "{env_dir}"

47
packs/core/runtimes/r.yaml Normal file
View File

@@ -0,0 +1,47 @@
ref: core.r
pack_ref: core
name: R
description: R runtime for statistical computing and data analysis scripts
distributions:
verification:
commands:
- binary: Rscript
args:
- "--version"
exit_code: 0
pattern: "\\d+\\.\\d+\\.\\d+"
priority: 1
min_version: "4.0.0"
recommended_version: "4.4.0"
installation:
package_managers:
- install.packages
- renv
interpreters:
- Rscript
- R
execution_config:
interpreter:
binary: Rscript
args:
- "--vanilla"
file_extension: ".R"
environment:
env_type: renv
dir_name: renv
create_command:
- sh
- "-c"
- "mkdir -p {env_dir}/library"
interpreter_path: null
dependencies:
manifest_file: renv.lock
install_command:
- sh
- "-c"
- "cd {pack_dir} && R_LIBS_USER={env_dir}/library Rscript -e \"if (file.exists('renv.lock')) renv::restore(library='{env_dir}/library', prompt=FALSE)\" 2>/dev/null || true"
env_vars:
R_LIBS_USER: "{env_dir}/library"

48
packs/core/runtimes/ruby.yaml Normal file
View File

@@ -0,0 +1,48 @@
ref: core.ruby
pack_ref: core
name: Ruby
description: Ruby runtime for script execution with automatic gem environment management
distributions:
verification:
commands:
- binary: ruby
args:
- "--version"
exit_code: 0
pattern: "ruby \\d+\\."
priority: 1
min_version: "2.7"
recommended_version: "3.2"
installation:
package_managers:
- gem
- bundler
interpreters:
- ruby
portable: false
execution_config:
interpreter:
binary: ruby
args: []
file_extension: ".rb"
environment:
env_type: gem_home
dir_name: gems
create_command:
- sh
- "-c"
- "mkdir -p {env_dir}/gems"
interpreter_path: null
dependencies:
manifest_file: Gemfile
install_command:
- sh
- "-c"
- "cd {pack_dir} && GEM_HOME={env_dir}/gems GEM_PATH={env_dir}/gems bundle install --quiet 2>/dev/null || true"
env_vars:
GEM_HOME: "{env_dir}/gems"
GEM_PATH: "{env_dir}/gems"
BUNDLE_PATH: "{env_dir}/gems"

122
scripts/attune-agent-wrapper.sh Executable file
View File

@@ -0,0 +1,122 @@
#!/bin/sh
# attune-agent-wrapper.sh — Bootstrap the Attune agent in any container
#
# This script provides a simple way to start the Attune universal worker agent
# in containers where the agent binary isn't available via a shared volume.
# It first checks for a volume-mounted binary, then falls back to downloading
# it from the Attune API.
#
# Usage:
# As a container entrypoint:
# entrypoint: ["/opt/attune/scripts/attune-agent-wrapper.sh"]
#
# With Docker Compose volume mount:
# volumes:
# - ./scripts/attune-agent-wrapper.sh:/opt/attune/scripts/attune-agent-wrapper.sh:ro
#
# Environment Variables:
# ATTUNE_AGENT_DIR - Directory for the agent binary (default: /opt/attune/agent)
# ATTUNE_AGENT_URL - URL to download the agent binary from
# (default: http://attune-api:8080/api/v1/agent/binary)
# ATTUNE_AGENT_TOKEN - Optional bootstrap token for authenticated downloads
# ATTUNE_AGENT_ARCH - Target architecture (default: auto-detected via uname -m)
#
set -e
AGENT_DIR="${ATTUNE_AGENT_DIR:-/opt/attune/agent}"
AGENT_BIN="$AGENT_DIR/attune-agent"
AGENT_URL="${ATTUNE_AGENT_URL:-http://attune-api:8080/api/v1/agent/binary}"
# SECURITY: The default URL uses plain HTTP, which is fine for internal Docker
# networking. For cross-network or production deployments, set ATTUNE_AGENT_URL
# to an HTTPS endpoint and consider setting ATTUNE_AGENT_TOKEN to authenticate.
AGENT_TOKEN="${ATTUNE_AGENT_TOKEN:-}"
# Auto-detect architecture if not specified
if [ -z "$ATTUNE_AGENT_ARCH" ]; then
MACHINE=$(uname -m)
case "$MACHINE" in
x86_64|amd64) ATTUNE_AGENT_ARCH="x86_64" ;;
aarch64|arm64) ATTUNE_AGENT_ARCH="aarch64" ;;
*)
echo "[attune] WARNING: Unknown architecture '$MACHINE', defaulting to x86_64" >&2
ATTUNE_AGENT_ARCH="x86_64"
;;
esac
fi
# Use volume-mounted binary if available
if [ -x "$AGENT_BIN" ]; then
echo "[attune] Agent binary found at $AGENT_BIN"
exec "$AGENT_BIN" "$@"
fi
# Download the agent binary
echo "[attune] Agent binary not found at $AGENT_BIN, downloading..."
echo "[attune] URL: $AGENT_URL"
echo "[attune] Architecture: $ATTUNE_AGENT_ARCH"
DOWNLOAD_URL="${AGENT_URL}?arch=${ATTUNE_AGENT_ARCH}"
mkdir -p "$AGENT_DIR"
# Build auth header if token is provided
AUTH_HEADER=""
if [ -n "$AGENT_TOKEN" ]; then
AUTH_HEADER="X-Agent-Token: $AGENT_TOKEN"
fi
# Download with retries (agent might start before API is ready)
MAX_RETRIES=10
RETRY_DELAY=5
ATTEMPT=0
while [ $ATTEMPT -lt $MAX_RETRIES ]; do
ATTEMPT=$((ATTEMPT + 1))
if command -v curl >/dev/null 2>&1; then
if [ -n "$AUTH_HEADER" ]; then
if curl -fsSL --retry 3 --retry-delay 2 -o "$AGENT_BIN" -H "$AUTH_HEADER" "$DOWNLOAD_URL" 2>/dev/null; then
break
fi
else
if curl -fsSL --retry 3 --retry-delay 2 -o "$AGENT_BIN" "$DOWNLOAD_URL" 2>/dev/null; then
break
fi
fi
elif command -v wget >/dev/null 2>&1; then
if [ -n "$AUTH_HEADER" ]; then
if wget -q -O "$AGENT_BIN" --header="$AUTH_HEADER" "$DOWNLOAD_URL" 2>/dev/null; then
break
fi
else
if wget -q -O "$AGENT_BIN" "$DOWNLOAD_URL" 2>/dev/null; then
break
fi
fi
else
echo "[attune] ERROR: Neither curl nor wget available. Cannot download agent." >&2
echo "[attune] Install curl or wget, or mount the agent binary via volume." >&2
exit 1
fi
if [ $ATTEMPT -lt $MAX_RETRIES ]; then
echo "[attune] Download attempt $ATTEMPT/$MAX_RETRIES failed, retrying in ${RETRY_DELAY}s..."
sleep $RETRY_DELAY
else
echo "[attune] ERROR: Failed to download agent binary after $MAX_RETRIES attempts." >&2
echo "[attune] Check that the API is running and ATTUNE_AGENT_URL is correct." >&2
exit 1
fi
done
chmod +x "$AGENT_BIN"
# Verify the binary works
if ! "$AGENT_BIN" --version >/dev/null 2>&1; then
echo "[attune] WARNING: Downloaded binary may not be compatible with this system." >&2
echo "[attune] Architecture: $ATTUNE_AGENT_ARCH ($(uname -m))" >&2
echo "[attune] File type: $(file "$AGENT_BIN" 2>/dev/null || echo 'unknown')" >&2
fi
echo "[attune] Agent binary downloaded successfully ($(wc -c < "$AGENT_BIN") bytes)."
echo "[attune] Starting agent..."
exec "$AGENT_BIN" "$@"

65
work-summary/2026-02-05-agent-phase2-registration-integration.md Normal file
View File

@@ -0,0 +1,65 @@
# Universal Worker Agent Phase 2: Runtime Detection ↔ Worker Registration Integration
**Date**: 2026-02-05
## Summary
Integrated the Phase 1 runtime auto-detection module with the worker registration system so that `attune-agent` workers register with rich interpreter metadata (binary paths, versions) in their capabilities, enabling the system to distinguish agents from standard workers and know exactly which interpreters are available and where.
## Changes
### 1. `crates/worker/src/runtime_detect.rs`
- Added `Serialize` and `Deserialize` derives to `DetectedRuntime` so instances can be stored as structured JSON in worker capabilities.
### 2. `crates/worker/src/registration.rs`
- Added `use crate::runtime_detect::DetectedRuntime` import.
- Added `set_detected_runtimes(&mut self, runtimes: Vec<DetectedRuntime>)` method that stores detected interpreter metadata under the `detected_interpreters` capability key as a JSON array of `{name, path, version}` objects.
- Added `set_agent_mode(&mut self, is_agent: bool)` method that sets an `agent_mode` boolean capability to distinguish agent workers from standard workers.
- Both methods are additive — the existing `runtimes` string list capability remains for backward compatibility.
### 3. `crates/worker/src/service.rs`
- Added `detected_runtimes: Option<Vec<DetectedRuntime>>` field to `WorkerService` (initialized to `None` in `new()`).
- Added `pub fn with_detected_runtimes(mut self, runtimes: Vec<DetectedRuntime>) -> Self` builder method that stores agent detection results for use during `start()`.
- Updated `start()` to call `registration.set_detected_runtimes()` and `registration.set_agent_mode(true)` before `register()` when detected runtimes are present.
- Standard `attune-worker` binary is completely unaffected — the field stays `None` and no agent-specific code runs.
### 4. `crates/worker/src/agent_main.rs`
- Added `agent_detected_runtimes: Option<Vec<DetectedRuntime>>` variable to stash detection results.
- After auto-detection runs and sets `ATTUNE_WORKER_RUNTIMES`, the detected `Vec` is saved into `agent_detected_runtimes`.
- After `WorkerService::new()`, calls `.with_detected_runtimes(detected)` if auto-detection ran, so the registration includes full interpreter metadata.
### 5. `crates/worker/src/lib.rs`
- Added `pub use runtime_detect::DetectedRuntime` re-export for convenient access.
## Capability Format
After Phase 2, an agent worker's `capabilities` JSON in the `worker` table looks like:
```json
{
"runtimes": ["shell", "python", "node"],
"max_concurrent_executions": 10,
"worker_version": "0.1.0",
"agent_mode": true,
"detected_interpreters": [
{"name": "shell", "path": "/bin/bash", "version": "5.2.15"},
{"name": "python", "path": "/usr/bin/python3", "version": "3.12.1"},
{"name": "node", "path": "/usr/bin/node", "version": "20.11.0"}
]
}
```
Standard `attune-worker` instances do NOT have `agent_mode` or `detected_interpreters` keys.
## Design Decisions
- **Builder pattern** (`with_detected_runtimes`) rather than a separate constructor — keeps the API surface minimal and avoids duplicating `new()` logic.
- **Explicit `set_agent_mode`** separate from `set_detected_runtimes` — allows independent control, though in practice they're always called together for agents.
- **JSON serialization via `serde_json::json!()` macro** rather than `serde_json::to_value(&runtimes)` — gives explicit control over the capability shape and avoids coupling the DB format to the Rust struct layout.
- **No changes to the `Worker` model or database schema** — `detected_interpreters` and `agent_mode` are stored inside the existing `capabilities` JSONB column.
## Verification
- `cargo check --workspace` — zero errors, zero warnings
- `cargo test -p attune-worker` — 139 tests pass (105 unit + 17 dependency isolation + 8 log truncation + 7 security + 2 doc-tests)
- Standard `attune-worker` binary path is completely unchanged

69
work-summary/2026-02-05-universal-agent-phase2.md Normal file
View File

@@ -0,0 +1,69 @@
# Universal Worker Agent — Phase 2: Runtime Auto-Detection Integration
**Date**: 2026-02-05
## Overview
Implemented Phase 2 of the Universal Worker Agent plan (`docs/plans/universal-worker-agent.md`), which integrates the runtime auto-detection module (built in Phase 1) with the worker registration system. Agent workers now register with rich interpreter metadata — binary paths and versions — alongside the simple runtime name list used for backward compatibility.
## Changes
### 1. `crates/worker/src/runtime_detect.rs`
- Added `Serialize` and `Deserialize` derives to `DetectedRuntime` so detection results can be stored as JSON in worker capabilities.
### 2. `crates/worker/src/registration.rs`
- Added `use crate::runtime_detect::DetectedRuntime` import.
- **`set_detected_runtimes(runtimes: Vec<DetectedRuntime>)`** — Stores interpreter metadata under the `detected_interpreters` capability key as a structured JSON array (each entry has `name`, `path`, `version`). This supplements the existing `runtimes` string list for backward compatibility.
- **`set_agent_mode(is_agent: bool)`** — Sets an `agent_mode` boolean capability so the system can distinguish agent-mode workers from standard workers.
### 3. `crates/worker/src/service.rs`
- Added `detected_runtimes: Option<Vec<DetectedRuntime>>` field to `WorkerService` (defaults to `None`).
- **`with_detected_runtimes(self, runtimes) -> Self`** — Builder method to pass agent-detected runtimes into the service. No-op for standard `attune-worker`.
- Updated `start()` to call `set_detected_runtimes()` + `set_agent_mode(true)` on the registration before `register()` when detected runtimes are present.
### 4. `crates/worker/src/agent_main.rs`
- Added `agent_detected_runtimes: Option<Vec<DetectedRuntime>>` variable to stash detection results.
- After auto-detection runs, the detected runtimes are saved (previously they were consumed by the env var setup and discarded).
- After `WorkerService::new()`, chains `.with_detected_runtimes(detected)` to pass the metadata through.
### 5. `crates/worker/src/lib.rs`
- Re-exported `DetectedRuntime` from `runtime_detect` module for external use.
## Worker Capabilities (Agent Mode)
When an `attune-agent` registers, its capabilities JSON now includes:
```json
{
"runtimes": ["shell", "python", "node"],
"detected_interpreters": [
{"name": "shell", "path": "/bin/bash", "version": "5.2.15"},
{"name": "python", "path": "/usr/bin/python3", "version": "3.12.1"},
{"name": "node", "path": "/usr/bin/node", "version": "20.11.0"}
],
"agent_mode": true,
"max_concurrent_executions": 10,
"worker_version": "0.1.0"
}
```
Standard `attune-worker` registrations are unchanged — no `detected_interpreters` or `agent_mode` keys.
## Phase 2 Sub-tasks
| Sub-task | Status | Notes |
|----------|--------|-------|
| 2.1 Interpreter Discovery Module | ✅ Done (Phase 1) | `runtime_detect.rs` already existed |
| 2.2 Integration with Worker Registration | ✅ Done | Rich capabilities + agent_mode flag |
| 2.3 Runtime Hints File | ⏭️ Deferred | Optional enhancement, not needed yet |
## Verification
- `cargo check --workspace` — zero errors, zero warnings
- `cargo test -p attune-worker` — all tests pass (unit, integration, doc-tests)
- No breaking changes to the standard `attune-worker` binary

72
work-summary/2026-02-05-universal-worker-agent-phase3.md Normal file
View File

@@ -0,0 +1,72 @@
# Universal Worker Agent — Phase 3: WorkerService Dual-Mode Refactor
**Date**: 2026-02-05
## Overview
Implemented Phase 3 of the Universal Worker Agent plan (`docs/plans/universal-worker-agent.md`): refactoring `WorkerService` for clean code reuse between the full `attune-worker` and the `attune-agent` binary, without code duplication.
## Changes
### 1. `StartupMode` Enum (`crates/worker/src/service.rs`)
Added a `StartupMode` enum that controls how the worker initializes its runtime environment:
- **`Worker`** — Full worker mode with proactive environment setup and full version verification sweep at startup. This is the existing behavior used by `attune-worker`.
- **`Agent { detected_runtimes }`** — Agent mode with lazy (on-demand) environment setup and deferred version verification. Used by `attune-agent`. Carries the auto-detected runtimes from Phase 2.
### 2. `WorkerService` Struct Refactoring (`crates/worker/src/service.rs`)
- Replaced the `detected_runtimes: Option<Vec<DetectedRuntime>>` field with `startup_mode: StartupMode`
- `new()` defaults to `StartupMode::Worker`
- `with_detected_runtimes()` now sets `StartupMode::Agent { detected_runtimes }` — the method signature is unchanged, so `agent_main.rs` requires no modifications
### 3. Conditional Startup in `start()` (`crates/worker/src/service.rs`)
The `start()` method now branches on `self.startup_mode`:
- **Worker mode**: Runs `verify_runtime_versions()` and `scan_and_setup_environments()` proactively (existing behavior, unchanged)
- **Agent mode**: Skips both with an info log — environments will be created lazily on first execution
Agent capability registration (`set_detected_runtimes()`, `set_agent_mode()`) also uses the `StartupMode` match instead of the old `Option` check.
### 4. Lazy Environment Setup (`crates/worker/src/runtime/process.rs`)
Updated `ProcessRuntime::execute()` to perform on-demand environment creation when the env directory is missing. Previously, a missing env dir produced a warning and fell back to the system interpreter. Now it:
1. Logs an info message about lazy setup
2. Creates a temporary `ProcessRuntime` with the effective config
3. Calls `setup_pack_environment()` to create the environment (venv, node_modules, etc.)
4. Falls back to the system interpreter only if creation fails
This is the primary code path for agent mode (where proactive startup setup is skipped) but also serves as a safety net for standard workers.
### 5. Re-export (`crates/worker/src/lib.rs`)
`StartupMode` is re-exported from the `attune_worker` crate root for external use.
## Files Modified
| File | Change |
|------|--------|
| `crates/worker/src/service.rs` | Added `StartupMode` enum, replaced `detected_runtimes` field, conditional startup logic |
| `crates/worker/src/runtime/process.rs` | Lazy on-demand environment creation in `execute()` |
| `crates/worker/src/lib.rs` | Re-export `StartupMode` |
| `AGENTS.md` | Updated development status (Phase 3 complete, Phases 4–7 in progress) |
## Test Results
All 139 tests pass:
- 105 unit tests
- 17 dependency isolation tests
- 8 log truncation tests
- 7 security tests
- 2 doc-tests
Zero compiler warnings across the workspace.
## Design Decisions
- **No code duplication**: The `StartupMode` enum parameterizes `WorkerService` rather than creating a separate `AgentService`. All execution machinery (runtimes, consumers, heartbeat, cancellation) is shared.
- **Lazy setup as safety net**: The on-demand environment creation in `ProcessRuntime::execute()` benefits both modes — agents rely on it as the primary path, while standard workers get it as a fallback if proactive setup missed something.
- **Backward compatible API**: `with_detected_runtimes()` keeps its signature, so `agent_main.rs` needed no changes.

77
work-summary/2026-02-agent-docker-compose.md Normal file
View File

@@ -0,0 +1,77 @@
# Universal Worker Agent: Phase 4 — Docker Compose Integration
**Date**: 2026-02-05
**Phase**: 4 of 7
**Scope**: Docker Compose integration for agent-based workers
## Summary
Added Docker Compose infrastructure to make it trivial to add agent-based workers to an Attune deployment. Users can now inject the statically-linked `attune-agent` binary into any container image via a shared volume, turning it into a fully functional Attune worker with auto-detected runtimes — no Dockerfiles, no Rust compilation.
## Changes
### docker-compose.yaml
- Added `init-agent` service between `init-packs` and `rabbitmq`
- Builds from `docker/Dockerfile.agent` (target: `agent-init`)
- Copies the statically-linked binary to the `agent_bin` volume at `/opt/attune/agent/attune-agent`
- Runs once (`restart: "no"`) and completes immediately
- Added `agent_bin` named volume to the volumes section
### docker-compose.agent.yaml (new)
- Override file with example agent-based worker services
- **Active (uncommented)**: `worker-ruby` using `ruby:3.3-slim`
- **Commented templates**: Python 3.12, NVIDIA CUDA GPU, and custom image workers
- All workers follow the same pattern: mount `agent_bin` read-only, use `attune-agent` as entrypoint, share standard volumes
### Makefile
- Added `docker-up-agent` target: `docker compose -f docker-compose.yaml -f docker-compose.agent.yaml up -d`
- Added `docker-down-agent` target: corresponding `down` command
- Updated `.PHONY` and help text
### docs/QUICKREF-agent-workers.md (new)
- Quick-reference guide for adding agent-based workers
- Covers: how it works, quick start (override file vs docker-compose.override.yaml), required volumes, required environment variables, runtime auto-detection, testing detection, examples (Ruby, Node.js, GPU, multi-runtime), comparison table (traditional vs agent workers), troubleshooting
## Usage
```bash
# Start everything including the Ruby agent worker
make docker-up-agent
# Or manually
docker compose -f docker-compose.yaml -f docker-compose.agent.yaml up -d
# Stop
make docker-down-agent
```
Adding a new runtime worker is ~12 lines of YAML in `docker-compose.override.yaml`:
```yaml
services:
worker-my-runtime:
image: my-org/my-image:latest
depends_on:
init-agent:
condition: service_completed_successfully
# ... standard health checks
entrypoint: ["/opt/attune/agent/attune-agent"]
volumes:
- agent_bin:/opt/attune/agent:ro
- packs_data:/opt/attune/packs:ro
- runtime_envs:/opt/attune/runtime_envs
- artifacts_data:/opt/attune/artifacts
- ${ATTUNE_DOCKER_CONFIG_PATH:-./config.docker.yaml}:/opt/attune/config/config.yaml:ro
networks:
- attune-network
```
## Dependencies
- **Requires**: Phase 1 (agent binary build infrastructure) — `docker/Dockerfile.agent` must exist
- **Requires**: Phase 3 (WorkerService dual-mode refactor) — agent auto-detection and lazy env setup
## Next Steps
- **Phase 5**: API binary download endpoint (`GET /api/v1/agent/binary`)
- **Phase 6**: Database runtime registry extensions (runtime template packs)
- **Phase 7**: Kubernetes support (InitContainer pattern, Helm chart)

100
work-summary/2026-02-universal-agent-phase6.md Normal file
View File

@@ -0,0 +1,100 @@
# Universal Worker Agent — Phase 6: Database & Runtime Registry Extensions
**Date**: 2026-02
**Phase**: 6 of 7 (Universal Worker Agent)
**Plan**: `docs/plans/universal-worker-agent.md`
## Overview
Phase 6 extends the runtime registry so that the universal worker agent (`attune-agent`) can work with arbitrary runtimes — including languages like Ruby, Go, Java, Perl, and R — without requiring every possible runtime to be pre-registered in the database by an administrator.
## Changes Made
### 6.1 Extended Runtime Detection Metadata
**Migration** (`migrations/20250101000012_agent_runtime_detection.sql`):
- Added `auto_detected BOOLEAN NOT NULL DEFAULT FALSE` column to `runtime` table — distinguishes agent-created runtimes from pack-loaded ones
- Added `detection_config JSONB NOT NULL DEFAULT '{}'` column — stores detection metadata (detected binary path, version, runtime name)
- Added index `idx_runtime_auto_detected` for efficient filtering
**Rust Model** (`crates/common/src/models.rs`):
- Added `auto_detected: bool` and `detection_config: JsonDict` fields to the `Runtime` struct
**Repository** (`crates/common/src/repositories/runtime.rs`):
- Added `SELECT_COLUMNS` constant centralising the column list for all runtime queries
- Added `auto_detected` and `detection_config` to `CreateRuntimeInput` and `UpdateRuntimeInput`
- Updated ALL 7 SELECT queries, 2 RETURNING clauses, and the INSERT statement to include the new columns
- Updated the `update` method to support setting `auto_detected` and `detection_config`
**External query sites updated**:
- `crates/common/src/runtime_detection.rs` — `detect_from_database()`
- `crates/common/src/pack_environment.rs` — `get_runtime()`
- `crates/worker/src/executor.rs` — `prepare_execution_context()`
**All `CreateRuntimeInput` construction sites updated** (7 files):
- `crates/api/src/routes/runtimes.rs`
- `crates/common/src/pack_registry/loader.rs`
- `crates/common/tests/helpers.rs`
- `crates/common/tests/repository_runtime_tests.rs`
- `crates/common/tests/repository_worker_tests.rs`
- `crates/executor/tests/fifo_ordering_integration_test.rs`
- `crates/executor/tests/policy_enforcer_tests.rs`
### 6.2 Runtime Template Packs
Added 5 new runtime YAML definitions in `packs/core/runtimes/`:
| File | Ref | Interpreter | Environment | Dependencies |
|------|-----|-------------|-------------|--------------|
| `ruby.yaml` | `core.ruby` | `ruby` (.rb) | GEM_HOME isolation | Gemfile → bundle install |
| `go.yaml` | `core.go` | `go run` (.go) | GOPATH isolation | go.mod → go mod download |
| `java.yaml` | `core.java` | `java` (.java) | None (simple) | None |
| `perl.yaml` | `core.perl` | `perl` (.pl) | local::lib isolation | cpanfile → cpanm |
| `r.yaml` | `core.r` | `Rscript --vanilla` (.R) | renv isolation | renv.lock → renv::restore() |
Each includes verification commands matching the auto-detection module's probe strategy.
### 6.3 Dynamic Runtime Registration
**New module** (`crates/worker/src/dynamic_runtime.rs`):
- `auto_register_detected_runtimes(pool, detected)` — main entry point called from `agent_main.rs` BEFORE `WorkerService::new()`
- For each detected runtime:
1. Alias-aware lookup in existing DB runtimes (via `normalize_runtime_name`)
2. If not found, looks for a template runtime by ref pattern `core.<name>`
3. If template found, clones it with `auto_detected = true` and substitutes the detected binary path
4. If no template, creates a minimal runtime with just the interpreter binary and file extension
5. Auto-registered runtimes use ref format `auto.<name>` (e.g., `auto.ruby`)
- Helper functions: `build_detection_config()`, `build_execution_config_from_template()`, `build_minimal_execution_config()`, `build_minimal_distributions()`, `capitalize_runtime_name()`
- 8 unit tests covering all helpers
**Agent entrypoint** (`crates/worker/src/agent_main.rs`):
- Added Phase 2b between config loading and `WorkerService::new()`
- Creates a temporary DB connection and calls `auto_register_detected_runtimes()` for all detected runtimes
- Non-fatal: registration failures are logged as warnings, agent continues
**Runtime name normalization** (`crates/common/src/runtime_detection.rs`):
- Extended `normalize_runtime_name()` with 5 new alias groups:
- `ruby`/`rb` → `ruby`
- `go`/`golang` → `go`
- `java`/`jdk`/`openjdk` → `java`
- `perl`/`perl5` → `perl`
- `r`/`rscript` → `r`
- Added 5 new unit tests + 6 new assertions in existing filter tests
## Architecture Decisions
1. **Dynamic registration before WorkerService::new()**: The `WorkerService` constructor loads runtimes from the DB into an immutable `RuntimeRegistry` wrapped in `Arc`. Rather than restructuring this, dynamic registration runs beforehand so the normal loading pipeline picks up the new entries.
2. **Template-based cloning**: Auto-detected runtimes clone their execution config from pack templates (e.g., `core.ruby`) when available, inheriting environment management, dependency installation, and env_vars configuration. Only the interpreter binary path is substituted with the actual detected path.
3. **Minimal fallback**: When no template exists, a bare-minimum runtime entry is created with just the interpreter binary. This enables immediate script execution without environment/dependency management.
4. **`auto.` ref prefix**: Auto-detected runtimes use `auto.<name>` refs to avoid collisions with pack-registered templates (which use `core.<name>` or `<pack>.<name>`).
## Test Results
- **Worker crate**: 114 passed, 0 failed, 3 ignored
- **Common crate**: 321 passed, 0 failed
- **API crate**: 110 passed, 0 failed, 1 ignored
- **Executor crate**: 115 passed, 0 failed, 1 ignored
- **Workspace check**: Zero errors, zero warnings

62
work-summary/2026-03-21-agent-binary-download-endpoint.md Normal file
View File

@@ -0,0 +1,62 @@
# Universal Worker Agent Phase 5: API Binary Download Endpoint
**Date**: 2026-03-21
**Phase**: Universal Worker Agent Phase 5
**Status**: Complete
## Overview
Implemented the API binary download endpoint for the Attune universal worker agent. This enables deployments where shared Docker volumes are impractical (Kubernetes, ECS, remote Docker hosts) by allowing containers to download the agent binary directly from the Attune API at startup.
## Changes
### New Files
- **`crates/api/src/routes/agent.rs`** — Two new unauthenticated API endpoints:
- `GET /api/v1/agent/binary` — Streams the statically-linked `attune-agent` binary as `application/octet-stream`. Supports `?arch=x86_64|aarch64|arm64` query parameter (defaults to `x86_64`). Tries arch-specific binary (`attune-agent-{arch}`) first, falls back to generic (`attune-agent`). Uses `ReaderStream` for memory-efficient streaming. Optional bootstrap token authentication via `X-Agent-Token` header or `token` query parameter.
- `GET /api/v1/agent/info` — Returns JSON metadata about available agent binaries (architectures, sizes, availability status, version).
- **`scripts/attune-agent-wrapper.sh`** — Bootstrap entrypoint script for containers without volume-mounted agent binary. Features:
- Auto-detects host architecture via `uname -m`
- Checks for volume-mounted binary first (zero-overhead fast path)
- Downloads from API with retry logic (10 attempts, 5s delay) using `curl` or `wget`
- Supports bootstrap token via `ATTUNE_AGENT_TOKEN` env var
- Verifies downloaded binary compatibility
- Configurable via `ATTUNE_AGENT_DIR`, `ATTUNE_AGENT_URL`, `ATTUNE_AGENT_ARCH` env vars
### Modified Files
- **`crates/common/src/config.rs`** — Added `AgentConfig` struct with `binary_dir` (path to agent binaries) and `bootstrap_token` (optional auth). Added `agent: Option<AgentConfig>` field to `Config`.
- **`crates/api/src/routes/mod.rs`** — Added `pub mod agent` and `pub use agent::routes as agent_routes`.
- **`crates/api/src/server.rs`** — Added `.merge(routes::agent_routes())` to the API v1 router.
- **`crates/api/src/openapi.rs`** — Registered both endpoints in OpenAPI paths, added `AgentBinaryInfo` and `AgentArchInfo` schemas, added `"agent"` tag. Updated endpoint count test assertions (+2 paths, +2 operations).
- **`config.docker.yaml`** — Added `agent.binary_dir: /opt/attune/agent` configuration.
- **`config.development.yaml`** — Added commented-out agent config pointing to local musl build output.
- **`docker-compose.yaml`** — API service now mounts `agent_bin` volume read-only at `/opt/attune/agent` and depends on `init-agent` service completing successfully.
- **`AGENTS.md`** — Updated development status (Phase 5 complete), updated agent_bin volume description, added agent config to Key Settings.
## Architecture Decisions
1. **Unauthenticated endpoint** — The agent needs to download its binary before it can authenticate with JWT. An optional lightweight bootstrap token (`agent.bootstrap_token`) provides security when needed.
2. **Streaming response** — Uses `tokio_util::io::ReaderStream` to stream the ~20MB binary without loading it entirely into memory.
3. **Architecture whitelist** — Only `x86_64`, `aarch64`, and `arm64` (alias) are accepted, preventing path traversal attacks.
4. **Graceful fallback** — Arch-specific binary (`attune-agent-x86_64`) → generic binary (`attune-agent`) → 404. This supports both multi-arch and single-arch deployments.
5. **Volume-first strategy** — The wrapper script checks for a volume-mounted binary before attempting download, so Docker Compose deployments with the `agent_bin` volume pay zero network overhead.
## Testing
- All 4 OpenAPI tests pass (including updated endpoint count: 59 paths, 83 operations)
- All 21 config tests pass (including `AgentConfig` integration)
- API crate compiles with zero warnings
- Common crate compiles with zero warnings

72
work-summary/2026-03-21-universal-worker-agent-phase1.md Normal file
View File

@@ -0,0 +1,72 @@
# Universal Worker Agent — Phase 1: Static Binary Build Infrastructure
**Date**: 2026-03-21
## Summary
Implemented Phase 1 of the Universal Worker Agent plan (`docs/plans/universal-worker-agent.md`), establishing the build infrastructure for a statically-linked `attune-agent` binary that can be injected into any container to turn it into an Attune worker.
## Problem
Adding support for new runtime environments (Ruby, Go, Java, R, etc.) required building custom Docker images for each combination. This meant modifying `Dockerfile.worker.optimized`, installing interpreters via apt, managing a combinatorial explosion of worker variants, and rebuilding images (~5 min) for every change.
## Solution
Phase 1 lays the groundwork for flipping the model: instead of baking the worker into custom images, a single static binary is injected into **any** container at startup. This phase delivers:
1. **TLS backend audit** — confirmed the worker crate has zero `native-tls` or `openssl` dependencies, making musl static linking viable without any TLS backend changes
2. **New binary target** — `attune-agent` alongside `attune-worker` in the same crate
3. **Runtime auto-detection module** — probes container environments for interpreters
4. **Dockerfile for static builds** — multi-stage musl cross-compilation
5. **Makefile targets** — local and Docker build commands
## Changes
### New Files
- **`crates/worker/src/agent_main.rs`** — Agent entrypoint with three-phase startup: (1) auto-detect runtimes or respect `ATTUNE_WORKER_RUNTIMES` override, (2) load config, (3) run `WorkerService`. Includes `--detect-only` flag for diagnostic probing.
- **`crates/worker/src/runtime_detect.rs`** — Database-free runtime detection module. Probes 8 interpreter families (shell, python, node, ruby, go, java, r, perl) via `which`-style PATH lookup with fallbacks. Captures version strings. 18 unit tests covering version parsing, display formatting, binary lookup, and detection pipeline.
- **`docker/Dockerfile.agent`** — Multi-stage Dockerfile:
- `builder` stage: cross-compiles with `x86_64-unknown-linux-musl` target, BuildKit cache mounts
- `agent-binary` stage: `FROM scratch` with just the static binary
- `agent-init` stage: busybox-based for Docker Compose/K8s init container volume population
### Modified Files
- **`crates/worker/Cargo.toml`** — Added second `[[bin]]` target for `attune-agent`
- **`crates/worker/src/lib.rs`** — Added `pub mod runtime_detect`
- **`Makefile`** — Added targets: `build-agent` (local musl build), `docker-build-agent`, `run-agent`, `run-agent-release`
- **`docker/Dockerfile.worker.optimized`** — Added `agent_main.rs` stub for second binary target
- **`docker/Dockerfile.optimized`** — Added `agent_main.rs` stub
- **`docker/Dockerfile.sensor.optimized`** — Added `agent_main.rs` stub
- **`docker/Dockerfile.pack-binaries`** — Added `agent_main.rs` stub
- **`AGENTS.md`** — Documented agent service, runtime auto-detection, Docker build, Makefile targets
## Key Design Decisions
1. **Same crate, new binary** — The agent lives as a second `[[bin]]` target in `crates/worker` rather than a separate crate. This gives zero code duplication and the same test suite covers both binaries. Can be split into a separate crate later if binary size becomes a concern.
2. **No TLS changes needed** — The plan anticipated needing to switch from `native-tls` to `rustls` workspace-wide. Audit revealed the worker crate already uses `rustls` exclusively (`native-tls` only enters via `tokio-tungstenite` in CLI and `ldap3` in API, neither of which the worker depends on).
3. **Database-free detection** — The `runtime_detect` module is deliberately separate from `attune_common::runtime_detection` (which queries the database). The agent must discover runtimes before any DB connectivity, using pure filesystem probing.
4. **All Dockerfiles updated** — Since the worker crate now has two binary targets, all Dockerfiles that create workspace stubs for `cargo fetch` need a stub for `agent_main.rs`. Missing this would break Docker builds.
## Verification
- `cargo check --all-targets --workspace` — zero warnings ✅
- `cargo test -p attune-worker` — all 37 tests pass (18 new runtime_detect tests + 19 existing) ✅
- `cargo run --bin attune-agent -- --detect-only` — successfully detected 6 runtimes on dev machine ✅
- `cargo run --bin attune-agent -- --help` — correct CLI documentation ✅
## Next Steps (Phases 2–7)
See `docs/plans/universal-worker-agent.md` for the remaining phases:
- **Phase 2**: Integration with worker registration (auto-detected runtimes → DB)
- **Phase 3**: Refactor `WorkerService` for dual modes (lazy env setup)
- **Phase 4**: Docker Compose init service for agent volume
- **Phase 5**: API binary download endpoint
- **Phase 6**: Database runtime registry extensions
- **Phase 7**: Kubernetes support (init containers, Helm chart)

46
work-summary/kubernetes-agent-workers.md Normal file
View File

@@ -0,0 +1,46 @@
# Universal Worker Agent Phase 7: Kubernetes Support
**Date**: 2026-02-05
## Summary
Implemented Kubernetes support for agent-based workers in the Attune Helm chart, completing Phase 7 of the Universal Worker Agent plan. Users can now deploy the `attune-agent` binary into any container image on Kubernetes using the InitContainer pattern — the same approach used by Tekton and Argo.
## Changes
### Helm Chart (`charts/attune/`)
- **`templates/agent-workers.yaml`** (new): Helm template that iterates over `agentWorkers[]` values and creates a Deployment per entry. Each Deployment includes:
- `agent-loader` init container — copies the statically-linked `attune-agent` binary from the `attune-agent` image into an `emptyDir` volume
- `wait-for-schema` init container — polls PostgreSQL until the Attune schema is ready
- `wait-for-packs` init container — waits for the core pack on the shared PVC
- Worker container — runs the user's chosen image with the agent binary as entrypoint
- Volumes: `agent-bin` (emptyDir), `config` (ConfigMap), `packs` (PVC, read-only), `runtime-envs` (PVC), `artifacts` (PVC)
- **`values.yaml`**: Added `images.agent` (repository, tag, pullPolicy) and `agentWorkers: []` with full documentation of supported fields: `name`, `image`, `replicas`, `runtimes`, `resources`, `env`, `imagePullPolicy`, `logLevel`, `runtimeClassName`, `nodeSelector`, `tolerations`, `stopGracePeriod`
- **`templates/NOTES.txt`**: Updated to list enabled agent workers on install/upgrade
### CI/CD (`.gitea/workflows/publish.yml`)
- Added `attune-agent` to the image build matrix (target: `agent-init`, dockerfile: `docker/Dockerfile.agent`) so the agent image is published alongside all other Attune images
### Documentation
- **`docs/QUICKREF-kubernetes-agent-workers.md`** (new): Quick-reference guide covering how agent workers work on Kubernetes, all supported Helm values fields, runtime auto-detection table, differences from the standard worker, and troubleshooting steps
- **`docs/deployment/gitea-registry-and-helm.md`**: Added `attune-agent` to the published images list
- **`docs/plans/universal-worker-agent.md`**: Marked Phase 7 as complete with implementation details
### AGENTS.md
- Moved Phase 7 from "In Progress" to "Complete" with a summary of what was implemented
## Design Decisions
1. **emptyDir volume** (not PVC) for the agent binary — each pod gets its own copy via the init container. This avoids needing a shared RWX volume just for a single static binary and follows the standard Kubernetes sidecar injection pattern used by Tekton, Argo, and Istio.
2. **Pod-level scheduling fields** — `runtimeClassName`, `nodeSelector`, and `tolerations` are exposed at the pod spec level (not container level) to support GPU scheduling via NVIDIA RuntimeClass and node affinity for specialized hardware.
3. **Runtime auto-detect by default** — when `runtimes` is empty (the default), the agent probes the container for interpreters. Users can override with an explicit list to skip detection and limit which runtimes are registered.
4. **Consistent patterns** — the template reuses the same `wait-for-schema` and `wait-for-packs` init containers, `envFrom` secret injection, and volume mount structure as the existing worker Deployment in `applications.yaml`.