Files

David Culbreth b43495b26d change capture

2026-02-26 14:34:02 -06:00

51 KiB

Raw Blame History

Attune Project Rules

Project Overview

Attune is an event-driven automation and orchestration platform built in Rust, similar to StackStorm. It enables building complex workflows triggered by events with multi-tenancy, RBAC, and human-in-the-loop capabilities.

Development Status: Pre-Production

This project is under active development with no users, deployments, or stable releases.

Breaking Changes Policy

Breaking changes are explicitly allowed and encouraged when they improve the architecture, API design, or developer experience
No backward compatibility required - there are no existing versions to support
Database migrations can be modified or consolidated - no production data exists
API contracts can change freely - no external integrations depend on them, only internal interfaces with other services and the web UI must be maintained.
Configuration formats can be redesigned - no existing config files need migration
Service interfaces can be refactored - no live deployments to worry about

When this project reaches v1.0 or gets its first production deployment, this section should be removed and replaced with appropriate stability guarantees and versioning policies.

Languages & Core Technologies

Primary Language: Rust 2021 edition
Database: PostgreSQL 16+ with TimescaleDB 2.17+ (primary data store + LISTEN/NOTIFY pub/sub + time-series history)
Message Queue: RabbitMQ 3.12+ (via lapin)
Cache: Redis 7.0+ (optional)
Web UI: TypeScript + React 19 + Vite
Async Runtime: Tokio
Web Framework: Axum 0.8
ORM: SQLx (compile-time query checking)

Project Structure (Cargo Workspace)

attune/
├── Cargo.toml                    # Workspace root
├── config.{development,test}.yaml # Environment configs
├── Makefile                      # Common dev tasks
├── crates/                       # Rust services
│   ├── common/                   # Shared library (models, db, repos, mq, config, error, template_resolver)
│   ├── api/                      # REST API service (8080)
│   ├── executor/                 # Execution orchestration service
│   ├── worker/                   # Action execution service (multi-runtime)
│   ├── sensor/                   # Event monitoring service
│   ├── notifier/                 # Real-time notification service
│   └── cli/                      # Command-line interface
├── migrations/                   # SQLx database migrations (19 tables)
├── web/                          # React web UI (Vite + TypeScript)
├── packs/                        # Pack bundles
│   └── core/                     # Core pack (timers, HTTP, etc.)
├── docs/                         # Technical documentation
├── scripts/                      # Helper scripts (DB setup, testing)
└── tests/                        # Integration tests

Service Architecture (Distributed Microservices)

attune-api: REST API gateway, JWT auth, all client interactions
attune-executor: Manages execution lifecycle, scheduling, policy enforcement
attune-worker: Executes actions in multiple runtimes (Python/Node.js/containers)
attune-sensor: Monitors triggers, generates events
attune-notifier: Real-time notifications via PostgreSQL LISTEN/NOTIFY + WebSocket

Communication: Services communicate via RabbitMQ for async operations

Docker Compose Orchestration

All Attune services run via Docker Compose.

Compose file: docker-compose.yaml (root directory)
Configuration: config.docker.yaml (Docker-specific settings)
Default user: test@attune.local / TestPass123! (auto-created)

Services:

Infrastructure: postgres (TimescaleDB), rabbitmq, redis
Init (run-once): migrations, init-user, init-packs
Application: api (8080), executor, worker-{shell,python,node,full}, sensor, notifier (8081), web (3000)

Commands:

docker compose up -d          # Start all services
docker compose down           # Stop all services
docker compose logs -f <svc>  # View logs

Key environment overrides: JWT_SECRET, ENCRYPTION_KEY (required for production)

Docker Build Optimization

Optimized Dockerfiles: docker/Dockerfile.optimized, docker/Dockerfile.worker.optimized, and docker/Dockerfile.sensor.optimized
Strategy: Selective crate copying - only copy crates needed for each service (not entire workspace)
Performance: 90% faster incremental builds (~30 sec vs ~5 min for code changes)
BuildKit cache mounts: Persist cargo registry and compilation artifacts between builds
- Cache strategy: sharing=shared for registry/git (concurrent-safe), service-specific IDs for target caches
- Parallel builds: 4x faster than old sharing=locked strategy - no serialization overhead
Documentation: See docs/docker-layer-optimization.md, docs/QUICKREF-docker-optimization.md, docs/QUICKREF-buildkit-cache-strategy.md

Docker Runtime Standardization

Base image: All worker and sensor runtime stages use debian:bookworm-slim (or debian:bookworm for worker-full)
Python: Always installed via apt-get install python3 python3-pip python3-venv → binary at /usr/bin/python3
Node.js: Always installed via NodeSource apt repo (setup_${NODE_VERSION}.x) → binary at /usr/bin/node
NEVER use python: or node: Docker images as base — they install binaries at /usr/local/bin/ which causes broken venv symlinks when multiple containers share the runtime_envs volume
UID: All containers use UID 1000 for the attune user
Venv creation: Uses --copies flag (python3 -m venv --copies) to avoid cross-container broken symlinks
Worker targets: worker-base (shell), worker-python (shell+python), worker-node (shell+node), worker-full (all)
Sensor targets: sensor-base (native only), sensor-full (native+python+node)

Packs Volume Architecture

Key Principle: Packs are NOT copied into Docker images - they are mounted as volumes
Volume Flow: Host ./packs/ → init-packs service → packs_data volume → mounted in all services
Benefits: Update packs with restart (~5 sec) instead of rebuild (~5 min)
Pack Binaries: Built separately with ./scripts/build-pack-binaries.sh (GLIBC compatibility)
Development: Use ./packs.dev/ for instant testing (direct bind mount, no restart needed)
Documentation: See docs/QUICKREF-packs-volumes.md

Runtime Environments Volume

Key Principle: Runtime environments (virtualenvs, node_modules) are stored OUTSIDE pack directories
Volume: runtime_envs named volume mounted at /opt/attune/runtime_envs in worker, sensor, and API containers
Path Pattern: {runtime_envs_dir}/{pack_ref}/{runtime_name} (e.g., /opt/attune/runtime_envs/python_example/python)
Creation: Worker creates environments proactively at startup and via pack.registered MQ events; lightweight existence check at execution time
Broken venv auto-repair: Worker detects broken interpreter symlinks (e.g., from mismatched container python paths) and automatically recreates the environment
API best-effort: API attempts environment setup during pack registration but logs and defers to worker on failure (Docker API containers lack interpreters)
Pack directories remain read-only: Packs mounted :ro in workers; all generated env files go to runtime_envs volume
Config: runtime_envs_dir setting in config YAML (default: /opt/attune/runtime_envs)

Domain Model & Event Flow

Critical Event Flow:

Sensor → Trigger fires → Event created → Rule evaluates →
Enforcement created → Execution scheduled → Worker executes Action

Key Entities (all in public schema, IDs are i64):

Pack: Bundle of automation components (actions, sensors, rules, triggers, runtimes)
Runtime: Unified execution environment definition (Python, Shell, Node.js, etc.) — used by both actions and sensors. Configured via execution_config JSONB (interpreter, environment setup, dependency management, env_vars). No type distinction; whether a runtime is executable is determined by its execution_config content.
RuntimeVersion: A specific version of a runtime (e.g., Python 3.12.1, Node.js 20.11.0). Each version has its own execution_config and distributions for version-specific interpreter paths, verification commands, and environment setup. Actions and sensors can declare an optional runtime_version_constraint (semver range) to select a compatible version at execution time.
Trigger: Event type definition (e.g., "webhook_received")
Sensor: Monitors for trigger conditions, creates events
Event: Instance of a trigger firing with payload
Action: Executable task with parameters
Rule: Links triggers to actions with conditional logic
Enforcement: Represents a rule activation
Execution: Single action run; supports parent-child relationships for workflows
- Workflow Tasks: Workflow-specific metadata stored in execution.workflow_task JSONB field
Inquiry: Human-in-the-loop async interaction (approvals, inputs)
Identity: User/service account with RBAC permissions
Key: Encrypted secrets storage

Key Tools & Libraries

Shared Dependencies (workspace-level)

Async: tokio, async-trait, futures
Web: axum, tower, tower-http
Database: sqlx (with postgres, json, chrono, uuid features)
Serialization: serde, serde_json, serde_yaml_ng
Version Matching: semver (with serde feature)
Logging: tracing, tracing-subscriber
Error Handling: anyhow, thiserror
Config: config crate (YAML + env vars)
Validation: validator
Auth: jsonwebtoken, argon2
CLI: clap
OpenAPI: utoipa, utoipa-swagger-ui
Message Queue: lapin (RabbitMQ)
HTTP Client: reqwest
Testing: mockall, tempfile, serial_test

Web UI Dependencies

Framework: React 19 + react-router-dom
State: Zustand, @tanstack/react-query
HTTP: axios (with generated OpenAPI client)
Styling: Tailwind CSS
Icons: lucide-react
Build: Vite, TypeScript

Configuration System

Primary: YAML config files (config.yaml, config.{env}.yaml)
Overrides: Environment variables with prefix ATTUNE__ and separator __
- Example: ATTUNE__DATABASE__URL, ATTUNE__SERVER__PORT, ATTUNE__RUNTIME_ENVS_DIR
Loading Priority: Base config → env-specific config → env vars
Required for Production: JWT_SECRET, ENCRYPTION_KEY (32+ chars)
Location: Root directory or ATTUNE_CONFIG env var path
Key Settings:
- packs_base_dir - Where pack files are stored (default: /opt/attune/packs)
- runtime_envs_dir - Where isolated runtime environments are created (default: /opt/attune/runtime_envs)

Authentication & Security

Auth Type: JWT (access tokens: 1h, refresh tokens: 7d)
Password Hashing: Argon2id
Protected Routes: Use RequireAuth(user) extractor in Axum
Secrets Storage: AES-GCM encrypted in key table with scoped ownership
User Info: Stored in identity table

Code Conventions & Patterns

General

Error Handling: Use attune_common::error::Error and Result<T> type alias
Async Everywhere: All I/O operations use async/await with Tokio
Module Structure: Public API exposed via mod.rs with pub use re-exports

Database Layer

Schema: All tables use unqualified names; schema determined by PostgreSQL search_path
Production: Always uses public schema (configured explicitly in config.production.yaml)
Tests: Each test uses isolated schema (e.g., test_a1b2c3d4) for true parallel execution
Schema Resolution: PostgreSQL search_path mechanism, NO hardcoded schema prefixes in queries
Models: Defined in common/src/models.rs with #[derive(FromRow)] for SQLx
Repositories: One per entity in common/src/repositories/, provides CRUD + specialized queries
Pattern: Services MUST interact with DB only through repository layer (no direct queries)
Transactions: Use SQLx transactions for multi-table operations
IDs: All IDs are i64 (BIGSERIAL in PostgreSQL)
Timestamps: created/updated columns auto-managed by DB triggers
JSON Fields: Use serde_json::Value for flexible attributes/parameters, including execution.workflow_task JSONB
Enums: PostgreSQL enum types mapped with #[sqlx(type_name = "...")]
Workflow Tasks: Stored as JSONB in execution.workflow_task (consolidated from separate table 2026-01-27)
FK ON DELETE Policy: Historical records (executions, events, enforcements) use ON DELETE SET NULL so they survive entity deletion while preserving text ref fields (action_ref, trigger_ref, etc.) for auditing. Pack-owned entities (actions, triggers, sensors, rules, runtimes) use ON DELETE CASCADE from pack. Workflow executions cascade-delete with their workflow definition.
Entity History Tracking (TimescaleDB): Append-only <table>_history hypertables track field-level changes to execution, worker, enforcement, and event tables. Populated by PostgreSQL AFTER INSERT OR UPDATE OR DELETE triggers — no Rust code changes needed for recording. Uses JSONB diff format (old_values/new_values) with a changed_fields TEXT[] column for efficient filtering. Worker heartbeat-only updates are excluded. See docs/plans/timescaledb-entity-history.md for full design.
History Large-Field Guardrails: The execution history trigger stores a compact digest summary instead of the full value for the result column (which can be arbitrarily large). The digest is produced by the _jsonb_digest_summary(JSONB) helper function and has the shape {"digest": "md5:<hex>", "size": <bytes>, "type": "<jsonb_typeof>"}. This preserves change-detection semantics while avoiding history table bloat. The full result is always available on the live execution row. When adding new large JSONB columns to history triggers, use _jsonb_digest_summary() instead of storing the raw value.
Nullable FK Fields: rule.action and rule.trigger are nullable (Option<Id> in Rust) — a rule with NULL action/trigger is non-functional but preserved for traceability. execution.action, execution.parent, execution.enforcement, and event.source are also nullable. Table Count: 22 tables total in the schema (including runtime_version and 4 *_history hypertables) Migration Count: 9 consolidated migrations (000001 through 000009) — see migrations/ directory
Pack Component Loading Order: Runtimes → Triggers → Actions → Sensors (dependency order). Both PackComponentLoader (Rust) and load_core_pack.py (Python) follow this order.

Pack File Loading & Action Execution

Pack Base Directory: Configured via packs_base_dir in config (defaults to /opt/attune/packs, development uses ./packs)
Pack Volume Strategy: Packs are mounted as volumes (NOT copied into Docker images)
- Host ./packs/ → packs_data volume via init-packs service → mounted at /opt/attune/packs in all services
- Development packs in ./packs.dev/ are bind-mounted directly for instant updates
Pack Binaries: Native binaries (sensors) built separately with ./scripts/build-pack-binaries.sh
Action Script Resolution: Worker constructs file paths as {packs_base_dir}/{pack_ref}/actions/{entrypoint}
Workflow File Storage: Visual workflow builder saves files to {packs_base_dir}/{pack_ref}/actions/workflows/{name}.workflow.yaml via POST /api/v1/packs/{pack_ref}/workflow-files and PUT /api/v1/workflows/{ref}/file endpoints
Task Model (Orquesta-aligned): Tasks are purely action invocations — there is no task type field or task-level when condition in the UI model. Parallelism is implicit (multiple do targets in a transition fan out into parallel branches). Conditions belong exclusively on transitions (next[].when). Each task has: name, action, input, next (transitions), delay, retry, timeout, with_items, batch_size, concurrency, join.
- The backend Task struct (crates/common/src/workflow/parser.rs) still supports type and task-level when for backward compatibility, but the UI never sets them.
Task Transition Model (Orquesta-style): Tasks use an ordered next array of transitions instead of flat on_success/on_failure/on_complete/on_timeout fields. Each transition has:
- when — condition expression (e.g., {{ succeeded() }}, {{ failed() }}, {{ timed_out() }}, or custom). Omit for unconditional.
- publish — key-value pairs to publish into the workflow context (e.g., - result: "{{ result() }}")
- do — list of next task names to invoke when the condition is met
- label — optional custom display label (overrides auto-derived label from when expression)
- color — optional custom CSS color for the transition edge (e.g., "#ff6600")
- edge_waypoints — optional Record<string, NodePosition[]> of intermediate routing points per target task name (chart-only, stored in __chart_meta__)
- label_positions — optional Record<string, NodePosition> of custom label positions per target task name (chart-only, stored in __chart_meta__)
- Example YAML:
```
next:
  - when: "{{ succeeded() }}"
    label: "main path"
    color: "#22c55e"
    publish:
      - msg: "task done"
    do:
      - log
      - next_task
  - when: "{{ failed() }}"
    do:
      - error_handler
```
- Legacy format support: The parser (crates/common/src/workflow/parser.rs) auto-converts legacy on_success/on_failure/on_complete/on_timeout/decision fields into next transitions during parsing. The canonical internal representation always uses next.
- Frontend types: TaskTransition in web/src/types/workflow.ts (includes edge_waypoints, label_positions for visual routing); TransitionPreset ("succeeded" | "failed" | "always") for quick-access drag handles; WorkflowEdge includes per-edge waypoints and labelPosition derived from the transition; SelectedEdgeInfo and EdgeHoverInfo (includes targetTaskId) in WorkflowEdges.tsx
- Backend types: TaskTransition in crates/common/src/workflow/parser.rs; GraphTransition in crates/executor/src/workflow/graph.rs
- NOT this (legacy format): on_success: task2 / on_failure: error_handler — still parsed for backward compat but normalized to next
Runtime YAML Loading: Pack registration reads runtimes/*.yaml files and inserts them into the runtime table. Runtime refs use format {pack_ref}.{name} (e.g., core.python, core.shell). If the YAML includes a versions array, each entry is inserted into the runtime_version table with its own execution_config, distributions, and optional is_default flag.
Runtime Version Constraints: Actions and sensors can declare runtime_version: ">=3.12" (or any semver constraint like ~3.12, ^3.12, >=3.12,<4.0) in their YAML. This is stored in the runtime_version_constraint column. At execution time the worker can select the highest available version satisfying the constraint. A bare version like "3.12" is treated as tilde (~3.12 → >=3.12.0, <3.13.0).
Version Matching Module: crates/common/src/version_matching.rs provides parse_version() (lenient semver parsing), parse_constraint(), matches_constraint(), select_best_version(), and extract_version_components(). Uses the semver crate internally.
Runtime Version Table: runtime_version stores version-specific execution configs per runtime. Each row has: runtime (FK), version (string), version_major/minor/patch (ints for range queries), execution_config (complete, not a diff), distributions (verification metadata), is_default, available, verified_at, meta. Unique on (runtime, version).
Runtime Selection: Determined by action's runtime field (e.g., "Shell", "Python") - compared case-insensitively; when an explicit runtime_name is set in execution context, it is authoritative (no fallback to extension matching). When the action also declares a runtime_version_constraint, the executor queries runtime_version rows, calls select_best_version(), and passes the selected version's execution_config as an override through ExecutionContext.runtime_config_override. The ProcessRuntime uses this override instead of its built-in config.
Worker Runtime Loading: Worker loads all runtimes from DB that have a non-empty execution_config (i.e., runtimes with an interpreter configured). Native runtimes (e.g., core.native with empty config) are automatically skipped since they execute binaries directly.
Worker Startup Sequence: (1) Connect to DB and MQ, (2) Load runtimes from DB → create ProcessRuntime instances, (3) Register worker and set up MQ infrastructure, (4) Verify runtime versions — run verification commands from distributions JSONB for each RuntimeVersion row and update available flag (crates/worker/src/version_verify.rs), (5) Set up runtime environments — create per-version environments for packs, (6) Start heartbeat, execution consumer, and pack registration consumer.
Runtime Name Normalization: The ATTUNE_WORKER_RUNTIMES filter (e.g., shell,node) uses alias-aware matching via normalize_runtime_name() in crates/common/src/runtime_detection.rs. This ensures that filter value "node" matches DB runtime name "Node.js" (lowercased to "node.js"). Alias groups: node/nodejs/node.js → node, python/python3 → python, shell/bash/sh → shell, native/builtin/standalone → native. Used in worker service runtime loading and environment setup.
Runtime Execution Environment Variables: RuntimeExecutionConfig.env_vars (HashMap<String, String>) specifies template-based environment variables injected during action execution. Example: {"NODE_PATH": "{env_dir}/node_modules"} ensures Node.js finds packages in the isolated environment. Template variables ({env_dir}, {pack_dir}, {interpreter}, {manifest_path}) are resolved at execution time by ProcessRuntime::execute.
Native Runtime Detection: Runtime detection is purely data-driven via execution_config in the runtime table. A runtime with empty execution_config (or empty interpreter.binary) is native — the entrypoint is executed directly without an interpreter. There is no special "builtin" runtime concept.
Sensor Runtime Assignment: Sensors declare their runner_type in YAML (e.g., python, native). The pack loader resolves this to the correct runtime from the database. Default is native (compiled binary, no interpreter). Legacy values standalone and builtin map to core.native.
Runtime Environment Setup: Worker creates isolated environments (virtualenvs, node_modules) proactively at startup and via pack.registered MQ events at {runtime_envs_dir}/{pack_ref}/{runtime_name}; setup is idempotent. Environment create_command and dependency install_command templates MUST use {env_dir} (not {pack_dir}) since pack directories are mounted read-only in Docker. For Node.js, create_command copies package.json to {env_dir} and install_command uses npm install --prefix {env_dir}.
Per-Version Environment Isolation: When runtime versions are registered, the worker creates per-version environments at {runtime_envs_dir}/{pack_ref}/{runtime_name}-{version} (e.g., python-3.12). This ensures different versions maintain isolated environments with their own interpreter binaries and installed dependencies. A base (unversioned) environment is also created for backward compatibility. The ExecutionContext.runtime_env_dir_suffix field controls which env dir the ProcessRuntime uses at execution time.
Runtime Version Verification: At worker startup, version_verify::verify_all_runtime_versions() runs each version's verification commands (from distributions.verification.commands JSONB) and updates the available and verified_at columns in the database. Only versions marked available = true are considered by select_best_version(). Verification respects the ATTUNE_WORKER_RUNTIMES filter.
Schema Format (Unified): ALL schemas (param_schema, out_schema, conf_schema) use the same flat format with required and secret inlined per-parameter (NOT standard JSON Schema). Stored as JSONB columns.
- Example YAML: parameters:\n url:\n type: string\n required: true\n token:\n type: string\n secret: true
- Stored JSON: {"url": {"type": "string", "required": true}, "token": {"type": "string", "secret": true}}
- NOT this (legacy JSON Schema): {"type": "object", "properties": {"url": {"type": "string"}}, "required": ["url"]}
- Web UI: extractProperties() in ParamSchemaForm.tsx is the single extraction function for all schema types. Only handles flat format.
- SchemaBuilder: Visual schema editor reads and writes flat format with required and secret checkboxes per parameter.
- Backend Validation: flat_to_json_schema() in crates/api/src/validation/params.rs converts flat format to JSON Schema internally for jsonschema crate validation. This conversion is an implementation detail — external interfaces always use flat format.
Parameter Delivery: Actions receive parameters via stdin as JSON (never environment variables)
Output Format: Actions declare output format (text/json/yaml) - json/yaml are parsed into execution.result JSONB
Standard Environment Variables: Worker provides execution context via ATTUNE_* environment variables:
- ATTUNE_ACTION - Action ref (always present)
- ATTUNE_EXEC_ID - Execution database ID (always present)
- ATTUNE_API_TOKEN - Execution-scoped API token (always present)
- ATTUNE_RULE - Rule ref (if triggered by rule)
- ATTUNE_TRIGGER - Trigger ref (if triggered by event/trigger)
Custom Environment Variables: Optional, set via execution.env_vars JSONB field (for debug flags, runtime config only)

API Service (`crates/api`)

Structure: routes/ (endpoints) + dto/ (request/response) + auth/ + middleware/
Responses: Standardized ApiResponse<T> wrapper with data field
Protected Routes: Apply RequireAuth middleware
OpenAPI: Documented with utoipa attributes (#[utoipa::path])
Error Handling: Custom ApiError type with proper HTTP status codes
Available at: http://localhost:8080 (dev), /api-spec/openapi.json for spec

Common Library (`crates/common`)

Modules: models, repositories, db, config, error, mq, crypto, utils, workflow, pack_registry, template_resolver, version_matching, runtime_detection
Exports: Commonly used types re-exported from lib.rs
Repository Layer: All DB access goes through repositories in repositories/
Message Queue: Abstractions in mq/ for RabbitMQ communication
Template Resolver: Resolves {{ }} template variables in rule action_params during enforcement creation. Re-exported from attune_common::{TemplateContext, resolve_templates}.

Template Variable Syntax

Rule action_params support Jinja2-style {{ source.path }} templates resolved at enforcement creation time:

Namespace	Example	Description
`event.payload.*`	`{{ event.payload.service }}`	Event payload fields
`event.id`	`{{ event.id }}`	Event database ID
`event.trigger`	`{{ event.trigger }}`	Trigger ref that generated the event
`event.created`	`{{ event.created }}`	Event creation timestamp (RFC 3339)
`pack.config.*`	`{{ pack.config.api_token }}`	Pack configuration values
`system.*`	`{{ system.timestamp }}`	System variables (timestamp, rule info)

Implementation: crates/common/src/template_resolver.rs (also re-exported from attune_sensor::template_resolver)
Integration: crates/executor/src/event_processor.rs calls resolve_templates() in create_enforcement()
IMPORTANT: The old trigger.payload.* syntax was renamed to event.payload.* — the payload data comes from the Event, not the Trigger

Web UI (`web/`)

Generated Client: OpenAPI client auto-generated from API spec
- Run: npm run generate:api (requires API running on :8080)
- Location: src/api/
State Management: Zustand for global state, TanStack Query for server state
Styling: Tailwind utility classes
Dev Server: npm run dev (typically :3000 or :5173)
Build: npm run build
Workflow Builder: Visual node-based workflow editor at /actions/workflows/new and /actions/workflows/:ref/edit
- Components in web/src/components/workflows/ (ActionPalette, WorkflowCanvas, TaskNode, WorkflowEdges, TaskInspector)
- Types and conversion utilities in web/src/types/workflow.ts
- Hooks in web/src/hooks/useWorkflows.ts
- Saves workflow files to {packs_base_dir}/{pack_ref}/actions/workflows/{name}.workflow.yaml via dedicated API endpoints
- Visual / Raw YAML toggle: Toolbar has a segmented toggle to switch between the visual node-based builder and a full-width read-only YAML preview (generated via js-yaml). Raw YAML mode replaces the canvas, palette, and inspector with the effective workflow definition.
- Drag-handle connections: TaskNode has output handles (green=succeeded, red=failed, gray=always) and an input handle (top). Drag from an output handle to another node's input handle to create a transition.
- Transition customization: Users can rename transitions (custom label) and assign custom colors (CSS color string or preset swatches) via the TaskInspector. Custom colors/labels are persisted in the workflow YAML and rendered on the canvas edges.
- Edge waypoints & label dragging: Transition edges support intermediate waypoints for custom routing. Click an edge to select it, then:
  - Drag existing waypoint handles (colored circles) to reposition the edge path
  - Hover near the midpoint of any edge segment to reveal a "+" handle; click or drag it to insert a new waypoint
  - Drag the transition label to reposition it independently of the edge path
  - Double-click a waypoint to remove it; double-click a label to reset its position
  - Waypoints and label positions are stored per-edge (keyed by target task name) in TaskTransition.edge_waypoints and TaskTransition.label_positions, serialized via __chart_meta__ in the workflow YAML
  - Edge selection state (SelectedEdgeInfo) is managed in WorkflowCanvas; only the selected edge shows interactive handles
  - Multi-segment paths use Catmull-Rom → cubic Bezier conversion for smooth curves through waypoints (buildSmoothPath in WorkflowEdges.tsx)
- Orquesta-style next transitions: Tasks use a next: TaskTransition[] array instead of flat on_success/on_failure fields. Each transition has when (condition), publish (variables), do (target tasks), plus optional label, color, edge_waypoints, and label_positions. See "Task Transition Model" above.
- No task type or task-level condition: The UI does not expose task type or task-level when — all tasks are actions (workflows are also actions), and conditions belong on transitions. Parallelism is implicit via multiple do targets.

Development Workflow

Common Commands (Makefile)

make build              # Build all services
make build-release      # Release build
make test               # Run all tests
make test-integration   # Run integration tests
make fmt                # Format code
make clippy             # Run linter
make lint               # fmt + clippy

make run-api            # Run API service
make run-executor       # Run executor service
make run-worker         # Run worker service
make run-sensor         # Run sensor service
make run-notifier       # Run notifier service

make db-create          # Create database
make db-migrate         # Run migrations
make db-reset           # Drop & recreate DB

Database Operations

Migrations: Located in migrations/, applied via sqlx migrate run
Test DB: Separate attune_test database, setup with make db-test-setup
Schema: All tables in public schema with auto-updating timestamps
Core Pack: Load with ./scripts/load-core-pack.sh after DB setup

Testing

Architecture: Schema-per-test isolation (each test gets unique test_<uuid> schema)
Parallel Execution: Tests run concurrently without #[serial] constraints (4-8x faster)
Unit Tests: In module files alongside code
Integration Tests: In tests/ directory
Test DB Required: Use make db-test-setup before integration tests
Run: cargo test or make test (parallel by default)
Verbose: cargo test -- --nocapture --test-threads=1
Cleanup: Schemas auto-dropped on test completion; orphaned schemas cleaned via ./scripts/cleanup-test-schemas.sh
SQLx Offline Mode: Enabled for compile-time query checking without live DB; regenerate with cargo sqlx prepare

CLI Tool

cargo install --path crates/cli  # Install CLI
attune auth login                # Login
attune pack list                 # List packs
attune action execute <ref> --param key=value
attune execution list            # Monitor executions

Test Failure Protocol

Proactively investigate and fix test failures when discovered, even if unrelated to the current task.

Guidelines:

ALWAYS report test failures to the user with relevant error output
ALWAYS run tests after making changes: make test or cargo test
DO fix immediately if the cause is obvious and fixable in 1-2 attempts
DO ask the user if the failure is complex, requires architectural changes, or you're unsure of the cause
NEVER silently ignore test failures or skip tests without approval
Gather context: Run with cargo test -- --nocapture --test-threads=1 for details

Priority:

Critical (build/compile failures): Fix immediately
Related (affects current work): Fix before proceeding
Unrelated: Report and ask if you should fix now or defer

When reporting, ask: "Should I fix this first or continue with [original task]?"

Code Quality: Zero Warnings Policy

Maintain zero compiler warnings across the workspace. Clean builds ensure new issues are immediately visible.

Workflow

Check after changes: cargo check --all-targets --workspace
Before completing work: Fix or document any warnings introduced
End of session: Verify zero warnings before finishing

Handling Warnings

Fix first: Remove dead code, unused imports, unnecessary variables
Prefix _: For intentionally unused variables that document intent
Use #[allow(dead_code)]: For API methods intended for future use (add doc comment explaining why)
Never ignore blindly: Every suppression needs a clear rationale

Conservative Approach

Preserve methods that complete a logical API surface
Keep test helpers that are part of shared infrastructure
When uncertain about removal, ask the user

Red Flags

❌ Introducing new warnings
❌ Blanket #[allow(warnings)] without specific justification
❌ Accumulating warnings over time

File Naming & Location Conventions

When Adding Features:

New API Endpoint:
- Route handler in crates/api/src/routes/<domain>.rs
- DTO in crates/api/src/dto/<domain>.rs
- Update routes/mod.rs and main router
New Domain Model:
- Add to crates/common/src/models.rs
- Create migration in migrations/YYYYMMDDHHMMSS_description.sql
- Add repository in crates/common/src/repositories/<entity>.rs
New Service: Add to crates/ and update workspace Cargo.toml members
Configuration: Update crates/common/src/config.rs with serde defaults
Documentation: Add to docs/ directory

Important Files

crates/common/src/models.rs - All domain models
crates/common/src/error.rs - Error types
crates/common/src/config.rs - Configuration structure
crates/api/src/routes/mod.rs - API routing
config.development.yaml - Dev configuration
Cargo.toml - Workspace dependencies
Makefile - Development commands
docker/Dockerfile.optimized - Optimized service builds (api, executor, notifier)
docker/Dockerfile.worker.optimized - Optimized worker builds (shell, python, node, full)
docker/Dockerfile.sensor.optimized - Optimized sensor builds (base, full)
docker/Dockerfile.pack-binaries - Separate pack binary builder
scripts/build-pack-binaries.sh - Build pack binaries script

Common Pitfalls to Avoid

NEVER bypass repositories - always use the repository layer for DB access
NEVER forget RequireAuth middleware on protected endpoints
NEVER hardcode service URLs - use configuration
NEVER commit secrets in config files (use env vars in production)
NEVER hardcode schema prefixes in SQL queries - rely on PostgreSQL search_path mechanism
NEVER copy packs into Dockerfiles - they are mounted as volumes
ALWAYS use PostgreSQL enum type mappings for custom enums
ALWAYS use transactions for multi-table operations
ALWAYS start with attune/ or correct crate name when specifying file paths
ALWAYS convert runtime names to lowercase for comparison (database may store capitalized)
ALWAYS use optimized Dockerfiles for new services (selective crate copying)
REMEMBER IDs are i64, not i32 or uuid
REMEMBER schema is determined by search_path, not hardcoded in queries (production uses attune, development uses public)
REMEMBER to regenerate SQLx metadata after schema-related changes: cargo sqlx prepare
REMEMBER packs are volumes - update with restart, not rebuild
REMEMBER to build pack binaries separately: ./scripts/build-pack-binaries.sh
REMEMBER when adding mutable columns to execution, worker, enforcement, or event, add a corresponding IS DISTINCT FROM check to the entity's history trigger function in the TimescaleDB migration
REMEMBER for large JSONB columns in history triggers (like execution.result), use _jsonb_digest_summary() instead of storing the raw value — see migration 000009_timescaledb_history

Deployment

Target: Distributed deployment with separate service instances
Docker: Dockerfiles for each service (planned in docker/ dir)
Config: Use environment variables for secrets in production
Database: PostgreSQL 14+ with connection pooling
Message Queue: RabbitMQ required for service communication
Web UI: Static files served separately or via API service

Current Development Status

✅ Complete: Database migrations (22 tables, 9 consolidated migration files), API service (most endpoints), common library, message queue infrastructure, repository layer, JWT auth, CLI tool, Web UI (basic + workflow builder), Executor service (core functionality), Worker service (shell/Python execution), Runtime version data model, constraint matching, worker version selection pipeline, version verification at startup, per-version environment isolation, TimescaleDB entity history tracking (execution, worker, enforcement, event), History API endpoints (generic + entity-specific with pagination & filtering), History UI panels on entity detail pages (execution, enforcement, event), TimescaleDB continuous aggregates (5 hourly rollup views with auto-refresh policies), Analytics API endpoints (7 endpoints under /api/v1/analytics/ — dashboard, execution status/throughput/failure-rate, event volume, worker status, enforcement volume), Analytics dashboard widgets (bar charts, stacked status charts, failure rate ring gauge, time range selector)
🔄 In Progress: Sensor service, advanced workflow features, Python runtime dependency management, API/UI endpoints for runtime version management
📋 Planned: Notifier service, execution policies, monitoring, pack registry system, configurable retention periods via admin settings, export/archival to external storage

Quick Reference

Start Development Environment

# Start PostgreSQL and RabbitMQ
# Load core pack: ./scripts/load-core-pack.sh
# Start API: make run-api
# Start Web UI: cd web && npm run dev

File Path Examples

Models: attune/crates/common/src/models.rs
API routes: attune/crates/api/src/routes/actions.rs
Repositories: attune/crates/common/src/repositories/execution.rs
Migrations: attune/migrations/*.sql
Web UI: attune/web/src/
Config: attune/config.development.yaml

Documentation Locations

API docs: attune/docs/api-*.md
Configuration: attune/docs/configuration.md
Architecture: attune/docs/*-architecture.md, attune/docs/*-service.md
Testing: attune/docs/testing-*.md, attune/docs/running-tests.md, attune/docs/schema-per-test.md
Docker optimization: attune/docs/docker-layer-optimization.md, attune/docs/QUICKREF-docker-optimization.md, attune/docs/QUICKREF-buildkit-cache-strategy.md
Packs architecture: attune/docs/QUICKREF-packs-volumes.md, attune/docs/DOCKER-OPTIMIZATION-SUMMARY.md
AI Agent Work Summaries: attune/work-summary/*.md
Deployment: attune/docs/production-deployment.md
DO NOT create additional documentation files in the root of the project. all new documentation describing how to use the system should be placed in the attune/docs directory, and documentation describing the work performed should be placed in the attune/work-summary directory.

Work Summary & Reporting

Avoid redundant summarization - summarize changes once at completion, not continuously.

Guidelines:

Report progress during work: brief status updates, blockers, questions
Summarize once at completion: consolidated overview of all changes made
Work summaries: Write to attune/work-summary/*.md only at task completion, not incrementally
Avoid duplication: Don't re-explain the same changes multiple times in different formats
What changed, not how: Focus on outcomes and impacts, not play-by-play narration

Good Pattern:

[Making changes with tool calls and brief progress notes]
...
[At completion]
"I've completed the task. Here's a summary of changes: [single consolidated overview]"

Bad Pattern:

[Makes changes]
"So I changed X, Y, and Z..."
[More changes]
"To summarize, I modified X, Y, and Z..."
[Writes work summary]
"In this session I updated X, Y, and Z..."

Maintaining the AGENTS.md file

IMPORTANT: Keep this file up-to-date as the project evolves.

After making changes to the project, you MUST update this AGENTS.md file if any of the following occur:

New dependencies added or major dependencies removed (check package.json, Cargo.toml, requirements.txt, etc.)
Project structure changes: new directories/modules created, existing ones renamed or removed
Architecture changes: new layers, patterns, or major refactoring that affects how components interact
New frameworks or tools adopted (e.g., switching from REST to GraphQL, adding a new testing framework)
Deployment or infrastructure changes (new CI/CD pipelines, different hosting, containerization added)
New major features that introduce new subsystems or significantly change existing ones
Style guide or coding convention updates

`AGENTS.md` Content inclusion policy

DO NOT simply summarize changes in the AGENTS.md file. If there are existing sections that need updating due to changes in the application architecture or project structure, update them accordingly.
When relevant, work summaries should instead be written to attune/work-summary/*.md

Update procedure:

After completing your changes, review if they affect any section of AGENTS.md
If yes, immediately update the relevant sections
Add a brief comment at the top of AGENTS.md with the date and what was updated (optional but helpful)

Update format:

When updating, be surgical - modify only the affected sections rather than rewriting the entire file. Maintain the existing structure and tone.

Treat AGENTS.md as living documentation. An outdated AGENTS.md file is worse than no AGENTS.md file, as it will mislead future AI agents and waste time.

Project Documentation Index

[Attune Project Documentation Index] |root: ./ |IMPORTANT: Prefer retrieval-led reasoning over pre-training-led reasoning |IMPORTANT: This index provides a quick overview - use grep/read_file for details | | Format: path/to/dir:{file1,file2,...} | '...' indicates truncated file list - use grep/list_directory for full contents | | To regenerate this index: make generate-agents-index | |docs:{MIGRATION-queue-separation-2026-02-03.md,QUICKREF-containerized-workers.md,QUICKREF-rabbitmq-queues.md,QUICKREF-sensor-worker-registration.md,QUICKREF-unified-runtime-detection.md,README.md,docker-deployment.md,pack-runtime-environments.md,worker-containerization.md,worker-containers-quickstart.md} |docs/api:{api-actions.md,api-completion-plan.md,api-events-enforcements.md,api-executions.md,api-inquiries.md,api-pack-testing.md,api-pack-workflows.md,api-packs.md,api-rules.md,api-secrets.md,api-triggers-sensors.md,api-workflows.md,openapi-client-generation.md,openapi-spec-completion.md} |docs/architecture:{executor-service.md,notifier-service.md,pack-management-architecture.md,queue-architecture.md,sensor-service.md,trigger-sensor-architecture.md,web-ui-architecture.md,webhook-system-architecture.md,worker-service.md} |docs/authentication:{auth-quick-reference.md,authentication.md,secrets-management.md,security-review-2024-01-02.md,service-accounts.md,token-refresh-quickref.md,token-rotation.md} |docs/cli:{cli-profiles.md,cli.md} |docs/configuration:{CONFIG_README.md,config-troubleshooting.md,configuration.md,env-to-yaml-migration.md} |docs/dependencies:{dependency-deduplication-results.md,dependency-deduplication.md,dependency-isolation.md,dependency-management.md,http-client-consolidation-complete.md,http-client-consolidation-plan.md,sea-query-removal.md,serde-yaml-migration.md,workspace-dependency-compliance-audit.md} |docs/deployment:{ops-runbook-queues.md,production-deployment.md} |docs/development:{QUICKSTART-vite.md,WORKSPACE_SETUP.md,agents-md-index.md,compilation-notes.md,dead-code-cleanup.md,documentation-organization.md,vite-dev-setup.md} |docs/examples:{complete-workflow.yaml,pack-test-demo.sh,registry-index.json,rule-parameter-examples.md,simple-workflow.yaml} |docs/guides:{QUICKREF-timer-happy-path.md,quick-start.md,quickstart-example.md,quickstart-timer-demo.md,timer-sensor-quickstart.md,workflow-quickstart.md} |docs/migrations:{workflow-task-execution-consolidation.md} |docs/packs:{PACK_TESTING.md,QUICKREF-git-installation.md,core-pack-integration.md,pack-install-testing.md,pack-installation-git.md,pack-registry-cicd.md,pack-registry-spec.md,pack-structure.md,pack-testing-framework.md} |docs/performance:{QUICKREF-performance-optimization.md,log-size-limits.md,performance-analysis-workflow-lists.md,performance-before-after-results.md,performance-context-cloning-diagram.md} |docs/plans:{schema-per-test-refactor.md,timescaledb-entity-history.md} |docs/sensors:{CHECKLIST-sensor-worker-registration.md,COMPLETION-sensor-worker-registration.md,SUMMARY-database-driven-detection.md,database-driven-runtime-detection.md,native-runtime.md,sensor-authentication-overview.md,sensor-interface.md,sensor-lifecycle-management.md,sensor-runtime.md,sensor-service-setup.md,sensor-worker-registration.md} |docs/testing:{e2e-test-plan.md,running-tests.md,schema-per-test.md,test-user-setup.md,testing-authentication.md,testing-dashboard-rules.md,testing-status.md} |docs/web-ui:{web-ui-pack-testing.md,websocket-usage.md} |docs/webhooks:{webhook-manual-testing.md,webhook-testing.md} |docs/workflows:{dynamic-parameter-forms.md,execution-hierarchy.md,inquiry-handling.md,parameter-mapping-status.md,rule-parameter-mapping.md,rule-trigger-params.md,workflow-execution-engine.md,workflow-implementation-plan.md,workflow-orchestration.md,workflow-summary.md} |scripts:{check-workspace-deps.sh,cleanup-test-schemas.sh,create-test-user.sh,create_test_user.sh,generate-python-client.sh,generate_agents_md_index.py,load-core-pack.sh,load_core_pack.py,quick-test-happy-path.sh,seed_core_pack.sql,seed_runtimes.sql,setup-db.sh,setup-e2e-db.sh,setup_timer_echo_rule.sh,start-all-services.sh,start-e2e-services.sh,start_services_test.sh,status-all-services.sh,stop-all-services.sh,stop-e2e-services.sh,...} |work-summary:{2025-01-console-logging-cleanup.md,2025-01-token-refresh-improvements.md,2025-01-websocket-duplicate-connection-fix.md,2026-02-02-unified-runtime-verification.md,2026-02-03-canonical-message-types.md,2026-02-03-inquiry-queue-separation.md,2026-02-04-event-generation-fix.md,README.md,auto-populate-ref-from-label.md,buildkit-cache-implementation.md,collapsible-navigation-implementation.md,containerized-workers-implementation.md,docker-build-race-fix.md,docker-containerization-complete.md,docker-migrations-startup-fix.md,empty-pack-creation-ui.md,git-pack-installation.md,pack-runtime-environments.md,sensor-service-cleanup-standalone-only.md,sensor-worker-registration.md,...} |work-summary/changelogs:{API-COMPLETION-SUMMARY.md,CHANGELOG.md,CLEANUP_SUMMARY_2026-01-27.md,FIFO-ORDERING-COMPLETE.md,MIGRATION_CONSOLIDATION_SUMMARY.md,cli-integration-tests-summary.md,core-pack-setup-summary.md,web-ui-session-summary.md,webhook-phase3-summary.md,webhook-testing-summary.md,workflow-loader-summary.md} |work-summary/features:{AUTOMATIC-SCHEMA-CLEANUP-ENHANCEMENT.md,TESTING-TIMER-DEMO.md,e2e-test-schema-issues.md,openapi-spec-verification.md,sensor-runtime-implementation.md,sensor-service-implementation.md} |work-summary/migrations:{2026-01-17-orquesta-refactoring.md,2026-01-24-generated-client-migration.md,2026-01-27-workflow-migration.md,DEPLOYMENT-READY-performance-optimization.md,MIGRATION_NEXT_STEPS.md,migration_comparison.txt,migration_consolidation_status.md} |work-summary/phases:{2025-01-policy-ordering-plan.md,2025-01-secret-passing-fix-plan.md,2025-01-workflow-performance-analysis.md,PHASE-5-COMPLETE.md,PHASE_1_1_SUMMARY.txt,PROBLEM.md,Pitfall-Resolution-Plan.md,SENSOR_SERVICE_README.md,StackStorm-Lessons-Learned.md,StackStorm-Pitfalls-Analysis.md,orquesta-refactor-plan.md,phase-1-1-complete.md,phase-1.2-models-repositories-complete.md,phase-1.2-repositories-summary.md,phase-1.3-test-infrastructure-summary.md,phase-1.3-yaml-validation-complete.md,phase-1.4-COMPLETE.md,phase-1.4-loader-registration-progress.md,phase-1.5-COMPLETE.md,phase-1.6-pack-integration-complete.md,...} |work-summary/sessions:{2024-01-13-event-enforcement-endpoints.md,2024-01-13-inquiry-endpoints.md,2024-01-13-integration-testing-setup.md,2024-01-13-route-conflict-fix.md,2024-01-13-secret-management-api.md,2024-01-17-sensor-runtime.md,2024-01-17-sensor-service-session.md,2024-01-20-core-pack-unit-tests.md,2024-01-20-pack-testing-framework-phase1.md,2024-01-21-pack-registry-phase1.md,2024-01-21-pack-registry-phase2.md,2024-01-22-pack-registry-phase3.md,2024-01-22-pack-registry-phase4.md,2024-01-22-pack-registry-phase5.md,2024-01-22-pack-registry-phase6.md,2025-01-13-phase-1.4-session.md,2025-01-13-yaml-configuration.md,2025-01-16_migration_consolidation.md,2025-01-17-performance-optimization-complete.md,2025-01-18-timer-triggers.md,...} |work-summary/status:{ACCOMPLISHMENTS.md,COMPILATION_STATUS.md,FIFO-ORDERING-STATUS.md,FINAL_STATUS.md,PROGRESS.md,SENSOR_STATUS.md,TEST-STATUS.md,TODO.OLD.md,TODO.md}

51 KiB Raw Blame History

Attune Project Rules

Project Overview

Development Status: Pre-Production

Breaking Changes Policy

Languages & Core Technologies

Project Structure (Cargo Workspace)

Service Architecture (Distributed Microservices)

Docker Compose Orchestration

Docker Build Optimization

Docker Runtime Standardization

Packs Volume Architecture

Runtime Environments Volume

Domain Model & Event Flow

Key Tools & Libraries

Shared Dependencies (workspace-level)

Web UI Dependencies

Configuration System

Authentication & Security

Code Conventions & Patterns

General

Database Layer

Pack File Loading & Action Execution

API Service (crates/api)

Common Library (crates/common)

Template Variable Syntax

Web UI (web/)

Development Workflow

Common Commands (Makefile)

Database Operations

Testing

CLI Tool

Test Failure Protocol

Guidelines:

Priority:

Code Quality: Zero Warnings Policy

Workflow

Handling Warnings

Conservative Approach

Red Flags

File Naming & Location Conventions

When Adding Features:

Important Files

Common Pitfalls to Avoid

Deployment

Current Development Status

Quick Reference

Start Development Environment

File Path Examples

Documentation Locations

Work Summary & Reporting

Guidelines:

Good Pattern:

Bad Pattern:

Maintaining the AGENTS.md file

AGENTS.md Content inclusion policy

Update procedure:

Update format:

Project Documentation Index

51 KiB

Raw Blame History

API Service (`crates/api`)

Common Library (`crates/common`)

Web UI (`web/`)

`AGENTS.md` Content inclusion policy