attune-system/attune
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

node running, runtime version awareness

Browse Source
This commit is contained in:
David Culbreth
2026-02-25 23:24:07 -06:00
parent e89b5991ec
commit 495b81236a
54 changed files with 4308 additions and 246 deletions
Download Patch File Download Diff File Expand all files Collapse all files

58
work-summary/2026-02-25-workflow-companion-action-fix.md Normal file
View File

@@ -0,0 +1,58 @@
# Workflow Companion Action Fix
**Date**: 2026-02-25
**Issue**: Workflows created via the web Workflow Builder did not appear in the action list or action palette.
## Root Cause
When a workflow was saved via the Workflow Builder UI (`POST /api/v1/packs/{pack_ref}/workflow-files`), the API handler:
1. ✅ Wrote the workflow YAML file to disk
2. ✅ Created a `workflow_definition` record in the database
3. ❌ Did **not** create a corresponding `action` record
The action palette and action list both query the `action` table. Since no `action` record was created for workflows, they were invisible in those views despite existing in the `workflow_definition` table.
The same gap existed in:
- `POST /api/v1/workflows` (create workflow via API)
- `PUT /api/v1/workflows/{ref}` (update workflow via API)
- `PUT /api/v1/workflows/{ref}/file` (update workflow file)
- `WorkflowRegistrar` in both `crates/common` and `crates/executor` (pack-based workflow loading)
## Fix
### Companion Action Records
Every workflow definition now gets a **companion action record** with:
- `is_workflow = true`
- `workflow_def` FK pointing to the workflow definition
- Same `ref`, `label`, `description`, `param_schema`, `out_schema` as the workflow
- `entrypoint` set to the workflow YAML file path (e.g., `workflows/{name}.workflow.yaml`)
- `runtime = NULL` (workflows don't use a runtime — the executor reads the definition from DB)
### Files Modified
**`crates/api/src/routes/workflows.rs`**:
- `save_workflow_file()`: Now creates a companion action after creating the workflow definition
- `create_workflow()`: Now creates a companion action after creating the workflow definition
- `update_workflow()`: Now updates the companion action's metadata to stay in sync
- `update_workflow_file()`: Uses `ensure_companion_action()` to update or backfill the action
- `delete_workflow()`: No change needed — the `action.workflow_def` FK has `ON DELETE CASCADE`, so deleting the workflow definition automatically deletes the companion action
- Added three helper functions: `create_companion_action()`, `update_companion_action()`, `ensure_companion_action()`
**`crates/common/src/workflow/registrar.rs`**:
- `register_workflow()`: Creates companion action on new workflows, ensures/updates on existing
- Added `create_companion_action()` and `ensure_companion_action()` methods
**`crates/executor/src/workflow/registrar.rs`**:
- Same changes as the common crate registrar (this is a duplicate used by the executor service)
### Backfill Support
The `ensure_companion_action()` function handles workflows that were created before this fix. When updating such a workflow, if no companion action exists, it creates one. This means existing workflows will get their companion action on the next update/re-registration.
## Testing
- All workspace lib tests pass (86 API, 160 common, 85 executor, 82 worker)
- Zero compiler warnings across the workspace
- Compilation clean after `cargo clean` + full rebuild

136
work-summary/2026-02-26-runtime-version-worker-integration.md Normal file
View File

@@ -0,0 +1,136 @@
# Runtime Version Worker Pipeline Integration
**Date**: 2026-02-26
## Summary
Integrated the runtime version system into the worker execution pipeline, enabling version-aware action execution. When an action declares a `runtime_version_constraint` (e.g., `">=3.12"`), the worker now automatically selects the best matching runtime version and uses its version-specific interpreter, environment commands, and configuration.
## Changes Made
### 1. Version-Aware Execution Context (`crates/worker/src/runtime/mod.rs`)
Added three new fields to `ExecutionContext`:
- `runtime_config_override: Option<RuntimeExecutionConfig>` — version-specific execution config that overrides the parent runtime's config
- `runtime_env_dir_suffix: Option<String>` — directory suffix for per-version environment isolation (e.g., `"python-3.12"`)
- `selected_runtime_version: Option<String>` — selected version string for logging/diagnostics
### 2. Runtime Version Resolution in Executor (`crates/worker/src/executor.rs`)
Added `resolve_runtime_version()` method to `ActionExecutor` that:
- Queries `runtime_version` rows for the action's runtime from the database
- Calls `select_best_version()` with the action's `runtime_version_constraint`
- Returns the selected version's `execution_config`, env dir suffix, and version string
- Gracefully handles missing versions, failed queries, and unmatched constraints with warnings
The method is called during `prepare_execution_context()` and the results are passed through the `ExecutionContext` to the `ProcessRuntime`.
### 3. Version-Aware ProcessRuntime (`crates/worker/src/runtime/process.rs`)
Modified `ProcessRuntime::execute()` to:
- Use `effective_config` (either `context.runtime_config_override` or `self.config`)
- Compute version-specific environment directories when `runtime_env_dir_suffix` is present
- Use version-specific interpreter resolution, environment variables, and interpreter args
- Recreate broken environments using the effective (possibly version-specific) config
- Log the selected version in all execution diagnostics
### 4. Runtime Version Verification (`crates/worker/src/version_verify.rs`)
New module that verifies which runtime versions are available on the system at worker startup:
- Extracts verification commands from `distributions.verification.commands` JSONB
- Runs each command with proper timeout handling (10s per command)
- Matches exit codes and output patterns (regex-based)
- Updates `available` and `verified_at` columns in the database
- Respects `ATTUNE_WORKER_RUNTIMES` filter (alias-aware)
- Falls back to `binary --version` check when no explicit verification commands exist
- Includes 8 unit tests for command extraction and verification logic
### 5. Version-Aware Environment Setup (`crates/worker/src/env_setup.rs`)
Extended the proactive environment setup system to create per-version environments:
- At startup scan: creates environments at `{runtime_envs_dir}/{pack_ref}/{runtime_name}-{version}` for each available version
- On `pack.registered` MQ events: creates per-version environments alongside base environments
- Each version environment uses the version's own `execution_config` (different interpreter binary, venv create command, etc.)
- Base (unversioned) environments are still created for backward compatibility
### 6. Worker Startup Sequence (`crates/worker/src/service.rs`)
Updated the startup sequence to include version verification before environment setup:
1. Connect to DB and MQ
2. Load runtimes → create ProcessRuntime instances
3. Register worker and set up MQ infrastructure
4. **NEW: Verify runtime versions** (run verification commands, update `available` flags)
5. Set up runtime environments (now version-aware)
6. Start heartbeat, execution consumer, pack registration consumer
### 7. Test Updates
Updated all `ExecutionContext` initializations across the workspace to include the new fields:
- `crates/worker/src/runtime/process.rs` — 8 test contexts
- `crates/worker/src/runtime/local.rs` — 2 test contexts
- `crates/worker/src/runtime/python.rs` — 4 test contexts
- `crates/worker/src/runtime/shell.rs` — 8 test contexts
- `crates/worker/tests/dependency_isolation_test.rs` — 1 test context
- `crates/worker/tests/log_truncation_test.rs` — 3 test contexts
- `crates/worker/tests/security_tests.rs` — 8 test contexts
Also fixed pre-existing missing `env_vars` fields in `RuntimeExecutionConfig` initializations in test helper functions.
## Architecture
```
Action with runtime_version_constraint=">=3.12"
│
▼
ActionExecutor::prepare_execution_context()
│
├── Load runtime model (e.g., "Python")
├── Query RuntimeVersion rows for that runtime
├── select_best_version(versions, ">=3.12")
│ └── Returns Python 3.13 (highest available matching >=3.12)
├── Set runtime_config_override = version's execution_config
├── Set runtime_env_dir_suffix = "python-3.13"
└── Set selected_runtime_version = "3.13"
│
▼
ProcessRuntime::execute(context)
│
├── effective_config = context.runtime_config_override (python3.13 config)
├── env_dir = /opt/attune/runtime_envs/{pack}/python-3.13
├── interpreter = python3.13 (from version-specific config)
└── Execute action with version-specific setup
```
## Environment Directory Layout
```
/opt/attune/runtime_envs/
└── my_pack/
├── python/ # Base (unversioned) environment
├── python-3.11/ # Python 3.11 specific environment
├── python-3.12/ # Python 3.12 specific environment
├── python-3.13/ # Python 3.13 specific environment
├── node/ # Base Node.js environment
├── node-18/ # Node.js 18 specific environment
├── node-20/ # Node.js 20 specific environment
└── node-22/ # Node.js 22 specific environment
```
## Dependencies Added
- `regex` (workspace dependency, already in `Cargo.toml`) added to `attune-worker` for verification pattern matching
## Test Results
- All 93 worker unit tests pass
- All 17 dependency isolation tests pass
- All 8 log truncation tests pass
- All 7 security tests pass
- All 33 version matching tests + 2 doc tests pass
- Zero compiler warnings across the workspace
## What's Next
- API/UI endpoints for runtime version management (list, verify, toggle availability)
- Sensor service integration with version-aware runtime selection
- Runtime version auto-detection at pack install time (not just worker startup)

92
work-summary/2026-02-26-runtime-versions.md Normal file
View File

@@ -0,0 +1,92 @@
# Runtime Versions Feature
**Date**: 2026-02-26
**Scope**: Data model, repositories, version matching, pack loader, API, core pack definitions
## Summary
Added support for multiple versions of the same runtime (e.g., Python 3.11, 3.12, 3.13 or Node.js 18, 20, 22). Actions and sensors can now declare a semver version constraint (e.g., `>=3.12`, `~18.0`, `>=3.12,<4.0`) to specify which runtime version they require. The system selects the best matching available version at execution time.
The design is fully data-driven — no hard-coded handling for any specific runtime. All version detection, constraint matching, and execution configuration is driven by database records and YAML definitions.
## Changes
### New Migration: `20260226000000_runtime_versions.sql`
- **`runtime_version` table**: Stores version-specific execution configurations per runtime
- `runtime` (FK → runtime.id), `runtime_ref`, `version` (semver string)
- `version_major`, `version_minor`, `version_patch` (ints for efficient range queries)
- `execution_config` JSONB — complete standalone config (not a diff) replacing the parent runtime's config when this version is selected
- `distributions` JSONB — version-specific verification commands
- `is_default` (at most one per runtime), `available`, `verified_at`
- `meta` JSONB for arbitrary metadata (EOL dates, LTS codenames, etc.)
- Unique constraint on `(runtime, version)`
- **`action.runtime_version_constraint`** — new nullable TEXT column for semver constraints
- **`sensor.runtime_version_constraint`** — same as above
### New Module: `crates/common/src/version_matching.rs`
- `parse_version()` — lenient semver parsing (`"3.12"` → `3.12.0`, `"v20.11"` → `20.11.0`)
- `parse_constraint()` — constraint parsing with bare version support (`"3.12"` → `~3.12`)
- `matches_constraint()` — check if a version satisfies a constraint
- `select_best_version()` — pick the highest available version matching a constraint, with default-version preference when no constraint is specified
- `extract_version_components()` — split version string into (major, minor, patch) for DB columns
- 33 unit tests covering all constraint operators, edge cases, and selection logic
### New Repository: `crates/common/src/repositories/runtime_version.rs`
- Full CRUD (FindById, List, Create, Update, Delete)
- `find_by_runtime()` — all versions for a runtime, ordered newest-first
- `find_by_runtime_ref()` — same, by runtime ref string
- `find_available_by_runtime()` — only available versions
- `find_default_by_runtime()` — the default version
- `find_by_runtime_and_version()` — exact version lookup
- `clear_default_for_runtime()` — helper for changing defaults
- `set_availability()` — mark available/unavailable with timestamp
- `delete_by_runtime()` — bulk delete
### Model Changes
- **`RuntimeVersion`** struct in `models::runtime` module with `parsed_execution_config()` method
- **`Action`** — added `runtime_version_constraint: Option<String>`
- **`Sensor`** — added `runtime_version_constraint: Option<String>`
### Pack Loader Updates (`crates/common/src/pack_registry/loader.rs`)
- `load_runtimes()` now calls `load_runtime_versions()` after creating each runtime
- `load_runtime_versions()` parses the `versions` array from runtime YAML and creates `runtime_version` rows
- `load_actions()` reads `runtime_version` from action YAML → stored as `runtime_version_constraint`
- `load_sensors()` reads `runtime_version` from sensor YAML → same
### Repository Updates
- **Action repository**: All SELECT/INSERT/UPDATE/RETURNING queries updated to include `runtime_version_constraint`
- **Sensor repository**: Same — all queries updated
- **Input structs**: `CreateActionInput`, `UpdateActionInput`, `CreateSensorInput`, `UpdateSensorInput` all include the new field
### API Updates
- **Action DTOs**: `CreateActionRequest`, `UpdateActionRequest`, `ActionResponse`, `ActionSummary` include `runtime_version_constraint`
- **Route handlers**: `create_action`, `update_action` pass the field through
- OpenAPI annotations added for the new field
### Core Pack Runtime YAML Updates
- **`packs/core/runtimes/python.yaml`**: Added `versions` array with Python 3.11, 3.12 (default), 3.13 — each with version-specific interpreter binary (`python3.11`, `python3.12`, `python3.13`), venv commands, and verification patterns
- **`packs/core/runtimes/nodejs.yaml`**: Added `versions` array with Node.js 18, 20 (default), 22 — each with version-specific binary, verification commands, and LTS metadata
### Dependency Addition
- **`semver` 1.0** (with serde feature) added to workspace and `attune-common` Cargo.toml
### Test Fixes
- All callsites constructing `CreateActionInput`, `CreateSensorInput`, `UpdateActionInput`, `UpdateSensorInput` across the workspace updated with `runtime_version_constraint: None` (approximately 20 files touched)
## Architecture Decisions
1. **Full execution_config per version, not diffs**: Each `runtime_version` row stores a complete `execution_config` rather than overrides. This avoids merge complexity and makes each version self-contained.
2. **Constraint stored as text, matched at runtime**: The `runtime_version_constraint` column stores the raw semver string. Matching is done in Rust code using the `semver` crate rather than in SQL, because semver range logic is complex and better handled in application code.
3. **Bare version = tilde range**: A constraint like `"3.12"` is interpreted as `~3.12` (>=3.12.0, <3.13.0), which is the most common developer expectation for "compatible with 3.12".
4. **No hard-coded runtime handling**: The entire version system is data-driven through the `runtime_version` table and YAML definitions. Any runtime can define versions with arbitrary verification commands and execution configs.
## What's Next
- **Worker integration**: The worker execution pipeline should query `runtime_version` rows when an action has a `runtime_version_constraint`, use `select_best_version()` to pick the right version, and use that version's `execution_config` instead of the parent runtime's.
- **Runtime version detection**: The `RuntimeDetector` should verify individual version availability by running each version's verification commands and updating the `available`/`verified_at` fields.
- **Environment isolation per version**: The `runtime_envs_dir` path pattern may need to include the version (e.g., `{runtime_envs_dir}/{pack_ref}/{runtime_name}-{version}`) to support multiple Python versions with separate virtualenvs.
- **API endpoints**: CRUD endpoints for `runtime_version` management (list versions for a runtime, register new versions, mark availability).
- **Web UI**: Display version information in runtime/action views, version constraint field in action editor.

94
work-summary/2026-02-nodejs-runtime-fix.md Normal file
View File

@@ -0,0 +1,94 @@
# Node.js Runtime Fix
**Date**: 2026-02
**Scope**: Worker runtime loading, environment setup, action execution
## Problem
Node.js actions from installed packs (e.g., `nodejs_example`) were stuck in "In Progress" status and never completing. Three root causes were identified:
### 1. Runtime Name Mismatch (Blocking)
The `ATTUNE_WORKER_RUNTIMES` env var for the node worker is `shell,node`, but the database runtime name is "Node.js" which lowercases to `"node.js"`. The worker's filter used exact string matching (`filter.contains(&rt_name)`), so `"node.js"` was never found in `["shell", "node"]`. **The Node.js ProcessRuntime was never registered**, meaning no worker could handle Node.js executions.
### 2. Broken Environment Setup
The Node.js `execution_config` in `packs/core/runtimes/nodejs.yaml` had:
- `create_command: ["npm", "init", "-y"]` — ran in the pack directory (read-only in Docker), didn't create anything at `{env_dir}`
- `install_command: ["npm", "install", "--prefix", "{pack_dir}"]` — tried to write `node_modules` into the read-only pack directory
### 3. Missing NODE_PATH
Even with correct installation, Node.js couldn't find modules installed at `{env_dir}/node_modules` because Node resolves modules relative to the script location, not the environment directory. No mechanism existed to set `NODE_PATH` during execution.
## Solution
### Runtime Name Normalization
Added `normalize_runtime_name()` and `runtime_in_filter()` to `crates/common/src/runtime_detection.rs`. These functions map common aliases to canonical names:
- `node` / `nodejs` / `node.js` → `node`
- `python` / `python3` → `python`
- `shell` / `bash` / `sh` → `shell`
- `native` / `builtin` / `standalone` → `native`
Updated `crates/worker/src/service.rs` and `crates/worker/src/env_setup.rs` to use `runtime_in_filter()` instead of exact string matching.
### RuntimeExecutionConfig.env_vars
Added an `env_vars: HashMap<String, String>` field to `RuntimeExecutionConfig` in `crates/common/src/models.rs`. Values support the same template variables as other fields (`{env_dir}`, `{pack_dir}`, `{interpreter}`, `{manifest_path}`).
In `ProcessRuntime::execute` (`crates/worker/src/runtime/process.rs`), runtime env_vars are resolved and injected into the action's environment before building the command.
### Fixed Node.js Runtime Config
Updated `packs/core/runtimes/nodejs.yaml` and `scripts/seed_runtimes.sql`:
```yaml
execution_config:
interpreter:
binary: node
args: []
file_extension: ".js"
environment:
env_type: node_modules
dir_name: node_modules
create_command:
- sh
- "-c"
- "mkdir -p {env_dir} && cp {manifest_path} {env_dir}/ 2>/dev/null || true"
interpreter_path: null
dependencies:
manifest_file: package.json
install_command:
- npm
- install
- "--prefix"
- "{env_dir}"
env_vars:
NODE_PATH: "{env_dir}/node_modules"
```
## Files Changed
| File | Change |
|------|--------|
| `crates/common/src/models.rs` | Added `env_vars` field to `RuntimeExecutionConfig` |
| `crates/common/src/runtime_detection.rs` | Added `normalize_runtime_name()`, `runtime_matches_filter()`, `runtime_in_filter()` with tests |
| `crates/worker/src/service.rs` | Use `runtime_in_filter()` for ATTUNE_WORKER_RUNTIMES matching |
| `crates/worker/src/env_setup.rs` | Use `runtime_in_filter()` for runtime filter matching in env setup |
| `crates/worker/src/runtime/process.rs` | Inject `env_vars` into action environment during execution |
| `crates/worker/src/runtime/local.rs` | Added `env_vars` field to fallback config |
| `packs/core/runtimes/nodejs.yaml` | Fixed `create_command`, `install_command`, added `env_vars` |
| `scripts/seed_runtimes.sql` | Fixed Node.js execution_config to match YAML |
| `crates/worker/tests/*.rs` | Added `env_vars` field to test configs |
| `AGENTS.md` | Documented runtime name normalization, env_vars, and env setup requirements |
## Testing
- All 424 unit tests pass
- Zero compiler warnings
- New tests for `normalize_runtime_name`, `runtime_matches_filter`, `runtime_in_filter`
## Deployment Notes
After deploying, the Node.js runtime config in the database needs to be updated. This happens automatically when packs are re-registered (the pack loader reads the updated YAML). Alternatively, run the updated `seed_runtimes.sql` script.