attune-system/attune
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[WIP] workflow builder

Browse Source
This commit is contained in:
David Culbreth
2026-02-23 20:45:10 -06:00
parent d629da32fa
commit 53a3fbb6b1
66 changed files with 7887 additions and 1608 deletions
Download Patch File Download Diff File Expand all files Collapse all files

133
work-summary/2026-02-04-orquesta-style-transitions.md Normal file
View File

@@ -0,0 +1,133 @@
# Orquesta-Style Task Transition Model
**Date**: 2026-02-04
## Overview
Refactored the workflow builder's task transition model from flat `on_success`/`on_failure`/`on_complete`/`on_timeout` fields to an Orquesta-style ordered `next` array of transitions. Each transition can specify a `when` condition, `publish` directives, and multiple `do` targets — enabling far more expressive workflow definitions.
Also added visual drag handles to task nodes in the workflow builder for creating transitions via drag-and-drop.
## Motivation
The previous model only allowed a single target task per transition type and had no way to:
- Route to multiple tasks from a single transition
- Attach per-transition variable publishing
- Use custom condition expressions beyond the four fixed types
- Publish variables without transitioning to another task
The Orquesta model (from StackStorm) solves all of these with a simple, ordered list of conditional transitions.
## Changes
### Frontend (`web/`)
#### `web/src/types/workflow.ts`
- **Added** `TaskTransition` type: `{ when?: string; publish?: PublishDirective[]; do?: string[] }`
- **Added** `TransitionPreset` type and constants (`PRESET_WHEN`, `PRESET_LABELS`) for the three common quick-access patterns: succeeded, failed, always
- **Added** `classifyTransitionWhen()` and `transitionLabel()` for edge visualization
- **Added** `EdgeType` — simplified to `"success" | "failure" | "complete" | "custom"`
- **Added** helper functions: `findOrCreateTransition()`, `addTransitionTarget()`, `removeTaskFromTransitions()`
- **Removed** `on_success`, `on_failure`, `on_complete`, `on_timeout`, `decision`, `publish` fields from `WorkflowTask`
- **Removed** `DecisionBranch` type (subsumed by `TaskTransition.when`)
- **Updated** `WorkflowYamlTask` to use `next?: WorkflowYamlTransition[]`
- **Updated** `builderStateToDefinition()` to serialize `next` array
- **Updated** `definitionToBuilderState()` to load both new `next` format and legacy flat fields (auto-converts)
- **Updated** `deriveEdges()` to iterate `task.next[].do[]`
- **Updated** `validateWorkflow()` to validate `next[].do[]` targets
#### `web/src/components/workflows/TaskNode.tsx`
- **Redesigned output handles**: Three color-coded drag handles at bottom (green=succeeded, red=failed, gray=always)
- **Added input handle**: Neutral circle at top center as drop target, highlights purple during active connection
- **Removed** old footer link-icon buttons
- **Added** transition summary in node body (e.g., "2 targets via 1 transition")
- **Added** custom transitions badge
#### `web/src/components/workflows/WorkflowEdges.tsx`
- Updated edge colors for new `EdgeType` values
- Preview line color now uses `TransitionPreset` mapping
- Dynamic label width based on text content
#### `web/src/components/workflows/WorkflowCanvas.tsx`
- Updated to use `TransitionPreset` instead of `TransitionType`
- Added `onMouseUp` handler for drag-to-cancel on canvas background
#### `web/src/components/workflows/TaskInspector.tsx`
- **Replaced** four fixed `TransitionField` dropdowns with a dynamic transition list editor
- Each transition card shows: `when` expression (editable), `do` target list (add/remove), `publish` key-value pairs (add/remove)
- Quick-set buttons for common `when` presets (On Success, On Failure, Always)
- Add transition buttons: "On Success", "On Failure", "Custom transition"
- **Moved** publish variables from task-level section to per-transition
- **Removed** old `TransitionField` component
- **Added** Join section for barrier configuration
#### `web/src/pages/actions/WorkflowBuilderPage.tsx`
- Updated `handleSetConnection` to use `addTransitionTarget()` with `TransitionPreset`
- Updated `handleDeleteTask` to use `removeTaskFromTransitions()`
### Backend (`crates/`)
#### `crates/common/src/workflow/parser.rs`
- **Added** `TaskTransition` struct: `{ when, publish, do }`
- **Added** `Task::normalize_transitions()` — converts legacy fields into `next` array
- **Added** `Task::all_transition_targets()` — collects all referenced task names
- **Updated** `parse_workflow_yaml()` to call `normalize_all_transitions()` after parsing
- **Updated** `validate_task()` to use `all_transition_targets()` instead of checking individual fields
- Legacy fields (`on_success`, `on_failure`, `on_complete`, `on_timeout`, `decision`) retained for deserialization but cleared after normalization
- **Added** 12 new tests covering both new and legacy formats
#### `crates/common/src/workflow/validator.rs`
- Updated `build_graph()` and `find_entry_points()` to use `task.all_transition_targets()`
#### `crates/common/src/workflow/mod.rs`
- Exported new `TaskTransition` type
#### `crates/executor/src/workflow/graph.rs`
- **Replaced** `TaskTransitions` struct (flat fields) with `Vec<GraphTransition>`
- **Added** `GraphTransition`: `{ when, publish: Vec<PublishVar>, do_tasks: Vec<String> }`
- **Added** `PublishVar`: `{ name, expression }` — preserves both key and value
- **Added** `TransitionKind` enum and `GraphTransition::kind()` classifier
- **Added** `TaskGraph::matching_transitions()` — returns full transition objects for coordinators
- **Added** `TaskGraph::all_transition_targets()` — all target names from a task
- **Updated** `next_tasks()` to evaluate transitions by `TransitionKind`
- **Updated** `compute_inbound_edges()` to iterate `GraphTransition.do_tasks`
- **Updated** `extract_publish_vars()` to return `Vec<PublishVar>` instead of `Vec<String>`
- **Added** 12 new tests
#### `crates/executor/src/workflow/task_executor.rs`
- Updated variable publishing to extract from matching transitions instead of removed `task.publish` field
## YAML Format
### New (canonical)
```yaml
tasks:
- name: task1
action: core.echo
next:
- when: "{{ succeeded() }}"
publish:
- result: "{{ result() }}"
do:
- task2
- log
- when: "{{ failed() }}"
do:
- error_handler
```
### Legacy (still parsed, auto-converted)
```yaml
tasks:
- name: task1
action: core.echo
on_success: task2
on_failure: error_handler
```
## Test Results
- **Parser tests**: 37 passed (includes 12 new)
- **Graph tests**: 12 passed (includes 10 new)
- **TypeScript**: Zero errors
- **Rust workspace**: Zero warnings

57
work-summary/2026-02-05-pack-reinstall-preserve-adhoc-rules.md Normal file
View File

@@ -0,0 +1,57 @@
# Pack Reinstallation: Preserve Ad-Hoc Rules
**Date**: 2026-02-05
## Problem
When reinstalling a pack (force=true), user-created (ad-hoc) rules belonging to that pack were being permanently deleted. This happened because the reinstallation flow performed a hard `PackRepository::delete()` before recreating the pack, and the `rule.pack` foreign key uses `ON DELETE CASCADE` — destroying all rules owned by the pack, including custom ones created by users through the API or UI.
Additionally, rules from *other* packs that referenced triggers or actions from the reinstalled pack would have their `action` and `trigger` FK columns set to `NULL` (via `ON DELETE SET NULL`) when the old pack's entities were cascade-deleted, but were never re-linked after the new entities were created.
## Root Cause
In `register_pack_internal()` (`crates/api/src/routes/packs.rs`), the force-reinstall path was:
```
1. Delete existing pack (CASCADE deletes ALL rules, actions, triggers, sensors, runtimes)
2. Create new pack + components
```
No distinction was made between pack-defined rules (`is_adhoc = false`) and user-created rules (`is_adhoc = true`).
## Solution
### Repository Changes (`crates/common/src/repositories/rule.rs`)
Added four new methods/types:
- **`RestoreRuleInput`** — Like `CreateRuleInput` but with `Option<Id>` for action and trigger, since referenced entities may not exist after reinstallation.
- **`find_adhoc_by_pack()`** — Queries ad-hoc rules (`is_adhoc = true`) belonging to a specific pack.
- **`restore_rule()`** — Inserts a rule with optional action/trigger FK IDs, always setting `is_adhoc = true`.
- **`relink_action_by_ref()` / `relink_trigger_by_ref()`** — Updates rules with NULL action/trigger FKs, matching by the text `_ref` field to re-establish the link.
### Pack Registration Changes (`crates/api/src/routes/packs.rs`)
Modified `register_pack_internal()` to add two phases after component loading:
**Phase 1 — Save & Restore Ad-Hoc Rules:**
- Before deleting the old pack, queries and saves all ad-hoc rules
- After the new pack and components are created, restores each saved rule with the new pack ID
- Resolves action/trigger FKs by looking up entities by ref; if not found, the rule is preserved with NULL FKs (non-functional but not lost)
**Phase 2 — Re-link Orphaned Rules from Other Packs:**
- Iterates over all newly created actions and triggers
- For each, updates any rules (from any pack) that have a matching `_ref` but a NULL FK
## Files Changed
| File | Change |
|------|--------|
| `crates/common/src/repositories/rule.rs` | Added `RestoreRuleInput`, `find_adhoc_by_pack()`, `restore_rule()`, `relink_action_by_ref()`, `relink_trigger_by_ref()` |
| `crates/api/src/routes/packs.rs` | Save ad-hoc rules before pack deletion; restore them and re-link orphaned cross-pack rules after component loading |
## Testing
- Zero compiler warnings across the workspace
- All unit tests pass
- Integration test failures are pre-existing (no `attune_test` database configured)

96
work-summary/2026-02-22-stackstorm-param-schema.md Normal file
View File

@@ -0,0 +1,96 @@
# StackStorm-Style Parameter Schema Migration
**Date**: 2026-02-22
## Summary
Migrated `param_schema` format from standard JSON Schema to StackStorm-style flat parameter maps with `required` and `secret` inlined per-parameter. This makes parameter definitions more readable and eliminates the clunky top-level `required` array pattern from JSON Schema.
## Format Change
### Before (JSON Schema)
```yaml
parameters:
type: object
properties:
url:
type: string
description: "Target URL"
token:
type: string
secret: true
required:
- url
```
### After (StackStorm-style)
```yaml
parameters:
url:
type: string
description: "Target URL"
required: true
token:
type: string
secret: true
```
The `type: object` / `properties:` wrapper is removed. `required` moves from a top-level array to an inline boolean per-parameter. `secret` was already inline and remains unchanged.
## Scope
- **`param_schema`** (action, trigger, sensor, workflow parameters): Converted to StackStorm-style
- **`out_schema`** (output schemas): Left as standard JSON Schema — `required`/`secret` are not meaningful for outputs
- **Database**: No migration needed — columns are JSONB, the JSON shape just changes
- **Backward compatibility**: Web UI `extractProperties()` handles both formats during transition
## Files Modified
### Pack YAML Files (13 files)
- `packs/core/actions/echo.yaml`
- `packs/core/actions/sleep.yaml`
- `packs/core/actions/noop.yaml`
- `packs/core/actions/http_request.yaml`
- `packs/core/actions/download_packs.yaml`
- `packs/core/actions/register_packs.yaml`
- `packs/core/actions/build_pack_envs.yaml`
- `packs/core/actions/get_pack_dependencies.yaml`
- `packs/core/triggers/intervaltimer.yaml`
- `packs/core/triggers/crontimer.yaml`
- `packs/core/triggers/datetimetimer.yaml`
- `packs/core/workflows/install_packs.yaml`
- `packs/examples/actions/list_example.yaml`
### Web UI (7 files)
- `web/src/components/common/ParamSchemaForm.tsx` — New `ParamSchemaProperty` interface with inline `required`/`secret`/`position`, new exported `extractProperties()` utility, updated `validateParamSchema()` logic
- `web/src/components/common/ParamSchemaDisplay.tsx` — Imports shared `extractProperties`, removed duplicate type definitions
- `web/src/components/common/ExecuteActionModal.tsx` — Uses shared `extractProperties` for parameter initialization
- `web/src/components/common/SchemaBuilder.tsx` — Produces StackStorm-style flat format, added Secret checkbox, handles both formats on input
- `web/src/components/forms/TriggerForm.tsx` — Updated empty-schema check for flat format
- `web/src/pages/actions/ActionsPage.tsx` — Uses `extractProperties`, added Secret badges
- `web/src/pages/triggers/TriggersPage.tsx` — Uses `extractProperties`, added Secret badges
### API DTOs (3 files)
- `crates/api/src/dto/action.rs` — Updated OpenAPI examples and doc comments
- `crates/api/src/dto/trigger.rs` — Updated OpenAPI examples and doc comments
- `crates/api/src/dto/workflow.rs` — Updated OpenAPI examples and doc comments
### Documentation
- `AGENTS.md` — Added Parameter Schema Format documentation in Pack File Loading section
## Key Design Decisions
1. **Shared `extractProperties()` utility**: Single exported function in `ParamSchemaForm.tsx` handles both StackStorm-style and legacy JSON Schema formats. All consumers import from one place instead of duplicating logic.
2. **Backward compatibility in Web UI**: The `extractProperties()` function detects the old format (presence of `type: "object"` + `properties` wrapper) and normalizes it to the flat format, merging the top-level `required` array into per-parameter `required: true` flags. This means existing database records in the old format will still render correctly.
3. **No Rust model changes needed**: `param_schema` is stored as `Option<JsonValue>` (aliased as `JsonSchema`). The Rust code doesn't deeply inspect the schema structure — it passes it through as opaque JSONB. The format change is transparent to the backend.
4. **Pack loaders unchanged**: Both `loader.rs` and `load_core_pack.py` read `data.get("parameters")` and serialize it to JSONB as-is. Since we changed the YAML format, the stored format automatically changes to match.
## Verification
- Rust: `cargo check --all-targets --workspace` — zero warnings
- Rust: `cargo test --workspace --lib` — 82 tests passed
- TypeScript: `npx tsc --noEmit` — clean
- Vite: `npx vite build` — successful production build

93
work-summary/2026-02-23-workflow-builder-ui.md Normal file
View File

@@ -0,0 +1,93 @@
# Workflow Builder UI Implementation
**Date:** 2026-02-23
## Summary
Implemented a visual workflow builder interface for creating and editing workflow actions. The builder is accessible from the Actions page and provides a node-based canvas for constructing workflows using installed actions as task building blocks.
## Changes
### Frontend (Web UI)
#### New Pages
- **`web/src/pages/actions/WorkflowBuilderPage.tsx`** — Main workflow builder page with:
- Top toolbar with pack selector, workflow name/label/version inputs, save button
- Description, tags, and enabled toggle in a secondary row
- Three-panel layout: action palette (left), canvas (center), task inspector (right)
- Definition JSON preview panel (toggleable)
- Validation error display
- Support for both create (`/actions/workflows/new`) and edit (`/actions/workflows/:ref/edit`) modes
#### New Components (`web/src/components/workflows/`)
- **`ActionPalette.tsx`** — Searchable sidebar listing all available actions grouped by pack. Clicking an action adds it as a task to the canvas with auto-populated input parameters from the action's schema.
- **`WorkflowCanvas.tsx`** — Visual canvas with:
- Draggable task nodes with absolute positioning
- SVG edge rendering for task transitions
- Interactive connection mode: click a port on one node, then click another node to create success/failure transitions
- Grid background, empty state with guidance
- Floating "add task" button
- **`TaskNode.tsx`** — Individual task node component showing task name, action reference, input count, badges for conditions/retry/iteration, and connection/configure/delete action buttons
- **`WorkflowEdges.tsx`** — SVG overlay rendering curved bezier edges between connected nodes with color-coded and dash-styled lines per transition type (success=green, failure=red dashed, complete=indigo, timeout=amber, decision=violet). Includes arrow markers and edge labels.
- **`TaskInspector.tsx`** — Right-side property panel with collapsible sections for:
- Basic settings (name, type, condition)
- Action selection (dropdown of all actions) with auto-populate from schema
- Transitions (on_success, on_failure, on_complete, on_timeout dropdowns)
- Iteration (with_items, batch_size, concurrency)
- Retry & timeout configuration
- Publish variables (key=value pairs for workflow variable publishing)
#### New Types & Utilities (`web/src/types/workflow.ts`)
- TypeScript types for workflow builder state, tasks, edges, parameters, YAML definition format
- `builderStateToDefinition()` — Converts builder state to the YAML-compatible definition format
- `definitionToBuilderState()` — Converts existing workflow definitions back to builder state (for edit mode)
- `deriveEdges()` — Extracts visual edges from task transition properties
- `validateWorkflow()` — Client-side validation (name, label, version, pack, task names, action assignments, transition references)
- Utility functions: `generateTaskId()`, `createEmptyTask()`, `generateUniqueTaskName()`
#### New Hooks (`web/src/hooks/useWorkflows.ts`)
- `useWorkflows()` — List workflows with filtering
- `useWorkflow()` — Get single workflow by ref
- `useCreateWorkflow()` / `useUpdateWorkflow()` / `useDeleteWorkflow()` — Standard CRUD mutations
- `useSaveWorkflowFile()` — Calls `POST /api/v1/packs/{pack_ref}/workflow-files` to save workflow file to disk
- `useUpdateWorkflowFile()` — Calls `PUT /api/v1/workflows/{ref}/file` to update workflow file on disk
#### Modified Files
- **`web/src/pages/actions/ActionsPage.tsx`** — Added "Workflow" button in the header that navigates to `/actions/workflows/new`
- **`web/src/App.tsx`** — Added lazy-loaded routes for `WorkflowBuilderPage` at `/actions/workflows/new` and `/actions/workflows/:ref/edit`
### Backend (API)
#### New Endpoints
- **`POST /api/v1/packs/{pack_ref}/workflow-files`** — Saves a new workflow:
1. Validates the request and checks the pack exists
2. Checks for duplicate workflow ref
3. Writes `{name}.workflow.yaml` to `{packs_base_dir}/{pack_ref}/actions/workflows/`
4. Creates the `workflow_definition` record in the database
5. Returns the workflow response
- **`PUT /api/v1/workflows/{ref}/file`** — Updates an existing workflow:
1. Validates the request and finds the existing workflow
2. Overwrites the YAML file on disk
3. Updates the database record
4. Returns the updated workflow response
#### New DTO
- **`SaveWorkflowFileRequest`** in `crates/api/src/dto/workflow.rs` — Request body with name, label, description, version, pack_ref, definition (JSON), param_schema, out_schema, tags, enabled
#### Modified Files
- **`crates/api/src/routes/workflows.rs`** — Added `save_workflow_file`, `update_workflow_file` handlers and helper function `write_workflow_yaml`. Updated routes to include new endpoints. Added unit tests.
- **`crates/api/src/dto/workflow.rs`** — Added `SaveWorkflowFileRequest` DTO
## Workflow File Storage
Workflow files are saved to: `{packs_base_dir}/{pack_ref}/actions/workflows/{name}.workflow.yaml`
This is a new path (`actions/workflows/`) distinct from the existing `workflows/` directory used by the pack sync mechanism. The definition is serialized as YAML and simultaneously persisted to both disk and database.
## Testing
- All 89 existing unit tests pass
- 2 new unit tests added for `SaveWorkflowFileRequest` validation
- TypeScript compilation passes with zero errors from new code
- Rust workspace compilation passes with zero warnings

88
work-summary/2026-02-unified-schema-format.md Normal file
View File

@@ -0,0 +1,88 @@
# Unified Schema Format: Flat Format for All Schema Types
**Date**: 2026-02-05
## Summary
Unified all schema types (`param_schema`, `out_schema`, `conf_schema`) to use the same flat StackStorm-style format with inline `required` and `secret` per parameter. Previously, `param_schema` used flat format while `out_schema` and `conf_schema` used standard JSON Schema (`{ type: "object", properties: { ... }, required: [...] }`). This inconsistency prevented features like `secret` badges from working on output and configuration schemas.
## Motivation
- No reason for `conf_schema` and `out_schema` to use a different format than `param_schema`
- Users should be able to mark `secret` and `required` inline on any schema type
- Eliminates dual-format shim logic in the web UI (`extractProperties` backward compatibility branch)
- Project is pre-production — no data migration needed, just adjust configurations
## Changes
### Pack YAML Files (12 files)
Converted all top-level `type: object` + `properties` wrappers to flat format, moving `required` array entries inline:
- `packs/core/pack.yaml` — `conf_schema`
- `packs/examples/pack.yaml` — `conf_schema`
- `packs/core/sensors/interval_timer_sensor.yaml` — `parameters`
- `packs/core/triggers/intervaltimer.yaml` — `output`
- `packs/core/triggers/crontimer.yaml` — `output`
- `packs/core/triggers/datetimetimer.yaml` — `output`
- `packs/core/actions/http_request.yaml` — `output_schema`
- `packs/core/actions/build_pack_envs.yaml` — `output_schema`
- `packs/core/actions/download_packs.yaml` — `output_schema`
- `packs/core/actions/get_pack_dependencies.yaml` — `output_schema`
- `packs/core/actions/register_packs.yaml` — `output_schema`
- `packs/core/workflows/install_packs.yaml` — `output_schema`
- `packs/examples/actions/list_example.yaml` — `output_schema`
Nested structures (e.g., `items: { type: object, properties: { ... } }` within array parameters) remain unchanged — only the top-level wrapper was converted.
### Web UI (6 files)
- **`ParamSchemaForm.tsx`** — Removed legacy JSON Schema branch from `extractProperties()`. Removed `extractJsonSchemaProperties()` (no longer needed). Single `extractProperties()` handles all schema types.
- **`ParamSchemaDisplay.tsx`** — Updated doc comment, tightened `schema` prop type from `ParamSchema | any` to `ParamSchema`.
- **`SchemaBuilder.tsx`** — Removed legacy JSON Schema reading from both `useEffect` initializer and `handleRawJsonChange`. Only reads/writes flat format.
- **`PackForm.tsx`** — Updated `confSchema` initial state from JSON Schema to `{}`. Updated `hasSchemaProperties` check (no longer looks for `.properties` sub-key). Updated config sync logic, validation, schema examples (API/Database/Webhook examples now use flat format with `secret` and `required` inline), and `ParamSchemaForm` pass-through (passes `confSchema` directly instead of `confSchema.properties`).
- **`TriggerForm.tsx`** — Updated `paramSchema` and `outSchema` initial states from JSON Schema to `{}`.
- **`TriggersPage.tsx`** — Uses `extractProperties()` for both `param_schema` and `out_schema`.
### Backend Rust (5 files)
- **`crates/api/src/validation/params.rs`** — Added `flat_to_json_schema()` function that converts flat format to JSON Schema internally before passing to `jsonschema::Validator`. Updated `validate_trigger_params()` and `validate_action_params()` to call the converter. Converted all 29 test schemas from JSON Schema to flat format. Added 4 unit tests for `flat_to_json_schema()` and 1 test for secret field validation.
- **`crates/api/src/dto/action.rs`** — Updated `out_schema` doc comment and utoipa example.
- **`crates/api/src/dto/trigger.rs`** — Updated `out_schema` and sensor `param_schema` doc comments and utoipa examples.
- **`crates/api/src/dto/workflow.rs`** — Updated `out_schema` doc comment and utoipa example.
- **`crates/api/src/dto/pack.rs`** — Updated `conf_schema` doc comment and utoipa example.
- **`crates/api/src/dto/inquiry.rs`** — Updated `response_schema` doc comment and utoipa example.
### Documentation
- **`AGENTS.md`** — Updated "Parameter Schema Format" section to "Schema Format (Unified)", reflecting that all schema types now use the same flat format.
## Test Results
- All 29 backend validation tests pass (converted to flat format schemas)
- TypeScript compilation clean (zero errors)
- Rust workspace compilation clean (zero warnings)
## Format Reference
**Before** (JSON Schema for out_schema/conf_schema):
```yaml
output:
type: object
properties:
fired_at:
type: string
format: date-time
required:
- fired_at
```
**After** (unified flat format):
```yaml
output:
fired_at:
type: string
format: date-time
required: true
secret: false # optional, can mark outputs as secret too
```