[wip] cli capability parity

AGENTS.md

@@ -160,7 +160,7 @@ Completion listener advances workflow → Schedules successor tasks → Complete
   - **Workflow Tasks**: Workflow-specific metadata stored in `execution.workflow_task` JSONB field
 - **Inquiry**: Human-in-the-loop async interaction (approvals, inputs)
 - **Identity**: User/service account with RBAC permissions
-- **Key**: Encrypted secrets storage
+- **Key**: Secrets/config storage. The `value` column is JSONB — keys can store strings, objects, arrays, numbers, or booleans. Keys are **unencrypted by default**; use `--encrypt`/`-e` (CLI) or `"encrypted": true` (API) to encrypt. When encrypted, the JSON value is serialised to a compact string, encrypted with AES-256-GCM, and stored as a JSON string; decryption reverses this. The `encrypt_json`/`decrypt_json` helpers in `attune_common::crypto` handle this — **all services use this single shared implementation** (the worker's `SecretManager` delegates directly to `attune_common::crypto::decrypt_json`; it no longer has its own bespoke encryption code). The ciphertext format is `BASE64(nonce_bytes ++ ciphertext_bytes)` everywhere. The worker's `SecretManager` returns `HashMap<String, JsonValue>` and secrets are merged directly into action parameters (no `Value::String` wrapping). The workflow `keystore` namespace already uses `JsonValue`, so structured secrets are natively accessible (e.g., `{{ keystore.db_credentials.password }}`). The CLI `key show` command displays a SHA-256 hash of the value by default; pass `--decrypt`/`-d` to reveal the actual value.
 - **Artifact**: Tracked output from executions (files, logs, progress indicators). Metadata + optional structured `data` (JSONB). Linked to execution via plain BIGINT (no FK). Supports retention policies (version-count or time-based). File-type artifacts (FileBinary, FileDataTable, FileImage, FileText) use disk-based storage on a shared volume; Progress and Url artifacts use DB storage. Each artifact has a `visibility` field (`ArtifactVisibility` enum: `public` or `private`, DB default `private`). Public artifacts are viewable by all authenticated users; private artifacts are restricted based on the artifact's `scope` (Identity, Pack, Action, Sensor) and `owner` fields. **Type-aware API default**: when `visibility` is omitted from `POST /api/v1/artifacts`, the API defaults to `public` for Progress artifacts (informational status indicators anyone watching an execution should see) and `private` for all other types. Callers can always override by explicitly setting `visibility`. Full RBAC enforcement is deferred — the column and basic filtering are in place for future permission checks.
 - **ArtifactVersion**: Immutable content snapshot for an artifact. File-type versions store a `file_path` (relative path on shared volume) with `content` BYTEA left NULL. DB-stored versions use `content` BYTEA and/or `content_json` JSONB. Version number auto-assigned via `next_artifact_version()`. Retention trigger auto-deletes oldest versions beyond limit. Invariant: exactly one of `content`, `content_json`, or `file_path` should be non-NULL per row.
 
@@ -208,7 +208,7 @@ Completion listener advances workflow → Schedules successor tasks → Complete
 - **Auth Type**: JWT (access tokens: 1h, refresh tokens: 7d)
 - **Password Hashing**: Argon2id
 - **Protected Routes**: Use `RequireAuth(user)` extractor in Axum
-- **Secrets Storage**: AES-GCM encrypted in `key` table with scoped ownership
+- **Secrets Storage**: AES-GCM encrypted in `key` table (JSONB `value` column) with scoped ownership. Supports structured values (objects, arrays) in addition to plain strings. All encryption/decryption goes through `attune_common::crypto` (`encrypt_json`/`decrypt_json`) — the worker's `SecretManager` no longer has its own crypto implementation, eliminating a prior ciphertext format incompatibility between the API (`BASE64(nonce++ciphertext)`) and the old worker code (`BASE64(nonce):BASE64(ciphertext)`). The worker stores the raw encryption key string and passes it to the shared crypto module, which derives the AES-256 key internally via SHA-256.
 - **User Info**: Stored in `identity` table
 
 ## Code Conventions & Patterns
@@ -240,7 +240,7 @@ Completion listener advances workflow → Schedules successor tasks → Complete
 - **History Large-Field Guardrails**: The `execution` history trigger stores a compact **digest summary** instead of the full value for the `result` column (which can be arbitrarily large). The digest is produced by the `_jsonb_digest_summary(JSONB)` helper function and has the shape `{"digest": "md5:<hex>", "size": <bytes>, "type": "<jsonb_typeof>"}`. This preserves change-detection semantics while avoiding history table bloat. The full result is always available on the live `execution` row. When adding new large JSONB columns to history triggers, use `_jsonb_digest_summary()` instead of storing the raw value.
 - **Nullable FK Fields**: `rule.action` and `rule.trigger` are nullable (`Option<Id>` in Rust) — a rule with NULL action/trigger is non-functional but preserved for traceability. `execution.action`, `execution.parent`, `execution.enforcement`, `execution.started_at`, and `event.source` are also nullable. `enforcement.event` is nullable but has no FK constraint (event is a hypertable). `execution.enforcement` is nullable but has no FK constraint (enforcement is a hypertable). All FK columns on the execution table (`action`, `parent`, `original_execution`, `enforcement`, `executor`, `workflow_def`) have no FK constraints (execution is a hypertable). `inquiry.execution` and `workflow_execution.execution` also have no FK constraints. `enforcement.resolved_at` is nullable — `None` while status is `created`, set when resolved. `execution.started_at` is nullable — `None` until the worker sets status to `running`.
 **Table Count**: 21 tables total in the schema (including `runtime_version`, `artifact_version`, 2 `*_history` hypertables, and the `event`, `enforcement`, + `execution` hypertables)
-**Migration Count**: 10 migrations (`000001` through `000010`) — see `migrations/` directory
+**Migration Count**: 11 migrations (`000001` through `000011`) — see `migrations/` directory
 - **Artifact System**: The `artifact` table stores metadata + structured data (progress entries via JSONB `data` column). The `artifact_version` table stores immutable content snapshots — either on disk (via `file_path` column) or in DB (via `content` BYTEA / `content_json` JSONB). Version numbering is auto-assigned via `next_artifact_version()` SQL function. A DB trigger (`enforce_artifact_retention`) auto-deletes oldest versions when count exceeds the artifact's `retention_limit`. `artifact.execution` is a plain BIGINT (no FK — execution is a hypertable). Progress-type artifacts use `artifact.data` (atomic JSON array append); file-type artifacts use `artifact_version` rows with `file_path` set. Binary content is excluded from default queries for performance (`SELECT_COLUMNS` vs `SELECT_COLUMNS_WITH_CONTENT`). **Visibility**: Each artifact has a `visibility` column (`artifact_visibility_enum`: `public` or `private`, DB default `private`). The `CreateArtifactRequest` DTO accepts `visibility` as `Option<ArtifactVisibility>` — when omitted the API route handler applies a **type-aware default**: `public` for Progress artifacts (informational status indicators), `private` for all other types. Callers can always override explicitly. Public artifacts are viewable by all authenticated users; private artifacts are restricted based on the artifact's `scope` (Identity, Pack, Action, Sensor) and `owner` fields. The visibility field is filterable via the search/list API (`?visibility=public`). Full RBAC enforcement is deferred — the column and basic query filtering are in place for future permission checks. **Notifications**: `artifact_created` and `artifact_updated` DB triggers (in migration `000008`) fire PostgreSQL NOTIFY with entity_type `artifact` and include `visibility` in the payload. The `artifact_updated` trigger extracts a progress summary (`progress_percent`, `progress_message`, `progress_entries`) from the last entry of the `data` JSONB array for progress-type artifacts. The Web UI `ExecutionProgressBar` component (`web/src/components/executions/ExecutionProgressBar.tsx`) renders an inline progress bar in the Execution Details card using the `useArtifactStream` hook (`web/src/hooks/useArtifactStream.ts`) for real-time WebSocket updates, with polling fallback via `useExecutionArtifacts`.
 - **File-Based Artifact Storage**: File-type artifacts (FileBinary, FileDataTable, FileImage, FileText) use a shared filesystem volume instead of PostgreSQL BYTEA. The `artifact_version.file_path` column stores the relative path from the `artifacts_dir` root (e.g., `mypack/build_log/v1.txt`). Pattern: `{ref_with_dots_as_dirs}/v{version}.{ext}`. The artifact ref (globally unique) is used as the directory key — no execution ID in the path, so artifacts can outlive executions and be shared across them. **Endpoint**: `POST /api/v1/artifacts/{id}/versions/file` allocates a version number and file path without any file content; the execution process writes the file to `$ATTUNE_ARTIFACTS_DIR/{file_path}`. **Download**: `GET /api/v1/artifacts/{id}/download` and version-specific downloads check `file_path` first (read from disk), fall back to DB BYTEA/JSON. **Finalization**: After execution exits, the worker stats all file-backed versions for that execution and updates `size_bytes` on both `artifact_version` and parent `artifact` rows via direct DB access. **Cleanup**: Delete endpoints remove disk files before deleting DB rows; empty parent directories are cleaned up. **Backward compatible**: Existing DB-stored artifacts (`file_path = NULL`) continue to work unchanged.
 - **Pack Component Loading Order**: Runtimes → Triggers → Actions (+ workflow definitions) → Sensors (dependency order). Both `PackComponentLoader` (Rust) and `load_core_pack.py` (Python) follow this order. When an action YAML contains a `workflow_file` field, the loader creates/updates the referenced `workflow_definition` record and links it to the action during the Actions phase.
@@ -377,7 +377,7 @@ Workflow templates (`{{ expr }}`) support a full expression language for evaluat
 | `workflow` | `{{ workflow.counter }}` | Mutable workflow-scoped variables (set via `publish`) |
 | `task` | `{{ task.fetch.result.data }}` | Completed task results keyed by task name |
 | `config` | `{{ config.api_token }}` | Pack configuration values (read-only) |
-| `keystore` | `{{ keystore.secret_key }}` | Encrypted secrets from the key store (read-only) |
+| `keystore` | `{{ keystore.secret_key }}` | Encrypted secrets from the key store (read-only). Values are `JsonValue` — strings, objects, arrays, etc. Access nested fields with dot notation: `{{ keystore.db_credentials.password }}` |
 | `item` | `{{ item }}` / `{{ item.name }}` | Current element in a `with_items` loop |
 | `index` | `{{ index }}` | Zero-based iteration index in a `with_items` loop |
 | `system` | `{{ system.workflow_start }}` | System-provided variables |
@@ -387,7 +387,7 @@ Backward-compatible aliases (kept for existing workflow definitions):
 - `tasks` → same as `task`
 - Bare variable names (e.g. `{{ my_var }}`) resolve against the `workflow` variable store as a last-resort fallback.
 
-**IMPORTANT**: New workflow definitions should always use the canonical namespace names. The `config` and `keystore` namespaces are populated by the scheduler from the pack's `config` JSONB column and decrypted `key` table entries respectively. If not populated, they resolve to `null`.
+**IMPORTANT**: New workflow definitions should always use the canonical namespace names. The `config` and `keystore` namespaces are populated by the scheduler from the pack's `config` JSONB column and decrypted `key` table entries (JSONB values) respectively. If not populated, they resolve to `null`. Keystore values preserve their JSON type — a key storing `{"host":"db.example.com","port":5432}` is accessible as `{{ keystore.db_config.host }}` and `{{ keystore.db_config.port }}` (the latter resolves to integer `5432`, not string `"5432"`).
 
 **Operators** (lowest to highest precedence):
 1. `or` — logical OR (short-circuit)
@@ -535,12 +535,45 @@ attune pack upload ./path/to/pack  # Upload local pack to API (works with Docker
 attune pack register /opt/attune/packs/mypak  # Register from API-visible path
 attune action execute <ref> --param key=value
 attune execution list            # Monitor executions
+attune key list                  # List all keys (values redacted)
+attune key list --owner-type pack  # Filter keys by owner type
+attune key show my_token         # Show key details (value shown as SHA-256 hash)
+attune key show my_token -d      # Show key details with decrypted/actual value
+attune key create --ref my_token --name "My Token" --value "secret123"  # Create unencrypted string key (default)
+attune key create --ref my_token --name "My Token" --value '{"user":"admin","pass":"s3cret"}' # Create unencrypted structured key
+attune key create --ref my_token --name "My Token" --value "secret123" -e  # Create encrypted string key
+attune key create --ref my_token --name "My Token" --value "secret123" --encrypt --owner-type pack --owner-pack-ref core  # Create encrypted pack-scoped key
+attune key update my_token --value "new_secret"  # Update key value (string)
+attune key update my_token --value '{"host":"db.example.com","port":5432}'  # Update key value (structured)
+attune key update my_token --name "Renamed Token"  # Update key name
+attune key delete my_token       # Delete a key (with confirmation)
+attune key delete my_token --yes # Delete without confirmation
 attune workflow upload actions/deploy.yaml  # Upload workflow action to existing pack
 attune workflow upload actions/deploy.yaml --force  # Update existing workflow
 attune workflow list             # List all workflows
 attune workflow list --pack core # List workflows in a pack
 attune workflow show core.install_packs  # Show workflow details + task summary
 attune workflow delete core.my_workflow --yes  # Delete a workflow
+attune artifact list                 # List all artifacts
+attune artifact list --type file_text --visibility public  # Filter artifacts
+attune artifact list --execution 42  # List artifacts for an execution
+attune artifact show 1               # Show artifact by ID
+attune artifact show mypack.build_log  # Show artifact by ref
+attune artifact create --ref mypack.build_log --scope action --owner mypack.deploy --type file_text --name "Build Log"
+attune artifact upload 1 ./output.log  # Upload file as new version
+attune artifact upload 1 ./data.json --content-type application/json --created-by "cli"
+attune artifact download 1           # Download latest version to auto-named file
+attune artifact download 1 -V 3     # Download specific version
+attune artifact download 1 -o ./local.txt  # Download to specific path
+attune artifact download 1 -o -     # Download to stdout
+attune artifact delete 1             # Delete artifact (with confirmation)
+attune artifact delete 1 --yes       # Delete without confirmation
+attune artifact version list 1       # List all versions of artifact 1
+attune artifact version show 1 3     # Show details of version 3
+attune artifact version upload 1 ./new-file.txt  # Upload file as new version
+attune artifact version create-json 1 '{"key":"value"}'  # Create JSON version
+attune artifact version download 1 2 -o ./v2.txt  # Download version 2
+attune artifact version delete 1 2 --yes  # Delete version 2
 ```
 
 **Pack Upload vs Register**:
@@ -668,7 +701,7 @@ When reporting, ask: "Should I fix this first or continue with [original task]?"
 - **Web UI**: Static files served separately or via API service
 
 ## Current Development Status
-- ✅ **Complete**: Database migrations (21 tables, 10 migration files), API service (most endpoints), common library, message queue infrastructure, repository layer, JWT auth, CLI tool, Web UI (basic + workflow builder + workflow timeline DAG), Executor service (core functionality + workflow orchestration), Worker service (shell/Python execution), Runtime version data model, constraint matching, worker version selection pipeline, version verification at startup, per-version environment isolation, TimescaleDB entity history tracking (execution, worker), Event, enforcement, and execution tables as TimescaleDB hypertables (time-series with retention/compression), History API endpoints (generic + entity-specific with pagination & filtering), History UI panels on entity detail pages (execution), TimescaleDB continuous aggregates (6 hourly rollup views with auto-refresh policies), Analytics API endpoints (7 endpoints under `/api/v1/analytics/` — dashboard, execution status/throughput/failure-rate, event volume, worker status, enforcement volume), Analytics dashboard widgets (bar charts, stacked status charts, failure rate ring gauge, time range selector), Workflow execution orchestration (scheduler detects workflow actions, creates child task executions, completion listener advances workflow via transitions), Workflow template resolution (type-preserving `{{ }}` rendering in task inputs), Workflow `with_items` expansion (parallel child executions per item), Workflow `with_items` concurrency limiting (sliding-window dispatch with pending items stored in workflow variables), Workflow `publish` directive processing (variable propagation between tasks), Workflow function expressions (`result()`, `succeeded()`, `failed()`, `timed_out()`), Workflow expression engine (full arithmetic/comparison/boolean/membership operators, 30+ built-in functions, recursive-descent parser), Canonical workflow namespaces (`parameters`, `workflow`, `task`, `config`, `keystore`, `item`, `index`, `system`), Artifact content system (versioned file/JSON storage, progress-append semantics, binary upload/download, retention enforcement, execution-linked artifacts, 18 API endpoints under `/api/v1/artifacts/`, file-backed disk storage via shared volume for file-type artifacts), CLI `--wait` flag (WebSocket-first with polling fallback — connects to notifier on port 8081, subscribes to execution, returns immediately on terminal status; falls back to exponential-backoff REST polling if WS unavailable; polling always gets at least 10s budget regardless of how long WS path ran), Workflow Timeline DAG visualization (Prefect-style time-aligned Gantt+DAG on execution detail page, pure SVG, transition-aware edge coloring from workflow definition metadata, hover tooltips, click-to-highlight path, zoom/pan)
+- ✅ **Complete**: Database migrations (21 tables, 10 migration files), API service (most endpoints), common library, message queue infrastructure, repository layer, JWT auth, CLI tool, Web UI (basic + workflow builder + workflow timeline DAG), Executor service (core functionality + workflow orchestration), Worker service (shell/Python execution), Runtime version data model, constraint matching, worker version selection pipeline, version verification at startup, per-version environment isolation, TimescaleDB entity history tracking (execution, worker), Event, enforcement, and execution tables as TimescaleDB hypertables (time-series with retention/compression), History API endpoints (generic + entity-specific with pagination & filtering), History UI panels on entity detail pages (execution), TimescaleDB continuous aggregates (6 hourly rollup views with auto-refresh policies), Analytics API endpoints (7 endpoints under `/api/v1/analytics/` — dashboard, execution status/throughput/failure-rate, event volume, worker status, enforcement volume), Analytics dashboard widgets (bar charts, stacked status charts, failure rate ring gauge, time range selector), Workflow execution orchestration (scheduler detects workflow actions, creates child task executions, completion listener advances workflow via transitions), Workflow template resolution (type-preserving `{{ }}` rendering in task inputs), Workflow `with_items` expansion (parallel child executions per item), Workflow `with_items` concurrency limiting (sliding-window dispatch with pending items stored in workflow variables), Workflow `publish` directive processing (variable propagation between tasks), Workflow function expressions (`result()`, `succeeded()`, `failed()`, `timed_out()`), Workflow expression engine (full arithmetic/comparison/boolean/membership operators, 30+ built-in functions, recursive-descent parser), Canonical workflow namespaces (`parameters`, `workflow`, `task`, `config`, `keystore`, `item`, `index`, `system`), Artifact content system (versioned file/JSON storage, progress-append semantics, binary upload/download, retention enforcement, execution-linked artifacts, 18 API endpoints under `/api/v1/artifacts/`, file-backed disk storage via shared volume for file-type artifacts), CLI artifact management (`attune artifact list/show/create/upload/download/delete` + `attune artifact version list/show/upload/create-json/download/delete` — full CRUD for artifacts and their versions with multipart file upload, binary download, JSON version creation, auto-detected MIME types, human-readable size formatting, and pagination), CLI `--wait` flag (WebSocket-first with polling fallback — connects to notifier on port 8081, subscribes to execution, returns immediately on terminal status; falls back to exponential-backoff REST polling if WS unavailable; polling always gets at least 10s budget regardless of how long WS path ran), Workflow Timeline DAG visualization (Prefect-style time-aligned Gantt+DAG on execution detail page, pure SVG, transition-aware edge coloring from workflow definition metadata, hover tooltips, click-to-highlight path, zoom/pan)
 - 🔄 **In Progress**: Sensor service, advanced workflow features (nested workflow context propagation), Python runtime dependency management, API/UI endpoints for runtime version management, Artifact UI (web UI for browsing/downloading artifacts), Notifier service WebSocket (functional but lacks auth — the WS connection is unauthenticated; the subscribe filter controls visibility)
 - 📋 **Planned**: Execution policies, monitoring, pack registry system, configurable retention periods via admin settings, export/archival to external storage