attune-system/attune
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

more internal polish, resilient workers

Browse Source
This commit is contained in:
David Culbreth
2026-02-09 18:32:34 -06:00
parent 588b319fec
commit e31ecb781b
62 changed files with 9872 additions and 584 deletions
Download Patch File Download Diff File Expand all files Collapse all files

374
packs/core/actions/README.md
View File

@@ -2,19 +2,31 @@
## Overview
All actions in the core pack follow Attune's secure-by-design architecture:
- **Parameter delivery:** stdin (JSON format) - never environment variables
- **Output format:** Explicitly declared (text, json, or yaml)
- **Output schema:** Describes structured data shape (json/yaml only)
- **Execution metadata:** Automatically captured (stdout/stderr/exit_code)
All actions in the core pack are implemented as **pure POSIX shell scripts** with **zero external dependencies** (except `curl` for HTTP actions). This design ensures maximum portability and minimal runtime requirements.
**Key Principles:**
- **POSIX shell only** - No bash-specific features, works everywhere
- **DOTENV parameter format** - Simple key=value format, no JSON parsing needed
- **No jq/yq/Python/Node.js** - Core pack depends only on standard POSIX utilities
- **Stdin parameter delivery** - Secure, never exposed in process list
- **Explicit output formats** - text, json, or yaml
## Parameter Delivery Method
**All actions:**
- Read parameters from **stdin** as JSON
- Use `parameter_delivery: stdin` and `parameter_format: json` in their YAML definitions
**All actions use stdin with DOTENV format:**
- Parameters read from **stdin** in `key=value` format
- Use `parameter_delivery: stdin` and `parameter_format: dotenv` in YAML
- Terminated with `---ATTUNE_PARAMS_END---` delimiter
- **DO NOT** use environment variables for parameters
**Example DOTENV input:**
```
message="Hello World"
seconds=5
enabled=true
---ATTUNE_PARAMS_END---
```
## Output Format
**All actions must specify an `output_format`:**
@@ -48,170 +60,160 @@ The worker automatically provides these environment variables to all action exec
- Creating child executions
- Accessing secrets via API
**Example:**
```bash
#!/bin/bash
# Log with context
echo "[$ATTUNE_ACTION] [Exec: $ATTUNE_EXEC_ID] Processing..." >&2
# Call Attune API
curl -s -H "Authorization: Bearer $ATTUNE_API_TOKEN" \
"$ATTUNE_API_URL/api/v1/executions/$ATTUNE_EXEC_ID"
# Conditional behavior
if [ -n "$ATTUNE_RULE" ]; then
echo "Triggered by rule: $ATTUNE_RULE" >&2
fi
```
See [Execution Environment Variables](../../../docs/QUICKREF-execution-environment.md) for complete documentation.
### Custom Environment Variables (Optional)
Custom environment variables can be set via `execution.env_vars` field for:
- **Debug/logging controls** (e.g., `DEBUG=1`, `LOG_LEVEL=debug`)
- **Runtime configuration** (e.g., custom paths, feature flags)
- **Action-specific context** (non-sensitive execution context)
Environment variables should **NEVER** be used for:
- Action parameters (use stdin instead)
- Action parameters (use stdin DOTENV instead)
- Secrets or credentials (use `ATTUNE_API_TOKEN` to fetch from key vault)
- User-provided data (use stdin parameters)
## Implementation Patterns
## Implementation Pattern
### Bash/Shell Actions
### POSIX Shell Actions (Standard Pattern)
Shell actions read JSON from stdin using `jq`:
All core pack actions follow this pattern:
```sh
#!/bin/sh
# Action Name - Core Pack
# Brief description
#
# This script uses pure POSIX shell without external dependencies like jq.
# It reads parameters in DOTENV format from stdin until the delimiter.
```bash
#!/bin/bash
set -e
set -o pipefail
# Read JSON parameters from stdin
INPUT=$(cat)
# Initialize variables with defaults
param1=""
param2="default_value"
# Parse parameters using jq
PARAM1=$(echo "$INPUT" | jq -r '.param1 // "default_value"')
PARAM2=$(echo "$INPUT" | jq -r '.param2 // ""')
# Read DOTENV-formatted parameters from stdin
while IFS= read -r line; do
case "$line" in
*"---ATTUNE_PARAMS_END---"*) break ;;
esac
[ -z "$line" ] && continue
# Check for null values (optional parameters)
if [ -n "$PARAM2" ] && [ "$PARAM2" != "null" ]; then
echo "Param2 provided: $PARAM2"
fi
key="${line%%=*}"
value="${line#*=}"
# Use the parameters
echo "Param1: $PARAM1"
```
# Remove quotes if present
case "$value" in
\"*\") value="${value#\"}"; value="${value%\"}" ;;
\'*\') value="${value#\'}"; value="${value%\'}" ;;
esac
### Advanced Bash Actions
For more complex bash actions (like http_request.sh), use `curl` or other standard utilities:
```bash
#!/bin/bash
set -e
set -o pipefail
# Read JSON parameters from stdin
INPUT=$(cat)
# Parse parameters
URL=$(echo "$INPUT" | jq -r '.url // ""')
METHOD=$(echo "$INPUT" | jq -r '.method // "GET"')
# Process parameters
case "$key" in
param1) param1="$value" ;;
param2) param2="$value" ;;
esac
done
# Validate required parameters
if [ -z "$URL" ]; then
echo "ERROR: url parameter is required" >&2
if [ -z "$param1" ]; then
echo "ERROR: param1 is required" >&2
exit 1
fi
# Make HTTP request with curl
RESPONSE=$(curl -s -X "$METHOD" "$URL")
# Action logic
echo "Processing: $param1"
# Output result as JSON
jq -n \
--arg body "$RESPONSE" \
--argjson success true \
'{body: $body, success: $success}'
exit 0
```
### Boolean Normalization
```sh
case "$bool_param" in
true|True|TRUE|yes|Yes|YES|1) bool_param="true" ;;
*) bool_param="false" ;;
esac
```
### Numeric Validation
```sh
case "$number" in
''|*[!0-9]*)
echo "ERROR: must be a number" >&2
exit 1
;;
esac
```
## Core Pack Actions
### Simple Actions
1. **echo.sh** - Outputs a message
1. **echo.sh** - Outputs a message (reference implementation)
2. **sleep.sh** - Pauses execution for a specified duration
3. **noop.sh** - Does nothing (useful for testing)
3. **noop.sh** - Does nothing (useful for testing and placeholder workflows)
### HTTP Action
4. **http_request.sh** - Makes HTTP requests with authentication support (curl-based)
4. **http_request.sh** - Makes HTTP requests with full feature support:
- Multiple HTTP methods (GET, POST, PUT, PATCH, DELETE, etc.)
- Custom headers and query parameters
- Authentication (basic, bearer token)
- SSL verification control
- Redirect following
- JSON output with parsed response
### Pack Management Actions (API Wrappers)
These actions wrap API endpoints and pass parameters to the Attune API:
These actions wrap Attune API endpoints for pack management:
5. **download_packs.sh** - Downloads packs from git/HTTP/registry
6. **build_pack_envs.sh** - Builds runtime environments for packs
7. **register_packs.sh** - Registers packs in the database
8. **get_pack_dependencies.sh** - Analyzes pack dependencies
All API wrappers:
- Accept parameters via DOTENV format
- Build JSON request bodies manually (no jq)
- Make authenticated API calls with curl
- Extract response data using simple sed patterns
- Return structured JSON output
## Testing Actions Locally
You can test actions locally by piping JSON to stdin:
Test actions by echoing DOTENV format to stdin:
```bash
# Test echo action
echo '{"message": "Hello from stdin!"}' | ./echo.sh
printf 'message="Hello World"\n---ATTUNE_PARAMS_END---\n' | ./echo.sh
# Test echo with no message (outputs empty line)
echo '{}' | ./echo.sh
# Test with empty parameters
printf '---ATTUNE_PARAMS_END---\n' | ./echo.sh
# Test sleep action
echo '{"seconds": 2, "message": "Sleeping..."}' | ./sleep.sh
printf 'seconds=2\nmessage="Sleeping..."\n---ATTUNE_PARAMS_END---\n' | ./sleep.sh
# Test http_request action
echo '{"url": "https://api.github.com", "method": "GET"}' | ./http_request.sh
printf 'url="https://api.github.com"\nmethod="GET"\n---ATTUNE_PARAMS_END---\n' | ./http_request.sh
# Test with file input
cat params.json | ./echo.sh
cat params.dotenv | ./echo.sh
```
## Migration Summary
**Before (using environment variables):**
```bash
MESSAGE="${ATTUNE_ACTION_MESSAGE:-}"
```
**After (using stdin JSON):**
```bash
INPUT=$(cat)
MESSAGE=$(echo "$INPUT" | jq -r '.message // ""')
```
## Security Benefits
1. **No process exposure** - Parameters never appear in `ps`, `/proc/<pid>/environ`
2. **Secure by default** - All actions use stdin, no special configuration needed
3. **Clear separation** - Action parameters vs. environment configuration
4. **Audit friendly** - All sensitive data flows through stdin, not environment
## YAML Configuration
All action YAML files explicitly declare parameter delivery and output format:
## YAML Configuration Example
```yaml
name: example_action
ref: core.example_action
label: "Example Action"
description: "Example action demonstrating DOTENV format"
enabled: true
runner_type: shell
entry_point: example.sh
# Parameter delivery: stdin for secure parameter passing (no env vars)
# IMPORTANT: Use DOTENV format for POSIX shell compatibility
parameter_delivery: stdin
parameter_format: json
parameter_format: dotenv
# Output format: text, json, or yaml
output_format: text
@@ -221,51 +223,75 @@ parameters:
properties:
message:
type: string
description: "Message to output (empty string if not provided)"
required: []
# Output schema: not applicable for text output format
# For json/yaml formats, describe the structure of data your action outputs
# Do NOT include stdout/stderr/exit_code - those are captured automatically
# Do NOT include generic "status" or "result" wrappers - output your data directly
description: "Message to output"
default: ""
count:
type: integer
description: "Number of times to repeat"
default: 1
required:
- message
```
## Dependencies
**Core pack has ZERO runtime dependencies:**
✅ **Required (universally available):**
- POSIX-compliant shell (`/bin/sh`)
- `curl` (for HTTP actions only)
- Standard POSIX utilities: `sed`, `mktemp`, `cat`, `printf`, `sleep`
❌ **NOT Required:**
- `jq` - Eliminated (was used for JSON parsing)
- `yq` - Never used
- Python - Not used in core pack actions
- Node.js - Not used in core pack actions
- bash - Scripts are POSIX-compliant
- Any other external tools or libraries
This makes the core pack **maximally portable** and suitable for minimal containers (Alpine, distroless, etc.).
## Security Benefits
1. **No process exposure** - Parameters never appear in `ps`, `/proc/<pid>/environ`
2. **Secure by default** - All actions use stdin, no special configuration needed
3. **Clear separation** - Action parameters vs. environment configuration
4. **Audit friendly** - All sensitive data flows through stdin, not environment
5. **Minimal attack surface** - No external dependencies to exploit
## Best Practices
### Parameters
1. **Always use stdin** for action parameters
2. **Use jq for bash** scripts to parse JSON
3. **Handle null values** - Use jq's `// "default"` operator to provide defaults
4. **Provide sensible defaults** - Use empty string, 0, false, or empty array/object as appropriate
5. **Validate required params** - Exit with error if required parameters are missing (when truly required)
6. **Mark secrets** - Use `secret: true` in YAML for sensitive parameters
7. **Never use env vars for parameters** - Parameters come from stdin, not environment
1. **Always use stdin with DOTENV format** for action parameters
2. **Handle quoted values** - Remove both single and double quotes
3. **Provide sensible defaults** - Use empty string, 0, false as appropriate
4. **Validate required params** - Exit with error if truly required parameters missing
5. **Mark secrets** - Use `secret: true` in YAML for sensitive parameters
6. **Never use env vars for parameters** - Parameters come from stdin only
### Environment Variables
1. **Use standard ATTUNE_* variables** - Worker provides execution context
2. **Access API with ATTUNE_API_TOKEN** - Execution-scoped authentication
3. **Log with context** - Include `ATTUNE_ACTION` and `ATTUNE_EXEC_ID` in logs
4. **Custom env vars via execution.env_vars** - For debug flags and configuration only
5. **Never log ATTUNE_API_TOKEN** - Security sensitive
6. **Check ATTUNE_RULE/ATTUNE_TRIGGER** - Conditional behavior for automated vs manual
7. **Use env vars for runtime context** - Not for user data or parameters
4. **Never log ATTUNE_API_TOKEN** - Security sensitive
5. **Use env vars for runtime config only** - Not for user data or parameters
### Output Format
1. **Specify output_format** - Always set to "text", "json", or "yaml"
2. **Use text for simple output** - Messages, logs, unstructured data
3. **Use json for structured data** - API responses, complex results
4. **Use yaml for readable config** - Human-readable structured output
5. **Define schema for structured output** - Only for json/yaml formats
6. **Don't include execution metadata** - No stdout/stderr/exit_code in schema
7. **Use stderr for errors** - Diagnostic messages go to stderr, not stdout
8. **Return proper exit codes** - 0 for success, non-zero for failure
4. **Define schema for structured output** - Only for json/yaml formats
5. **Use stderr for diagnostics** - Error messages go to stderr, not stdout
6. **Return proper exit codes** - 0 for success, non-zero for failure
## Dependencies
All core pack actions have **zero runtime dependencies**:
- **Bash actions**: Require `jq` (for JSON parsing) and `curl` (for HTTP requests)
- Both `jq` and `curl` are standard utilities available in all Attune worker containers
- **No Python, Node.js, or other runtime dependencies required**
### Shell Script Best Practices
1. **Use `#!/bin/sh`** - POSIX shell, not bash
2. **Use `set -e`** - Exit on error
3. **Quote all variables** - `"$var"` not `$var`
4. **Use `case` not `if`** - More portable for pattern matching
5. **Clean up temp files** - Use trap handlers
6. **Avoid bash-isms** - No `[[`, `${var^^}`, `=~`, arrays, etc.
## Execution Metadata (Automatic)
@@ -278,44 +304,66 @@ The following are **automatically captured** by the worker and should **NOT** be
These are execution system concerns, not action output concerns.
## Example: Using Environment Variables and Parameters
## Example: Complete Action
```sh
#!/bin/sh
# Example Action - Core Pack
# Demonstrates DOTENV parameter parsing and environment variable usage
#
# This script uses pure POSIX shell without external dependencies like jq.
```bash
#!/bin/bash
set -e
set -o pipefail
# Standard environment variables (provided by worker)
echo "[$ATTUNE_ACTION] [Exec: $ATTUNE_EXEC_ID] Starting execution" >&2
# Log execution start
echo "[$ATTUNE_ACTION] [Exec: $ATTUNE_EXEC_ID] Starting" >&2
# Read action parameters from stdin
INPUT=$(cat)
URL=$(echo "$INPUT" | jq -r '.url // ""')
# Initialize variables
url=""
timeout="30"
if [ -z "$URL" ]; then
echo "ERROR: url parameter is required" >&2
# Read DOTENV parameters
while IFS= read -r line; do
case "$line" in
*"---ATTUNE_PARAMS_END---"*) break ;;
esac
[ -z "$line" ] && continue
key="${line%%=*}"
value="${line#*=}"
case "$value" in
\"*\") value="${value#\"}"; value="${value%\"}" ;;
esac
case "$key" in
url) url="$value" ;;
timeout) timeout="$value" ;;
esac
done
# Validate
if [ -z "$url" ]; then
echo "ERROR: url is required" >&2
exit 1
fi
# Log execution context
if [ -n "$ATTUNE_RULE" ]; then
echo "Triggered by rule: $ATTUNE_RULE" >&2
fi
# Execute
echo "Fetching: $url" >&2
result=$(curl -s --max-time "$timeout" "$url")
# Make request
RESPONSE=$(curl -s "$URL")
# Output
echo "$result"
# Output result
echo "$RESPONSE"
echo "[$ATTUNE_ACTION] [Exec: $ATTUNE_EXEC_ID] Completed successfully" >&2
echo "[$ATTUNE_ACTION] [Exec: $ATTUNE_EXEC_ID] Completed" >&2
exit 0
```
## Future Considerations
## Further Documentation
- Consider adding a bash library for common parameter parsing patterns
- Add parameter validation helpers
- Create templates for new actions in different languages
- Add output schema validation tooling
- Add helper functions for API interaction using ATTUNE_API_TOKEN
- **Pattern Reference:** `docs/QUICKREF-dotenv-shell-actions.md`
- **Pack Structure:** `docs/pack-structure.md`
- **Example Actions:**
- `echo.sh` - Simplest reference implementation
- `http_request.sh` - Complex action with full HTTP client
- `register_packs.sh` - API wrapper with JSON construction