attune/work-summary/changelogs/core-pack-setup-summary.md

Core Pack Setup Implementation Summary

Date: 2025-01-20
Task: Core Pack Setup - Loader Script & Documentation
Status: ✅ Complete

Overview

Created a comprehensive system for loading the Attune core pack from the filesystem into the database. This enables the built-in system pack to be properly initialized with all its actions, triggers, and sensors.

What Was Built

1. Python Loader Script (scripts/load_core_pack.py)

Purpose: Parse YAML pack definitions and load them into the database

Features:

• Reads pack.yaml for pack metadata
• Loads triggers from triggers/*.yaml
• Loads actions from actions/*.yaml
• Loads sensors from sensors/*.yaml
• Creates runtime entries for actions and sensors
• Idempotent operations (can be run multiple times safely)
• Transaction-based (all-or-nothing)
• Command-line arguments for flexibility
• Comprehensive error handling and reporting

Usage:

# Basic usage
python3 scripts/load_core_pack.py

# With custom database URL
python3 scripts/load_core_pack.py --database-url "postgresql://..."

# With custom pack directory
python3 scripts/load_core_pack.py --pack-dir ./packs

Key Functions:

• upsert_pack() - Create/update pack metadata
• upsert_triggers() - Load all trigger definitions
• upsert_actions() - Load all action definitions
• upsert_sensors() - Load all sensor definitions
• ensure_shell_runtime() - Create shell runtime for actions
• ensure_sensor_runtime() - Create built-in sensor runtime

Database Operations:

• Uses ON CONFLICT ... DO UPDATE for upserts
• Maintains referential integrity (pack → triggers/actions → sensors)
• Returns IDs for cross-referencing
• Proper transaction handling with rollback on error

2. Shell Wrapper Script (scripts/load-core-pack.sh)

Purpose: User-friendly wrapper with prerequisite checking

Features:

• Prerequisites validation (Python, packages, database)
• Interactive package installation
• Database connectivity testing
• Colored output for better UX
• Help documentation
• Environment variable support
• Verbose mode for debugging

Checks Performed:

1. ✓ Python 3 is installed
2. ✓ Required Python packages (psycopg2-binary, pyyaml)
3. ✓ Database is accessible
4. ✓ Core pack directory exists
5. ✓ pack.yaml file is present

Usage:

# Basic usage
./scripts/load-core-pack.sh

# With options
./scripts/load-core-pack.sh --database-url "postgresql://..." --verbose

# Dry run
./scripts/load-core-pack.sh --dry-run

3. Comprehensive Documentation

packs/core/SETUP.md (305 lines)

Complete setup guide including:

• Overview - What the core pack provides
• Prerequisites - Requirements before loading
• Loading Methods - 3 different approaches:
  1. Python loader script (recommended)
  2. SQL seed script
  3. CLI (future)
• Verification - How to confirm successful loading
• Testing - Testing the loaded pack
• Updating - Re-running after changes
• Troubleshooting - Common issues and solutions
• Development Workflow - Adding new components
• Environment Variables - Configuration options
• CI/CD Integration - Automated deployment examples

Updated README.md

Added new section "3. Load the Core Pack":

• Quick start instructions
• Verification commands
• Link to detailed setup guide
• Renumbered subsequent sections

4. Existing Assets

Already Present:

• ✅ packs/core/pack.yaml - Pack metadata
• ✅ packs/core/actions/*.yaml - Action definitions (echo, sleep, noop, http_request)
• ✅ packs/core/actions/*.sh - Shell action implementations
• ✅ packs/core/actions/*.py - Python action implementations
• ✅ packs/core/triggers/*.yaml - Trigger definitions (intervaltimer, crontimer, datetimetimer)
• ✅ packs/core/sensors/*.yaml - Sensor definitions
• ✅ packs/core/sensors/*.py - Sensor implementations
• ✅ packs/core/README.md - Component documentation
• ✅ packs/core/TESTING.md - Testing procedures
• ✅ packs/core/test_core_pack.sh - Test suite
• ✅ scripts/seed_core_pack.sql - SQL seed script (legacy)

Core Pack Contents

Pack Metadata

• Ref: core
• Version: 1.0.0
• Type: System pack (built-in)

Triggers (3)

1. core.intervaltimer - Regular interval-based timer
2. core.crontimer - Cron schedule-based timer
3. core.datetimetimer - One-shot datetime timer

Actions (4)

1. core.echo - Echo message to stdout
2. core.sleep - Sleep for N seconds
3. core.noop - No operation (testing)
4. core.http_request - Make HTTP requests

Sensors (1)

1. core.interval_timer_sensor - Built-in interval timer sensor

Technical Details

Database Schema Usage

Tables Populated:

• attune.pack - Core pack entry
• attune.runtime - Shell and sensor runtimes
• attune.trigger - Trigger type definitions
• attune.action - Action definitions
• attune.sensor - Sensor instances

Key Features:

• UPSERT operations for idempotency
• Proper foreign key relationships
• JSON schema storage for parameters/output
• Runtime type discrimination (action vs sensor)

Python Dependencies

Required packages:

• psycopg2-binary - PostgreSQL database adapter
• pyyaml - YAML parsing

Environment Variables

• DATABASE_URL - PostgreSQL connection string

  • Default: postgresql://postgres:postgres@localhost:5432/attune

• ATTUNE_PACKS_DIR - Base directory for packs

  • Default: ./packs

Usage Workflow

First-Time Setup

# 1. Ensure database is running and migrations applied
createdb attune
sqlx migrate run

# 2. Install Python dependencies
pip install psycopg2-binary pyyaml

# 3. Load core pack
./scripts/load-core-pack.sh

# 4. Verify
attune pack show core

Development Workflow

# 1. Edit pack files (actions, triggers, sensors)
vim packs/core/actions/new_action.yaml
vim packs/core/actions/new_action.sh

# 2. Make scripts executable
chmod +x packs/core/actions/new_action.sh

# 3. Test locally
export ATTUNE_ACTION_FOO="bar"
./packs/core/actions/new_action.sh

# 4. Reload into database
./scripts/load-core-pack.sh

# 5. Restart services
# API, executor, worker, sensor services

Updating Existing Components

# 1. Modify YAML definitions
vim packs/core/actions/echo.yaml

# 2. Re-run loader (upsert mode)
./scripts/load-core-pack.sh

# 3. Changes take effect immediately for new executions

Testing

Loader Script Testing

# Syntax check
python3 -m py_compile scripts/load_core_pack.py

# Dry run
python3 scripts/load_core_pack.py --dry-run

# Actual load
python3 scripts/load_core_pack.py

Verification Queries

-- Check pack exists
SELECT * FROM attune.pack WHERE ref = 'core';

-- Count components
SELECT 
    (SELECT COUNT(*) FROM attune.trigger WHERE pack_ref = 'core') as triggers,
    (SELECT COUNT(*) FROM attune.action WHERE pack_ref = 'core') as actions,
    (SELECT COUNT(*) FROM attune.sensor WHERE pack_ref = 'core') as sensors;

-- List all core components
SELECT ref, label FROM attune.trigger WHERE pack_ref = 'core';
SELECT ref, label FROM attune.action WHERE pack_ref = 'core';
SELECT ref, label FROM attune.sensor WHERE pack_ref = 'core';

End-to-End Test

# 1. Load pack
./scripts/load-core-pack.sh

# 2. Create a test rule
attune rule create \
  --name "test_timer" \
  --trigger "core.intervaltimer" \
  --trigger-config '{"unit":"seconds","interval":10}' \
  --action "core.echo" \
  --action-params '{"message":"Hello!"}' \
  --enabled

# 3. Monitor executions
attune execution list --limit 5

Files Created

• ✅ scripts/load_core_pack.py (478 lines)
• ✅ scripts/load-core-pack.sh (231 lines)
• ✅ packs/core/SETUP.md (305 lines)
• ✅ work-summary/core-pack-setup-summary.md (this file)

Files Modified

• ✅ README.md - Added "Load the Core Pack" section

Benefits

For Users

• Simple Setup - One command to load entire pack
• Clear Documentation - Step-by-step guides
• Error Messages - Helpful troubleshooting info
• Verification Tools - Easy to confirm success

For Developers

• Flexible Loading - Multiple methods available
• Idempotent - Safe to run multiple times
• Version Control - YAML definitions in git
• Easy Updates - Change YAML and reload
• Extensible - Easy to add new components

For Operations

• Automated - Script-based for CI/CD
• Transactional - All-or-nothing updates
• Logged - Clear output for debugging
• Configurable - Environment variable support

Future Enhancements

Short Term

• Add --force flag to delete and recreate
• Add validation mode (check YAML without loading)
• Generate migration SQL from YAML changes
• Support loading multiple packs at once

Medium Term

• CLI integration: attune pack load ./packs/core
• Pack versioning and upgrades
• Dependency resolution between packs
• Pack marketplace/registry support

Long Term

• Hot-reload without service restart
• Pack development mode with file watching
• Pack testing framework
• Pack distribution as archives

Integration Points

Services That Use Core Pack

1. Sensor Service - Runs interval timer sensor
2. Worker Service - Executes core actions
3. Executor Service - Schedules core action executions
4. API Service - Serves core pack metadata

Other Packs Can Depend On Core

Example in custom-pack/pack.yaml:

runtime_deps:
  - core

Success Criteria

• ✅ Core pack can be loaded with one command
• ✅ All components (triggers, actions, sensors) are created
• ✅ Idempotent operation (safe to re-run)
• ✅ Clear error messages on failure
• ✅ Comprehensive documentation
• ✅ Verification commands work
• ✅ Compatible with existing pack structure

Next Steps

After loading the core pack:

1. Start Services - Run executor, worker, sensor services
2. Create Rules - Use core triggers and actions
3. Test Automation - Verify timer triggers fire
4. Build Custom Packs - Create domain-specific automation
5. Monitor Executions - Observe core actions running

Related Documentation

• packs/core/README.md - Component reference
• packs/core/SETUP.md - Detailed setup guide
• packs/core/TESTING.md - Testing procedures
• docs/pack-development.md - Creating custom packs
• docs/api-packs.md - Pack management API

---

Core Pack Setup Status: ✅ COMPLETE AND PRODUCTION-READY

The core pack can now be easily loaded into any Attune installation with comprehensive documentation and tooling support.