379 lines
10 KiB
Markdown
379 lines
10 KiB
Markdown
# Core Pack Setup Implementation Summary
|
|
|
|
**Date:** 2025-01-20
|
|
**Task:** Core Pack Setup - Loader Script & Documentation
|
|
**Status:** ✅ Complete
|
|
|
|
## Overview
|
|
|
|
Created a comprehensive system for loading the Attune core pack from the filesystem into the database. This enables the built-in system pack to be properly initialized with all its actions, triggers, and sensors.
|
|
|
|
## What Was Built
|
|
|
|
### 1. Python Loader Script (`scripts/load_core_pack.py`)
|
|
|
|
**Purpose:** Parse YAML pack definitions and load them into the database
|
|
|
|
**Features:**
|
|
- Reads `pack.yaml` for pack metadata
|
|
- Loads triggers from `triggers/*.yaml`
|
|
- Loads actions from `actions/*.yaml`
|
|
- Loads sensors from `sensors/*.yaml`
|
|
- Creates runtime entries for actions and sensors
|
|
- Idempotent operations (can be run multiple times safely)
|
|
- Transaction-based (all-or-nothing)
|
|
- Command-line arguments for flexibility
|
|
- Comprehensive error handling and reporting
|
|
|
|
**Usage:**
|
|
```bash
|
|
# Basic usage
|
|
python3 scripts/load_core_pack.py
|
|
|
|
# With custom database URL
|
|
python3 scripts/load_core_pack.py --database-url "postgresql://..."
|
|
|
|
# With custom pack directory
|
|
python3 scripts/load_core_pack.py --pack-dir ./packs
|
|
```
|
|
|
|
**Key Functions:**
|
|
- `upsert_pack()` - Create/update pack metadata
|
|
- `upsert_triggers()` - Load all trigger definitions
|
|
- `upsert_actions()` - Load all action definitions
|
|
- `upsert_sensors()` - Load all sensor definitions
|
|
- `ensure_shell_runtime()` - Create shell runtime for actions
|
|
- `ensure_sensor_runtime()` - Create built-in sensor runtime
|
|
|
|
**Database Operations:**
|
|
- Uses `ON CONFLICT ... DO UPDATE` for upserts
|
|
- Maintains referential integrity (pack → triggers/actions → sensors)
|
|
- Returns IDs for cross-referencing
|
|
- Proper transaction handling with rollback on error
|
|
|
|
### 2. Shell Wrapper Script (`scripts/load-core-pack.sh`)
|
|
|
|
**Purpose:** User-friendly wrapper with prerequisite checking
|
|
|
|
**Features:**
|
|
- Prerequisites validation (Python, packages, database)
|
|
- Interactive package installation
|
|
- Database connectivity testing
|
|
- Colored output for better UX
|
|
- Help documentation
|
|
- Environment variable support
|
|
- Verbose mode for debugging
|
|
|
|
**Checks Performed:**
|
|
1. ✓ Python 3 is installed
|
|
2. ✓ Required Python packages (psycopg2-binary, pyyaml)
|
|
3. ✓ Database is accessible
|
|
4. ✓ Core pack directory exists
|
|
5. ✓ pack.yaml file is present
|
|
|
|
**Usage:**
|
|
```bash
|
|
# Basic usage
|
|
./scripts/load-core-pack.sh
|
|
|
|
# With options
|
|
./scripts/load-core-pack.sh --database-url "postgresql://..." --verbose
|
|
|
|
# Dry run
|
|
./scripts/load-core-pack.sh --dry-run
|
|
```
|
|
|
|
### 3. Comprehensive Documentation
|
|
|
|
#### `packs/core/SETUP.md` (305 lines)
|
|
|
|
Complete setup guide including:
|
|
- **Overview** - What the core pack provides
|
|
- **Prerequisites** - Requirements before loading
|
|
- **Loading Methods** - 3 different approaches:
|
|
1. Python loader script (recommended)
|
|
2. SQL seed script
|
|
3. CLI (future)
|
|
- **Verification** - How to confirm successful loading
|
|
- **Testing** - Testing the loaded pack
|
|
- **Updating** - Re-running after changes
|
|
- **Troubleshooting** - Common issues and solutions
|
|
- **Development Workflow** - Adding new components
|
|
- **Environment Variables** - Configuration options
|
|
- **CI/CD Integration** - Automated deployment examples
|
|
|
|
#### Updated `README.md`
|
|
|
|
Added new section "3. Load the Core Pack":
|
|
- Quick start instructions
|
|
- Verification commands
|
|
- Link to detailed setup guide
|
|
- Renumbered subsequent sections
|
|
|
|
### 4. Existing Assets
|
|
|
|
**Already Present:**
|
|
- ✅ `packs/core/pack.yaml` - Pack metadata
|
|
- ✅ `packs/core/actions/*.yaml` - Action definitions (echo, sleep, noop, http_request)
|
|
- ✅ `packs/core/actions/*.sh` - Shell action implementations
|
|
- ✅ `packs/core/actions/*.py` - Python action implementations
|
|
- ✅ `packs/core/triggers/*.yaml` - Trigger definitions (intervaltimer, crontimer, datetimetimer)
|
|
- ✅ `packs/core/sensors/*.yaml` - Sensor definitions
|
|
- ✅ `packs/core/sensors/*.py` - Sensor implementations
|
|
- ✅ `packs/core/README.md` - Component documentation
|
|
- ✅ `packs/core/TESTING.md` - Testing procedures
|
|
- ✅ `packs/core/test_core_pack.sh` - Test suite
|
|
- ✅ `scripts/seed_core_pack.sql` - SQL seed script (legacy)
|
|
|
|
## Core Pack Contents
|
|
|
|
### Pack Metadata
|
|
- **Ref:** `core`
|
|
- **Version:** `1.0.0`
|
|
- **Type:** System pack (built-in)
|
|
|
|
### Triggers (3)
|
|
1. `core.intervaltimer` - Regular interval-based timer
|
|
2. `core.crontimer` - Cron schedule-based timer
|
|
3. `core.datetimetimer` - One-shot datetime timer
|
|
|
|
### Actions (4)
|
|
1. `core.echo` - Echo message to stdout
|
|
2. `core.sleep` - Sleep for N seconds
|
|
3. `core.noop` - No operation (testing)
|
|
4. `core.http_request` - Make HTTP requests
|
|
|
|
### Sensors (1)
|
|
1. `core.interval_timer_sensor` - Built-in interval timer sensor
|
|
|
|
## Technical Details
|
|
|
|
### Database Schema Usage
|
|
|
|
**Tables Populated:**
|
|
- `attune.pack` - Core pack entry
|
|
- `attune.runtime` - Shell and sensor runtimes
|
|
- `attune.trigger` - Trigger type definitions
|
|
- `attune.action` - Action definitions
|
|
- `attune.sensor` - Sensor instances
|
|
|
|
**Key Features:**
|
|
- UPSERT operations for idempotency
|
|
- Proper foreign key relationships
|
|
- JSON schema storage for parameters/output
|
|
- Runtime type discrimination (action vs sensor)
|
|
|
|
### Python Dependencies
|
|
|
|
Required packages:
|
|
- `psycopg2-binary` - PostgreSQL database adapter
|
|
- `pyyaml` - YAML parsing
|
|
|
|
### Environment Variables
|
|
|
|
- `DATABASE_URL` - PostgreSQL connection string
|
|
- Default: `postgresql://postgres:postgres@localhost:5432/attune`
|
|
|
|
- `ATTUNE_PACKS_DIR` - Base directory for packs
|
|
- Default: `./packs`
|
|
|
|
## Usage Workflow
|
|
|
|
### First-Time Setup
|
|
```bash
|
|
# 1. Ensure database is running and migrations applied
|
|
createdb attune
|
|
sqlx migrate run
|
|
|
|
# 2. Install Python dependencies
|
|
pip install psycopg2-binary pyyaml
|
|
|
|
# 3. Load core pack
|
|
./scripts/load-core-pack.sh
|
|
|
|
# 4. Verify
|
|
attune pack show core
|
|
```
|
|
|
|
### Development Workflow
|
|
```bash
|
|
# 1. Edit pack files (actions, triggers, sensors)
|
|
vim packs/core/actions/new_action.yaml
|
|
vim packs/core/actions/new_action.sh
|
|
|
|
# 2. Make scripts executable
|
|
chmod +x packs/core/actions/new_action.sh
|
|
|
|
# 3. Test locally
|
|
export ATTUNE_ACTION_FOO="bar"
|
|
./packs/core/actions/new_action.sh
|
|
|
|
# 4. Reload into database
|
|
./scripts/load-core-pack.sh
|
|
|
|
# 5. Restart services
|
|
# API, executor, worker, sensor services
|
|
```
|
|
|
|
### Updating Existing Components
|
|
```bash
|
|
# 1. Modify YAML definitions
|
|
vim packs/core/actions/echo.yaml
|
|
|
|
# 2. Re-run loader (upsert mode)
|
|
./scripts/load-core-pack.sh
|
|
|
|
# 3. Changes take effect immediately for new executions
|
|
```
|
|
|
|
## Testing
|
|
|
|
### Loader Script Testing
|
|
```bash
|
|
# Syntax check
|
|
python3 -m py_compile scripts/load_core_pack.py
|
|
|
|
# Dry run
|
|
python3 scripts/load_core_pack.py --dry-run
|
|
|
|
# Actual load
|
|
python3 scripts/load_core_pack.py
|
|
```
|
|
|
|
### Verification Queries
|
|
```sql
|
|
-- Check pack exists
|
|
SELECT * FROM attune.pack WHERE ref = 'core';
|
|
|
|
-- Count components
|
|
SELECT
|
|
(SELECT COUNT(*) FROM attune.trigger WHERE pack_ref = 'core') as triggers,
|
|
(SELECT COUNT(*) FROM attune.action WHERE pack_ref = 'core') as actions,
|
|
(SELECT COUNT(*) FROM attune.sensor WHERE pack_ref = 'core') as sensors;
|
|
|
|
-- List all core components
|
|
SELECT ref, label FROM attune.trigger WHERE pack_ref = 'core';
|
|
SELECT ref, label FROM attune.action WHERE pack_ref = 'core';
|
|
SELECT ref, label FROM attune.sensor WHERE pack_ref = 'core';
|
|
```
|
|
|
|
### End-to-End Test
|
|
```bash
|
|
# 1. Load pack
|
|
./scripts/load-core-pack.sh
|
|
|
|
# 2. Create a test rule
|
|
attune rule create \
|
|
--name "test_timer" \
|
|
--trigger "core.intervaltimer" \
|
|
--trigger-config '{"unit":"seconds","interval":10}' \
|
|
--action "core.echo" \
|
|
--action-params '{"message":"Hello!"}' \
|
|
--enabled
|
|
|
|
# 3. Monitor executions
|
|
attune execution list --limit 5
|
|
```
|
|
|
|
## Files Created
|
|
|
|
- ✅ `scripts/load_core_pack.py` (478 lines)
|
|
- ✅ `scripts/load-core-pack.sh` (231 lines)
|
|
- ✅ `packs/core/SETUP.md` (305 lines)
|
|
- ✅ `work-summary/core-pack-setup-summary.md` (this file)
|
|
|
|
## Files Modified
|
|
|
|
- ✅ `README.md` - Added "Load the Core Pack" section
|
|
|
|
## Benefits
|
|
|
|
### For Users
|
|
- **Simple Setup** - One command to load entire pack
|
|
- **Clear Documentation** - Step-by-step guides
|
|
- **Error Messages** - Helpful troubleshooting info
|
|
- **Verification Tools** - Easy to confirm success
|
|
|
|
### For Developers
|
|
- **Flexible Loading** - Multiple methods available
|
|
- **Idempotent** - Safe to run multiple times
|
|
- **Version Control** - YAML definitions in git
|
|
- **Easy Updates** - Change YAML and reload
|
|
- **Extensible** - Easy to add new components
|
|
|
|
### For Operations
|
|
- **Automated** - Script-based for CI/CD
|
|
- **Transactional** - All-or-nothing updates
|
|
- **Logged** - Clear output for debugging
|
|
- **Configurable** - Environment variable support
|
|
|
|
## Future Enhancements
|
|
|
|
### Short Term
|
|
- [ ] Add `--force` flag to delete and recreate
|
|
- [ ] Add validation mode (check YAML without loading)
|
|
- [ ] Generate migration SQL from YAML changes
|
|
- [ ] Support loading multiple packs at once
|
|
|
|
### Medium Term
|
|
- [ ] CLI integration: `attune pack load ./packs/core`
|
|
- [ ] Pack versioning and upgrades
|
|
- [ ] Dependency resolution between packs
|
|
- [ ] Pack marketplace/registry support
|
|
|
|
### Long Term
|
|
- [ ] Hot-reload without service restart
|
|
- [ ] Pack development mode with file watching
|
|
- [ ] Pack testing framework
|
|
- [ ] Pack distribution as archives
|
|
|
|
## Integration Points
|
|
|
|
### Services That Use Core Pack
|
|
|
|
1. **Sensor Service** - Runs interval timer sensor
|
|
2. **Worker Service** - Executes core actions
|
|
3. **Executor Service** - Schedules core action executions
|
|
4. **API Service** - Serves core pack metadata
|
|
|
|
### Other Packs Can Depend On Core
|
|
|
|
Example in `custom-pack/pack.yaml`:
|
|
```yaml
|
|
runtime_deps:
|
|
- core
|
|
```
|
|
|
|
## Success Criteria
|
|
|
|
- ✅ Core pack can be loaded with one command
|
|
- ✅ All components (triggers, actions, sensors) are created
|
|
- ✅ Idempotent operation (safe to re-run)
|
|
- ✅ Clear error messages on failure
|
|
- ✅ Comprehensive documentation
|
|
- ✅ Verification commands work
|
|
- ✅ Compatible with existing pack structure
|
|
|
|
## Next Steps
|
|
|
|
After loading the core pack:
|
|
|
|
1. **Start Services** - Run executor, worker, sensor services
|
|
2. **Create Rules** - Use core triggers and actions
|
|
3. **Test Automation** - Verify timer triggers fire
|
|
4. **Build Custom Packs** - Create domain-specific automation
|
|
5. **Monitor Executions** - Observe core actions running
|
|
|
|
## Related Documentation
|
|
|
|
- `packs/core/README.md` - Component reference
|
|
- `packs/core/SETUP.md` - Detailed setup guide
|
|
- `packs/core/TESTING.md` - Testing procedures
|
|
- `docs/pack-development.md` - Creating custom packs
|
|
- `docs/api-packs.md` - Pack management API
|
|
|
|
---
|
|
|
|
**Core Pack Setup Status: ✅ COMPLETE AND PRODUCTION-READY**
|
|
|
|
The core pack can now be easily loaded into any Attune installation with comprehensive documentation and tooling support. |