680 lines
20 KiB
Markdown
680 lines
20 KiB
Markdown
# Core Pack Integration Guide
|
|
|
|
**Last Updated**: 2026-01-20
|
|
**Status**: Implementation Guide
|
|
|
|
---
|
|
|
|
## Overview
|
|
|
|
This document outlines the steps required to integrate the filesystem-based core pack with the Attune platform. The core pack has been implemented in `packs/core/` and needs to be loaded into the system during startup or installation.
|
|
|
|
---
|
|
|
|
## Current State
|
|
|
|
### ✅ Completed
|
|
|
|
- **Pack Structure**: Complete filesystem-based pack in `packs/core/`
|
|
- **Actions**: 4 actions implemented (echo, sleep, noop, http_request)
|
|
- **Triggers**: 3 trigger type definitions (intervaltimer, crontimer, datetimetimer)
|
|
- **Sensors**: 1 sensor implementation (interval_timer_sensor)
|
|
- **Documentation**: Comprehensive README and pack structure docs
|
|
- **Testing**: Manual validation of action execution
|
|
|
|
### ⏳ Pending Integration
|
|
|
|
- **Pack Loader**: Service to parse and register pack components
|
|
- **Database Registration**: Insert pack metadata and components into PostgreSQL
|
|
- **Worker Integration**: Execute actions from pack directory
|
|
- **Sensor Integration**: Load and run sensors from pack directory
|
|
- **Startup Process**: Automatic pack loading on service startup
|
|
|
|
---
|
|
|
|
## Integration Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ Attune Services │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ ┌──────────────────────────────────────────────────────┐ │
|
|
│ │ Pack Loader Service │ │
|
|
│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │
|
|
│ │ │ Parse │ │ Validate │ │ Register │ │ │
|
|
│ │ │ pack.yaml│→ │ Schemas │→ │ in DB │ │ │
|
|
│ │ └──────────┘ └──────────┘ └──────────┘ │ │
|
|
│ └──────────────────────────────────────────────────────┘ │
|
|
│ ↓ │
|
|
│ ┌──────────────────────────────────────────────────────┐ │
|
|
│ │ PostgreSQL Database │ │
|
|
│ │ ┌──────┐ ┌────────┐ ┌─────────┐ ┌─────────┐ │ │
|
|
│ │ │ Pack │ │ Action │ │ Trigger │ │ Sensor │ │ │
|
|
│ │ │ Meta │ │ Meta │ │ Meta │ │ Meta │ │ │
|
|
│ │ └──────┘ └────────┘ └─────────┘ └─────────┘ │ │
|
|
│ └──────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ ┌──────────────────┐ ┌──────────────────┐ │
|
|
│ │ Worker Service │ │ Sensor Service │ │
|
|
│ │ ┌────────────┐ │ │ ┌────────────┐ │ │
|
|
│ │ │ Execute │ │ │ │ Run │ │ │
|
|
│ │ │ Actions │ │ │ │ Sensors │ │ │
|
|
│ │ │ from Pack │ │ │ │ from Pack │ │ │
|
|
│ │ └────────────┘ │ │ └────────────┘ │ │
|
|
│ └──────────────────┘ └──────────────────┘ │
|
|
│ ↑ ↑ │
|
|
└───────────┼──────────────────────────────────┼──────────────┘
|
|
│ │
|
|
└──────────────┬───────────────────┘
|
|
↓
|
|
┌─────────────────┐
|
|
│ Filesystem │
|
|
│ packs/core/ │
|
|
│ - actions/ │
|
|
│ - sensors/ │
|
|
│ - triggers/ │
|
|
└─────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Implementation Steps
|
|
|
|
### Phase 1: Pack Loader Service
|
|
|
|
Create a pack loader service in `crates/common/src/pack_loader.rs` (or as a separate crate).
|
|
|
|
#### 1.1 Pack Parser
|
|
|
|
```rust
|
|
// Parse pack.yaml manifest
|
|
pub struct PackLoader {
|
|
pack_dir: PathBuf,
|
|
}
|
|
|
|
pub struct PackManifest {
|
|
pub ref_: String,
|
|
pub label: String,
|
|
pub description: String,
|
|
pub version: String,
|
|
pub author: Option<String>,
|
|
pub email: Option<String>,
|
|
pub system: bool,
|
|
pub enabled: bool,
|
|
pub conf_schema: Option<serde_json::Value>,
|
|
pub config: Option<serde_json::Value>,
|
|
pub meta: Option<serde_json::Value>,
|
|
pub tags: Vec<String>,
|
|
pub runtime_deps: Vec<String>,
|
|
}
|
|
|
|
impl PackLoader {
|
|
pub fn load_manifest(&self) -> Result<PackManifest> {
|
|
// Parse packs/{pack_name}/pack.yaml
|
|
}
|
|
}
|
|
```
|
|
|
|
#### 1.2 Component Parsers
|
|
|
|
```rust
|
|
pub struct ActionMetadata {
|
|
pub name: String,
|
|
pub ref_: String,
|
|
pub description: String,
|
|
pub runner_type: String,
|
|
pub entry_point: String,
|
|
pub enabled: bool,
|
|
pub parameters: Option<serde_json::Value>,
|
|
pub output_schema: Option<serde_json::Value>,
|
|
pub tags: Vec<String>,
|
|
}
|
|
|
|
pub struct TriggerMetadata {
|
|
pub name: String,
|
|
pub ref_: String,
|
|
pub description: String,
|
|
pub type_: String,
|
|
pub enabled: bool,
|
|
pub parameters_schema: Option<serde_json::Value>,
|
|
pub payload_schema: Option<serde_json::Value>,
|
|
pub tags: Vec<String>,
|
|
}
|
|
|
|
pub struct SensorMetadata {
|
|
pub name: String,
|
|
pub ref_: String,
|
|
pub description: String,
|
|
pub runner_type: String,
|
|
pub entry_point: String,
|
|
pub trigger_types: Vec<String>,
|
|
pub enabled: bool,
|
|
pub parameters: Option<serde_json::Value>,
|
|
pub poll_interval: Option<i32>,
|
|
pub tags: Vec<String>,
|
|
}
|
|
|
|
impl PackLoader {
|
|
pub fn load_actions(&self) -> Result<Vec<ActionMetadata>> {
|
|
// Parse actions/*.yaml files
|
|
}
|
|
|
|
pub fn load_triggers(&self) -> Result<Vec<TriggerMetadata>> {
|
|
// Parse triggers/*.yaml files
|
|
}
|
|
|
|
pub fn load_sensors(&self) -> Result<Vec<SensorMetadata>> {
|
|
// Parse sensors/*.yaml files
|
|
}
|
|
}
|
|
```
|
|
|
|
#### 1.3 Database Registration
|
|
|
|
```rust
|
|
impl PackLoader {
|
|
pub async fn register_pack(
|
|
&self,
|
|
pool: &PgPool,
|
|
manifest: &PackManifest,
|
|
) -> Result<i64> {
|
|
// Insert into attune.pack table
|
|
// Returns pack ID
|
|
}
|
|
|
|
pub async fn register_actions(
|
|
&self,
|
|
pool: &PgPool,
|
|
pack_id: i64,
|
|
actions: &[ActionMetadata],
|
|
) -> Result<()> {
|
|
// Insert into attune.action table
|
|
}
|
|
|
|
pub async fn register_triggers(
|
|
&self,
|
|
pool: &PgPool,
|
|
pack_id: i64,
|
|
triggers: &[TriggerMetadata],
|
|
) -> Result<()> {
|
|
// Insert into attune.trigger table
|
|
}
|
|
|
|
pub async fn register_sensors(
|
|
&self,
|
|
pool: &PgPool,
|
|
pack_id: i64,
|
|
sensors: &[SensorMetadata],
|
|
) -> Result<()> {
|
|
// Insert into attune.sensor table
|
|
}
|
|
}
|
|
```
|
|
|
|
#### 1.4 Pack Loading Function
|
|
|
|
```rust
|
|
pub async fn load_pack(
|
|
pack_dir: PathBuf,
|
|
pool: &PgPool,
|
|
) -> Result<()> {
|
|
let loader = PackLoader::new(pack_dir);
|
|
|
|
// Parse pack manifest
|
|
let manifest = loader.load_manifest()?;
|
|
|
|
// Register pack
|
|
let pack_id = loader.register_pack(pool, &manifest).await?;
|
|
|
|
// Load and register components
|
|
let actions = loader.load_actions()?;
|
|
loader.register_actions(pool, pack_id, &actions).await?;
|
|
|
|
let triggers = loader.load_triggers()?;
|
|
loader.register_triggers(pool, pack_id, &triggers).await?;
|
|
|
|
let sensors = loader.load_sensors()?;
|
|
loader.register_sensors(pool, pack_id, &sensors).await?;
|
|
|
|
info!("Pack '{}' loaded successfully", manifest.ref_);
|
|
Ok(())
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
### Phase 2: Worker Service Integration
|
|
|
|
Update the worker service to execute actions from the filesystem.
|
|
|
|
#### 2.1 Action Execution Path Resolution
|
|
|
|
```rust
|
|
pub struct ActionExecutor {
|
|
packs_dir: PathBuf,
|
|
}
|
|
|
|
impl ActionExecutor {
|
|
pub fn resolve_action_path(
|
|
&self,
|
|
pack_ref: &str,
|
|
entry_point: &str,
|
|
) -> Result<PathBuf> {
|
|
// packs/{pack_ref}/actions/{entry_point}
|
|
let path = self.packs_dir
|
|
.join(pack_ref)
|
|
.join("actions")
|
|
.join(entry_point);
|
|
|
|
if !path.exists() {
|
|
return Err(Error::ActionNotFound(entry_point.to_string()));
|
|
}
|
|
|
|
Ok(path)
|
|
}
|
|
}
|
|
```
|
|
|
|
#### 2.2 Environment Variable Setup
|
|
|
|
```rust
|
|
pub fn prepare_action_env(
|
|
params: &HashMap<String, serde_json::Value>,
|
|
) -> HashMap<String, String> {
|
|
let mut env = HashMap::new();
|
|
|
|
for (key, value) in params {
|
|
let env_key = format!("ATTUNE_ACTION_{}", key.to_uppercase());
|
|
let env_value = match value {
|
|
serde_json::Value::String(s) => s.clone(),
|
|
serde_json::Value::Number(n) => n.to_string(),
|
|
serde_json::Value::Bool(b) => b.to_string(),
|
|
_ => serde_json::to_string(value).unwrap(),
|
|
};
|
|
env.insert(env_key, env_value);
|
|
}
|
|
|
|
env
|
|
}
|
|
```
|
|
|
|
#### 2.3 Action Execution
|
|
|
|
```rust
|
|
pub async fn execute_action(
|
|
&self,
|
|
action: &Action,
|
|
params: HashMap<String, serde_json::Value>,
|
|
) -> Result<ExecutionResult> {
|
|
// Resolve action script path
|
|
let script_path = self.resolve_action_path(
|
|
&action.pack_ref,
|
|
&action.entrypoint,
|
|
)?;
|
|
|
|
// Prepare environment variables
|
|
let env = prepare_action_env(¶ms);
|
|
|
|
// Execute based on runner type
|
|
let output = match action.runtime_type.as_str() {
|
|
"shell" => self.execute_shell_action(script_path, env).await?,
|
|
"python" => self.execute_python_action(script_path, env).await?,
|
|
_ => return Err(Error::UnsupportedRuntime(action.runtime_type.clone())),
|
|
};
|
|
|
|
Ok(output)
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
### Phase 3: Sensor Service Integration
|
|
|
|
Update the sensor service to load and run sensors from the filesystem.
|
|
|
|
#### 3.1 Sensor Path Resolution
|
|
|
|
```rust
|
|
pub struct SensorManager {
|
|
packs_dir: PathBuf,
|
|
}
|
|
|
|
impl SensorManager {
|
|
pub fn resolve_sensor_path(
|
|
&self,
|
|
pack_ref: &str,
|
|
entry_point: &str,
|
|
) -> Result<PathBuf> {
|
|
// packs/{pack_ref}/sensors/{entry_point}
|
|
let path = self.packs_dir
|
|
.join(pack_ref)
|
|
.join("sensors")
|
|
.join(entry_point);
|
|
|
|
if !path.exists() {
|
|
return Err(Error::SensorNotFound(entry_point.to_string()));
|
|
}
|
|
|
|
Ok(path)
|
|
}
|
|
}
|
|
```
|
|
|
|
#### 3.2 Sensor Environment Setup
|
|
|
|
```rust
|
|
pub fn prepare_sensor_env(
|
|
sensor: &Sensor,
|
|
trigger_instances: &[TriggerInstance],
|
|
) -> HashMap<String, String> {
|
|
let mut env = HashMap::new();
|
|
|
|
// Add sensor config
|
|
for (key, value) in &sensor.config {
|
|
let env_key = format!("ATTUNE_SENSOR_{}", key.to_uppercase());
|
|
env.insert(env_key, value.to_string());
|
|
}
|
|
|
|
// Add trigger instances as JSON array
|
|
let triggers_json = serde_json::to_string(trigger_instances).unwrap();
|
|
env.insert("ATTUNE_SENSOR_TRIGGERS".to_string(), triggers_json);
|
|
|
|
env
|
|
}
|
|
```
|
|
|
|
#### 3.3 Sensor Execution
|
|
|
|
```rust
|
|
pub async fn run_sensor(
|
|
&self,
|
|
sensor: &Sensor,
|
|
trigger_instances: Vec<TriggerInstance>,
|
|
) -> Result<()> {
|
|
// Resolve sensor script path
|
|
let script_path = self.resolve_sensor_path(
|
|
&sensor.pack_ref,
|
|
&sensor.entrypoint,
|
|
)?;
|
|
|
|
// Prepare environment
|
|
let env = prepare_sensor_env(sensor, &trigger_instances);
|
|
|
|
// Start sensor process
|
|
let mut child = Command::new(&script_path)
|
|
.envs(env)
|
|
.stdout(Stdio::piped())
|
|
.stderr(Stdio::piped())
|
|
.spawn()?;
|
|
|
|
// Read stdout line by line (JSON events)
|
|
let stdout = child.stdout.take().unwrap();
|
|
let reader = BufReader::new(stdout);
|
|
|
|
for line in reader.lines() {
|
|
let event_json = line?;
|
|
let event: SensorEvent = serde_json::from_str(&event_json)?;
|
|
|
|
// Create event in database
|
|
self.create_event_from_sensor(sensor, event).await?;
|
|
}
|
|
|
|
Ok(())
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
### Phase 4: Service Startup Integration
|
|
|
|
Add pack loading to service initialization.
|
|
|
|
#### 4.1 API Service Startup
|
|
|
|
```rust
|
|
// In crates/api/src/main.rs
|
|
|
|
#[tokio::main]
|
|
async fn main() -> Result<()> {
|
|
// ... existing initialization ...
|
|
|
|
// Load core pack
|
|
let packs_dir = PathBuf::from("packs");
|
|
let core_pack_dir = packs_dir.join("core");
|
|
|
|
if core_pack_dir.exists() {
|
|
info!("Loading core pack...");
|
|
pack_loader::load_pack(core_pack_dir, &pool).await?;
|
|
info!("Core pack loaded successfully");
|
|
}
|
|
|
|
// ... continue with server startup ...
|
|
}
|
|
```
|
|
|
|
#### 4.2 Worker Service Startup
|
|
|
|
```rust
|
|
// In crates/worker/src/main.rs
|
|
|
|
#[tokio::main]
|
|
async fn main() -> Result<()> {
|
|
// ... existing initialization ...
|
|
|
|
// Set packs directory for action execution
|
|
let packs_dir = PathBuf::from("packs");
|
|
let executor = ActionExecutor::new(packs_dir);
|
|
|
|
// ... continue with worker initialization ...
|
|
}
|
|
```
|
|
|
|
#### 4.3 Sensor Service Startup
|
|
|
|
```rust
|
|
// In crates/sensor/src/main.rs
|
|
|
|
#[tokio::main]
|
|
async fn main() -> Result<()> {
|
|
// ... existing initialization ...
|
|
|
|
// Set packs directory for sensor execution
|
|
let packs_dir = PathBuf::from("packs");
|
|
let sensor_manager = SensorManager::new(packs_dir);
|
|
|
|
// Load and start sensors
|
|
let sensors = load_enabled_sensors(&pool).await?;
|
|
for sensor in sensors {
|
|
sensor_manager.start_sensor(sensor).await?;
|
|
}
|
|
|
|
// ... continue with sensor service ...
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Configuration
|
|
|
|
Add pack-related configuration to `config.yaml`:
|
|
|
|
```yaml
|
|
packs:
|
|
# Directory containing packs
|
|
directory: "./packs"
|
|
|
|
# Auto-load packs on startup
|
|
auto_load:
|
|
- core
|
|
|
|
# Pack-specific configuration
|
|
core:
|
|
max_action_timeout: 300
|
|
enable_debug_logging: false
|
|
```
|
|
|
|
---
|
|
|
|
## Database Schema Updates
|
|
|
|
The existing database schema already supports packs. Ensure these tables are used:
|
|
|
|
- `attune.pack` - Pack metadata
|
|
- `attune.action` - Action definitions
|
|
- `attune.trigger` - Trigger type definitions
|
|
- `attune.sensor` - Sensor definitions
|
|
- `attune.runtime` - Runtime definitions
|
|
|
|
**Note**: The current `scripts/seed_core_pack.sql` inserts data directly. This should be replaced or complemented by the filesystem-based loader.
|
|
|
|
---
|
|
|
|
## Migration Strategy
|
|
|
|
### Option 1: Replace SQL Seed Script
|
|
|
|
Remove `scripts/seed_core_pack.sql` and load from filesystem exclusively.
|
|
|
|
**Pros**: Single source of truth (filesystem)
|
|
**Cons**: Requires pack loader to be implemented first
|
|
|
|
### Option 2: Dual Approach (Recommended)
|
|
|
|
Keep SQL seed script for initial setup, add filesystem loader for development/updates.
|
|
|
|
**Pros**: Works immediately, smooth migration path
|
|
**Cons**: Need to maintain both during transition
|
|
|
|
**Implementation**:
|
|
1. Keep existing SQL seed script for now
|
|
2. Implement pack loader service
|
|
3. Add CLI command: `attune pack reload core`
|
|
4. Eventually replace SQL seed with filesystem loading
|
|
|
|
---
|
|
|
|
## Testing Plan
|
|
|
|
### Unit Tests
|
|
|
|
- Pack manifest parsing
|
|
- Component metadata parsing
|
|
- Path resolution
|
|
- Environment variable preparation
|
|
|
|
### Integration Tests
|
|
|
|
1. **Pack Loading**
|
|
- Load core pack from filesystem
|
|
- Verify database registration
|
|
- Validate component metadata
|
|
|
|
2. **Action Execution**
|
|
- Execute `core.echo` with parameters
|
|
- Execute `core.http_request` with mock server
|
|
- Verify environment variable passing
|
|
- Capture stdout/stderr correctly
|
|
|
|
3. **Sensor Execution**
|
|
- Run `core.interval_timer_sensor`
|
|
- Verify event emission
|
|
- Check trigger firing logic
|
|
|
|
### End-to-End Tests
|
|
|
|
- Create rule with `core.intervaltimer` trigger
|
|
- Verify rule fires and executes `core.echo` action
|
|
- Check execution logs and results
|
|
|
|
---
|
|
|
|
## Dependencies
|
|
|
|
### Rust Crates
|
|
|
|
```toml
|
|
[dependencies]
|
|
serde_yaml = "0.9" # Parse YAML files
|
|
walkdir = "2.4" # Traverse pack directories
|
|
tokio = { version = "1", features = ["process"] } # Async process execution
|
|
```
|
|
|
|
### System Dependencies
|
|
|
|
- Shell (bash/sh) for shell actions
|
|
- Python 3.8+ for Python actions
|
|
- Python packages: `requests>=2.28.0`, `croniter>=1.4.0`
|
|
|
|
---
|
|
|
|
## Rollout Plan
|
|
|
|
### Week 1: Pack Loader Implementation
|
|
- [ ] Create `pack_loader` module in `attune_common`
|
|
- [ ] Implement manifest and component parsers
|
|
- [ ] Add database registration functions
|
|
- [ ] Write unit tests
|
|
|
|
### Week 2: Worker Integration
|
|
- [ ] Add action path resolution
|
|
- [ ] Implement environment variable preparation
|
|
- [ ] Update action execution to use filesystem
|
|
- [ ] Add integration tests
|
|
|
|
### Week 3: Sensor Integration
|
|
- [ ] Add sensor path resolution
|
|
- [ ] Implement sensor process management
|
|
- [ ] Update event creation from sensor output
|
|
- [ ] Add integration tests
|
|
|
|
### Week 4: Testing & Documentation
|
|
- [ ] End-to-end testing
|
|
- [ ] CLI commands for pack management
|
|
- [ ] Update deployment documentation
|
|
- [ ] Performance testing
|
|
|
|
---
|
|
|
|
## Success Criteria
|
|
|
|
- ✅ Core pack loaded from filesystem on startup
|
|
- ✅ Actions execute successfully from pack directory
|
|
- ✅ Sensors run and emit events correctly
|
|
- ✅ Environment variables passed properly to actions/sensors
|
|
- ✅ Database contains correct metadata for all components
|
|
- ✅ No regression in existing functionality
|
|
- ✅ Integration tests pass
|
|
- ✅ Documentation updated
|
|
|
|
---
|
|
|
|
## Related Documentation
|
|
|
|
- `packs/core/README.md` - Core pack usage guide
|
|
- `docs/pack-structure.md` - Pack structure reference
|
|
- `docs/pack-management-architecture.md` - Architecture overview
|
|
- `docs/worker-service.md` - Worker service documentation
|
|
- `docs/sensor-service.md` - Sensor service documentation
|
|
|
|
---
|
|
|
|
## Open Questions
|
|
|
|
1. **Runtime Registration**: Should we create runtime entries in the database for each runner type (shell, python)?
|
|
2. **Pack Versioning**: How to handle pack updates? Replace existing entries or keep version history?
|
|
3. **Pack Dependencies**: How to handle dependencies between packs?
|
|
4. **Pack Registry**: Future external pack registry integration?
|
|
5. **Hot Reload**: Should packs be hot-reloadable without service restart?
|
|
|
|
---
|
|
|
|
## Conclusion
|
|
|
|
Integrating the filesystem-based core pack requires:
|
|
1. Pack loader service to parse and register components
|
|
2. Worker service updates to execute actions from filesystem
|
|
3. Sensor service updates to run sensors from filesystem
|
|
4. Startup integration to load packs automatically
|
|
|
|
The implementation can be phased, starting with the pack loader, then worker integration, then sensor integration. The existing SQL seed script can remain as a fallback during the transition. |