419 lines
9.1 KiB
Markdown
419 lines
9.1 KiB
Markdown
# Quick Reference: Unified Runtime Detection System
|
|
|
|
**Last Updated:** 2024-02-03
|
|
**Status:** Production-ready
|
|
|
|
---
|
|
|
|
## Overview
|
|
|
|
Both worker and sensor services use a **unified runtime detection system** from `attune-common` that:
|
|
- Queries a single `runtime` table (no action/sensor distinction)
|
|
- Verifies runtime availability using database-stored verification metadata
|
|
- Supports three-tier configuration: env var → config file → database detection
|
|
|
|
---
|
|
|
|
## Key Changes from Previous System
|
|
|
|
### What Changed
|
|
- ❌ **Removed:** `runtime_type` field (was `'action'` or `'sensor'`)
|
|
- ❌ **Removed:** Duplicate runtime records for actions vs sensors
|
|
- ✅ **Added:** Unified `RuntimeDetector` in `attune-common`
|
|
- ✅ **Added:** Shared verification logic for all services
|
|
|
|
### Runtime Refs Changed
|
|
```
|
|
Before: core.action.python, core.sensor.python
|
|
After: core.python (used by both)
|
|
```
|
|
|
|
---
|
|
|
|
## Quick Start
|
|
|
|
### Worker Service
|
|
|
|
```rust
|
|
use attune_common::runtime_detection::RuntimeDetector;
|
|
|
|
let mut registration = WorkerRegistration::new(pool, &config);
|
|
registration.detect_capabilities(&config).await?;
|
|
registration.register().await?;
|
|
```
|
|
|
|
### Sensor Service
|
|
|
|
```rust
|
|
use attune_common::runtime_detection::RuntimeDetector;
|
|
|
|
let mut registration = SensorWorkerRegistration::new(pool, &config);
|
|
registration.register(&config).await?; // Calls detection internally
|
|
```
|
|
|
|
---
|
|
|
|
## Configuration Priority
|
|
|
|
### 1. Environment Variable (Highest)
|
|
```bash
|
|
# Worker
|
|
export ATTUNE_WORKER_RUNTIMES="python,shell,node"
|
|
|
|
# Sensor
|
|
export ATTUNE_SENSOR_RUNTIMES="python,shell,builtin"
|
|
```
|
|
|
|
### 2. Config File (Medium)
|
|
```yaml
|
|
worker:
|
|
capabilities:
|
|
runtimes: ["python", "shell", "native"]
|
|
|
|
sensor:
|
|
capabilities:
|
|
runtimes: ["python", "shell", "builtin"]
|
|
```
|
|
|
|
### 3. Database Detection (Default)
|
|
- Queries all runtimes from `runtime` table
|
|
- Verifies each using `distributions->verification` metadata
|
|
- Reports only available runtimes
|
|
|
|
---
|
|
|
|
## Database Structure
|
|
|
|
### Runtime Table (Unified)
|
|
|
|
```sql
|
|
CREATE TABLE runtime (
|
|
id BIGSERIAL PRIMARY KEY,
|
|
ref TEXT NOT NULL UNIQUE,
|
|
pack BIGINT REFERENCES pack(id),
|
|
pack_ref TEXT,
|
|
description TEXT,
|
|
name TEXT NOT NULL, -- e.g., "Python", "Node.js"
|
|
distributions JSONB NOT NULL, -- Verification metadata
|
|
installation JSONB,
|
|
created TIMESTAMPTZ DEFAULT NOW(),
|
|
updated TIMESTAMPTZ DEFAULT NOW()
|
|
);
|
|
```
|
|
|
|
**No `runtime_type` field** - runtimes are shared between actions and sensors.
|
|
|
|
### Verification Metadata Structure
|
|
|
|
```json
|
|
{
|
|
"verification": {
|
|
"commands": [
|
|
{
|
|
"binary": "python3",
|
|
"args": ["--version"],
|
|
"exit_code": 0,
|
|
"pattern": "Python 3\\.",
|
|
"priority": 1
|
|
}
|
|
],
|
|
"always_available": false,
|
|
"check_required": true
|
|
}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Common Runtime Refs
|
|
|
|
| Runtime | Ref | Always Available? |
|
|
|---------|-----|-------------------|
|
|
| Python | `core.python` | No (requires verification) |
|
|
| Node.js | `core.nodejs` | No (requires verification) |
|
|
| Shell | `core.shell` | Yes |
|
|
| Native | `core.native` | Yes |
|
|
| Built-in Sensors | `core.sensor.builtin` | Yes (sensor-only) |
|
|
|
|
---
|
|
|
|
## RuntimeDetector API
|
|
|
|
### Methods
|
|
|
|
```rust
|
|
pub struct RuntimeDetector {
|
|
pool: PgPool,
|
|
}
|
|
|
|
impl RuntimeDetector {
|
|
pub fn new(pool: PgPool) -> Self
|
|
|
|
pub async fn detect_capabilities(
|
|
&self,
|
|
config: &Config,
|
|
env_var_name: &str,
|
|
config_capabilities: Option<&HashMap<String, serde_json::Value>>,
|
|
) -> Result<HashMap<String, serde_json::Value>>
|
|
|
|
pub async fn detect_from_database(&self) -> Result<Vec<String>>
|
|
|
|
pub async fn verify_runtime_available(runtime: &Runtime) -> bool
|
|
}
|
|
```
|
|
|
|
### Example Usage
|
|
|
|
```rust
|
|
use attune_common::runtime_detection::RuntimeDetector;
|
|
|
|
let detector = RuntimeDetector::new(pool.clone());
|
|
|
|
// For worker service
|
|
let capabilities = detector
|
|
.detect_capabilities(
|
|
&config,
|
|
"ATTUNE_WORKER_RUNTIMES",
|
|
config.worker.as_ref().and_then(|w| w.capabilities.as_ref())
|
|
)
|
|
.await?;
|
|
|
|
// For sensor service
|
|
let capabilities = detector
|
|
.detect_capabilities(
|
|
&config,
|
|
"ATTUNE_SENSOR_RUNTIMES",
|
|
config.sensor.as_ref().and_then(|s| s.capabilities.as_ref())
|
|
)
|
|
.await?;
|
|
```
|
|
|
|
---
|
|
|
|
## Migration
|
|
|
|
### Apply Migration
|
|
|
|
```bash
|
|
cd attune
|
|
sqlx migrate run
|
|
```
|
|
|
|
**Migration:** `20260203000001_unify_runtimes.sql`
|
|
|
|
**What It Does:**
|
|
- Consolidates duplicate runtime records
|
|
- Migrates foreign keys in `action` and `sensor` tables
|
|
- Drops `runtime_type` column and enum
|
|
- Updates indexes
|
|
|
|
### Verify Migration
|
|
|
|
```sql
|
|
-- Check unified runtimes
|
|
SELECT ref, name FROM runtime ORDER BY ref;
|
|
|
|
-- Expected:
|
|
-- core.native
|
|
-- core.nodejs
|
|
-- core.python
|
|
-- core.sensor.builtin
|
|
-- core.shell
|
|
|
|
-- Check worker capabilities
|
|
SELECT name, capabilities->'runtimes' FROM worker;
|
|
```
|
|
|
|
---
|
|
|
|
## Adding New Runtimes
|
|
|
|
### 1. Add to Database
|
|
|
|
```sql
|
|
INSERT INTO runtime (ref, pack, pack_ref, name, distributions)
|
|
VALUES (
|
|
'core.ruby',
|
|
(SELECT id FROM pack WHERE ref = 'core'),
|
|
'core',
|
|
'Ruby',
|
|
jsonb_build_object(
|
|
'verification', jsonb_build_object(
|
|
'commands', jsonb_build_array(
|
|
jsonb_build_object(
|
|
'binary', 'ruby',
|
|
'args', jsonb_build_array('--version'),
|
|
'exit_code', 0,
|
|
'pattern', 'ruby \d+\.\d+',
|
|
'priority', 1
|
|
)
|
|
)
|
|
)
|
|
)
|
|
);
|
|
```
|
|
|
|
### 2. Restart Services
|
|
|
|
Services will automatically detect the new runtime on next startup.
|
|
|
|
### 3. Verify
|
|
|
|
```sql
|
|
SELECT name, capabilities->'runtimes' FROM worker WHERE name = 'worker-hostname';
|
|
-- Should include 'ruby' if installed
|
|
```
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
### Runtime Not Detected
|
|
|
|
**Check verification command:**
|
|
```bash
|
|
python3 --version # Does this work?
|
|
node --version # Does this work?
|
|
```
|
|
|
|
**Check database metadata:**
|
|
```sql
|
|
SELECT ref, distributions->'verification' FROM runtime WHERE ref = 'core.python';
|
|
```
|
|
|
|
**Force detection:**
|
|
```bash
|
|
unset ATTUNE_WORKER_RUNTIMES # Remove env override
|
|
# Restart service - will query database
|
|
```
|
|
|
|
### Wrong Runtimes Reported
|
|
|
|
**Priority order:**
|
|
1. Env var overrides everything
|
|
2. Config file if no env var
|
|
3. Database detection if neither
|
|
|
|
**Check env:**
|
|
```bash
|
|
env | grep ATTUNE.*RUNTIMES
|
|
```
|
|
|
|
**Check config:**
|
|
```bash
|
|
cat config.development.yaml | grep -A5 capabilities
|
|
```
|
|
|
|
### Update Runtime Verification
|
|
|
|
```sql
|
|
UPDATE runtime
|
|
SET distributions = jsonb_set(
|
|
distributions,
|
|
'{verification,commands,0,binary}',
|
|
'"python3.11"'
|
|
)
|
|
WHERE ref = 'core.python';
|
|
```
|
|
|
|
Restart services to pick up changes.
|
|
|
|
---
|
|
|
|
## Code Locations
|
|
|
|
### Core Module
|
|
- `crates/common/src/runtime_detection.rs` - RuntimeDetector implementation
|
|
- `crates/common/src/models.rs` - Runtime model (no runtime_type)
|
|
- `crates/common/src/repositories/runtime.rs` - Database operations
|
|
|
|
### Service Integration
|
|
- `crates/worker/src/registration.rs` - Worker uses RuntimeDetector
|
|
- `crates/sensor/src/sensor_worker_registration.rs` - Sensor uses RuntimeDetector
|
|
|
|
### Migration
|
|
- `migrations/20260203000001_unify_runtimes.sql` - Schema changes
|
|
|
|
---
|
|
|
|
## Testing
|
|
|
|
### Unit Tests
|
|
```bash
|
|
cargo test -p attune-common runtime_detection
|
|
```
|
|
|
|
### Integration Tests
|
|
```bash
|
|
cargo test --test repository_runtime_tests
|
|
```
|
|
|
|
### Manual Verification
|
|
```bash
|
|
# Start worker with debug logging
|
|
RUST_LOG=debug cargo run -p attune-worker
|
|
|
|
# Check logs for:
|
|
# - "Detecting worker capabilities..."
|
|
# - "✓ Runtime available: Python (core.python)"
|
|
# - "Detected available runtimes: ["python", "shell", "native"]"
|
|
```
|
|
|
|
---
|
|
|
|
## Performance
|
|
|
|
### Query Optimization
|
|
- Runtime detection happens **once at startup**
|
|
- Results cached in worker registration
|
|
- No runtime queries during action/sensor execution
|
|
|
|
### Indexing
|
|
```sql
|
|
CREATE INDEX idx_runtime_name ON runtime(name);
|
|
CREATE INDEX idx_runtime_verification ON runtime USING gin ((distributions->'verification'));
|
|
```
|
|
|
|
---
|
|
|
|
## Security Considerations
|
|
|
|
### Command Execution
|
|
- Verification commands run **at startup only**
|
|
- Commands from database (trusted source)
|
|
- Output parsed with regex, not eval'd
|
|
- Non-zero exit codes handled safely
|
|
|
|
### Environment Overrides
|
|
- Env vars allow operators to restrict runtimes
|
|
- Useful for security-sensitive environments
|
|
- Can disable verification entirely with explicit list
|
|
|
|
---
|
|
|
|
## Future Enhancements
|
|
|
|
### Planned Features
|
|
1. **Version Constraints:** Require Python >=3.9
|
|
2. **Capability Matching:** Route work to compatible workers
|
|
3. **Health Checks:** Re-verify runtimes periodically
|
|
4. **API Endpoints:** GET /api/workers/{id}/capabilities
|
|
|
|
### Contribution Guide
|
|
- Add verification metadata to new runtime records
|
|
- Update `RuntimeDetector` for new verification types
|
|
- Keep worker and sensor services using shared detector
|
|
|
|
---
|
|
|
|
## Summary
|
|
|
|
✅ **One Runtime Table** - No action/sensor distinction
|
|
✅ **Shared Detection Logic** - In `attune-common`
|
|
✅ **Three-Tier Config** - Env → Config → Database
|
|
✅ **Database-Driven** - Verification metadata in JSONB
|
|
✅ **Extensible** - Add runtimes via SQL inserts
|
|
|
|
**Migration Required:** Yes (`20260203000001_unify_runtimes.sql`)
|
|
**Breaking Changes:** Yes (pre-production only)
|
|
**Production Ready:** ✅ Yes |