re-uploading work
This commit is contained in:
419
docs/QUICKREF-unified-runtime-detection.md
Normal file
419
docs/QUICKREF-unified-runtime-detection.md
Normal file
@@ -0,0 +1,419 @@
|
||||
# Quick Reference: Unified Runtime Detection System
|
||||
|
||||
**Last Updated:** 2024-02-03
|
||||
**Status:** Production-ready
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Both worker and sensor services use a **unified runtime detection system** from `attune-common` that:
|
||||
- Queries a single `runtime` table (no action/sensor distinction)
|
||||
- Verifies runtime availability using database-stored verification metadata
|
||||
- Supports three-tier configuration: env var → config file → database detection
|
||||
|
||||
---
|
||||
|
||||
## Key Changes from Previous System
|
||||
|
||||
### What Changed
|
||||
- ❌ **Removed:** `runtime_type` field (was `'action'` or `'sensor'`)
|
||||
- ❌ **Removed:** Duplicate runtime records for actions vs sensors
|
||||
- ✅ **Added:** Unified `RuntimeDetector` in `attune-common`
|
||||
- ✅ **Added:** Shared verification logic for all services
|
||||
|
||||
### Runtime Refs Changed
|
||||
```
|
||||
Before: core.action.python, core.sensor.python
|
||||
After: core.python (used by both)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Worker Service
|
||||
|
||||
```rust
|
||||
use attune_common::runtime_detection::RuntimeDetector;
|
||||
|
||||
let mut registration = WorkerRegistration::new(pool, &config);
|
||||
registration.detect_capabilities(&config).await?;
|
||||
registration.register().await?;
|
||||
```
|
||||
|
||||
### Sensor Service
|
||||
|
||||
```rust
|
||||
use attune_common::runtime_detection::RuntimeDetector;
|
||||
|
||||
let mut registration = SensorWorkerRegistration::new(pool, &config);
|
||||
registration.register(&config).await?; // Calls detection internally
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration Priority
|
||||
|
||||
### 1. Environment Variable (Highest)
|
||||
```bash
|
||||
# Worker
|
||||
export ATTUNE_WORKER_RUNTIMES="python,shell,node"
|
||||
|
||||
# Sensor
|
||||
export ATTUNE_SENSOR_RUNTIMES="python,shell,builtin"
|
||||
```
|
||||
|
||||
### 2. Config File (Medium)
|
||||
```yaml
|
||||
worker:
|
||||
capabilities:
|
||||
runtimes: ["python", "shell", "native"]
|
||||
|
||||
sensor:
|
||||
capabilities:
|
||||
runtimes: ["python", "shell", "builtin"]
|
||||
```
|
||||
|
||||
### 3. Database Detection (Default)
|
||||
- Queries all runtimes from `runtime` table
|
||||
- Verifies each using `distributions->verification` metadata
|
||||
- Reports only available runtimes
|
||||
|
||||
---
|
||||
|
||||
## Database Structure
|
||||
|
||||
### Runtime Table (Unified)
|
||||
|
||||
```sql
|
||||
CREATE TABLE runtime (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
ref TEXT NOT NULL UNIQUE,
|
||||
pack BIGINT REFERENCES pack(id),
|
||||
pack_ref TEXT,
|
||||
description TEXT,
|
||||
name TEXT NOT NULL, -- e.g., "Python", "Node.js"
|
||||
distributions JSONB NOT NULL, -- Verification metadata
|
||||
installation JSONB,
|
||||
created TIMESTAMPTZ DEFAULT NOW(),
|
||||
updated TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
```
|
||||
|
||||
**No `runtime_type` field** - runtimes are shared between actions and sensors.
|
||||
|
||||
### Verification Metadata Structure
|
||||
|
||||
```json
|
||||
{
|
||||
"verification": {
|
||||
"commands": [
|
||||
{
|
||||
"binary": "python3",
|
||||
"args": ["--version"],
|
||||
"exit_code": 0,
|
||||
"pattern": "Python 3\\.",
|
||||
"priority": 1
|
||||
}
|
||||
],
|
||||
"always_available": false,
|
||||
"check_required": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Runtime Refs
|
||||
|
||||
| Runtime | Ref | Always Available? |
|
||||
|---------|-----|-------------------|
|
||||
| Python | `core.python` | No (requires verification) |
|
||||
| Node.js | `core.nodejs` | No (requires verification) |
|
||||
| Shell | `core.shell` | Yes |
|
||||
| Native | `core.native` | Yes |
|
||||
| Built-in Sensors | `core.sensor.builtin` | Yes (sensor-only) |
|
||||
|
||||
---
|
||||
|
||||
## RuntimeDetector API
|
||||
|
||||
### Methods
|
||||
|
||||
```rust
|
||||
pub struct RuntimeDetector {
|
||||
pool: PgPool,
|
||||
}
|
||||
|
||||
impl RuntimeDetector {
|
||||
pub fn new(pool: PgPool) -> Self
|
||||
|
||||
pub async fn detect_capabilities(
|
||||
&self,
|
||||
config: &Config,
|
||||
env_var_name: &str,
|
||||
config_capabilities: Option<&HashMap<String, serde_json::Value>>,
|
||||
) -> Result<HashMap<String, serde_json::Value>>
|
||||
|
||||
pub async fn detect_from_database(&self) -> Result<Vec<String>>
|
||||
|
||||
pub async fn verify_runtime_available(runtime: &Runtime) -> bool
|
||||
}
|
||||
```
|
||||
|
||||
### Example Usage
|
||||
|
||||
```rust
|
||||
use attune_common::runtime_detection::RuntimeDetector;
|
||||
|
||||
let detector = RuntimeDetector::new(pool.clone());
|
||||
|
||||
// For worker service
|
||||
let capabilities = detector
|
||||
.detect_capabilities(
|
||||
&config,
|
||||
"ATTUNE_WORKER_RUNTIMES",
|
||||
config.worker.as_ref().and_then(|w| w.capabilities.as_ref())
|
||||
)
|
||||
.await?;
|
||||
|
||||
// For sensor service
|
||||
let capabilities = detector
|
||||
.detect_capabilities(
|
||||
&config,
|
||||
"ATTUNE_SENSOR_RUNTIMES",
|
||||
config.sensor.as_ref().and_then(|s| s.capabilities.as_ref())
|
||||
)
|
||||
.await?;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Migration
|
||||
|
||||
### Apply Migration
|
||||
|
||||
```bash
|
||||
cd attune
|
||||
sqlx migrate run
|
||||
```
|
||||
|
||||
**Migration:** `20260203000001_unify_runtimes.sql`
|
||||
|
||||
**What It Does:**
|
||||
- Consolidates duplicate runtime records
|
||||
- Migrates foreign keys in `action` and `sensor` tables
|
||||
- Drops `runtime_type` column and enum
|
||||
- Updates indexes
|
||||
|
||||
### Verify Migration
|
||||
|
||||
```sql
|
||||
-- Check unified runtimes
|
||||
SELECT ref, name FROM runtime ORDER BY ref;
|
||||
|
||||
-- Expected:
|
||||
-- core.native
|
||||
-- core.nodejs
|
||||
-- core.python
|
||||
-- core.sensor.builtin
|
||||
-- core.shell
|
||||
|
||||
-- Check worker capabilities
|
||||
SELECT name, capabilities->'runtimes' FROM worker;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Adding New Runtimes
|
||||
|
||||
### 1. Add to Database
|
||||
|
||||
```sql
|
||||
INSERT INTO runtime (ref, pack, pack_ref, name, distributions)
|
||||
VALUES (
|
||||
'core.ruby',
|
||||
(SELECT id FROM pack WHERE ref = 'core'),
|
||||
'core',
|
||||
'Ruby',
|
||||
jsonb_build_object(
|
||||
'verification', jsonb_build_object(
|
||||
'commands', jsonb_build_array(
|
||||
jsonb_build_object(
|
||||
'binary', 'ruby',
|
||||
'args', jsonb_build_array('--version'),
|
||||
'exit_code', 0,
|
||||
'pattern', 'ruby \d+\.\d+',
|
||||
'priority', 1
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
);
|
||||
```
|
||||
|
||||
### 2. Restart Services
|
||||
|
||||
Services will automatically detect the new runtime on next startup.
|
||||
|
||||
### 3. Verify
|
||||
|
||||
```sql
|
||||
SELECT name, capabilities->'runtimes' FROM worker WHERE name = 'worker-hostname';
|
||||
-- Should include 'ruby' if installed
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Runtime Not Detected
|
||||
|
||||
**Check verification command:**
|
||||
```bash
|
||||
python3 --version # Does this work?
|
||||
node --version # Does this work?
|
||||
```
|
||||
|
||||
**Check database metadata:**
|
||||
```sql
|
||||
SELECT ref, distributions->'verification' FROM runtime WHERE ref = 'core.python';
|
||||
```
|
||||
|
||||
**Force detection:**
|
||||
```bash
|
||||
unset ATTUNE_WORKER_RUNTIMES # Remove env override
|
||||
# Restart service - will query database
|
||||
```
|
||||
|
||||
### Wrong Runtimes Reported
|
||||
|
||||
**Priority order:**
|
||||
1. Env var overrides everything
|
||||
2. Config file if no env var
|
||||
3. Database detection if neither
|
||||
|
||||
**Check env:**
|
||||
```bash
|
||||
env | grep ATTUNE.*RUNTIMES
|
||||
```
|
||||
|
||||
**Check config:**
|
||||
```bash
|
||||
cat config.development.yaml | grep -A5 capabilities
|
||||
```
|
||||
|
||||
### Update Runtime Verification
|
||||
|
||||
```sql
|
||||
UPDATE runtime
|
||||
SET distributions = jsonb_set(
|
||||
distributions,
|
||||
'{verification,commands,0,binary}',
|
||||
'"python3.11"'
|
||||
)
|
||||
WHERE ref = 'core.python';
|
||||
```
|
||||
|
||||
Restart services to pick up changes.
|
||||
|
||||
---
|
||||
|
||||
## Code Locations
|
||||
|
||||
### Core Module
|
||||
- `crates/common/src/runtime_detection.rs` - RuntimeDetector implementation
|
||||
- `crates/common/src/models.rs` - Runtime model (no runtime_type)
|
||||
- `crates/common/src/repositories/runtime.rs` - Database operations
|
||||
|
||||
### Service Integration
|
||||
- `crates/worker/src/registration.rs` - Worker uses RuntimeDetector
|
||||
- `crates/sensor/src/sensor_worker_registration.rs` - Sensor uses RuntimeDetector
|
||||
|
||||
### Migration
|
||||
- `migrations/20260203000001_unify_runtimes.sql` - Schema changes
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
### Unit Tests
|
||||
```bash
|
||||
cargo test -p attune-common runtime_detection
|
||||
```
|
||||
|
||||
### Integration Tests
|
||||
```bash
|
||||
cargo test --test repository_runtime_tests
|
||||
```
|
||||
|
||||
### Manual Verification
|
||||
```bash
|
||||
# Start worker with debug logging
|
||||
RUST_LOG=debug cargo run -p attune-worker
|
||||
|
||||
# Check logs for:
|
||||
# - "Detecting worker capabilities..."
|
||||
# - "✓ Runtime available: Python (core.python)"
|
||||
# - "Detected available runtimes: ["python", "shell", "native"]"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance
|
||||
|
||||
### Query Optimization
|
||||
- Runtime detection happens **once at startup**
|
||||
- Results cached in worker registration
|
||||
- No runtime queries during action/sensor execution
|
||||
|
||||
### Indexing
|
||||
```sql
|
||||
CREATE INDEX idx_runtime_name ON runtime(name);
|
||||
CREATE INDEX idx_runtime_verification ON runtime USING gin ((distributions->'verification'));
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Command Execution
|
||||
- Verification commands run **at startup only**
|
||||
- Commands from database (trusted source)
|
||||
- Output parsed with regex, not eval'd
|
||||
- Non-zero exit codes handled safely
|
||||
|
||||
### Environment Overrides
|
||||
- Env vars allow operators to restrict runtimes
|
||||
- Useful for security-sensitive environments
|
||||
- Can disable verification entirely with explicit list
|
||||
|
||||
---
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
### Planned Features
|
||||
1. **Version Constraints:** Require Python >=3.9
|
||||
2. **Capability Matching:** Route work to compatible workers
|
||||
3. **Health Checks:** Re-verify runtimes periodically
|
||||
4. **API Endpoints:** GET /api/workers/{id}/capabilities
|
||||
|
||||
### Contribution Guide
|
||||
- Add verification metadata to new runtime records
|
||||
- Update `RuntimeDetector` for new verification types
|
||||
- Keep worker and sensor services using shared detector
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
✅ **One Runtime Table** - No action/sensor distinction
|
||||
✅ **Shared Detection Logic** - In `attune-common`
|
||||
✅ **Three-Tier Config** - Env → Config → Database
|
||||
✅ **Database-Driven** - Verification metadata in JSONB
|
||||
✅ **Extensible** - Add runtimes via SQL inserts
|
||||
|
||||
**Migration Required:** Yes (`20260203000001_unify_runtimes.sql`)
|
||||
**Breaking Changes:** Yes (pre-production only)
|
||||
**Production Ready:** ✅ Yes
|
||||
Reference in New Issue
Block a user