attune/docs/QUICKREF-unified-runtime-detection.md

Quick Reference: Unified Runtime Detection System

Last Updated: 2024-02-03
Status: Production-ready

---

Overview

Both worker and sensor services use a unified runtime detection system from attune-common that:

• Queries a single runtime table (no action/sensor distinction)
• Verifies runtime availability using database-stored verification metadata
• Supports three-tier configuration: env var → config file → database detection

---

Key Changes from Previous System

What Changed

• ❌ Removed: runtime_type field (was 'action' or 'sensor')
• ❌ Removed: Duplicate runtime records for actions vs sensors
• ✅ Added: Unified RuntimeDetector in attune-common
• ✅ Added: Shared verification logic for all services

Runtime Refs Changed

Before: core.action.python, core.sensor.python
After:  core.python (used by both)

---

Quick Start

Worker Service

use attune_common::runtime_detection::RuntimeDetector;

let mut registration = WorkerRegistration::new(pool, &config);
registration.detect_capabilities(&config).await?;
registration.register().await?;

Sensor Service

use attune_common::runtime_detection::RuntimeDetector;

let mut registration = SensorWorkerRegistration::new(pool, &config);
registration.register(&config).await?;  // Calls detection internally

---

Configuration Priority

1. Environment Variable (Highest)

# Worker
export ATTUNE_WORKER_RUNTIMES="python,shell,node"

# Sensor
export ATTUNE_SENSOR_RUNTIMES="python,shell,builtin"

2. Config File (Medium)

worker:
  capabilities:
    runtimes: ["python", "shell", "native"]

sensor:
  capabilities:
    runtimes: ["python", "shell", "builtin"]

3. Database Detection (Default)

• Queries all runtimes from runtime table
• Verifies each using distributions->verification metadata
• Reports only available runtimes

---

Database Structure

Runtime Table (Unified)

CREATE TABLE runtime (
    id BIGSERIAL PRIMARY KEY,
    ref TEXT NOT NULL UNIQUE,
    pack BIGINT REFERENCES pack(id),
    pack_ref TEXT,
    description TEXT,
    name TEXT NOT NULL,                    -- e.g., "Python", "Node.js"
    distributions JSONB NOT NULL,          -- Verification metadata
    installation JSONB,
    created TIMESTAMPTZ DEFAULT NOW(),
    updated TIMESTAMPTZ DEFAULT NOW()
);

No runtime_type field - runtimes are shared between actions and sensors.

Verification Metadata Structure

{
  "verification": {
    "commands": [
      {
        "binary": "python3",
        "args": ["--version"],
        "exit_code": 0,
        "pattern": "Python 3\\.",
        "priority": 1
      }
    ],
    "always_available": false,
    "check_required": true
  }
}

---

Common Runtime Refs

Runtime	Ref	Always Available?
Python	core.python	No (requires verification)
Node.js	core.nodejs	No (requires verification)
Shell	core.shell	Yes
Native	core.native	Yes
Built-in Sensors	core.sensor.builtin	Yes (sensor-only)

---

RuntimeDetector API

Methods

pub struct RuntimeDetector {
    pool: PgPool,
}

impl RuntimeDetector {
    pub fn new(pool: PgPool) -> Self

    pub async fn detect_capabilities(
        &self,
        config: &Config,
        env_var_name: &str,
        config_capabilities: Option<&HashMap<String, serde_json::Value>>,
    ) -> Result<HashMap<String, serde_json::Value>>

    pub async fn detect_from_database(&self) -> Result<Vec<String>>

    pub async fn verify_runtime_available(runtime: &Runtime) -> bool
}

Example Usage

use attune_common::runtime_detection::RuntimeDetector;

let detector = RuntimeDetector::new(pool.clone());

// For worker service
let capabilities = detector
    .detect_capabilities(
        &config,
        "ATTUNE_WORKER_RUNTIMES",
        config.worker.as_ref().and_then(|w| w.capabilities.as_ref())
    )
    .await?;

// For sensor service
let capabilities = detector
    .detect_capabilities(
        &config,
        "ATTUNE_SENSOR_RUNTIMES",
        config.sensor.as_ref().and_then(|s| s.capabilities.as_ref())
    )
    .await?;

---

Migration

Apply Migration

cd attune
sqlx migrate run

Migration: 20260203000001_unify_runtimes.sql

What It Does:

• Consolidates duplicate runtime records
• Migrates foreign keys in action and sensor tables
• Drops runtime_type column and enum
• Updates indexes

Verify Migration

-- Check unified runtimes
SELECT ref, name FROM runtime ORDER BY ref;

-- Expected:
-- core.native
-- core.nodejs
-- core.python
-- core.sensor.builtin
-- core.shell

-- Check worker capabilities
SELECT name, capabilities->'runtimes' FROM worker;

---

Adding New Runtimes

1. Add to Database

INSERT INTO runtime (ref, pack, pack_ref, name, distributions)
VALUES (
    'core.ruby',
    (SELECT id FROM pack WHERE ref = 'core'),
    'core',
    'Ruby',
    jsonb_build_object(
        'verification', jsonb_build_object(
            'commands', jsonb_build_array(
                jsonb_build_object(
                    'binary', 'ruby',
                    'args', jsonb_build_array('--version'),
                    'exit_code', 0,
                    'pattern', 'ruby \d+\.\d+',
                    'priority', 1
                )
            )
        )
    )
);

2. Restart Services

Services will automatically detect the new runtime on next startup.

3. Verify

SELECT name, capabilities->'runtimes' FROM worker WHERE name = 'worker-hostname';
-- Should include 'ruby' if installed

---

Troubleshooting

Runtime Not Detected

Check verification command:

python3 --version  # Does this work?
node --version     # Does this work?

Check database metadata:

SELECT ref, distributions->'verification' FROM runtime WHERE ref = 'core.python';

Force detection:

unset ATTUNE_WORKER_RUNTIMES  # Remove env override
# Restart service - will query database

Wrong Runtimes Reported

Priority order:

1. Env var overrides everything
2. Config file if no env var
3. Database detection if neither

Check env:

env | grep ATTUNE.*RUNTIMES

Check config:

cat config.development.yaml | grep -A5 capabilities

Update Runtime Verification

UPDATE runtime
SET distributions = jsonb_set(
    distributions,
    '{verification,commands,0,binary}',
    '"python3.11"'
)
WHERE ref = 'core.python';

Restart services to pick up changes.

---

Code Locations

Core Module

• crates/common/src/runtime_detection.rs - RuntimeDetector implementation
• crates/common/src/models.rs - Runtime model (no runtime_type)
• crates/common/src/repositories/runtime.rs - Database operations

Service Integration

• crates/worker/src/registration.rs - Worker uses RuntimeDetector
• crates/sensor/src/sensor_worker_registration.rs - Sensor uses RuntimeDetector

Migration

• migrations/20260203000001_unify_runtimes.sql - Schema changes

---

Testing

Unit Tests

cargo test -p attune-common runtime_detection

Integration Tests

cargo test --test repository_runtime_tests

Manual Verification

# Start worker with debug logging
RUST_LOG=debug cargo run -p attune-worker

# Check logs for:
# - "Detecting worker capabilities..."
# - "✓ Runtime available: Python (core.python)"
# - "Detected available runtimes: ["python", "shell", "native"]"

---

Performance

Query Optimization

• Runtime detection happens once at startup
• Results cached in worker registration
• No runtime queries during action/sensor execution

Indexing

CREATE INDEX idx_runtime_name ON runtime(name);
CREATE INDEX idx_runtime_verification ON runtime USING gin ((distributions->'verification'));

---

Security Considerations

Command Execution

• Verification commands run at startup only
• Commands from database (trusted source)
• Output parsed with regex, not eval'd
• Non-zero exit codes handled safely

Environment Overrides

• Env vars allow operators to restrict runtimes
• Useful for security-sensitive environments
• Can disable verification entirely with explicit list

---

Future Enhancements

Planned Features

1. Version Constraints: Require Python >=3.9
2. Capability Matching: Route work to compatible workers
3. Health Checks: Re-verify runtimes periodically
4. API Endpoints: GET /api/workers/{id}/capabilities

Contribution Guide

• Add verification metadata to new runtime records
• Update RuntimeDetector for new verification types
• Keep worker and sensor services using shared detector

---

Summary

✅ One Runtime Table - No action/sensor distinction
✅ Shared Detection Logic - In attune-common
✅ Three-Tier Config - Env → Config → Database
✅ Database-Driven - Verification metadata in JSONB
✅ Extensible - Add runtimes via SQL inserts

Migration Required: Yes (20260203000001_unify_runtimes.sql)
Breaking Changes: Yes (pre-production only)
Production Ready: ✅ Yes