attune/work-summary/unified-runtime-detection.md

Work Summary: Unified Runtime Detection System

Date: 2024-02-03
Status: ✅ Complete
Related Documentation:

• docs/sensors/database-driven-runtime-detection.md
• docs/sensors/sensor-worker-registration.md

---

Overview

Consolidated runtime detection logic from sensor and worker services into a unified system in the attune-common crate. Removed the redundant runtime_type distinction between action and sensor runtimes, as both use identical binaries and verification processes.

---

Key Changes

1. Removed runtime_type Field from Database Schema

Migration: migrations/20260203000001_unify_runtimes.sql

• Dropped runtime_type enum column from runtime table
• Dropped runtime_type_enum PostgreSQL enum type
• Consolidated duplicate runtime records:
  • core.action.python + core.sensor.python → core.python
  • core.action.nodejs + core.sensor.nodejs → core.nodejs
  • core.action.shell + core.sensor.shell → core.shell
  • core.action.native + core.sensor.native → core.native
• Migrated all foreign key references in action and sensor tables
• Updated indexes to remove runtime_type references
• Kept core.sensor.builtin as sensor-specific (for internal timers/triggers)

2. Created Unified Runtime Detection Module

New File: crates/common/src/runtime_detection.rs

Provides a single RuntimeDetector service used by both worker and sensor services:

pub struct RuntimeDetector {
    pool: PgPool,
}

impl RuntimeDetector {
    pub async fn detect_capabilities(
        &self,
        config: &Config,
        env_var_name: &str,
        config_capabilities: Option<&HashMap<String, serde_json::Value>>,
    ) -> Result<HashMap<String, serde_json::Value>>
}

Features:

• Three-tier priority configuration:
  1. Environment variable override (e.g., ATTUNE_WORKER_RUNTIMES, ATTUNE_SENSOR_RUNTIMES)
  2. Config file specification
  3. Database-driven detection with verification
• Queries unified runtime table (no runtime_type filter needed)
• Uses verification metadata from database to check runtime availability
• Supports verification commands, exit codes, regex patterns, and "always available" flags
• Returns detected capabilities as HashMap for worker registration

3. Updated Runtime Model

File: crates/common/src/models.rs

• Removed runtime_type field from Runtime struct
• Removed RuntimeType enum from enums module
• Updated all SQLx FromRow derives to exclude the field

4. Updated Runtime Repository

File: crates/common/src/repositories/runtime.rs

• Removed runtime_type from CreateRuntimeInput
• Updated all SQL queries to exclude runtime_type column
• Removed find_by_type() method (no longer needed)
• Updated find_by_pack() to work without type filtering

5. Refactored Worker Service

File: crates/worker/src/registration.rs

Before:

• Hardcoded auto_detect_runtimes() function
• Checked for binaries using Command::new("python3").arg("--version")
• No database integration

After:

• Uses RuntimeDetector::detect_capabilities() from common crate
• Calls detection in detect_capabilities() async method
• Worker service calls this before registration in start() method
• Removed hardcoded detection logic

6. Refactored Sensor Service

File: crates/sensor/src/sensor_worker_registration.rs

Before:

• Inline runtime detection with verify_runtime_available() and try_verification_command()
• Duplicated verification logic from what would be needed in worker

After:

• Uses RuntimeDetector::detect_capabilities() from common crate
• Removed 200+ lines of duplicate verification code
• Simplified to just call shared detector
• Updated register() signature to accept &Config parameter

File: crates/sensor/src/service.rs

• Changed _config to config (stored, not discarded)
• Passes config to registration during start()

7. Updated Test Infrastructure

Files:

• crates/common/tests/helpers.rs - Removed RuntimeType from RuntimeFixture
• crates/common/tests/repository_runtime_tests.rs - Updated all tests to remove runtime_type parameter

Changes:

• RuntimeFixture::new() signature: removed runtime_type parameter
• Updated runtime ref format from core.action.python to core.python
• Fixed 15+ test functions to work without runtime type

---

Rationale: Why Unify Runtimes?

Problem with Separate Runtime Types

1. Duplicate Records: Same binary (e.g., python3) had separate database records for actions and sensors
2. Duplicate Logic: Worker and sensor services had identical verification code
3. Maintenance Burden: Changes to runtime detection required updates in 2+ places
4. Conceptual Mismatch: Runtime type (action vs sensor) describes usage, not capability

Benefits of Unified System

1. Single Source of Truth: One database record per actual runtime binary
2. Code Reuse: Shared detection logic in common crate
3. Easier Extension: Adding new runtimes (Ruby, Go, etc.) requires only database records
4. Clearer Semantics: Runtime describes what it is (Python, Node.js), not what uses it

Example: Before vs After

Before:

-- Two separate records for the same Python binary
INSERT INTO runtime (ref, runtime_type, name) VALUES ('core.action.python', 'action', 'Python');
INSERT INTO runtime (ref, runtime_type, name) VALUES ('core.sensor.python', 'sensor', 'Python');

After:

-- One unified record
INSERT INTO runtime (ref, name) VALUES ('core.python', 'Python');
-- Used by both actions and sensors

---

Configuration Examples

Environment Variable Override

# Worker runtimes
export ATTUNE_WORKER_RUNTIMES="python,shell,node"

# Sensor runtimes  
export ATTUNE_SENSOR_RUNTIMES="python,shell,builtin"

Config File

worker:
  capabilities:
    runtimes: ["python", "shell", "node"]
    
sensor:
  capabilities:
    runtimes: ["python", "shell", "builtin"]

Database-Driven (Default)

If no env var or config, services query runtime table and verify each:

• Check distributions->verification->always_available
• Or execute verification commands and check exit codes/patterns
• Report only available runtimes in worker capabilities

---

Migration Path

For Existing Deployments

1. Run Migration: migrations/20260203000001_unify_runtimes.sql

   • Automatically consolidates runtime records
   • Migrates all foreign key references
   • No manual intervention required

2. Restart Services: Worker and sensor services pick up changes automatically

3. Verify: Check worker capabilities in database:

   SELECT name, capabilities->'runtimes' FROM worker;
   

For New Deployments

• Migration runs as part of standard setup
• Core pack loads unified runtimes (core.python, core.shell, etc.)
• Services auto-detect capabilities on first start

---

Testing

Compilation

cargo check --workspace
# ✅ All services compile successfully

Unit Tests

cargo test -p attune-common runtime
# ✅ All runtime repository tests pass

Integration Tests

cargo test --test repository_runtime_tests
# ✅ CRUD operations work without runtime_type

---

Files Changed

New Files

• migrations/20260203000001_unify_runtimes.sql (338 lines)
• crates/common/src/runtime_detection.rs (338 lines)

Modified Files

• crates/common/src/lib.rs - Added runtime_detection module export
• crates/common/src/models.rs - Removed RuntimeType enum and field
• crates/common/src/repositories/runtime.rs - Removed runtime_type column references
• crates/common/Cargo.toml - Added regex dependency
• crates/worker/src/registration.rs - Replaced auto-detection with RuntimeDetector
• crates/worker/src/service.rs - Call detect_capabilities before registration
• crates/sensor/src/sensor_worker_registration.rs - Replaced inline detection with RuntimeDetector
• crates/sensor/src/service.rs - Store config and pass to registration
• crates/common/tests/helpers.rs - Updated RuntimeFixture
• crates/common/tests/repository_runtime_tests.rs - Removed runtime_type from tests

Lines Changed

• Added: ~676 lines (migration + runtime_detection module)
• Removed: ~550 lines (duplicate detection code, enum, test updates)
• Net: +126 lines (mostly migration documentation)

---

Breaking Changes

Database Schema

• runtime_type column removed from runtime table
• runtime_type_enum PostgreSQL type dropped
• Runtime refs changed format: core.action.python → core.python

Rust API

• RuntimeType enum removed from attune_common::models::enums
• CreateRuntimeInput no longer has runtime_type field
• RuntimeRepository::find_by_type() method removed
• RuntimeFixture::new() signature changed (removed runtime_type parameter)

Acceptable Because

• Project is pre-production (no deployments, no users)
• Migration automatically handles data transformation
• No external API contracts affected (internal services only)

---

Next Steps

Immediate

• ✅ Code compiles and tests pass
• ✅ Migration tested and verified
• ✅ Documentation updated

Future Enhancements

1. Version Detection: Parse runtime versions from verification output
2. Version Constraints: Allow actions/sensors to require specific versions
3. Distributed Scheduling: Route executions to workers with compatible runtimes
4. Health Monitoring: Periodic re-verification of runtime availability
5. API Endpoints: Expose runtime capabilities via REST API

---

Conclusion

Successfully unified runtime detection across worker and sensor services by:

1. Removing artificial runtime_type distinction
2. Consolidating detection logic into shared module
3. Enabling database-driven runtime configuration
4. Reducing code duplication and maintenance burden

The system is now more maintainable, extensible, and better reflects the underlying reality that runtimes are shared infrastructure used by multiple service types.

Compilation Status: ✅ Clean
Test Status: ✅ Passing
Migration Status: ✅ Ready for deployment