re-uploading work

2026-02-04 17:46:30 -06:00
commit 3b14c65998
1388 changed files with 381262 additions and 0 deletions
--- a/work-summary/phases/2025-01-secret-passing-fix-plan.md
+++ b/work-summary/phases/2025-01-secret-passing-fix-plan.md
@@ -0,0 +1,515 @@
+# Secret Passing Fix - Implementation Plan
+
+**Date:** 2025-01-XX  
+**Priority:** P0 - BLOCKING (Security Critical)  
+**Estimated Time:** 3-5 days  
+**Status:** 🔄 IN PROGRESS
+
+## Problem Statement
+
+**Current Implementation:**
+- Secrets are passed to actions via environment variables (using `prepare_secret_env()`)
+- Environment variables are visible in `/proc/[pid]/environ` and `ps` output
+- This is a **critical security vulnerability** - any user on the system can read secrets
+
+**Example of the vulnerability:**
+```bash
+# Current behavior - INSECURE
+$ ps auxe | grep python
+user  1234  ... SECRET_API_KEY=sk_live_abc123 SECRET_DB_PASSWORD=super_secret ...
+
+$ cat /proc/1234/environ
+SECRET_API_KEY=sk_live_abc123
+SECRET_DB_PASSWORD=super_secret
+```
+
+## Solution Design
+
+**New Approach:**
+- Pass secrets via **stdin as JSON** instead of environment variables
+- Secrets never appear in process table or environment
+- Wrapper scripts read JSON from stdin before executing action code
+
+**Security Benefits:**
+1. ✅ Secrets not visible in `ps` output
+2. ✅ Secrets not visible in `/proc/[pid]/environ`
+3. ✅ Secrets not visible in process monitoring tools
+4. ✅ Secrets only accessible to the running process itself
+
+## Implementation Steps
+
+### Phase 1: Update Data Structures (1-2 hours)
+
+#### 1.1 Update `ExecutionContext` struct
+**File:** `crates/worker/src/runtime/mod.rs`
+
+```rust
+pub struct ExecutionContext {
+    pub execution_id: i64,
+    pub action_ref: String,
+    pub parameters: HashMap<String, serde_json::Value>,
+    pub env: HashMap<String, String>,
+    
+    // NEW: Separate secrets field
+    pub secrets: HashMap<String, String>,  // ← ADD THIS
+    
+    pub timeout: Option<u64>,
+    pub working_dir: Option<PathBuf>,
+    pub entry_point: String,
+    pub code: Option<String>,
+    pub code_path: Option<PathBuf>,
+    pub runtime_name: Option<String>,
+}
+```
+
+**Changes:**
+- Add `secrets: HashMap<String, String>` field
+- Secrets stored separately from `env`
+- No more mixing secrets with environment variables
+
+#### 1.2 Update `ActionExecutor::prepare_execution_context()`
+**File:** `crates/worker/src/executor.rs` (lines 166-308)
+
+**Current code (INSECURE):**
+```rust
+// Fetch and inject secrets
+match self.secret_manager.fetch_secrets_for_action(action).await {
+    Ok(secrets) => {
+        let secret_env = self.secret_manager.prepare_secret_env(&secrets);
+        env.extend(secret_env);  // ← INSECURE: adds to env vars
+    }
+    // ...
+}
+```
+
+**New code (SECURE):**
+```rust
+// Fetch secrets (but don't add to env)
+let secrets = match self.secret_manager.fetch_secrets_for_action(action).await {
+    Ok(secrets) => {
+        debug!("Fetched {} secrets for action", secrets.len());
+        secrets
+    }
+    Err(e) => {
+        warn!("Failed to fetch secrets: {}", e);
+        HashMap::new()
+    }
+};
+
+// Add secrets to context (not env)
+let context = ExecutionContext {
+    execution_id: execution.id,
+    action_ref: execution.action_ref.clone(),
+    parameters,
+    env,
+    secrets,  // ← NEW: separate field
+    timeout,
+    working_dir: None,
+    entry_point,
+    code,
+    code_path: None,
+    runtime_name,
+};
+```
+
+### Phase 2: Update Python Runtime (2-3 hours)
+
+#### 2.1 Update Python wrapper script generation
+**File:** `crates/worker/src/runtime/python.rs` (function `generate_wrapper_script`)
+
+**Current wrapper (simplified):**
+```python
+#!/usr/bin/env python3
+import sys
+import json
+
+# Parameters exported as env vars
+# Secrets exported as env vars (INSECURE)
+
+# Execute action code
+```
+
+**New wrapper (SECURE):**
+```python
+#!/usr/bin/env python3
+import sys
+import json
+import os
+
+# Read secrets from stdin BEFORE executing action
+secrets_json = sys.stdin.readline().strip()
+if secrets_json:
+    secrets = json.loads(secrets_json)
+    # Store in process-local dict, NOT in os.environ
+    _attune_secrets = secrets
+else:
+    _attune_secrets = {}
+
+# Helper function for action code to access secrets
+def get_secret(name):
+    """Get a secret value by name"""
+    return _attune_secrets.get(name)
+
+# Parameters (exported as usual)
+# ... rest of wrapper code ...
+
+# Execute action code
+```
+
+**Key points:**
+- Read JSON from stdin FIRST (before action runs)
+- Store in Python dict `_attune_secrets`, NOT `os.environ`
+- Provide `get_secret()` helper function for action code
+- Stdin is consumed, so action can't read it again (one-time use)
+
+#### 2.2 Update `PythonRuntime::execute_python_code()`
+**File:** `crates/worker/src/runtime/python.rs`
+
+**Add stdin injection:**
+```rust
+async fn execute_python_code(
+    &self,
+    script: String,
+    secrets: &HashMap<String, String>,  // ← NEW parameter
+    env: &std::collections::HashMap<String, String>,
+    timeout_secs: Option<u64>,
+) -> RuntimeResult<ExecutionResult> {
+    // ... setup code ...
+    
+    let mut cmd = Command::new(&self.python_path);
+    cmd.arg(&script_file)
+        .stdin(Stdio::piped())  // ← Enable stdin
+        .stdout(Stdio::piped())
+        .stderr(Stdio::piped());
+    
+    // Add environment variables
+    for (key, value) in env {
+        cmd.env(key, value);
+    }
+    
+    // Spawn process
+    let mut child = cmd.spawn()?;
+    
+    // Write secrets to stdin as JSON
+    if let Some(mut stdin) = child.stdin.take() {
+        let secrets_json = serde_json::to_string(&secrets)?;
+        stdin.write_all(secrets_json.as_bytes()).await?;
+        stdin.write_all(b"\n").await?;
+        drop(stdin);  // Close stdin
+    }
+    
+    // Wait for output
+    let output = child.wait_with_output().await?;
+    
+    // ... process results ...
+}
+```
+
+#### 2.3 Update `PythonRuntime::execute()` signature
+```rust
+async fn execute(&self, context: ExecutionContext) -> RuntimeResult<ExecutionResult> {
+    // Generate wrapper with secret access helper
+    let script = self.generate_wrapper_script(&context)?;
+    
+    // Pass secrets separately
+    self.execute_python_code(
+        script,
+        &context.secrets,  // ← NEW: pass secrets
+        &context.env,
+        context.timeout,
+    ).await
+}
+```
+
+### Phase 3: Update Shell Runtime (2-3 hours)
+
+#### 3.1 Update Shell wrapper script generation
+**File:** `crates/worker/src/runtime/shell.rs` (function `generate_wrapper_script`)
+
+**New wrapper approach:**
+```bash
+#!/bin/bash
+set -e
+
+# Read secrets from stdin into associative array
+read -r ATTUNE_SECRETS_JSON
+declare -A ATTUNE_SECRETS
+
+if [ -n "$ATTUNE_SECRETS_JSON" ]; then
+    # Parse JSON secrets (requires jq or Python)
+    # Option A: Use Python to parse JSON
+    eval "$(echo "$ATTUNE_SECRETS_JSON" | python3 -c "
+import sys, json
+secrets = json.load(sys.stdin)
+for key, value in secrets.items():
+    safe_value = value.replace(\"'\", \"'\\\\''\")
+    print(f\"ATTUNE_SECRETS['{key}']='{safe_value}'\")
+")"
+fi
+
+# Helper function to get secrets
+get_secret() {
+    echo "${ATTUNE_SECRETS[$1]}"
+}
+
+# Export parameters as environment variables
+# ... (existing parameter export code) ...
+
+# Execute action code
+# ... (existing action code) ...
+```
+
+**Alternative (simpler but requires temp file):**
+```bash
+#!/bin/bash
+set -e
+
+# Read secrets from stdin into temp file
+SECRETS_FILE=$(mktemp)
+trap "rm -f $SECRETS_FILE" EXIT
+cat > "$SECRETS_FILE"
+
+# Helper function to get secrets (reads from temp file)
+get_secret() {
+    local name="$1"
+    python3 -c "import sys, json; secrets=json.load(open('$SECRETS_FILE')); print(secrets.get('$name', ''))"
+}
+
+# Export parameters
+# ... rest of wrapper ...
+```
+
+#### 3.2 Update `ShellRuntime::execute_shell_code()`
+Similar pattern to Python runtime - pipe secrets to stdin as JSON.
+
+### Phase 4: Remove Deprecated Method (30 minutes)
+
+#### 4.1 Deprecate `SecretManager::prepare_secret_env()`
+**File:** `crates/worker/src/secrets.rs`
+
+```rust
+/// Prepare secrets as environment variables
+///
+/// **DEPRECATED**: This method is insecure as it exposes secrets in the process environment.
+/// Secrets should be passed via stdin instead.
+#[deprecated(
+    since = "0.2.0",
+    note = "Use direct secret passing via stdin instead of environment variables"
+)]
+pub fn prepare_secret_env(&self, secrets: &HashMap<String, String>) -> HashMap<String, String> {
+    // ... existing implementation ...
+}
+```
+
+**Action:** Mark as deprecated, plan to remove in future version.
+
+### Phase 5: Security Testing (1-2 hours)
+
+#### 5.1 Create security test suite
+**File:** `crates/worker/tests/security_tests.rs` (NEW FILE)
+
+```rust
+/// Test that secrets are NOT visible in process environment
+#[tokio::test]
+async fn test_secrets_not_in_process_environ() {
+    // Create action with secret
+    let context = ExecutionContext {
+        secrets: {
+            let mut s = HashMap::new();
+            s.insert("api_key".to_string(), "super_secret_key_123".to_string());
+            s
+        },
+        // ... other fields ...
+    };
+    
+    // Execute action that spawns child process
+    // Child process should write its /proc/self/environ to stdout
+    
+    let result = runtime.execute(context).await.unwrap();
+    
+    // Verify secret is NOT in environ output
+    assert!(!result.stdout.contains("super_secret_key_123"));
+    assert!(!result.stdout.contains("SECRET_API_KEY"));
+}
+
+/// Test that secrets ARE accessible to action code
+#[tokio::test]
+async fn test_secrets_accessible_in_action() {
+    let context = ExecutionContext {
+        secrets: {
+            let mut s = HashMap::new();
+            s.insert("api_key".to_string(), "test_key_456".to_string());
+            s
+        },
+        code: Some("print(get_secret('api_key'))".to_string()),
+        // ... other fields ...
+    };
+    
+    let result = runtime.execute(context).await.unwrap();
+    
+    // Verify secret IS accessible via get_secret()
+    assert!(result.stdout.contains("test_key_456"));
+}
+
+/// Test that ps output doesn't show secrets
+#[tokio::test]
+async fn test_secrets_not_in_ps_output() {
+    // This test spawns a long-running action
+    // While it's running, capture ps output
+    // Verify secrets don't appear
+    
+    // Implementation requires:
+    // 1. Spawn action with sleep
+    // 2. Run `ps auxe` while action is running
+    // 3. Verify secret not in output
+    // 4. Wait for action to complete
+}
+```
+
+#### 5.2 Test action code patterns
+**File:** `crates/worker/tests/secret_access_tests.rs` (NEW FILE)
+
+Test that action code can access secrets via helper functions:
+
+**Python:**
+```python
+api_key = get_secret('api_key')
+print(f"Using key: {api_key}")
+```
+
+**Shell:**
+```bash
+api_key=$(get_secret 'api_key')
+echo "Using key: $api_key"
+```
+
+### Phase 6: Documentation (1-2 hours)
+
+#### 6.1 Update action development guide
+**File:** `docs/action-development.md` (NEW or UPDATE)
+
+```markdown
+## Accessing Secrets in Actions
+
+### Python Actions
+
+Secrets are available via the `get_secret()` function:
+
+```python
+def run(params):
+    api_key = get_secret('api_key')
+    db_password = get_secret('db_password')
+    
+    # Use secrets...
+    return {"status": "success"}
+```
+
+**Important:** Do NOT access secrets via `os.environ` - they are not stored there
+for security reasons.
+
+### Shell Actions
+
+Secrets are available via the `get_secret` function:
+
+```bash
+#!/bin/bash
+
+api_key=$(get_secret 'api_key')
+db_password=$(get_secret 'db_password')
+
+# Use secrets...
+echo "Connected successfully"
+```
+
+**Security Note:** Secrets are passed securely via stdin and never appear in
+process listings or environment variables.
+```
+
+#### 6.2 Update security documentation
+**File:** `docs/security.md` (UPDATE)
+
+Document the security improvements and rationale.
+
+### Phase 7: Migration Guide (1 hour)
+
+#### 7.1 Create migration guide for existing packs
+**File:** `docs/migrations/secret-access-migration.md` (NEW)
+
+```markdown
+# Migrating to Secure Secret Access
+
+## What Changed
+
+As of version 0.2.0, secrets are no longer passed via environment variables.
+This improves security by preventing secrets from appearing in process listings.
+
+## Migration Steps
+
+### Before (Insecure)
+```python
+import os
+api_key = os.environ.get('SECRET_API_KEY')
+```
+
+### After (Secure)
+```python
+api_key = get_secret('api_key')
+```
+
+### Backward Compatibility
+
+For a transitional period, you can support both methods:
+
+```python
+api_key = get_secret('api_key') or os.environ.get('SECRET_API_KEY')
+```
+
+However, we recommend migrating fully to `get_secret()`.
+```
+
+## Testing Checklist
+
+- [ ] Unit tests pass for ExecutionContext with secrets field
+- [ ] Python runtime injects secrets via stdin
+- [ ] Shell runtime injects secrets via stdin
+- [ ] Actions can access secrets via `get_secret()`
+- [ ] Secrets NOT in `/proc/[pid]/environ`
+- [ ] Secrets NOT in `ps auxe` output
+- [ ] Existing actions continue to work (backward compat)
+- [ ] Documentation updated
+- [ ] Migration guide created
+
+## Success Criteria
+
+1. ✅ All secrets passed via stdin (not environment)
+2. ✅ Security tests confirm secrets not visible externally
+3. ✅ Action code can still access secrets easily
+4. ✅ No breaking changes for users (helper functions added)
+5. ✅ Documentation complete
+6. ✅ All tests passing
+
+## Timeline
+
+- **Day 1:** Phase 1-2 (Data structures + Python runtime)
+- **Day 2:** Phase 3-4 (Shell runtime + deprecation)
+- **Day 3:** Phase 5-7 (Testing + documentation)
+- **Day 4-5:** Buffer for edge cases and refinement
+
+## Risks & Mitigation
+
+**Risk:** Breaking existing actions that access `os.environ['SECRET_*']`
+**Mitigation:** Provide backward compatibility period and clear migration guide
+
+**Risk:** Stdin approach may not work for all action types
+**Mitigation:** Test with various action patterns, provide alternative temp file approach if needed
+
+**Risk:** JSON parsing in shell may be fragile
+**Mitigation:** Use Python for JSON parsing in shell wrapper (Python always available)
+
+## Next Steps After Completion
+
+1. Announce change to users
+2. Provide migration examples
+3. Set deprecation timeline for old method
+4. Monitor for issues
+5. Remove deprecated `prepare_secret_env()` in v0.3.0