attune-system/attune
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

this is all of the changes now

Browse Source
This commit is contained in:
David Culbreth
2026-02-18 18:43:42 -06:00
parent 77cf18c02f
commit a1b9b8d2b1
22 changed files with 619 additions and 535 deletions
Download Patch File Download Diff File Expand all files Collapse all files

393
docs/workflows/parameter-mapping-status.md
View File

@@ -2,22 +2,20 @@
## Quick Reference
This document provides a quick overview of what exists and what needs to be implemented for rule parameter mapping.
This document tracks the implementation status of rule parameter mapping — the system that resolves `{{ }}` template variables in rule `action_params` before passing them to action executions.
---
## ✅ What Already Exists
## ✅ Completed
### Database Schema
- **Migration:** `migrations/20240103000003_add_rule_action_params.sql`
- **Column:** `attune.rule.action_params` (JSONB, default `{}`)
- **Column:** `rule.action_params` (JSONB, default `{}`)
- **Index:** `idx_rule_action_params_gin` (GIN index for efficient querying)
- **Status:** ✅ Complete
### Data Models
- **File:** `crates/common/src/models.rs`
- **Struct:** `rule::Rule` has `pub action_params: JsonValue` field
- **Status:** ✅ Complete
### API Layer
- **File:** `crates/api/src/dto/rule.rs`
@@ -26,232 +24,115 @@ This document provides a quick overview of what exists and what needs to be impl
- `UpdateRuleRequest.action_params` (optional)
- **Response DTOs:**
- `RuleResponse.action_params`
- **Status:** ✅ Complete
- `RuleSummary.action_params`
### Repository Layer
- **File:** `crates/common/src/repositories/rule.rs`
- **Operations:**
- `CreateRuleInput.action_params` included in INSERT
- `UpdateRuleInput.action_params` handled in UPDATE
- All SELECT queries include `action_params` column
- **Status:** ✅ Complete
- **Operations:** CREATE, UPDATE, and SELECT all handle `action_params`
### API Routes
- **File:** `crates/api/src/routes/rules.rs`
- **Handlers:**
- `create_rule()` accepts `action_params` from request
- `update_rule()` updates `action_params` if provided
- **Status:** ✅ Complete
### Template Resolver Module
- **File:** `crates/common/src/template_resolver.rs`
- **Struct:** `TemplateContext` with `event`, `pack_config`, and `system_vars` fields
- **Function:** `resolve_templates()` — recursively resolves `{{ }}` templates in JSON values
- **Re-exported** from `attune_common::template_resolver` and `attune_common::{TemplateContext, resolve_templates}`
- **Also re-exported** from `attune_sensor::template_resolver` for backward compatibility
- **20 unit tests** covering all template features
### Data Flow (Static Parameters)
### Template Syntax
**Available Sources:**
| Namespace | Example | Description |
|-----------|---------|-------------|
| `event.payload.*` | `{{ event.payload.service }}` | Event payload data |
| `event.id` | `{{ event.id }}` | Event database ID |
| `event.trigger` | `{{ event.trigger }}` | Trigger ref that generated the event |
| `event.created` | `{{ event.created }}` | Event creation timestamp (RFC 3339) |
| `pack.config.*` | `{{ pack.config.api_token }}` | Pack configuration values |
| `system.*` | `{{ system.timestamp }}` | System-provided variables |
### Integration in Executor
- **File:** `crates/executor/src/event_processor.rs`
- **Method:** `resolve_action_params()` builds a `TemplateContext` from the event and rule, then calls `resolve_templates()`
- **Context includes:**
- `event.id`, `event.trigger`, `event.created`, `event.payload.*` from the `Event` model
- `system.timestamp` (current time), `system.rule.id`, `system.rule.ref`
- **Called during:** enforcement creation in `create_enforcement()`
### Data Flow
```
Rule.action_params (static JSON)
↓
Enforcement.config (copied verbatim)
Rule.action_params (templates)
↓ resolve_templates() in EventProcessor
Enforcement.config (resolved values)
↓
Execution.config (passed through)
↓
Worker (receives as action parameters)
```
- **Status:** ✅ Working for static values
### Template Features
- ✅ Static values pass through unchanged
- ✅ Single-template type preservation (numbers, booleans, objects, arrays)
- ✅ String interpolation with multiple templates
- ✅ Nested object access via dot notation (`event.payload.metadata.host`)
- ✅ Array element access by index (`event.payload.tags.0`)
- ✅ Missing values resolve to `null` with warning logged
- ✅ Empty/null action_params handled gracefully
### Documentation
- ✅ `docs/workflows/rule-parameter-mapping.md` — comprehensive user guide
- ✅ `docs/examples/rule-parameter-examples.md` — real-world examples
- ✅ `docs/api/api-rules.md` — API documentation
- ✅ Inline code documentation in `template_resolver.rs`
---
## ❌ What's Missing
### Template Resolution Logic
- **Needed:** Parse and resolve `{{ }}` templates in `action_params`
- **Location:** `crates/sensor/src/` (new module needed)
- **Status:** ❌ Not implemented
### Template Resolver Module
```rust
// NEW FILE: crates/sensor/src/template_resolver.rs
pub struct TemplateContext {
pub trigger_payload: JsonValue,
pub pack_config: JsonValue,
pub system_vars: JsonValue,
}
pub fn resolve_templates(
params: &JsonValue,
context: &TemplateContext
) -> Result<JsonValue> {
// Implementation needed
}
```
- **Status:** ❌ Does not exist
## 🔄 Partially Implemented
### Pack Config Loading
- **Needed:** Load pack configuration from database
- **Current:** Rule matcher doesn't load pack config
- **Required for:** `{{ pack.config.* }}` templates
- **Status:** ❌ Not implemented
### Integration in Rule Matcher
- **File:** `crates/sensor/src/rule_matcher.rs`
- **Method:** `create_enforcement()`
- **Current code (line 309):**
```rust
let config = Some(&rule.action_params);
```
- **Needed code:**
```rust
// Load pack config
let pack_config = self.load_pack_config(&rule.pack_ref).await?;
// Build template context
let context = TemplateContext {
trigger_payload: event.payload.clone().unwrap_or_default(),
pack_config,
system_vars: self.build_system_vars(rule, event),
};
// Resolve templates
let resolved_params = resolve_templates(&rule.action_params, &context)?;
let config = Some(resolved_params);
```
- **Status:** ❌ Not implemented
### Unit Tests
- **File:** `crates/sensor/src/template_resolver.rs` (tests module)
- **Needed tests:**
- Simple string substitution
- Nested object access
- Array element access
- Type preservation
- Missing value handling
- Pack config reference
- System variables
- Multiple templates in one string
- Invalid syntax handling
- **Status:** ❌ Not implemented
### Integration Tests
- **Needed:** End-to-end test of template resolution
- **Scenario:** Create rule with templates → fire event → verify enforcement has resolved params
- **Status:** ❌ Not implemented
- **Current:** Executor passes empty `{}` for `pack.config` context
- **Needed:** Load pack configuration from database before template resolution
- **Impact:** `{{ pack.config.* }}` templates resolve to `null` until implemented
- **TODO comment** in `event_processor.rs` marks the location
---
## 📋 Implementation Checklist
## 📋 Remaining Work
### Phase 1: MVP (2-3 days)
### Phase 1: Complete Core (Short-term)
- [ ] **Create template resolver module**
- [ ] Define `TemplateContext` struct
- [ ] Implement `resolve_templates()` function
- [ ] Regex pattern matching for `{{ }}`
- [ ] JSON path extraction with dot notation
- [ ] Type preservation logic
- [ ] Error handling for missing values
- [ ] Unit tests (9+ test cases)
- [ ] **Pack config loading** — Load pack config from database for `{{ pack.config.* }}` resolution
- [ ] **Integration tests** — End-to-end test: create rule with templates → fire event → verify enforcement has resolved params
- [ ] **Add pack config loading**
- [ ] Add method to load pack config from database
- [ ] Implement in-memory cache with TTL
- [ ] Handle missing pack config gracefully
### Phase 2: Advanced Features (Future)
- [ ] **Integrate with rule matcher**
- [ ] Update `create_enforcement()` method
- [ ] Load pack config before resolution
- [ ] Build template context
- [ ] Call template resolver
- [ ] Handle resolution errors
- [ ] Log warnings for missing values
- [ ] **System variables**
- [ ] Build system context (timestamp, rule ID, event ID)
- [ ] Document available system variables
- [ ] **Testing**
- [ ] Unit tests for template resolver
- [ ] Integration test: end-to-end flow
- [ ] Test with missing values
- [ ] Test with nested objects
- [ ] Test with arrays
- [ ] Test performance (benchmark)
- [ ] **Documentation**
- [x] User documentation (`docs/rule-parameter-mapping.md`) ✅
- [x] API documentation updates (`docs/api-rules.md`) ✅
- [ ] Code documentation (inline comments)
- [ ] Update sensor service docs
### Phase 2: Advanced Features (1-2 days, future)
- [ ] **Default values**
- [ ] Parse `| default: 'value'` syntax
- [ ] Apply defaults when value is null/missing
- [ ] Unit tests
- [ ] **Filters**
- [ ] `upper` - Convert to uppercase
- [ ] `lower` - Convert to lowercase
- [ ] `trim` - Remove whitespace
- [ ] `date: <format>` - Format timestamp
- [ ] `truncate: <length>` - Truncate string
- [ ] `json` - Serialize to JSON string
- [ ] Unit tests for each filter
- [ ] **Performance optimization**
- [ ] Cache compiled regex patterns
- [ ] Skip resolution if no `{{ }}` found
- [ ] Parallel template resolution
- [ ] Benchmark improvements
- [ ] **Default values** — Parse `| default: 'value'` syntax for fallback values
- [ ] **Filters** — `upper`, `lower`, `trim`, `date`, `truncate`, `json`
- [ ] **Conditional templates** — `{% if event.payload.severity == 'critical' %}...{% endif %}`
- [ ] **Performance** — Skip resolution early if no `{{ }}` patterns detected in action_params
---
## 🔍 Key Implementation Details
## 🔍 Template Example
### Current Enforcement Creation (line 306-348)
```rust
async fn create_enforcement(&self, rule: &Rule, event: &Event) -> Result<Id> {
let payload = event.payload.clone().unwrap_or_default();
let config = Some(&rule.action_params); // ← This line needs to change
let enforcement_id = sqlx::query_scalar!(
r#"
INSERT INTO attune.enforcement
(rule, rule_ref, trigger_ref, config, event, status, payload, condition, conditions)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)
RETURNING id
"#,
Some(rule.id),
&rule.r#ref,
&rule.trigger_ref,
config, // ← Resolved params go here
Some(event.id),
EnforcementStatus::Created as EnforcementStatus,
payload,
EnforcementCondition::All as EnforcementCondition,
&rule.conditions
)
.fetch_one(&self.db)
.await?;
// ... rest of method
**Input (Rule `action_params`):**
```json
{
"message": "Error in {{ event.payload.service }}: {{ event.payload.message }}",
"channel": "{{ pack.config.alert_channel }}",
"severity": "{{ event.payload.severity }}",
"event_id": "{{ event.id }}",
"trigger": "{{ event.trigger }}"
}
```
### Template Examples
**Input (Rule):**
**Context (built from Event + Rule):**
```json
{
"action_params": {
"message": "Error in {{ trigger.payload.service }}: {{ trigger.payload.message }}",
"channel": "{{ pack.config.alert_channel }}",
"severity": "{{ trigger.payload.severity }}"
}
}
```
**Context:**
```json
{
"trigger": {
"event": {
"id": 456,
"trigger": "core.error_event",
"created": "2026-02-05T10:00:00Z",
"payload": {
"service": "api-gateway",
"message": "Connection timeout",
@@ -262,114 +143,30 @@ async fn create_enforcement(&self, rule: &Rule, event: &Event) -> Result<Id> {
"config": {
"alert_channel": "#incidents"
}
},
"system": {
"timestamp": "2026-02-05T10:00:01Z",
"rule": { "id": 42, "ref": "alerts.error_notification" }
}
}
```
**Output (Enforcement):**
**Output (Enforcement `config`):**
```json
{
"config": {
"message": "Error in api-gateway: Connection timeout",
"channel": "#incidents",
"severity": "critical"
}
"message": "Error in api-gateway: Connection timeout",
"channel": "#incidents",
"severity": "critical",
"event_id": 456,
"trigger": "core.error_event"
}
```
---
## 📊 Dependencies
## Related Documentation
### Existing (Already in Cargo.toml)
- `serde_json` - JSON manipulation ✅
- `regex` - Pattern matching ✅
- `anyhow` - Error handling ✅
- `sqlx` - Database access ✅
### New Dependencies
- **None required** - Can implement with existing dependencies
---
## 🎯 Success Criteria
- [ ] Static parameters continue to work unchanged
- [ ] Can reference `{{ trigger.payload.* }}` fields
- [ ] Can reference `{{ pack.config.* }}` fields
- [ ] Can reference `{{ system.* }}` variables
- [ ] Type preservation (strings, numbers, booleans, objects, arrays)
- [ ] Nested object access with dot notation works
- [ ] Array element access by index works
- [ ] Missing values handled gracefully (null + warning)
- [ ] Invalid syntax handled gracefully (literal + error)
- [ ] Unit tests pass (90%+ coverage)
- [ ] Integration tests pass
- [ ] Documentation accurate and complete
- [ ] No performance regression (<500µs overhead)
- [ ] Backward compatibility maintained (100%)
---
## 🚀 Getting Started
1. **Read documentation:**
- `docs/rule-parameter-mapping.md` - User guide
- `work-summary/2026-01-17-parameter-templating.md` - Technical spec
2. **Review current code:**
- `crates/sensor/src/rule_matcher.rs:306-348` - Where to integrate
- `crates/common/src/models.rs` - Rule model structure
- `migrations/20240103000003_add_rule_action_params.sql` - Schema
3. **Start implementation:**
- Create `crates/sensor/src/template_resolver.rs`
- Write unit tests first (TDD approach)
- Implement template parsing and resolution
- Integrate with rule_matcher
- Run integration tests
4. **Test thoroughly:**
- Unit tests for all edge cases
- Integration test with real database
- Manual testing with example rules
- Performance benchmarks
---
## 📚 Related Documentation
- [Rule Parameter Mapping Guide](./rule-parameter-mapping.md) - Complete user documentation
- [Rule Management API](./api-rules.md) - API reference with examples
- [Sensor Service Architecture](./sensor-service.md) - Service overview
- [Implementation Plan](../work-summary/2026-01-17-parameter-templating.md) - Technical specification
- [Session Summary](../work-summary/2026-01-17-session-parameter-mapping.md) - Discovery notes
---
## 🏷️ Status Summary
| Component | Status | Notes |
|-----------|--------|-------|
| Database schema | ✅ Complete | `action_params` column exists |
| Data models | ✅ Complete | Rule struct has field |
| API DTOs | ✅ Complete | Request/response support |
| API routes | ✅ Complete | CRUD operations work |
| Repository | ✅ Complete | All queries include field |
| Static parameters | ✅ Working | Flow end-to-end |
| Template resolver | ❌ Missing | Core implementation needed |
| Pack config loading | ❌ Missing | Required for `{{ pack.config }}` |
| Integration | ❌ Missing | Need to wire up resolver |
| Unit tests | ❌ Missing | Tests for resolver needed |
| Integration tests | ❌ Missing | E2E test needed |
| Documentation | ✅ Complete | User and tech docs done |
**Overall Status:** 📝 Documented, ⏳ Implementation Pending
**Priority:** P1 (High)
**Estimated Effort:** 2-3 days (MVP), 1-2 days (advanced features)
**Risk:** Low (backward compatible, well-scoped, clear requirements)
**Value:** High (unlocks production use cases, user expectation)
- [Rule Parameter Mapping Guide](./rule-parameter-mapping.md)
- [Rule Parameter Examples](../examples/rule-parameter-examples.md)
- [Rule Management API](../api/api-rules.md)
- [Executor Service Architecture](../architecture/executor-service.md)