proper sql filtering

work-summary/2026-02-28-expression-engine.md

@@ -0,0 +1,106 @@
+# Expression Engine for Workflow Templates
+
+**Date**: 2026-02-28
+**Scope**: `crates/common/src/workflow/expression/`, `crates/executor/src/workflow/context.rs`
+
+## Summary
+
+Implemented a complete expression evaluation engine for workflow templates, replacing the previous simple dot-path variable lookup and string-based condition evaluation with a full recursive-descent parser and evaluator operating over JSON values.
+
+## Motivation
+
+Workflows previously had no way to perform calculations, comparisons, or transformations within template expressions (`{{ }}`). The `evaluate_condition()` method could only check if a rendered string was truthy — it couldn't evaluate expressions like `result().code == 200 and succeeded()` or `length(items) > 3`. Publish directives and task inputs had no way to compute derived values.
+
+## Architecture
+
+The engine follows a classic three-phase interpreter design, located in `crates/common/src/workflow/expression/`:
+
+1. **Lexer** (`tokenizer.rs`) — converts expression strings into tokens
+2. **Parser** (`parser.rs`) — recursive-descent parser producing an AST
+3. **Evaluator** (`evaluator.rs`) — walks the AST against an `EvalContext` trait to produce `JsonValue` results
+
+The `EvalContext` trait decouples the expression engine from the `WorkflowContext`, allowing it to be used in other contexts (e.g., rule condition evaluation, template resolver) in the future.
+
+## Operator Precedence (lowest to highest)
+
+1. `or`
+2. `and`
+3. `not` (unary)
+4. `==`, `!=`, `<`, `>`, `<=`, `>=`, `in`
+5. `+`, `-`
+6. `*`, `/`, `%`
+7. Unary `-`
+8. Postfix: `.field`, `[index]`, `(args)`
+
+## Supported Operators
+
+### Arithmetic
+- `+` — number addition, string concatenation, array concatenation
+- `-`, `*`, `/`, `%` — standard numeric operations
+- Unary `-` — negation
+- Integer division returns float when not evenly divisible (e.g., `10 / 4` → `2.5`, `10 / 5` → `2`)
+
+### Comparison
+- `==`, `!=` — deep equality (recursive for objects/arrays, cross-type for int/float)
+- `>`, `<`, `>=`, `<=` — ordering for numbers, strings, and lists
+- No implicit type coercion: `"3" == 3` → `false`
+
+### Boolean
+- `and`, `or` — short-circuit evaluation
+- `not` — logical negation
+
+### Membership & Access
+- `.` — object property access
+- `[n]` — array index (supports negative indexing), object bracket access, string character access
+- `in` — membership test (item in array, key in object, substring in string)
+
+## Built-in Functions
+
+### Type Conversion
+`string(v)`, `number(v)`, `int(v)`, `bool(v)`
+
+### Introspection
+`type_of(v)`, `length(v)`, `keys(obj)`, `values(obj)`
+
+### Math
+`abs(n)`, `floor(n)`, `ceil(n)`, `round(n)`, `min(a,b)`, `max(a,b)`, `sum(arr)`
+
+### String
+`lower(s)`, `upper(s)`, `trim(s)`, `split(s, sep)`, `join(arr, sep)`, `replace(s, old, new)`, `starts_with(s, prefix)`, `ends_with(s, suffix)`, `match(pattern, s)` (regex)
+
+### Collection
+`contains(haystack, needle)`, `reversed(v)` (arrays and strings), `sort(arr)`, `unique(arr)`, `flat(arr)`, `zip(a, b)`, `range(n)` / `range(start, end)`, `slice(v, start, end)`, `index_of(haystack, needle)`, `count(haystack, needle)`, `merge(obj_a, obj_b)`, `chunks(arr, size)`
+
+### Workflow-specific (via `EvalContext`)
+`result()`, `succeeded()`, `failed()`, `timed_out()`
+
+## Design Decisions
+
+- **No implicit type coercion**: Arithmetic between strings and numbers is an error. Only int/float cross-comparison is allowed (e.g., `3 == 3.0` → `true`).
+- **Python-like truthiness**: `null`, `false`, `0`, `""`, `[]`, `{}` are falsy; everything else is truthy.
+- **Integer preservation**: Operations on two integers produce integers; operations involving any float produce floats. Integer division that isn't evenly divisible promotes to float.
+- **Array literals**: Supported in expressions: `[1, 2, 3]` (with optional trailing comma).
+- **Negative array indexing**: `arr[-1]` returns the last element.
+
+## Integration with WorkflowContext
+
+- `WorkflowContext` implements the `EvalContext` trait, bridging variable resolution (`parameters`, `item`, `index`, `system`, `task`, `variables`, and direct variable names) and workflow functions (`result()`, `succeeded()`, `failed()`, `timed_out()`)
+- `evaluate_expression()` now delegates entirely to the expression engine
+- `evaluate_condition()` tries the expression engine first (for bare expressions like `x > 5`), with fallback to template rendering for backward compatibility with `{{ }}` wrapper syntax
+- All existing tests pass without modification — backward compatibility is fully preserved
+
+## Test Coverage
+
+- **63 tests** in `crates/common/src/workflow/expression/` covering tokenizer, parser, and evaluator
+- **28 tests** in `crates/executor/src/workflow/context.rs` (17 existing + 11 new) covering integration with WorkflowContext
+- New tests cover: condition comparisons, boolean operators, `in` operator, function calls in conditions, `length()` in conditions, arithmetic in templates, string concatenation, nested function calls, bracket access, and type conversion
+
+## Files Changed
+
+- **Created**: `crates/common/src/workflow/expression/mod.rs` — module entry point and integration tests
+- **Created**: `crates/common/src/workflow/expression/ast.rs` — AST node types
+- **Created**: `crates/common/src/workflow/expression/tokenizer.rs` — lexer
+- **Created**: `crates/common/src/workflow/expression/parser.rs` — recursive-descent parser
+- **Created**: `crates/common/src/workflow/expression/evaluator.rs` — AST evaluator with all built-in functions
+- **Modified**: `crates/common/src/workflow/mod.rs` — added `expression` module
+- **Modified**: `crates/executor/src/workflow/context.rs` — integrated expression engine, replaced old evaluate_expression/evaluate_condition, implemented EvalContext trait, removed unused get_nested_value, added new tests