91 lines
5.0 KiB
Markdown
91 lines
5.0 KiB
Markdown
# `with_items` Concurrency Limiting Implementation
|
|
|
|
**Date**: 2026-02-27
|
|
**Scope**: `crates/executor/src/scheduler.rs`
|
|
|
|
## Problem
|
|
|
|
Workflow tasks with `with_items` and a `concurrency` limit dispatched all items simultaneously, ignoring the concurrency setting entirely. For example, a task with `concurrency: 3` and 20 items would dispatch all 20 at once instead of running at most 3 in parallel.
|
|
|
|
## Root Cause
|
|
|
|
The `dispatch_with_items_task` method iterated over all items in a single loop, creating a child execution and publishing it to the MQ for every item unconditionally. The `task_node.concurrency` value was logged but never used to gate dispatching.
|
|
|
|
## Solution
|
|
|
|
### Approach: DB-Based Sliding Window
|
|
|
|
All child execution records are created in the database up front (with fully-rendered inputs), but only the first `concurrency` items are published to the message queue. The remaining children stay at `Requested` status in the DB. As each item completes, `advance_workflow` queries for `Requested`-status siblings and publishes enough to refill the concurrency window.
|
|
|
|
This avoids the need for any auxiliary state in workflow variables — the database itself is the single source of truth for which items are pending vs in-flight.
|
|
|
|
### Initial Attempt: Workflow Variables (Abandoned)
|
|
|
|
The first implementation stored pending items as JSON metadata in `workflow_execution.variables` under `__pending_items__{task_name}`. This approach suffered from race conditions: when multiple items completed simultaneously, concurrent `advance_workflow` calls would read stale pending lists, pop the same item, and lose others. The result was that only the initial batch ever executed.
|
|
|
|
### Key Changes
|
|
|
|
#### 1. `dispatch_with_items_task` — Two-Phase Dispatch
|
|
|
|
- **Phase 1**: Creates ALL child execution records in the database. Each row has its input already rendered through the `WorkflowContext`, so no re-rendering is needed later.
|
|
- **Phase 2**: Publishes only the first `min(total, concurrency)` to the MQ via `publish_execution_requested`. The rest stay at `Requested` status.
|
|
|
|
#### 2. `publish_execution_requested` — New Helper
|
|
|
|
Publishes an `ExecutionRequested` MQ message for an existing execution row. Used both during initial dispatch (Phase 2) and when filling concurrency slots on completion.
|
|
|
|
#### 3. `publish_pending_with_items_children` — Fill Concurrency Slots
|
|
|
|
Replaces the old `dispatch_next_pending_with_items`. Queries the database for siblings at `Requested` status (ordered by `task_index`), limited to the number of free slots, and publishes them. No workflow variables involved — the DB query `status = 'requested'` is the authoritative source of undispatched items.
|
|
|
|
#### 4. `advance_workflow` — Concurrency-Aware Completion
|
|
|
|
The with_items completion branch now:
|
|
1. Counts **in-flight** siblings (`scheduling`, `scheduled`, `running` — NOT `requested`)
|
|
2. Reads the `concurrency` limit from the task graph
|
|
3. Calculates `free_slots = concurrency - in_flight`
|
|
4. Calls `publish_pending_with_items_children(free_slots)` to fill the window
|
|
5. Checks **all** non-terminal siblings (including `requested`) to decide whether to advance
|
|
|
|
## Concurrency Flow Example
|
|
|
|
For a task with 5 items and `concurrency: 3`:
|
|
|
|
```
|
|
Initial: Create items 0-4 in DB; publish items 0, 1, 2 to MQ
|
|
Items 3, 4 stay at Requested status in DB
|
|
|
|
Item 0 ✓: in_flight=2 (items 1,2), free_slots=1 → publish item 3
|
|
siblings_remaining=3 (items 1,2,3,4 minus terminal) → return early
|
|
|
|
Item 1 ✓: in_flight=2 (items 2,3), free_slots=1 → publish item 4
|
|
siblings_remaining=3 → return early
|
|
|
|
Item 2 ✓: in_flight=2 (items 3,4), free_slots=1 → no Requested items left
|
|
siblings_remaining=2 → return early
|
|
|
|
Item 3 ✓: in_flight=1 (item 4), free_slots=2 → no Requested items left
|
|
siblings_remaining=1 → return early
|
|
|
|
Item 4 ✓: in_flight=0, free_slots=3 → no Requested items left
|
|
siblings_remaining=0 → advance workflow to successor tasks
|
|
```
|
|
|
|
## Race Condition Handling
|
|
|
|
When multiple items complete simultaneously, concurrent `advance_workflow` calls may both query `status = 'requested'` and find the same pending items. The worst case is a brief over-dispatch (the same execution published to MQ twice). The scheduler handles this gracefully — the second message finds the execution already at `Scheduled`/`Running` status. This is a benign, self-correcting race that never loses items.
|
|
|
|
## Files Changed
|
|
|
|
- **`crates/executor/src/scheduler.rs`**:
|
|
- Rewrote `dispatch_with_items_task` with two-phase create-then-publish approach
|
|
- Added `publish_execution_requested` helper for publishing existing execution rows
|
|
- Added `publish_pending_with_items_children` for DB-query-based slot filling
|
|
- Rewrote `advance_workflow` with_items branch with in-flight counting and slot calculation
|
|
- Updated unit tests for the new approach
|
|
|
|
## Testing
|
|
|
|
- All 104 executor tests pass (102 + 2 ignored)
|
|
- 2 new unit tests for dispatch count and free slots calculations
|
|
- Clean workspace build with no new warnings |