0
Fork 0
mirror of https://github.com/mastra-ai/mastra.git synced 2026-11-17 08:44:46 +00:00

feat(core, pg, libsql, server): run evented workflows and durable agents on storage v2 (#29370)

## Description

Fifth PR in the workflow storage series. It moves the evented engine
onto storage v2 so evented messages stop carrying the whole run, and
adds an experimental `storageVersion` option to `createDurableAgent` and
`createEventedAgent`. On Postgres, an `EventedAgent` run with 16 tool
calls publishes 11 MB of messages on v2 against 98 MB on v1, writes 3.1
MB against 75 MB, and finishes in 2.6 s against 4.1 s. Both agents still
default to v1.

**API changes on storage v2.** v1 runs behave as before. The only
additions that reach them are `getState()` and `updateState()`;
`setState` keeps the contract it has on `main`.

| API | v1 | v2 | Status |
|---|---|---|---|
| `state` (steps, conditions, loop predicates, sleep and `.map`
functions, workflow tools) | Readable | Throws
`WORKFLOW_STORAGE_V2_SYNC_STATE`; use `await getState()` | New in this
PR on both engines. Default-engine v2 allowed it in `1.83.0-alpha.3` |
| `getState(keys?)` | Returns the state | Loads only the named top-level
keys | New in this PR, both versions |
| `setState(state)` | Validates the value it's given on its own and
replaces the step's pending state, so it takes the full state and a
second call in the same step drops the first one's keys | Same |
Unchanged |
| `updateState(partial)` | Merges the top-level keys it's given,
validates the merged state against `stateSchema`, and records only those
keys. A key set to `undefined` is removed | Same, and changes travel as
a delta, so parallel arms keep each other's keys | New in this PR, both
versions |
| `getStepResult(stepId)` | Works | Throws
`WORKFLOW_STORAGE_V2_SYNC_GET_STEP_RESULT` | Already on `main` for the
default engine. New for evented |
| `getStepResultAsync(stepId)` | Works | The way to read another step's
result | Already exists (in `1.82.0`) |
| `result.steps`, and `steps` in `onFinish` / `onError` | A record | `()
=> Promise<record>`. On the default engine, `steps()` throws
`WORKFLOW_STORAGE_V2_STEPS_STALE` if the run was resumed, restarted or
deleted since | Already on `main` for the default engine. New for
evented, where it doesn't read storage |
| `pruneSnapshot` | Applied | Can't be combined with `storageVersion:
2`. `autoStorageVersion` runs skip it | Already on `main`. Now applies
to evented |
| `autoStorageVersion` | — | Now also picks v2 for evented workflows, so
the rules above apply to them | New in this PR |
| Evented workflow state | Any value | Must convert to JSON. A cycle or
`BigInt` throws | New in this PR |
| `createDurableAgent` / `createEventedAgent` `storageVersion?: 1 \| 2`
| Default | Opt-in | New in this PR |

**Why the messages were so big.** Every evented step message carried the
run's whole state and every earlier step's result. Nested runs also
carried a copy of their parent, with its step graph and step results.
Each message grew with the run, so total traffic grew roughly with the
square of the number of steps. In the 16-tool-call agent run on `main`
(2 KB tool results), the parent copy was 45% of published bytes, step
results 30%, state 15% and the previous result 10%.

**Nested-run messages no longer copy the parent's step graph (v1 and
v2).** The copy was only read when a nested run handed back and couldn't
find the live parent workflow, and it couldn't help then: a serializing
pubsub strips the loop condition from it. Hand-back now uses the live
workflow only. On v1 this alone cuts the agent run's messages by more
than a quarter.

**Evented workflows run on v2.** The evented engine's ~55 direct store
calls now go through `RunStorage`, as the default engine's do, and the
v2 operations are implemented. On v2:
- Step messages carry the previous step's result and a state patch.
Start, resume, end, suspend and fail messages keep their full payload.
Nested-run messages leave out the parent's step results: a loop
hand-back loads them to check the loop condition, and the parent's step
end takes the nested step's input from its stored result.
- Step run and step end dispatches load the run's earlier successful
steps as summaries and fetch only the outputs of the entry they work on
(the step, its parallel branch, or its foreach items). Step code reads
other results with `getStepResultAsync`; sync `getStepResult` throws, as
on the default engine. Before a write, a summary in the result or in the
predecessor it retires is swapped for the stored result, and remote step
execution receives full results.
- Writes turn whole states into patches against the state the dispatch
loaded, so concurrent foreach iterations and parallel arms don't drop
each other's keys. A finishing parallel arm loads its siblings' results
only up to its own write, so when two arms finish together exactly one
of them runs the join.
- `result.steps` is an async function, and `onFinish` and `onError`
receive it too, as on the default engine. The finish message already
carries every result, so it resolves without reading storage.
- Runs skip `pruneSnapshot`, because later steps read earlier results
from storage. `EventedWorkflow.createRun` picks the version the way the
default engine does, including `autoStorageVersion`.
- Every message of a v2 run carries `storageVersion: 2`, including the
`workflow.fail` sent when a message runs out of delivery attempts, so
workers handle it against the v2 tables even when a broker such as Redis
delivers it outside the publisher's context.

`@mastra/server`'s step-execution endpoint takes the run's
`storageVersion`, so remote workers apply the same rules.

**Steps read state with `getState()` and can write only what changed
with `updateState()`.** Steps, conditions, loop predicates, sleep
functions, `.map` functions and workflow tools get `getState(keys?)` and
`updateState(partial)` on both versions. On v2, on both engines, reading
`state` throws and points to `await getState()`. `updateState` merges
the top-level keys it's given, validates the merged state against the
step's `stateSchema`, and records only those keys, so parallel arms that
set different keys don't overwrite each other; a key set to `undefined`
is removed. `setState` is untouched: it still takes the whole state,
validates only the value it's given, and a second call in the same step
replaces the first.

```ts
execute: async ({ getState, updateState }) => {
  const { count } = await getState(['count']); // loads only `count` on v2
  await updateState({ count: count + 1 });
}
```

**Agents choose their storage version.** `createDurableAgent` and
`createEventedAgent` take `storageVersion?: 1 | 2`, default `1`. With
`2`, new runs are stored as v2 on the in-memory, pg and libSQL stores;
on other stores the agent warns once and stores v1. `EventedAgent` no
longer pins its runs to v1. A stored run keeps its version, so a run
suspended on one version resumes in it from an agent set to the other.
Durable agent steps read and write only the state key they need: the
conversation or the step list.

```ts
const durableAgent = createDurableAgent({ agent, storageVersion: 2 });
```

**Stores accept v2 writes from several workers at once.** Evented
workers write the same run from different processes, so they can't send
a header they loaded earlier. The in-memory, pg and libSQL stores now
support:
- `appendWorkflowRunEntries` without a header, keeping the stored header
and merging `requestContext`
- `mergeWorkflowRunStepResult`, which merges a foreach iteration into
the step's latest result under the run's lock
- `loadWorkflowRunSteps({ upToSeq })` and `loadWorkflowRunState({ seq,
keys })`, so a step reads only what it needs, as of the point it ran
- a full state checkpoint after 100 patches or once the patches outgrow
the state, skipped if another state write landed first (`RunJournal`
follows the same rule)

Postgres also guards v2 step writes, as it does on v1: a guarded write
locks the run row, reads the guarded step and its predecessor, runs the
same `assertWorkflowStepGuard`, and appends only the step's entry, so a
redelivered message can't run a step twice or overwrite a finished
result. It reads the whole run only to report a conflict with a
completed step, so the read under the lock doesn't grow with the run: a
128-step sequential run takes 1.15 s, against 1.67 s when the guard read
the whole run. pg and libSQL `init` add the four state columns that
`@mastra/pg@1.36.0-alpha.0` and `@mastra/libsql@1.30.0-alpha.0` created
v2 run tables without.

**Also fixed:**
- An evented foreach whose iterations failed at the same time ended its
run twice. It now fails once, with the first failure's error.
- A durable agent could send the model a background tool's placeholder
instead of its result when the tool finished during `toModelOutput` or
the run was stored on v2.
- Time travel hung on an evented workflow whose `shouldPersistSnapshot`
skips running snapshots, when replaying from a step before a failed
foreach. Time travel now stores the reset step results either way.
- On v1, evented workflows merged the stored state like a step result,
so after parallel branches changed a state with a top-level `output`
array, the stored state kept the old array's items and a restarted run
continued with the wrong state. The state is now replaced whole.
- Workers starting together on a libSQL database that needed a new
column failed `init` with `duplicate column name`, all but the first.
They now start.

## Measurements

In-process pubsub (messages passed by reference), median of 5 runs after
a warm-up. Published is the JSON size of every published message.
Written is, on v1, the run's whole row after each write and, on v2, what
each write sends.

| v1 → v2 | time, in-memory | time, pg | published, pg | written, pg |
|---|---|---|---|---|
| agent, 16 tool calls, 2 KB results | 166 → 152 ms | 4.1 → 2.6 s | 98 →
11 MB | 75 → 3.1 MB |
| agent, 16 tool calls, 20 KB results | 366 → 184 ms | 9.9 → 3.2 s | 700
→ 76 MB | 509 → 15 MB |
| 16 parallel branches, 2 KB | 4 → 2 ms | 69 → 80 ms | 0.39 → 0.29 MB |
0.75 → 0.08 MB |
| 16 sequential steps, 2 KB | 2 → 1 ms | 99 → 92 ms | 1.39 → 0.32 MB |
1.76 → 0.17 MB |

Bytes read from storage per run (in-memory store, 2 KB outputs):

| | v1 | v2 |
|---|---|---|
| 16 sequential steps | 1,044 KB | 163 KB |
| foreach over 16 items | 1,683 KB | 1,625 KB |
| agent, 16 tool calls | 50.0 MB | 24.9 MB |
| 16 parallel branches | 199 KB | 493 KB |

**Before v2 becomes the default** (not in this PR):
- Parallel runs read more than on v1, because each finishing arm loads
its siblings' results to decide the join. On pg, 16 parallel branches
take 80 ms on v2 against 69 ms on v1.
- v2 makes about 7% more store calls (1,642 against 1,541 in the
16-tool-call agent run), mostly the duplicate-delivery check's header
read and the reads that fetch a dispatch's step outputs.
- End, fail and suspend dispatches still load the full run.

**Known issues, also on `main`** (not fixed here):
- The evented engine doesn't check `setState` against `stateSchema`;
`updateState` validates on both engines. The shared-suite case for
`setState` is skipped on evented.
- On Postgres, a loop restarted after its body finished ends early with
that iteration's output (v1 and v2).
- On v1, a loop restarted on another host re-runs the iteration that had
just finished. The result is still right; the test is marked `it.fails`.
- If a background tool's `toModelOutput` takes longer than
`waitTimeoutMs`, the next model call still gets the placeholder. The
result is added on the call after it.

## Testing

- The evented suite runs on v1 and v2, and fails if any v2 run writes to
the v1 tables. v1/v2 parity tests cover nested runs, a loop over a
nested workflow, a foreach over a nested workflow, loop iteration
counts, parallel branches' state changes staying out of step results,
and a nested workflow's `pruneSnapshot` being skipped on v2.
- New v2 tests check that evented steps and conditions throw on `state`
and sync `getStepResult`, locally and through remote step execution;
that the step-execution request carries `storageVersion`; and that
`result.steps` and `onFinish` get the same function, which reads nothing
from storage.
- The durable agent suites run on both storage-version projects. A case
suspends a run on one version and cold-resumes it with an agent set to
the other. The restart harness recovers a `DurableAgent` run and an
`EventedAgent` run in a fresh process on v2.
- The pg evented suite runs guarded writes on both versions. Store
conformance tests cover the new store calls.
- The shared workflow suite checks that `updateState` merges a partial
update, keeps the keys of two calls in one step and of two parallel
branches, and fails a step whose merged state breaks `stateSchema`, on
the default and evented engines on v1 and v2, and on Inngest.
- `workflows/step-state-writers.test.ts` and
`evented/update-state.test.ts` pin the split on v1 and v2: `setState`
validates only the value it's given, doesn't validate the state it
leaves behind, and its second call replaces the first; `updateState`
merges, validates the merged state, keeps keys it wasn't given, and
removes keys set to `undefined`.
- Regression tests cover time travel over a failed foreach with running
snapshots declined, a loop restarted on another host, a restart after
parallel branches replaced the state, a background task read failing
while a durable agent waits for its result, a workflow tool reading a
class-based context's getter, a v2 message running out of deliveries on
real Redis (top-level and nested), and four libSQL or pg stores
upgrading the runs table at once. Without its fix, each fails on the
version or store that had the bug.

Mutation checks (each change made on its own, and each made tests fail):

| Change | Tests failed |
|---|---|
| remove the bounded sibling load | 15 |
| load siblings past the arm's own write | parallel join test |
| stop storing the loop iteration count | loop test |
| keep the parent's step results in nested messages, or stop loading
them on hand-back | 1–2 each |
| remove the v2 `pruneSnapshot` guard | its test |
| ignore `autoStorageVersion` on the evented engine | v2 restart case |
| store a step result without swapping its summary for the stored result
| 6 (pg) |
| retire a predecessor without swapping its summary | 2 (pg) |
| send summaries to the step-execution strategy | remote step-execution
test |
| return v2 evented results as a record | v2 result test |
| report a completed-step conflict without reading the whole run | 3
(pg) |

Typecheck and lint are clean in core, pg and the Redis pubsub. The full
core unit suite (29,420 tests) and pg (4,255) pass at 4136e11803; the
two commits after it pass the core evented tests (1,025), libSQL (1,527)
and the Redis exhaustion tests. On Inngest, the shared workflow suite
passes except 5 tests that fail the same way on `main`. libSQL's `tsc`
reports errors in a cross-process test fixture, also on `main`. Not run:
evented v2 end to end on libSQL, and the shared suite on Temporal. After
the `updateState` change (`f610b639a4d`), core's workflow suites (3,195
non-evented, 1,161 evented) and the durable-agent suites (4,069) pass,
and `tsc --noEmit` is clean.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

ELI5: This change lets workflows store their data in a newer, smaller
format. Runs that use the new format send less data and write less data
to the database, so they finish faster. The old format stays the
default.

## Summary

This pull request adds an opt-in `storageVersion?: 1 | 2` option to
`createDurableAgent` and `createEventedAgent`. Both agents default to
version 1. Version 2 stores a run as a header plus one row per step,
with state tracked as patches and periodic full-state checkpoints.

The evented workflow engine now routes its store operations through
`RunStorage`. It supports version 2 in the in-memory, Postgres, and
libSQL stores. Remote step execution forwards the storage version.

### Behavior changes for workflow authors

- `getState()` is added to step, tool, `.map()`, condition, and sleep
contexts. It resolves state asynchronously. It accepts optional
top-level keys.
- `updateState()` is added. It merges top-level keys, validates the
merged state, and removes keys set to `undefined`.
- On version 2 runs, reading `state` throws. Synchronous
`getStepResult()` also throws. `result.steps` becomes an async function
(`await result.steps()`).
- On the evented engine, `setState` merges keys without schema
validation. Repeated calls accumulate.

### Other changes

- Nested evented runs inherit the parent's storage version through
`resolveRunStorage`.
- Concurrent `foreach` iterations merge their child run IDs. Concurrent
failures produce one `workflow.fail` event.
- Background durable-agent results are folded into the transcript when
they are available. The background-task wait is time-limited.
- The `storeRunsInV2` test helper and `storesRunsInJournal` getter are
removed. Tests set `storageVersion` directly.
- `EventedAgent` no longer overrides `storesRunsInJournal`.
- `WORKFLOW_RUNS_SCHEMA` adds `stateSeq`, `statePatchCount`,
`statePatchBytes`, and `stateBytes`. The libSQL and Postgres stores add
these columns during initialization.
- Stored runs from earlier alpha releases of `@mastra/pg` and
`@mastra/libsql` continue to work after startup adds the columns.

### Scope and risk

- Diff: 181 files changed, 9,141 lines added, 2,554 lines removed.
- Review effort: High. The change touches the evented engine, storage
contracts in core, libSQL, and Postgres, durable-agent steps, and public
types.
- Public API changes include `RunStorage.writeInitial` renamed to
`writeSnapshot`, new `RunStorage` methods, and new or changed storage
method signatures.

### Verification

Results come from the PR description. They are not re-run for this
summary.

- Core and pg typecheck and lint are reported clean.
- Full core and pg suites are reported passing at `4136e11803`.
- After `f610b639a4`, workflow and durable-agent suites are reported
passing, and `tsc --noEmit` is reported clean.
- Inngest shared suite: five failures are reported, which also occur on
`main`.
- libSQL: `tsc` errors in a cross-process test fixture are reported,
which also occur on `main`.
- Not run: evented v2 end-to-end testing on libSQL, and the shared suite
on Temporal.
- `packages/mcp` `server-tasks-background.test.ts` failure is attributed
to an unrelated `afterEach` timeout. Its status is not established here.

### Known gaps

- Evented `setState` does not validate against `stateSchema`.
- Postgres loop restart behavior has a known issue.
- A parent does not resume automatically when an evented child is
resumed directly.
- The parent receives the child's `StepResult` wrapper, not its output.
- Background placeholders can still reach the model when the wait times
out. The result is folded in later.

### Performance (author-reported)

For a Postgres evented-agent run with 16 tool calls and 2 KB results,
version 2 reduces published messages from 98 MB to 11 MB, writes from 75
MB to 3.1 MB, and runtime from 4.1 s to 2.6 s.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: review <review@local>
This commit is contained in:
Tony Kovanen 2026-11-16 18:23:06 +02:00 • committed by GitHub
commit 929c8aba1e
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
185 changed files with 9293 additions and 2570 deletions

Some files were not shown because too many files have changed in this diff Show more