## Description
On the evented engine, each iteration of a `dountil`/`dowhile` loop
whose body is a step now starts with `retryCount` 0 and gets the step's
full `retries`, as on the default engine.
### Why
The evented engine carried `retryCount` from one iteration into the
next, so iteration N started at N-1 plus every retry earlier iterations
had used. A body with `retries` ran out of budget on a later iteration:
with `retries: 1`, a single failure in iteration 3 failed the run
instead of being retried. Step code reading `retryCount` also saw that
inflated number. The default engine has counted retries per iteration
since #23593.
```ts
const body = createStep({ id: 'body', retries: 1, execute: async ({ retryCount }) => { /* ... */ } });
workflow.dountil(body, async ({ iterationCount }) => iterationCount >= 4).commit();
// retryCount seen by the body when iteration 3 fails once
// before: 0, 1, 2 and the run fails
// after: 0, 0, 0, 1, 0 and the run succeeds
```
### Behavior change
- Step code that reads `retryCount` inside an evented loop body now gets
the attempt within the current iteration. The iteration number is
available to the loop condition as `iterationCount`.
- The loop condition's `retryCount` is now the body's attempt in the
current iteration on every path. One path used to pass attempt + 1. The
default engine passes -1 to the condition; that difference is left as
is.
### Not affected
`foreach` loops, and loops whose body is a nested workflow (those
already started each iteration at 0).
### Tests
New case in the shared workflow suite
(`workflows/_test-utils/src/domains/run-count.ts`): a `dountil` body
with `retries: 1` that throws once in iteration 3. It runs on the
default and evented engines and is skipped on Inngest with the other
`retryCount` cases. Without the fix it fails on the evented engine with
`expected [ 0, 1, 2 ] to deeply equal [ 0, 0, 0, 1, 0 ]`.
## Related issue(s)
Linear PLTFRM-3202. No GitHub issue.
## Type of change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [ ] Test update
## Checklist
- [x] I have linked the related issue(s) in the description above
- [ ] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
ELI5: When a loop step fails and retries, the retry count from earlier
loop passes was carried into later passes. This could use up the retry
limit early. This change resets the count so each loop pass gets its
full retry limit.
## Summary
This pull request changes how the evented workflow engine tracks
`retryCount` for `dountil` and `dowhile` loops whose body is a step.
## Changes
-
`packages/core/src/workflows/evented/workflow-event-processor/loop.ts`:
When a loop iteration runs again, `processWorkflowLoop` passes
`retryCount: 0` instead of the current retry count. Each iteration
starts with the body's full retry budget.
-
`packages/core/src/workflows/evented/workflow-event-processor/index.ts`:
In `runLeafStep`, the loop-condition path passes `retryCount` to
`#processLoop` without adding 1.
- `.changeset/bright-terms-crash.md`: Adds a patch changeset for
`@mastra/core`. It documents that `dountil` and `dowhile` body retries
start at 0 on each iteration. It also states that `retryCount` is scoped
to the current iteration.
- `workflows/_test-utils/src/domains/run-count.ts`: Adds
`createRunCountWorkflows` and `createRunCountTests`. The new workflow is
a loop whose body allows one retry and throws once on the third
iteration. The test asserts that the run succeeds with recorded retry
counts `[0, 0, 0, 1, 0]` and final output `4`.
Diff size: 4 files, 63 lines added, 2 lines removed.
## Not changed
- The default engine's behavior is unchanged.
- `foreach` loops and loops with nested workflow bodies are not affected
by the diff shown.
## Verification
- The shared test case runs on the default and evented engines. It is
skipped on Inngest with the other `retryCount` cases.
- The author reports that, without the fix, the evented case fails with
`expected [ 0, 1, 2 ] to deeply equal [ 0, 0, 0, 1, 0 ]`. The author
reports this result. This review did not run the tests.
- CI results: not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
The core unit lane spends tens of seconds in `vi.clearAllMocks()` /
`vi.resetAllMocks()`, and almost all of it is caused by the workflow
test factory.
- **Mock registry built mocks on every reset.**
`workflows/_test-utils/src/factory.ts` resets every registry entry
before each test. Each domain shares one `MockRegistry`, and `reset()`
rebuilt every factory in it, so one run of `evented-workflow.test.ts`
created **722,585** `vi.fn()`s. `@vitest/spy` keeps every mock in a
module-level set for the life of the worker, so later `clearAllMocks`
calls in the same worker took about 440 ms each. `register()` and
`reset()` now only drop the cached mock, and `get()` builds it on first
use. Every read already goes through `get()`, so tests see the same
mocks. The same file now creates 579 mocks.
- **`evented-workflow.test.ts` Engine Specific block** used
`vi.resetAllMocks()` before each test. Every spy in that block is
created inside a test body and the file has no `vi.mock`, so it now uses
`vi.restoreAllMocks()`.
- **`durable-agent-background-tasks.test.ts`**: six fixed 500–1500 ms
sleeps are now `vi.waitFor` on the state the test asserts next. The
readOnly test keeps its sleep because it proves something does *not*
happen. No assertions changed.
## Numbers
Full `@mastra/core` unit lane, 5-CPU container, shared module cache
(stacked on #29134):
| | Base | This PR |
|---|---|---|
| Wall | 378.4 s | 339.7 s |
| Test time (worker-s) | 1,366.6 | 1,176.8 |
| `clearAllMocks` total | 37.6 s | 2.1 s |
| Result | 25,247 passed, 0 failed | 25,247 passed, 0 failed |
`workflow.test.ts` alone: 79 s → 24 s. The Inngest engine suite, which
uses the same factory, gives an identical result on the base and on this
branch: 222 passed, and the same 5 failures on both.
## Test plan
- [x] Full core unit lane: 25,247 passed, 0 failed, on both the base and
this branch.
- [x] `workflows/_test-utils`: 581/581.
- [x] Inngest engine suite: same result on base and branch.
- [x] oxlint and prettier clean.
Stacked on #29134 (core unit test wait knobs). I'll retarget it to main
once that merges.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## ELI5
This change makes tests faster and less fragile. Tests now wait for the
result they need instead of sleeping for a fixed time. Workflow test
mocks are also created only when a test first uses them.
## Changes
- `workflows/_test-utils/src/mock-registry.ts`: `register()` removes any
existing mock for a key. `reset()` clears all mocks. `get()` creates a
mock on first use, so registered factories no longer run during reset.
The Vitest imports are split into value and type imports.
- `packages/core/src/workflows/evented/evented-workflow.test.ts`: The
Engine Specific setup calls `vi.restoreAllMocks()` instead of
`vi.resetAllMocks()`.
-
`packages/core/src/agent/durable/__tests__/durable-agent-background-tasks.test.ts`:
Fixed sleeps are replaced with `vi.waitFor` checks for dispatch, tool
results, persisted results, and task status. The suspension test also
waits for `completed` after resuming. The waits use a 10-second timeout
and a 10 ms interval. The readOnly test keeps its sleep.
- `mastracode/factory/src/tools/messaging-tools.test.ts`: The
queued-message ordering test and the missing-thread test use
`vi.waitFor` at 20 ms intervals, with a timeout of twice
`QUEUE_WAIT_MS`. The missing-thread test waits for the dropped-delivery
audit record before it asserts that no message was sent.
- `mastracode/tui/src/tui/handlers/tool.test.ts`: `flushParser` advances
Vitest fake timers past the render-coalescing interval. With real
timers, it keeps the existing timeout wait. The compact-shell streaming
test uses fake timers and restores real timers in cleanup.
- `packages/core/src/agent/durable/__tests__/restart-agreement.test.ts`:
The T26 `echoContext` post-park delay drops from 200 ms to 20 ms. The
T55 `loggedStep` delay drops from 100 ms to 10 ms.
- `workflows/_test-utils/src/domains/abort.ts`: The immediate-abort
setup wait drops from 1 s to 50 ms. The in-step and nested-workflow
waits drop from 5 s to 50 ms. Comments now state that the in-step test
does not trigger an abort.
Diff size: 7 files, 85 lines added, 38 lines removed. No exported
declarations changed.
## Validation
Results below are reported by the PR author. This review did not run
them.
- Core unit lane: 25,247 passed and 0 failed on both the base and this
branch. Wall time fell from 378.4 s to 339.7 s. `clearAllMocks` time
fell from 37.6 s to 2.1 s.
- `workflow.test.ts`: runtime fell from 79 s to 24 s.
- `evented-workflow.test.ts`: mock creation fell from 722,585 to 579.
- Inngest engine suite: 222 passed and the same 5 failures on both the
base and this branch.
- `workflows/_test-utils`: 581 of 581 passed.
- oxlint and prettier: reported clean.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com>
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: yoko-reviewer[bot] <324360165+yoko-reviewer[bot]@users.noreply.github.com>
## Description
Evented workflows rejected valid input. `start()` and `startAsync()`
replaced an omitted or `null` `inputData` with `{}` before validating it
against the workflow `inputSchema`, so a top-level schema default never
applied and a nullable schema rejected an explicit `null`. `stream()`
calls `start()`, so it had the same bug. Both start paths in
`packages/core/src/workflows/evented/workflow.ts` now pass `inputData`
through unchanged, which is what the default engine already does.
Follow-up fixes on the same input path (from review):
- The evented step executor stored a step's `payload` as `input ?? {}`.
Resume feeds that payload back into the step, so a first step started
with `null` or no input got `{}` on resume and a nullable or optional
schema rejected it. It now stores the input unchanged, as the default
engine does. Nested resume uses the stored payload whenever the step
record exists, instead of `payload ?? parentInput`.
- Time travel to the first step built that step's payload as `stored
payload ?? {}`, so a run started with `null` or no input replayed `{}`
on the evented engine and a nullable or optional schema rejected it. The
evented engine also ignored new `inputData` passed to `timeTravel()` for
the first step. `createTimeTravelExecutionParams` (shared by all
engines) now gives a first-entry target the same input the default
engine gives it: new `inputData` if passed (an explicit `null`
included), else the stored workflow input, kept as `null` or
`undefined`. A nested workflow in the first entry replays the stored
input too, and when the evented engine finds no child snapshot it seeds
the child's input from what the parent passes in. Other steps keep their
existing payload fallback.
- With `null` input, `context.input` is `null`. Two scans read `.status`
on it: the evented suspended-step list, and
`hydrateSerializedStepErrors` (shared with Inngest). A suspend crashed
`start()`, and a step failure left it waiting forever. Both now skip the
`null` entry.
```diff
// evented start() and startAsync()
-const inputDataToUse = await this._validateInput(inputData ?? ({} as TInput));
+const inputDataToUse = await this._validateInput(inputData);
```
```ts
const input = z.object({ count: z.number() }).default({ count: 5 });
// evented engine, before: throws "Invalid input data: count: expected number, received undefined"
// evented engine, after: success, the first step and getInitData() both see { count: 5 }
await run.start({});
```
### Evidence
New cases, added to the shared workflow suite
(`workflows/_test-utils/src/domains/schema-validation.ts`), run on the
default, evented and Inngest engines. They cover an omitted input with
an object default, an explicit value winning over the default, a scalar
default, an explicit `null` with a nullable schema, `getInitData()`
returning the defaulted input, omitted input with `validateInputs:
false` reaching the step as `undefined`, and missing required input
still being rejected without the step running. A focused evented file
(`start-input-validation.test.ts`) covers `startAsync()`, which the
shared harness doesn't expose, and asserts on the persisted run
snapshot. It also covers suspend and resume with `null` and omitted
input, a step failure with `null` input, and a nested step resumed with
its stored `null` input.
- **Review follow-ups, before** (commit 80b7fd6851, same tests): 4
failed. `Cannot read properties of null (reading 'status')`, `expected
{} to be undefined`, a 15s timeout on the failing step, and the nested
step resumed with `'seed'` instead of `null`. **After:** 9 passed in
that file.
- **Before** (origin/main evented `workflow.ts`, same tests): `vitest
run start-input-validation.test.ts evented-workflow.test.ts
workflow.test.ts -t 29424` gave **10 failed | 9 passed**. All the
failures were on the evented engine, for example `Invalid input data: -
: Invalid input: expected number, received object`.
**After:** **19 passed**.
- **Time travel, before** (head 765d1f564f): the shared time-travel
cases for `null`, omitted, new `inputData` and nested `{}`/`null` replay
failed on the evented engine with `Step input validation failed`, or
`expected 'failed' to be 'success'`. **After:** all 9 pass on the
default engine, the default engine with storage v2, and the evented
engine.
- The Inngest harness had no `timeTravel` helper, so every shared
time-travel case skipped there. With the helper added, all 27 shared
time-travel cases pass on Inngest. Three of Inngest's older time-travel
tests (suspend and resume a time-travelled run, time travel a suspended
run, conditional chains) fail on this branch both with and without the
time-travel change; their diffs are about `resumePayload: undefined`
keys and a conditional sibling marked `skipped`, which this PR doesn't
touch.
- Inngest, whole Schema Validation domain: 13 passed, 1 expected failure
(see below).
- Regression check: every test in `src/workflows/evented/`, every core
test file that calls `timeTravel`, plus `workflow.test.ts`,
`storage-v2-suite.test.ts` and `utils.test.ts` passes (57 files, 1691
passed, 14 skipped). `pnpm --filter ./packages/core check` is clean.
`workflows/_test-utils` typecheck reports no errors in the touched
files; it has 14 existing errors elsewhere, the same on main. oxlint and
oxfmt are clean on the changed files.
- Built with a hill climb: the failing tests were locked first, and the
score went from 10 to 0 in one source step.
Inngest already passed input through unchanged, but the new shared cases
found two Inngest bugs:
- `getInitData()` returns `undefined` instead of the defaulted input
when `inputData` is omitted. Cause not yet investigated. Pinned with
`it.fails` (`schemaInitDataDefaultFails`).
- An explicit `null` input crashed the status poll in
`hydrateSerializedStepErrors`. Fixed by the shared `null` check above,
so that case now passes without a pin.
### Merge Danger
**Door:** two-way. Reverting the commit restores the old fallback.
**Blast radius:** callers of evented `start()`, `startAsync()` and
`stream()` that omit `inputData`. A workflow whose bare object schema
(no default) passed only because of the evented-only `{}` fallback now
fails validation, as it already does on the default engine. With no
`inputSchema` or with `validateInputs: false`, the first step now
receives `undefined` instead of `{}`. Evented step results now record a
`null` or `undefined` `payload` as is, instead of `{}`, as the default
engine does; code that reads `steps[id].payload` on the evented engine
can see those values. The changeset tells those callers to pass
`inputData: {}` or add a schema default. Scheduled runs are unchanged,
because the scheduler already sends `inputData ?? {}`. Nested workflows
don't go through `start()`. Time travel to the first step now replays a
stored `null` or `undefined` instead of `{}`, and on the evented engine
uses new `inputData` passed to `timeTravel()`, as the default engine
already did.
### Follow-ups (not in this PR)
- The Inngest `getInitData()` bug above needs its own issue.
- On every engine, time travel to a later step whose previous step
returned `null` or `undefined` still replays `{}`, and a falsy
`inputData` passed to `timeTravel()` for a step after the first one is
ignored (the `inputData &&` check in `createTimeTravelExecutionParams`).
Both predate this PR. For the first step, an explicit `null` `inputData`
now replaces the stored input.
## Related issue(s)
Closes#29424
## Type of change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [ ] Test update
## Checklist
- [x] I have linked the related issue(s) in the description above
- [ ] I have made corresponding changes to the documentation (if
applicable). Not applicable: the docs already say validation applies
schema defaults and `inputData` is optional.
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR
## Council review
Rounds: 4
| Seat | Round 1 | Round 2 | Round 3 | Round 4 |
| --- | --- | --- | --- | --- |
| Claude Opus 5.5 | not run | 9/10 | 9/10 | 9/10 |
| GPT-6.1 Sol | 9.5/10 | 9.5/10 | 9/10 | 9/10 |
| Grok 4.7 | empty seat | empty seat | empty seat | empty seat |
Every seat scored at least 9 in each category (correctness, tests,
scope, code quality) in rounds 2 to 4. Findings applied: clear the
`startAsync` test's timeout timer, describe the no-schema and
`validateInputs: false` change in the changeset, and add a shared case
for that change. Findings not applied:
- Add an Inngest branch to the required-input rejection case. That case
follows the same `schemaValidationThrows` skip as the existing
invalid-input cases.
- Persist an explicit `null` in the evented initial snapshot. The final
persisted run already stores `context.input = null`, and that assertion
passed without a code change.
Time-travel follow-up review (GPT-6.1 Sol, Correctness / Tests / Scope /
Code quality): round 1 8/8/10/9 (added the Inngest `timeTravel` helper),
round 2 7/8/10/9 (fixed a nested time-travel regression it found), round
3 9/9/10/9 with no findings.
Created with Claude Opus 5.5 in T3 Code.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## ELI5
Workflows now pass the input you provide—or leave it missing—through
validation instead of changing it to an empty object first. This lets
schemas apply defaults or accept `null`, and helps workflows keep the
same input when they resume or replay.
## Changes
- Evented `start()` and `startAsync()` pass input directly to schema
validation. `stream()` uses `start()`.
- Step payloads preserve `null` and `undefined`. Resume paths use stored
payloads, including for nested steps.
- Time travel to a workflow’s first step preserves its input. Supplied
`inputData` can replace the stored input. For a nested workflow without
a saved snapshot, replay uses the parent step’s successful output as the
child input.
- Null-safe status checks handle missing step results and context
entries. A step failure after a `null` start returns a failed result
instead of leaving `start()` waiting.
- Added evented and shared tests for defaults, nullable input, invalid
input, resume, time travel, `getInitData()`, and validation-disabled
behavior.
- Added a patch changeset that documents the behavior and compatibility
impact.
When no input schema is configured or validation is disabled, omitted
input reaches the first step as `undefined`. Callers that need `{}` can
pass it explicitly or add a schema default. The shared suite marks the
Inngest `getInitData()` default-input case as an expected failure.
**Test and check results:** not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Description
`run.timeTravel()` treated `false`, `0`, `''` and `null` resume data as
missing. The suspended step saw `undefined` and suspended again, even
when the value matched its `resumeSchema`. `run.resume()` already
delivered these values. This PR replaces three truthiness checks in
`@mastra/core` with presence checks:
```diff
// packages/core/src/workflows/utils.ts — validateStepResumeData
- if (!resumeData) {
+ if (resumeData === undefined) {
return { resumeData: undefined, validationError: undefined };
// packages/core/src/workflows/utils.ts — createTimeTravelExecutionParams
// (stopped collecting the suspended step's result when time travel starts at an earlier step)
- if (currentExecPathLength > 0 && !resumeData) {
+ if (currentExecPathLength > 0 && resumeData === undefined) {
// packages/core/src/workflows/evented/workflow-event-processor/index.ts
// (the default engine already uses !== undefined here, handlers/step.ts)
- if (timeTravelResumeData && !timeTravelResumeValidationError) {
+ if (timeTravelResumeData !== undefined && !timeTravelResumeValidationError) {
```
Omitted `resumeData` still leaves the step suspended. A falsy value that
fails the `resumeSchema` is still logged and not delivered. `null` now
reaches a step whose schema allows it, as it does with `run.resume()`.
### Evidence
New cases in the shared time-travel suite
(`workflows/_test-utils/src/domains/time-travel.ts`) start a run with a
fixed `runId`, wait for it to suspend, then time-travel that run with
`false`, `0`, `''` and `null`. Each value is tried at the suspended step
and from the step before it, plus controls for omitted and
schema-invalid values. They run on the default engine
(`workflow.test.ts`), the evented engine with real workers and
`EventEmitterPubSub` (`evented/evented-workflow.test.ts`), and
`storage-v2-suite.test.ts`, all on in-memory storage. A separate file,
`timetravel-nested-resume-data.test.ts`, covers steps inside a nested
workflow.
- **Before** (fix reverted): `vitest run workflow.test.ts
evented-workflow.test.ts storage-v2-suite.test.ts
timetravel-nested-resume-data.test.ts -t 29390` gives 28 failed, 9
passed, 5 expected fail. Every failure is `expected [ undefined ] to
deeply equal [ false | 0 | '' | null ]`: 24 flat cases plus 4 nested
ones on the default engine.
**After:** 37 passed, 5 expected fail, 0 failed.
- Existing time-travel and workflow suites (`utils`,
`timetravel-divergence`, `timetravel-foreach`, both
`concurrent-time-travel` files, `default`, `workflow`,
`evented-workflow`, `storage-v2-suite`): 1177 passed, 5 expected fail, 0
failed. `tsc --noEmit` for core and a strict typecheck of the new test
file both pass, and so do `oxlint` and `oxfmt` on the changed files.
- Not covered: Inngest calls the same core
`createTimeTravelExecutionParams` and default step handler, so it gets
the fix, but its suite sets `timeTravel: false` and skips the shared
time-travel domain. Nested time travel on the evented engine never
delivers resume data to the inner step, truthy or falsy, before or after
this change. That's a separate, pre-existing bug, not yet investigated,
and the 5 evented nested cases are pinned with `it.fails`.
- Added after CodeRabbit review: `evented/step-executor.ts` used `if
(params.resumeData)` when it records `resumePayload` and `resumedAt` on
the step result, so a falsy resume value was left off the stored result.
It now checks `!== undefined`. New `step-executor.test.ts` cases for
`false`, `0`, `''` and `null` fail 4/4 before and pass after; the whole
`src/workflows/evented` directory passes (43 files, 695 tests). This
doesn't add crash recovery for a time-travelled step: time travel still
writes no running record before the step runs, which is a separate,
pre-existing gap.
### Merge Danger
**Door:** two-way. Reverting the commit restores the old checks, and
nothing persisted changes shape.
**Blast radius:** `run.timeTravel()` with `resumeData` on the default,
evented and Inngest engines. A caller who passed `resumeData: null` to
mean "no data" now has `null` reach a step whose schema accepts it.
## Related issue(s)
Closes#29390
## Type of change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [x] Test update
## Checklist
- [x] I have linked the related issue(s) in the description above
- [ ] I have made corresponding changes to the documentation (if
applicable). No docs change is needed: the `timeTravel` reference
already says it accepts resume data without restricting it to truthy
values.
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR
## Council review
Scores out of 10: Correctness / Tests / Scope / Code quality.
| Round | GPT-6.1 Sol | Claude Opus 5.5 | Grok 4.7 |
|---|---|---|---|
| 1 | 9 / 9 / 10 / 7 | not seated yet | empty seat |
| 2 | 10 / 9 / 10 / 10 | not seated yet | empty seat |
| 3 | 10 / 9 / 10 / 10 | 9 / 8 / 10 / 9 | empty seat |
| 4 | 10 / 8 / 10 / 9 | 9 / 9 / 10 / 9 | empty seat |
| 5 | 10 / 9 / 10 / 10 | 9 / 9 / 10 / 9 | empty seat |
Created with Claude Opus 5.5 in T3 Code.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## ELI5
`run.timeTravel()` treated some valid answers as if the user had not
answered. This PR lets suspended steps receive valid values such as
`false`, `0`, an empty string, and `null`.
## What changed
- Updated time-travel validation and graph traversal to treat only
`undefined` as missing resume data.
- Updated the evented step executor to record defined resume data in
`resumePayload` and set `resumedAt`.
- Added regression tests for falsy resume values from the suspended step
and a preceding step.
- Added a patch changeset for `@mastra/core`.
## Known limitation
Evented workflows still do not deliver time-travel resume data to steps
inside nested workflows. Tests for this behavior remain marked as
expected failures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary
- persist nested parent coordinates for evented child runs
- reserve the shared suspended parent before accepting a direct child
resume
- reuse the event processor’s existing nested completion path to
continue the parent
- reject concurrent sibling resumes without changing the losing child
This PR is stacked on #25817.
## Test plan
- `pnpm vitest run src/workflows/evented` from `packages/core` — 413
passed, 5 skipped
- `pnpm exec tsc --noEmit` from `packages/core`
- `pnpm build:core` — 14/14 tasks passed
- `git diff --check`
## Proof (October 21, 2026)
- Forward-merged main at `e88b34defe` without rebasing; current head
`016fdce430` also fixes two reviewer-reported regressions from
`c8f6639039`.
- On the unfixed head, auth-token-persistence nested step failed and
restart-completed-boundary nested wrapper timed out. Persisting a live
parent step graph caused the regressions. Child snapshots now store
serializable continuation data without executable step graphs; direct
child resume restores graphs from the registered workflow.
- Reverting the fix in the working tree made three focused regression
tests fail; restoring it passed 6 targeted cases across 5 files (both
reviewer regressions, default/evented nested continuation, and
concurrent sibling/claim-release checks). Child snapshot test checks it
can be structured-cloned and contains no stepGraph.
- `npx tsc --noEmit` in `packages/core`, targeted oxlint/oxfmt, and
`pnpm build:core` (14/14 tasks) passed.
## Unverified / known limitation
- **Inngest is not covered by this fix.** A real Inngest dev server was
healthy at `localhost:4000/dev`; temporarily enabling only the focused
case reproduced the failure: the child completes but the parent stays
`suspended` (expected `success`). The `directNestedChildResume: true`
skip was restored. See #26071. The aimock normal/durable/fs variants are
agent-loop modes, not workflow execution engines.
- Full post-merge evented suite did not complete: both attempts passed
37/38 test files (413–414 tests passed, 5 skipped) but a Vitest worker
exhausted its ~2GB Node heap and exited. No assertion failure was
reported in those runs; this is not a full-suite green. The 413-passed
test-plan count above predates the forward merge.
## Risk: medium
Evented continuation is covered by red/green targeted regressions, but
the full suite is unverified after a worker OOM and the parent claim is
concurrency-sensitive. Inngest direct-child continuation remains
unsupported and explicitly skipped.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## ELI5
When someone resumes a paused child workflow, the parent workflow can
now continue after that child finishes. If another child is already
resuming under the same parent, the second resume is rejected and that
child stays paused.
## Changes
- Save parent workflow details with nested child runs. Include the
nested step ID and `forEach` index, then load the details from the
stored snapshot when creating a run.
- Pass the parent details in the `workflow.resume` event so the event
processor can continue the parent through its nested-workflow completion
path.
- Before resuming a child with a parent, claim the suspended parent. If
another resume has already claimed it, release the child’s resume claim
and return `WORKFLOW_RESUME_ALREADY_CLAIMED`.
- If child execution fails after the parent is claimed, attempt to
restore the parent to suspended.
- Add tests for concurrent sibling resumes and parent-workflow
authorization. Add a patch changeset for `@mastra/core`.
## Tests
The PR description reports that `pnpm vitest run src/workflows/evented`
completed with 413 passed and 5 skipped. It reports that `pnpm
build:core` completed with 14/14 tasks passed. The test plan also lists
`pnpm exec tsc --noEmit` and `git diff --check`, but does not report
their results.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
## Vanta 2026-10-30 security fixes (consolidated)
Direct dependency bumps and lockfile updates only. **No
overrides/resolutions added or raised.** The earlier override-based
edits on this branch were reverted; those findings are deferred below.
| Package | From → To | Advisory | Where |
|---|---|---|---|
| zod | ^3.25.76 → ^4.6.7 | zod <4.6.6 (High; no fixed 3.x) |
`client-sdks/react` (dev), `deployers/cloudflare` (dev),
`server-adapters/nestjs` (dev), `packages/editor` (dev),
`packages/playground-ui` (dep), `packages/playground`,
`workflows/_test-utils`, `examples/{durable-agents,inngest}`,
`explorations/longmemeval`,
`observability/_examples/otel-bridge/agent-hub`, e2e fixtures/templates
(`experiment-worker/fixtures/*`, `deployers/template/cloudflare`,
`no-bundling/template`, agent-builder `minimal-mastra-project`) + root
`pnpm-lock.yaml` |
| @hono/node-server | 1.19.x / 2.1.1 → 2.1.5 | CVE-2026-118663 /
GHSA-rmxm-3fg6-px4f (Medium) | direct deps: `examples/agent-builder`,
`otel-bridge/hono-multi/service-{one,two,mastra}`,
`templates/template-customer-refund-agent/client-demo-ui`,
`templates/template-kyc-customer-onboarding`,
`templates/template-security-incident-triage` |
Peer ranges (`^3.25.0 || ^4.0.0`) are unchanged, so consumers are
unaffected. Changeset: patch for `@mastra/playground-ui` (runtime zod
dep).
Supersedes the zod parts of #28075 (that PR also raises a pnpm override
and rewrites the `zod-v3` compat aliases to v4, so it is left as-is for
maintainers).
### Deferred (needs override, upstream release, or is intentional)
- zod v3 compat aliases (`zod-v3` in `packages/core`,
`packages/schema-compat`, `e2e-tests/type-check/template`;
`e2e-tests/client-js/zod-v3`): intentional zod 3 test coverage; no fixed
3.x release. Needs a decision to drop/dismiss.
- zod 3.25.76 transitive via `@arcadeai/arcadejs`, `@mcp-ui/client`,
firecrawl, `agent-browser`, `zod-to-json-schema`, MCP SDK paths:
upstream pins; override-only.
- @hono/node-server 1.19.x transitive via `@modelcontextprotocol/sdk`
(root, `examples/agent`, `packages/mcp/integration-tests`), and npm
lockfiles of `scripts/gh-issue-triage` (existing override, left alone)
and the two templates: override-only.
- graphql-scalars, raw-body, jose (templates/scripts): override-only.
- @azure/msal-node <7.1.0: 6.0.1 via `@azure/identity@4.14.0` (fixed in
4.14.1, blocked by the 1-day `minimumReleaseAge`; lockfile refresh
tomorrow), 5.6.0 via `@microsoft/teams.apps` (no fixed release).
- @opentelemetry/core <2.8.0: via `@google-cloud/pubsub@5` (fix needs
pubsub 6 major).
- @ai-sdk/provider-utils 2.2.8 / ai 4.x: pinned vendored AI SDK v4 line;
no fixed release in that major.
- extract-zip, deepmerge: no patched version.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## ELI5
This PR updates Zod and `@hono/node-server` versions in selected
packages, examples, and templates to address security advisories. It
also updates playground form code to work with Zod 4.
## Changes
- Updates selected `zod` dependencies to `^4.6.7`.
- Updates selected `@hono/node-server` dependencies to `2.1.5`.
- Updates the playground form’s IP address and record validators for Zod
4.
- Maps a specific Zod validation error to the message `Required`.
- Adds a patch changeset for `@mastra/playground-ui`.
## Validation
Test results are not provided in the supplied change summary.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Co-authored-by: shreyas-mastra <327047572+shreyas-mastra@users.noreply.github.com>
Enables `no-else-return: error` in the root Oxlint config and removes
186 redundant `else` branches across 111 source/test files. Existing
rule defaults are retained.
Reviewed every code diff for behavior preservation. An independent
TypeScript AST comparison confirms that changes are limited to removing
`else` after returning branches, with the same expressions, statement
order, return values, and `try`/`catch`/`finally` boundaries. All
181,158 identifier occurrences retain their bindings. The recorder
replay path keeps an explicit block: flattening it would shadow the
earlier request `body` with the response `body` and introduce a
temporal-dead-zone error.
Extends the existing monorepo optimized-dependency test to assert that a
raw TypeScript workspace package, including its enum and transitive
dependency, still executes in the production bundle.
Validation: the root rule-only scan passes on 15,273 files with zero
diagnostics; formatting and `git diff --check` pass. Schema
compatibility tests: 1,060 passed, 9 skipped. Recorder runtime tests: 21
passed. The core selection initially passed 3,513 tests; rerunning the
suites requiring local server/cache access passed 185 tests, with one
remaining cache-file assertion also reproduced on unchanged `main`.
Remaining validation limits: four asset-download cases cannot reach
`placehold.co`; the changed-file lint run reports six import-order
errors reproduced on unchanged files from `main`. Deployer unit tests
and the monorepo test could not run because this checkout lacks required
package dependencies (`fs-extra`/workspace links and `get-port`,
respectively). These limits are left visible for CI. No changeset is
included because this is a behavior-preserving lint cleanup.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## ELI5
This PR makes the code easier to read by removing `else` branches that
are not needed after a return. It also enables a rule that catches those
branches and adds a test for loading a raw TypeScript workspace package
in a production bundle.
## Summary
- Enables Oxlint’s `no-else-return` rule at error severity and removes
186 redundant `else` branches across 111 files.
- Extends the monorepo optimized-dependencies test to check a raw
TypeScript workspace package, its enum, and a transitive dependency.
- Updates the deployer’s transpile-package path pattern.
- In workflow event processing, a truthy branch index with no matching
child now falls through to publish a skipped-step end event. Previously,
this case published no event.
- Keeps the recorder replay block explicit to avoid request/response
`body` shadowing and a temporal-dead-zone error.
## Validation
Reported checks passed: the rule-only scan across 15,273 files,
formatting, `git diff --check`, schema compatibility tests (1,060
passed, 9 skipped), recorder runtime tests (21 passed), and an initial
core test selection (3,513 passed).
Some test limits remain. Four asset-download cases could not reach
`placehold.co`. Changed-file lint reported six import-order errors that
also occur on unchanged files from `main`. Deployer unit tests and the
monorepo test could not run because dependencies were unavailable. On
rerun, suites requiring local server/cache access had 185 passes; one
cache-file assertion also reproduced on unchanged `main`.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Enables all eleven requested Oxlint rules as errors in the root config
and migrates the 115 existing findings across 55 files. JSX curly-brace
simplification is limited to props (`props: 'never', children:
'ignore'`), boolean props use shorthand, and empty elements become
self-closing.
The manual edits preserve the exact continued-string and template
values, remove redundant assignment-side non-null assertions, and retain
the indexed-read assertion in fastembed's compound assignment. The
BigInt and octal conversions use native globals and exact values
supported by the existing compilation targets. No lint suppressions were
added.
Validation: the eleven-rule scan reports zero diagnostics; before/after
emitted-JavaScript comparison passes for all 55 migrated files,
including 31 JSX files, after normalizing only the reviewed syntax
rewrites. Formatting and whitespace checks pass. Core utility/clone
tests (146), fastembed pooling tests (9), the changed A2A signature test
(1), and fastembed build-config typechecking pass. The root lint scan
retains the same 1,244 pre-existing diagnostics with no new findings.
GitHub CI now also passes the full lint/format job, build, Mastra Code
typecheck, generated-file check, and bundle validation. Local UI/evals
runs were limited by missing dependencies; the CodeDiff folding failure
was reproduced on unchanged baseline code. A second audit confirmed
native unshadowed globals for all five literal conversions, pure targets
for all twelve assignment rewrites, no references or direct eval in all
eight simplified catches, and ordinary semantics for both computed keys.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## ELI5
This PR turns on more code-style checks and updates existing code to
pass them. The changes simplify syntax while preserving the intended
behavior.
## Changes
- Enabled eleven Oxlint rules as errors in the root configuration.
- Updated 115 findings across 55 files. Changes include simpler string
and JSX syntax, compound assignments, native BigInt and octal literals,
and catch clauses without unused error variables.
- Configured JSX curly-brace simplification for props only. JSX children
remain unchanged.
- Removed redundant assignment-side non-null assertions while retaining
the indexed-read assertion needed for fastembed’s compound assignment.
- Added no lint suppressions.
## Validation
The author reports zero diagnostics from the eleven-rule scan and no new
findings in the root lint scan, which retains 1,244 pre-existing
diagnostics. Before-and-after emitted-JavaScript comparisons reportedly
pass for all 55 changed files after normalizing the reviewed syntax
rewrites. Formatting and whitespace checks also pass.
Reported tests include 146 core utility and clone tests, 9 fastembed
pooling tests, 1 changed A2A signature test, and fastembed build-config
typechecking. Full workspace lint and the UI/evals suites were limited
by missing dependencies in the checkout. The CodeDiff folding test fails
on unchanged baseline code.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What
Agents get one identity concept: `scopes: string[]`. Each scope is a
`<type>:<value>` address, such as `org:acme`, `resource:user-123` or
`thread:conversation-123`. Types are arbitrary. Memory understands
`resource:` and `thread:`; other consumers can read the rest.
Scopes come from three places, and Mastra combines them:
- **per call:** on `stream`, `generate`, `network`, `sendMessage`,
`queueMessage` and `sendSignal`;
- **the Agent:** `new Agent({ scopes: ['team:core'] })` or `scopes: ({
requestContext }) => [...]`;
- **trusted server middleware:** the new reserved `MASTRA_SCOPES_KEY`
(`mastra__scopes`) request-context key.
```ts
// Before
await agent.generate('Hi', { memory: { resource: 'user-123', thread: 'conversation-123' } });
// After
await agent.generate('Hi', { scopes: ['resource:user-123', 'thread:conversation-123'] });
```
## Behavior
- **Memory** reads resource and thread from `resource:` / `thread:`
scopes, in addition to `memory.*`, `resourceId`/`threadId`, and the
reserved ID keys. If two sources name different values, the call throws
`AGENT_SCOPES_CONFLICT`, which the server returns as HTTP 400. Runs with
no scopes from any source behave exactly as before, including the
existing reserved-key precedence.
- **Downstream context:** tools, processors and delegated sub-agents get
a run-scoped copy of the request context. That copy holds only the
non-identity scopes, so nested agent calls (Observational Memory's
observer, tool-invoked agents, sub-agents) can pick their own thread
without a conflict. The caller's context is never modified.
- **Resume:** a resumed run keeps the scopes it started with. A resume
may re-supply any subset of them, but adding a new scope throws. This
holds on every resume path: approval resume, generate, network,
durable/evented and Inngest.
- **HTTP:** the server never accepts scopes from a request body,
including the top-level field and nested `ifIdle.streamOptions`. They
can only come from middleware. client-js no longer advertises `scopes`
on its stream params.
- **Older servers:** servers released before this change forward a
body's `scopes` and `requestContext.mastra__scopes` to the agent. Every
`@mastra/server` since 1.5.0 registers itself with Mastra, and the new
server marks itself as reserving scopes. While a server without that
mark is registered, agents refuse scopes from the call and the request
context with `AGENT_SCOPES_SERVER_OUTDATED` (HTTP 400). The NestJS
adapter carries the mark too. The Agent's own `scopes` still apply, and
apps with no server are unaffected. Scopes Core writes onto a run's
derived context, or restores from a durable run's persisted state, stay
trusted, so sub-agents, nested runs and cross-process resumes keep
working.
- **Server ownership checks** still read only `MASTRA_RESOURCE_ID_KEY` /
`MASTRA_THREAD_ID_KEY`. The docs tell middleware to set those keys
alongside `mastra__scopes`.
## Why
This gives Memory and upcoming consumers, such as the Subconscious
curator in the stacked follow-up PR, one agent-wide way to know which
addresses a run acts as. It replaces separate keys that each feature had
been reading on its own.
## Test plan
- Packed proof (real Hono server): base Core + old server lets body
`requestContext.mastra__scopes` reach the agent; branch Core + old
server returns 400 with the outdated-server error on generate and
stream, while requests without scopes run with the agent's own
`org:acme`; branch Core + branch server strips body scopes.
- [x] New tests:
- resolver unit tests (`agent-scopes.test.ts`);
- run-level stream/generate/resume/thread entry
points/sub-agents/until-idle (`agent-scopes-runs.test.ts`);
- legacy (`agent-scopes-legacy.test.ts`);
- durable + evented + preparation (`durable-agent-scopes.test.ts`);
- network resume (`agent-network.test.ts`);
- server boundary + 400 mapping (`agents.test.ts`, `utils.test.ts`);
- Inngest resume (`resume-scopes.test.ts`,
`create-inngest-agent.test.ts`).
- [x] Core unit `src/agent src/loop`: 7,613 passed. Core `pnpm check`
and lint are clean. Server handlers: 195 passed. Memory OM suite: 1,847
passed. Inngest: 101 passed.
- [x] Packed-tarball proof with a real Hono server:
- body `scopes` reach the agent with the base server and are stripped on
this branch;
- scopes-only memory saves and recalls through `@mastra/memory` +
LibSQL;
- conflicts throw on all five entry points;
- sub-agents see only non-identity scopes.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## ELI5
An agent can now use scopes such as an organization or conversation ID
to select memory and share useful context with tools. Trusted middleware
can set scopes, and a resumed run cannot add scopes that it did not have
before.
## What changed
- Added `scopes` to agent configuration and execution options.
Middleware can also set scopes through `MASTRA_SCOPES_KEY`
(`mastra__scopes`).
- Memory uses `resource:` and `thread:` scopes as identity values.
Conflicting scopes, memory options, or reserved request-context IDs
cause `AGENT_SCOPES_CONFLICT` with HTTP 400.
- Tools, processors, and delegated agents receive a copied request
context with non-identity scopes. The caller’s context remains
unchanged.
- Resumed runs retain their saved scopes. Resume requests can re-supply
compatible scopes but cannot add scopes the run did not hold.
- The server and NestJS adapter reserve scopes and remove body-supplied
scopes, including nested `ifIdle.streamOptions.scopes`. If a registered
server does not reserve scopes, call- and request-context-provided
scopes cause `AGENT_SCOPES_SERVER_OUTDATED` with HTTP 400.
Agent-configured scopes remain supported.
- Updated client types, documentation, and release notes. Server
ownership checks continue to use `MASTRA_RESOURCE_ID_KEY` and
`MASTRA_THREAD_ID_KEY`.
## Validation
The author reports 7,613 Core unit tests, 195 server-handler tests,
1,847 Memory OM tests, and 101 Inngest tests. Core `pnpm check` and lint
are reported clean. The author also reports a packed-tarball test with a
real Hono server covering old and new server behavior, memory save and
recall, scope conflicts, and sub-agent scope propagation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com>
Co-authored-by: Jamie Lee Nichols <10267569+CodingButter@users.noreply.github.com>
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
Adds `FileUploadProcessor`, an input processor that uploads the files a
user sends to the workspace sandbox and gives the model a path instead
of the bytes. Only the prompt sent to the model changes: the stored
messages keep their files, so the thread, Studio, and other agents still
have the original. Files are named after their content,
`uploads/<thread>/<sha256>.<ext>` under the directory where the sandbox
runs commands, so a file is written once and later turns only check it's
still there. Memory is optional.
```ts
const agent = new Agent({
// ...
workspace,
inputProcessors: [
new FileUploadProcessor({
// required: the workspace whose sandbox receives the files, the same one as the agent's
workspace,
// optional, default: every file is uploaded
// `file` is { fileName?, mimeType, extension? }
// true uploads the file, false leaves it to the model; it can be async, and runs before each model call
filter: ({ mimeType }) => !mimeType.startsWith('image/'),
// optional, default: 10 MB for every file
// receives the same `file` object and returns a limit in bytes
maxFileSize: ({ mimeType }) => (mimeType === 'application/pdf' ? 25 : 10) * 1024 * 1024,
}),
],
});
```
A file of the current turn that can't be uploaded stops the turn with a
tripwire code. A file of the thread history that can't be uploaded,
because it's too large or the sandbox can't take it, becomes a note
instead, so a stored file never blocks a thread. The processor never
downloads anything itself: a link the model fetches, or a provider file
ID, stays as is, while the bytes of a link Mastra downloads for the
model are handled like any other file.
Every agent also gets a new default error processor,
`UnsupportedFileHandler`. When the provider SDK refuses a file type, or
the provider returns an API error for a request that holds a file that
isn't an image, a PDF, or text (Gemini answers some with a 502), it
takes the file out of the prompt and calls the model again once,
uploading the file to the agent's sandbox when there is one, or giving
the model the `[Attachment unavailable: <name>]` placeholder Mastra
already uses for attachments it can't use. Before, the call failed, and
the stored file made every later turn fail too. To reach the sandbox,
`processLLMRequest` now receives the workspace of the step.
On the Studio side, the chat composer no longer refuses files it can't
read and sends them to the agent as file parts. It also puts the message
and its attachments back in the composer when the server can't have
stored it (no response, or a 4xx such as a body too large), keeping
anything typed in the meantime.
The upload path keeps nothing of the original file name on purpose: with
a path that looked like the name, `gpt-5-mini` sometimes retyped one
from the other and asked the sandbox for a file that didn't exist.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## ELI5
Studio can now send files it cannot read to the agent. An optional
processor can save inline file contents in the workspace and give the
model a path to those files. If a message does not reach the server,
Studio restores its text and attachments while keeping text typed during
the request.
## Changes
- Added `FileUploadProcessor` to upload filtered inline files to
`uploads/<thread>/<uuid>.<ext>`. It replaces uploaded file contents with
path notes and stops the turn with tripwire details when an upload
fails.
- Added per-file size limits, including a check of encoded base64 length
before decoding. URLs and provider file IDs remain unchanged and are not
downloaded.
- Updated Studio to accept files without checking whether it can read
them. Failed sends restore submitted text and attachments while
preserving text typed during the request.
- Added documentation, changesets, and tests for file uploads,
attachments, and failed-send recovery.
- The final change no longer includes the Chef agent example’s file
upload processor, E2B sandbox, or `xlsx` skill.
## Validation
No test results were supplied. Live Studio testing with the last two
commits remains outstanding, as noted in the supplied objectives.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## What this changes
Every agent created with `new Agent(...)` now recovers from three
provider-side failure classes
without the caller wiring any processors. Today those three fixes are
hand-copied into four
different places in the monorepo, and a plain `new Agent(...)` gets none
of them — `listErrorProcessors()`
returns `[]`, so a transient 500 or an assistant-prefill rejection ends
the run with an empty
response.
The three processors, added in this order when the caller has not
already supplied a processor with
the same `id`:
| Processor | Failure it recovers from |
|---|---|
| `ProviderHistoryCompat` | history the target provider rejects (foreign
signed thinking blocks, invalid tool-call IDs, provider-specific prompt
shapes) |
| `PrefillErrorHandler` | assistant-response prefill rejections (`does
not support assistant message prefill`, Gemini 3+ model-turn endings) |
| `StreamErrorRetryProcessor` | transient API/stream failures, bounded
by its own `maxRetries` budget |
The order is load-bearing and the retry processor is deliberately last.
Error processors stop at the
first one that returns `{ retry: true }`, and
`StreamErrorRetryProcessor`'s bad-request matcher
matches **any** `400`. Placed ahead of the two repairs it claims their
failures, resends the request
unchanged, spends its single bad-request retry, and then both it and the
repair decline — recovery
becomes unreachable. That is exactly the state the hand-copied stacks
were in: `ProviderHistoryCompat`
was last in three of the four, so its reactive repair could never run.
## How a caller stays in control
`errorProcessors` is now a base list, not a replacement for framework
capability:
```ts
// One tuned processor; the two defaults it does not name are added at the positions their ids give them.
new Agent({ ..., errorProcessors: [new StreamErrorRetryProcessor({ maxRetries: 5 })] })
// A completely custom set — name every id you want, and no defaults are added.
new Agent({ ..., errorProcessors: [new ProviderHistoryCompat(), new PrefillErrorHandler(), new StreamErrorRetryProcessor()] })
// Your list exactly as given — no defaults merged in, not even same-id slot replacement.
new Agent({ ..., errorProcessors: [myCustomProcessor], errorProcessorDefaults: false })
// No error processors at all.
new Agent({ ..., errorProcessorDefaults: false })
```
A processor is only added when nothing in your list carries its `id`, so
your own instance always
wins and is never duplicated. Added defaults go to the position their
`id` gives them rather than
being appended, so naming only the retry processor still resolves both
repairs ahead of it. Your own
processors are never reordered relative to each other.
`errorProcessorDefaults: false` skips the
merge entirely: only what you configure runs, and it is the only opt-out
— an empty
`errorProcessors: []` merges like any other list and still gets the
three defaults. A call-time `errorProcessors` option replaces the
resolved list — defaults included — in both the agentic and durable
execution paths.
`maxProcessorRetries` is still not defaulted on the `Agent` config — it
also governs
`processInputStep`/`processOutputStep` retries, and defaulting it would
silently convert an existing
abort-and-warn into a replay for every agent. When error processors
resolve to a non-empty list and
no cap is set, the runtime applies a safety cap of 3; the defaults all
stop below it on their own
(the repairs act only on the first attempt, the retry processor at its
own `maxRetries`), so it only
bites a processor that never stops asking. The "no explicit cap" warning
now fires only when *you*
configured error processors, not for the framework defaults.
## The structural fix underneath
An error processor placed in `errorProcessors` did not receive
`processLLMRequest`, so a
`ProviderHistoryCompat` there could only repair history *after* the
provider rejected it. Error-phase
processors now participate in the LLM request lane, so the processor
that handles a rejection also repairs
the outbound prompt preemptively. Error processors resolve once per run,
so a dynamic `errorProcessors`
function runs a single time and both lanes share its instances; durable
runs keep the same pipeline,
including a per-call `errorProcessors: []`, when a worker rebuilds them
after a restart. This is the change that lets the preemptive path exist
at all, and it is the structural
answer to #24340 — provider compatibility logic had three homes, two
activation policies, and one
duplicated rule.
## Duplication removed
The same three-processor stack was hand-built in four places. All four
now resolve from one shared
implementation in `packages/core/src/processors/stability-defaults.ts`,
exported through
`@mastra/core/processors`:
- `packages/core/src/coding-agent/index.ts`
- `packages/core/src/agent/goal/scorer.ts`
- `packages/editor/src/ee/agent-builder-agent.ts`
(`DEFAULT_BUILDER_ERROR_PROCESSORS` is still exported)
- `mastracode/sdk/src/index.ts`
Each consumer resolves the same three processors as before, in the
corrected order. Mastra Code keeps
its own tuned `StreamErrorRetryProcessor` matchers and its
`maxProcessorRetries: 64` ceiling (which
its account-rotation processor shares), naming only its two custom
processors — the shared defaults
supply the two repairs ahead of them, so the bad-request matcher no
longer runs ahead of history
repair. `createCodingAgent` additionally opts into `retryUnknownErrors:
true`, preserving its shipped
behavior of retrying unmatched provider failures.
## Proof
Both transcripts are produced against packed tarballs with a
hand-written mock v2 model — no network
calls, no credentials. `without.txt` is the commit this branches from;
`with.txt` is this branch:
```
lists: errorProcessors=[] # without
transient-500: FAIL (text="threw: internal server error") # without
VERDICT: RED (failed: lists, transient-500, prefill-400, caller-list, error-lane-prompt)
lists: errorProcessors=["provider-history-compat","prefill-error-handler","stream-error-retry-processor"]
transient-500: PASS (text="recovered") # with
prefill-400: PASS (text="recovered") # with
caller-list: PASS (expected caller processors kept, missing defaults added in order, [] empty)
# (proof predates the flag-only opt-out; an empty list now resolves to the defaults)
error-lane-prompt: system="You are a proof agent. <memory-context>internal note</memory-context>"
VERDICT: GREEN
```
The `error-lane-prompt` check places `ProviderHistoryCompat` **only** in
`errorProcessors` and
observes the rewritten outbound system prompt, which is a direct
observation of the lane widening.
To rerun:
```bash
pnpm build:core
pnpm --filter @mastra/core pack --pack-destination /tmp/pack
bash .mastracode/plans/enable-stability-processors-by-default.proof/demo.sh /tmp/pack/mastra-core-*.tgz /tmp/with.txt
```
## Changesets
`@mastra/core` minor, `@mastra/editor` patch, `@mastra/code-sdk` patch.
## Tests
The blast radius was large and was handled by isolating tests to their
own subject with
`errorProcessorDefaults: false` or an explicit zero-retry budget — never
by weakening an assertion:
| Suite | Result |
|---|---|
| 18-file core processor/error battery | 555 passed, 2 skipped |
| mock-provider scenarios (`loop/test-utils/aimock`, 91 files) | 548
passed |
| broad battery (`agent.test`, `runner`, `llm-execution-step`,
`src/agent/durable`, 107 files) | 1096 passed, 5 skipped, 14 todo |
| editor `agent-builder` | 33 passed |
| mastracode sdk `index.test.ts` | 58 passed |
| `pnpm --filter ./packages/core check`, editor typecheck,
`build:mastracode`, `oxfmt --check` | exit 0 |
New coverage: `stability-defaults.test.ts` (13 tests pinning the stack's
ids, order, fresh instances,
and each matcher's retry budget), 9 tests in `agent-processor.test.ts`
for the merge rule
(caller-first, no duplicate id, canonical insertion position, empty list
merges, function-form resolution
once per run), durable regression tests for resolver failure and cold
rebuilds,
4 tests for the request lane, 3 tests in `agent-builder.test.ts`, and a
rewritten goal-judge test.
## Notes for reviewers
- The built-in compatibility rules that rewrite the outbound prompt (the
Azure system-reminder
transform, the reasoning-content strips, the foreign-tool strips) never
touch persisted history.
The Anthropic tool-ID rule is the exception **on this PR's head**: it
still repairs reactively
through `fix()`, which rewrites stored message ids after a `400`. The
preemptive replacement for
that path ships in the stacked PR #24505, which sanitizes the ids in the
outbound request for
Anthropic models and keeps the reactive hooks as the fallback for
providers it doesn't recognize.
- The shared retry processor does **not** retry unmatched errors
(`retryUnknownErrors: false`), so a
structured-output schema-validation failure reaches the JSON fallback
immediately.
`createCodingAgent` opts into `retryUnknownErrors: true` to preserve its
shipped behavior; known
terminal authorization failures (401/403) are excluded in both cases.
- Because the request lane now includes an error processor, a
`provider-history-compat`
`processor_run` span appears under every model step in traces. The trace
snapshots in
`observability/mastra` were regenerated to match; the internal
moderation agent is opted out of the
defaults so its trace shape is unchanged.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## ELI5
Agents now have built-in helpers that can repair some request problems
and retry selected failures. This helps agents recover from common
errors without requiring callers to add each helper themselves.
## Summary
- Added shared default error processors in this order:
`ProviderHistoryCompat`, `PrefillErrorHandler`, and
`StreamErrorRetryProcessor`. The resolver adds each default only when no
configured processor has the same ID.
- Set `errorProcessorDefaults: false` to disable default merging; it is
the only opt-out, and an empty `errorProcessors` list still gets the
defaults. A call-time `errorProcessors` list replaces the resolved list.
- Error processors that implement `processLLMRequest()` can run in the
LLM request lane. This lets `ProviderHistoryCompat` modify outbound
requests. If a processor is in both lanes, it runs once in its input
position.
- The implicit retry cap for error-processor retries is 3 when
applicable. Input and output processor retries require an explicit
limit. Framework defaults do not trigger the configured-processor
warning.
- Durable execution now applies call-time processor lists and
distinguishes caller-configured processors from framework defaults.
- Updated coding-agent, editor, and Mastra Code integrations. The coding
agent opts into retries for unmatched errors and bad requests. Mastra
Code retains tuned retry settings.
- Updated tests, documentation, and trace snapshots for the defaults and
request-lane behavior. Test execution results were not provided.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com>
## Summary
Fixes#22636
Re-enables the evented workflow engine by running it on the same durable
agent loop as every other engine, and adds a shared cross-engine test
surface proving that all four ways of executing an agent — in-process,
durable, evented, Inngest — behave identically.
Previously `EventedAgent` inherited `DurableAgent.createWorkflow()` and
silently ran on the **default in-process engine** — the evented engine
was disabled after it crashed on enablement (#22636), so "evented"
agents weren't actually evented. This PR routes `EventedAgent` through
the real evented engine on the shared durable loop builder, adds crash
recovery for suspended and in-flight evented runs, and closes six real
behavior gaps that the new test surface uncovered:
- **Actor identity dropped on evented** (security-relevant): the actor
signal now rides every event envelope (start, resume, restart,
time-travel, step events), so tools doing actor-based authorization see
the caller's identity on every engine.
- **`bail()` one turn late on evented / never stopping on Inngest**: the
delegation bail flag now travels on serialized step output instead of
shared in-process state, and the Inngest continuation predicate honors
it.
- **`maxRetries` ignored on durable/evented**: agent-level and call-time
retry config now serialize through workflow options with the same
precedence as the in-process loop.
- **`requestContextSchema` skipped on durable/evented**: validation now
runs at the durable preparation boundary, matching plain
`Agent.stream()`.
- **`providerOptions` missing from `DurableAgentStreamOptions`**:
declared the field the runtime already forwarded.
- **Transcript-redaction race**: all persistence paths now apply
redaction before saving.
## Effect on the plain `Agent`
The shared cores are wired with per-engine policy parameters so the
default loop keeps its released contract byte-for-byte: every
engine-divergent behavior (continuation semantics — including
delegation-bail ordering and resurrection-feedback handling —,
errored-iteration grading, fatal
completion-scorer/signal-drain/`toModelOutput` errors, dispatch-failure
propagation) is gated behind the durable policy, and the default ladder
is verified line-for-line against the merge-base predicate (`a436d6c5`).
Three loop fixes that would have changed the plain `Agent`'s released
behavior (delegation-bail ordering, feedback preserved on resurrection,
errored-iteration grading skip) were pulled out of this PR and will ship
in a follow-up with their own product decision (material preserved on
the `loop-behavior-fixes-material` branch).
Two plain-`Agent` fixes do ship in this PR, each with a changeset:
- **readOnly background flush**: background tool results no longer flush
to memory when the run is read-only.
- **Transcript redaction on the history save path**: the redaction race
above also affected the default loop's history persistence.
## Test surface
- **Conformance factory** (`workflows/_test-utils`): ~48 behavior
domains executed through the `DurableAgentLike` interface across four
legs (DurableAgent, EventedAgent, plain Agent, Inngest) — 580 tests. New
domains: fga, network delegation, agent-id-context,
request-context-schema, stream-until-idle.
- **Loop scenario matrix**: recorded-fixture scenarios parameterized
over four engines — 101 files / 800 tests, including new scenario files
and previously-skipped actor-identity and delegation-complete-bail now
green on evented.
- **Shared-core policy tests**: the per-engine policy split is pinned by
dedicated unit suites for continuation, task-completion, background
dispatch, background result application, and signal draining.
- **Recorded e2e**: durable legs added to the provider-boundary subset
(tool-call finish reasons across 8 models, reasoning summaries, image
prompts), replaying the same cassettes as the default engine.
Ported tests keep their source assertions verbatim — every leg failure
became either a product fix above or a documented skip with a WHY
comment.
## Test plan
- [x] Conformance factory: 580/580 across all four legs (Inngest leg
against the docker dev server, 252 tests)
- [x] Loop scenario matrix: 101 files / 800 tests across normal,
durable, evented, fs
- [x] Durable suite: 119 files / 967 tests; evented suite: 17 files /
382 tests
- [x] Recorded e2e (replay mode): 167 passed / 45 skipped
- [x] `pnpm --filter @mastra/core typecheck` clean
- [x] Changesets: 31 patch (no minor), scoped per package
(`@mastra/core` ×31, `@mastra/inngest` ×16)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## ELI5
This PR makes durable and evented agents behave like regular agents. It
also makes evented runs recover after a crash and adds broad tests to
verify the same behavior across all execution engines.
## What changed
- Re-enabled the evented workflow engine for `EventedAgent`.
- Added crash recovery for suspended and in-flight evented runs.
- Shared loop construction and behavior across in-process, durable,
evented, and Inngest engines.
- Propagated actor identity through evented workflow execution.
- Preserved delegation `bail()` across serialized workflow steps.
- Added durable support for:
- Call-time retry settings.
- Total run timeouts.
- Request-context schema validation.
- `providerOptions`.
- Provider-executed and deferred tool results.
- Tool-result processor hooks.
- Processor-generated data chunks.
- Transcript redaction on every persistence path.
- Read-only memory behavior.
- Agent version pinning during resume and recovery.
- Fixed terminal finish handling, failed-run stream errors, and falsy
resume data. `toModelOutput` mapping errors are non-fatal on the durable
engine only; the plain `Agent` still throws, as on main.
- Improved evented streaming with split-pubsub delivery, step-event
controls, host validation, mid-step snapshots, and preserved running
history.
- Removed obsolete durable tool-execution helpers and centralized shared
tool-loop utilities.
## Testing
- Added cross-engine conformance coverage for in-process, durable,
evented, and Inngest execution.
- Added 580 conformance tests and 796 loop-scenario tests.
- Added durable and evented recovery, streaming, retry, timeout, tool,
transform, actor, and request-context coverage.
- Added recorded provider-boundary tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Mastra Code (anthropic/claude-fable-5) <noreply@mastra.ai>
Co-authored-by: NikAiyer <nick.aiyer@gmail.com>
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: Abhi Aiyer <abhiaiyer91@gmail.com>
Workflow runs that completed through `resume()` stayed in each engine's
in-memory `runs` map after reaching a terminal status. This releases
terminal resumed runs in the default, evented, and Inngest engines while
keeping suspended runs available for later resumes.
Before: `resume()` → terminal result → run remains in `workflow.runs`
After: `resume()` → terminal result → run is released
Shared regression coverage verifies both one-shot and repeated
suspend/resume lifecycles across the engines. The core tests and checks,
Inngest build, and focused Inngest lifecycle tests pass. The full
Inngest workflow suite still has four existing failures that reproduce
on the unmodified base revision.
Fixes#22390.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## ELI5
When a workflow finishes after `resume()`, the system now removes the
completed run from memory. Suspended runs remain available for later
resumes, which prevents memory growth.
## Changes
- Cleaned up completed resumed runs in the default, evented, and Inngest
engines.
- Added tests for single and repeated suspend/resume cycles.
- Added patch changesets for `@mastra/core` and `@mastra/inngest`.
## Verification
- Core checks, the Inngest build, and focused lifecycle tests passed.
- Four existing failures remain in the full Inngest workflow suite.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
## Summary
- Migrates the agent's internal agentic loop onto the evented workflow
engine. Each LLM call and tool call now runs as a workflow step, sharing
suspend/resume and observability with the rest of the workflow system.
No user-visible behavior change yet — this is the structural
prerequisite for future durability and event-based observability
features.
- Fixes a nested-workflow bug where `resumeLabel` wasn't propagated up
through a parent's suspendedPaths, so resuming a nested workflow by
label silently failed.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
ELI5: This change moves the agent's internal loop to run as a series of
steps in the evented workflow engine so each model call and tool call is
tracked like a tiny workflow. That makes suspend/resume, observability,
and future durability much easier and also fixes a bug that broke
resuming nested workflows by label.
- What it does
- Migrates the agentic loop so each LLM and tool call executes as
evented workflow steps (uses createEventedWorkflow /
createEventedWorkflow factory).
- Agents can run without a user-provided Mastra by creating a
lazily-initialized ephemeral Mastra (InMemoryStore + EventEmitterPubSub)
and starting ephemeral workers on first use.
- Adds a per-run runScope (PrepareStreamRunScope) to thread
non-JSON-safe runtime state (MessageList, processorStates,
convertedTools, loopOptions) across prepare-stream workflow steps
instead of passing those objects through step inputs/outputs.
- Introduces runId-scoped internal workflow registration/unregistration
to avoid collisions for concurrent/nested runs.
- Bug fixes / behavioral corrections
- Fixes nested-workflow resume-by-label: resumeLabel is propagated
through parent suspendedPaths and nested suspend metadata so resuming a
nested workflow by label works reliably.
- Preserves and aggregates suspendPayloads (not just resumeLabels)
across suspended foreach iterations and when storage persistence is
skipped/absent.
- Ensures persisted snapshots are retained for pending/paused/suspended
states (not only suspended).
- Tightens internal workflow lookup to prefer runId-scoped internal
workflow instances, preventing wrong closure/registration resolution in
nested/concurrent cases.
- Implementation details / notable API surface changes
- New helper: createEventedWorkflow(...) exported from workflows
entrypoint (throws if evented factory not loaded).
- Mastra changes:
- Defaults to InMemoryStore when storage not provided (with warning) and
augments the store.
- Internal workflow registry supports runId-scoped keys; added
__unregisterInternalWorkflow and updated __registerInternalWorkflow /
__hasInternalWorkflow / __getInternalWorkflow signatures.
- Evented processor and execution engine:
- Nested workflow resolution now passes runId to getNestedWorkflow and
resolveWorkflow.
- Workflow event processor rebuilt resumeLabels and suspend payload
propagation; adjusted foreach aggregation and merge behavior when
storage updates are absent.
- In-memory workflow storage replaced JSON-serialize deep-copy with a
cycle-safe deep clone preserving Dates, Errors, Maps/Sets, prototypes,
typed arrays, etc.
- Tests / tooling
- Added suspend/resume test to validate resuming a suspended nested
workflow step by label and updated many tests to run against an evented
Mastra (InMemoryStore + EventEmitterPubSub) with worker lifecycle
management helpers (createTestMastra, start/stop workers).
- Multiple test updates to use real InMemoryStore + evented pubsub
instead of mocked storage or missing worker lifecycles.
- Minor / UX
- Updated background-task generated system prompt to clarify _background
behavior and placeholder result guidance.
- stream/loop runtime: distinguishes "pre-run" queued signals from
normal "pending" signals and exposes drainPendingSignals(runId, scope?)
with scope 'pending'|'pre-run'; stream lifecycle registers/unregisters
run-scoped internal workflows to prevent leakage on suspend.
- Risk / review notes
- Structural and high-impact refactor touching core agent loop, workflow
engine integration, storage cloning, and Mastra internals—review effort:
high in multiple areas (agent, workflows, mastra, storage, evented
processor).
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/mastra-ai/mastra/pull/16796?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Mastra Code (anthropic/claude-opus-4-6) <noreply@mastra.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Abhi Aiyer <abhi@mastra.ai>
Co-authored-by: Abhi Aiyer <abhiaiyer91@gmail.com>
This fills in missing package-local lint-staged configs so staged files
in those workspaces get formatted/fixed before CI catches import
ordering.
It also switches the Husky hook to Corepack so the hook can resolve the
repo's pnpm version consistently. On my machine, plain `pnpm` was
failing in the hook env and the fallback made the hook silently skip
lint-staged.
Before:
```sh
pnpm --version || exit 0
pnpm lint-staged --quiet
```
After:
```sh
corepack pnpm@10.29.3 --version || exit 0
corepack pnpm@10.29.3 lint-staged --quiet
```
I checked workspace coverage and verified package-local lint-staged runs
cleanly against the branch diff with:
```sh
corepack pnpm@10.29.3 exec lint-staged --diff HEAD --quiet
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## ELI5
This PR makes sure every package in the repo automatically formats and
fixes staged files before you commit, and it makes the pre-commit hook
run a specific, consistent pnpm version so those fixes actually run
reliably.
## Changes to Pre-Commit Hook
- Updated .husky/pre-commit to run pnpm via Corepack with an explicit
pin: corepack pnpm@10.29.3 --version || exit 0 and corepack pnpm@10.29.3
lint-staged --quiet (replacing plain pnpm). This ensures the hook
resolves the repository's pnpm version consistently and prevents
lint-staged from being skipped in environments where a global pnpm may
be absent or incompatible.
## Workspace Package lint-staged Configs
- Added package-local lint-staged configuration files
(lint-staged.config.js or .mjs) across ~40+ workspace packages and
subpackages.
- Two consistent patterns applied:
- TypeScript/TSX-focused packages: run eslint --fix --max-warnings=0
then prettier --write on *.{ts,tsx}, and run prettier --write on
*.{js,jsx} and *.{json,md,yml,yaml}.
- Prettier-only packages (test-utils and simpler packages): run prettier
--write for *.{ts,tsx,js,jsx,json,md,yml,yaml}.
- These package-local configs ensure staged files within each workspace
are formatted and lint-fixed (including TS import ordering via eslint
--fix) before commit, reducing CI failures and manual formatting fixes.
## Other Notes
- Author verified workspace coverage and confirmed package-local
lint-staged runs cleanly against the branch diff using: corepack
pnpm@10.29.3 exec lint-staged --diff HEAD --quiet.
- Commit message notes an adjustment to align with pnpm 11 in a
follow-up commit (chore: align lint-staged hook with pnpm 11).
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/mastra-ai/mastra/pull/17220?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Mastra Code (openai/gpt-5.5) <noreply@mastra.ai>