0
Fork 0
mirror of https://github.com/mastra-ai/mastra.git synced 2026-11-17 08:44:46 +00:00
Commit graph

237 commits

Author SHA1 Message Date
dane-ai-mastra[bot]
d60ba71be0 chore: version - exit prerelease mode 2026-11-12 21:12:29 +00:00
dane-ai-mastra[bot]
055a8c3b82 chore: version packages 2026-11-12 04:17:40 +00:00
mastra-platform[bot]
743a02786e
feat(valkey): identify Mastra-created clients with a clientInfoTag (#30156) 2026-11-11 17:57:11 -08:00
Dane - AI (Mastra)
c014d187bb
chore(deps): update docker images (major) (#19788) 2026-11-11 17:08:49 -08:00
Arthur Fiorette
ce03237b34
fix(core): send only the response messages that changed on agent stream events (#29971)
## Description

Follow-up to #29965 (merged). After that PR, each plain Agent
`step-finish` on `agent.stream.<runId>` still carried the run's response
messages so far, so a long run stored every tool output once per later
step: the log still grew with the square of the step count, mostly from
large tool results.

The publisher now keeps a cursor of message fingerprints and sends only
the messages from the first one that changed:

```ts
// step 1: { from: 0, seq: 0, messages: [A1, T1] }
// step 2: { from: 2, seq: 1, messages: [A2, T2] }
// step 3: { from: 4, seq: 2, messages: [A3] }
_durableResponseMessagesDiff: { producer, seq, from, base, hash, messages }
```

"From the first changed index" (not "only new messages") because earlier
entries change: a step's assistant message grows across steps, and a
processor retry rolls the list back.

Readers rebuild the full list per step in the stream adapter, before
chunks reach `MastraModelOutput`, so
`onStepFinish(step).response.messages`, `output.steps` and `onFinish`
still see the whole list:

- `agent.observe(runId, { offset })` first folds the step-finish events
before the offset from the run's history (bounded wait, doesn't delay
the subscription).
- A diff only applies to the list it was made against (`base` must equal
the hash of the list the reader holds). One that can't be placed (a
missed or reordered delivery, a late copy) is never applied, and the
reader reports no messages for that step instead of a partial list.
- The publisher resends the whole list once the diffs since the last
whole list add up to its size, so whole lists cost at most about as much
as the diffs and the log stays linear. A failed publish or diff makes
the next one a whole list, so a reader that missed one recovers.
- On a bus that keeps no history of the run's topic (`cache: false`, or
a `CachingPubSub` whose `shouldCache` leaves the topic out), every
step-finish is a whole list, so an observer that attaches mid-run can
place it. An internal `PubSub.__replaysTopic(topic)` answers this per
topic, since `CachingPubSub.supportsOffsets` is true even for topics it
doesn't cache. Transports without offsets (Redis and Valkey Streams
without a cache) get whole lists too: an `observe()` from an offset
there follows the stream from its tail.
- Each diff carries its publisher's id: a run resumed after approval
publishes from a new publisher that starts with the whole list. Readers
keep the list of the current and previous publisher apart, so a late
event from the previous one doesn't break the next diff.
- The diff replaces `_durableResponseMessages`, so an older reader
during a rolling upgrade reports no messages rather than a partial list.
DurableAgent and EventedAgent still send whole lists, which the adapter
handles as before; they move to diffs in a follow-up.

## Related issue(s)

Part of COR-1532 (plain Agent; DurableAgent/EventedAgent and Inngest
follow in the next PRs of this stack). Follow-up to #29965 (production
Redis OOM from plain Agent run logs). Related: #27475.

## Type of change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [x] Performance improvement
- [x] Test update

## Checklist

- [x] I have linked the related issue(s) in the description above
- [ ] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR

Tests: `agent-stream-log.scenario.test.ts` (all engines) adds "an early
tool result is stored as often as the last one" (fails before this PR on
the plain Agent; durable engines pinned as the known gap), an observer
resuming mid-run at an offset (fails without the history fold), an
observer joining mid-run on a bus without history (fails without the
whole-list rule), and an approval suspend/resume replay compared step by
step. `response-messages.test.ts` covers the diff and apply rules
(edits, shrinking lists, duplicates, lost diffs, producer switches, a
late whole list from the previous producer, the size-based whole list).
The real-Redis test asserts the tool-result counts and that a 20-step
run stores less than 2.5x the bytes of a 10-step one (2.1x with this PR,
2.9x before it).



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

Each step used to save the full list of response messages again, even
when most messages had not changed. This change saves only the updates
and rebuilds the full list for observers, which can reduce storage
during long runs.

## Summary

- Plain Agent stream events now publish response-message diffs. The
publisher sends a full list first and periodically resends one as diffs
accumulate.
- Readers apply diffs when they match the current message list. If a
reader cannot rebuild the list, it reports no response messages for that
step rather than a partial list.
- When `agent.observe()` starts at a numeric offset, it replays earlier
events to rebuild response messages before delivering later events.
Observers continue to receive the full list for each step.
- DurableAgent and EventedAgent continue to publish full lists. Tests
add coverage for diff replay, offset replay, approval resumption,
tool-result repetition, and Redis stream storage growth.
- Older observers report no response messages for steps published with
diffs. The changeset recommends upgrading agent and observer processes
together.

Test results: not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-10 23:13:38 -03:00
Arthur Fiorette
749c09cf91
fix(core): publish lean agent stream events instead of the loop's chunks (#29965)
## Description

A registered Agent publishes every run on `agent.stream.<runId>` so
`agent.observe()` and reconnecting clients can replay it, and on Redis
that log is stored until the run is cleaned up. The publisher forwarded
the loop's in-process chunks as they were: every `step-start` carried
the provider request, and every `step-finish` carried the prompt again
(`metadata.request`, `messages.all`/`user`) plus every earlier step with
its own prompt (`output.steps`). The log grew roughly with the cube of
the step count; in production single runs of a plain Agent took 0.3 to
3.4 GB of Redis and filled it.

The publisher now builds `step-start`, `step-finish` and `finish` field
by field (`agent/streams/wire.ts`) with what readers use, which is what
DurableAgent already publishes:

```ts
// step-finish payload on the topic, before
{ stepResult, metadata: { ...response, request: { body: '<whole prompt>' } },
  output: { text, toolCalls, usage, steps: [/* every step, each with its prompt and response */] },
  messages: { all, user, nonUser } }

// after
{ stepResult, metadata: { id, timestamp, modelId, headers, providerMetadata, ... },
  output: { usage /*, steps: [{ tripwire }] when the step was rejected */ },
  _durableStepContent: [/* this step's content */],
  _durableResponseMessages: [/* response messages so far */] }
```

- The caller's own stream and callbacks are unchanged; only the
published copy is reduced.
- Observers of a plain Agent get better replays: each step's content was
`[]` before, and `onFinish` lost its text over JSON transports and threw
on in-memory ones (the finish steps carried AI SDK-shaped `toolCalls`
the reader can't read). Finish steps now carry `text`, and tool
calls/results in the record shape DurableAgent uses.
- A delegated agent's chunks inside `tool-output` /
`workflow-step-output` are slimmed by the plain publisher. Durable
engines still publish them as-is, since their callers read their own run
back from the topic; that case is pinned with `it.fails`.
- Step content is read from the run's message list when the step
finishes (one extra response-to-UI conversion per step, the same work
the caller's output does).

Still to come, stacked on this PR: each `step-finish` still sends the
response messages so far in full; follow-ups send only what changed, for
the plain Agent first and then DurableAgent/EventedAgent and Inngest
(superseding #27522).

## Related issue(s)

Fixes COR-1530. Production incident: shipyard Redis OOM from
`agent.stream.<runId>` keys of plain Agent runs (Factory). Related:
#27475 (the DurableAgent side of the per-step growth) and COR-1532
(per-step response messages, the follow-up stack), introduced with
#28424.

## Type of change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [x] Performance improvement
- [x] Test update

## Checklist

- [x] I have linked the related issue(s) in the description above
- [ ] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR

Tests: `agent-stream-log.scenario.test.ts` runs on every engine (prompt
copies on the log, delegated agent prompt, observer replay over
in-memory and JSON transports, a rejected and retried reply), and
`pubsub/redis-streams/src/agent-stream-size.test.ts` checks what real
Redis holds in both cache modes. The new plain-agent cases fail on main.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

Agent stream logs repeat old prompt and step data, which can make them
large. This change publishes smaller event records while keeping the
caller’s stream intact and giving observers step content they need.

## Summary

- The publisher creates reduced `step-start`, `step-finish`, and
`finish` events. It removes provider requests and repeated step data.
- For `step-finish`, the publisher reads the current step’s content from
the run’s message list. It omits content for retry steps and logs a
warning if it cannot read the content.
- The publisher reduces nested agent chunks inside `tool-output` and
`workflow-step-output`. DurableAgent and EventedAgent continue to
publish their delegated-agent steps in full.
- Observer replay gains step content and richer finish data, including
run text and tool calls, and step reasoning, sources, and files. The
event still includes response messages on each `step-finish`.
- Added scenarios cover prompt retention, delegated-agent prompts,
observer replay over in-memory and JSON transports, and rejected and
retried replies. A Redis integration test checks stored stream size and
observer replay with both cache modes.

## Validation

Completed CI, build, and test results: not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-10 19:51:50 -03:00
mastra-platform[bot]
d77893aed3
fix(core): resume evented agent observe() at the offset on Redis Streams with the default cache (#29798)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9

## What

Fixes #28324. On an evented agent (`createEventedAgent()`) using a
cross-process PubSub such as `RedisStreamsPubSub` with the default
in-memory server cache, `observe(runId, { offset })` ignored the offset.
A client reconnecting mid-run got earlier events again, and an offset
past the end of a finished run replayed nearly the whole run.

## Why it happened

Evented step frames went out unnumbered whenever an in-memory cache sat
behind a cross-process transport. The concern was that a worker in
another process would number the run from 0 in its own cache, and
observers would then drop its frames as duplicates. That skipped
numbering in the common case too, where every step runs in the process
that started the run.

## Change

- `CachingPubSub` gains two `@internal` options,
`extendExistingLogsOnly` and `onUnlogged`. With
`extendExistingLogsOnly`, a publish only appends to a topic whose log
this cache already holds. Any other publish goes out live and
unnumbered, and `onUnlogged` is called.
- `StepExecutor` now always wraps the step bus when a server cache
exists. If the cache is process-local and the bus crosses processes, it
turns on `extendExistingLogsOnly`. The caller writes the run's `start`
frame before any step runs, so steps in the caller's process extend that
log and get numbered. Steps on another worker find no log, stay
live-only as before, and log a one-time warning that recommends
`RedisServerCache`.
- `StepExecutor` now picks up the Mastra logger when it is constructed.
Workflow event processors create executors without registering them on
Mastra, so without this the warning was dropped.
- Docs (`durable-agent.mdx`) list the two setups that still need a
shared cache: steps running on a dedicated worker, and observers
reconnecting through another replica.

## Tests

- `pubsub/redis-streams/src/observe-offset.test.ts`: the evented +
default cache row now passes the exact-resume checks (resume at offset,
offset past end returns nothing, mid-run replay). The full matrix passes
12/12 against Redis 7.
- `step-executor.test.ts`:
  - When the caller holds the run's log, step frames are numbered.
- When a worker has no log, its frames stay unnumbered and one warning
reaches the logger configured on `Mastra`, using an executor that was
never registered on Mastra.
  - Both cases fail on main.
- `caching-pubsub.test.ts`: tests `extendExistingLogsOnly` both with and
without an existing log.
- `stores/libsql` cross-process suite, including `xproc-input-tripwire`:
48 passed, 0 failed.
- Core `tsc` is clean. oxlint and prettier pass on the changed source
files.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## ELI5
When an evented run starts, the process that started it can now number
its events even when events travel through Redis. That lets a client
resume from an offset without seeing old events again. Events from a
different worker remain unnumbered when only a local cache is available,
and Mastra warns about this limitation.

## Changes
- Added internal `CachingPubSub` options to append only to existing
cached logs and report events that cannot be numbered.
- Updated `StepExecutor` to use those options with a process-local cache
behind cross-process PubSub. It numbers events for runs whose log exists
locally and warns once when a worker publishes events for a run it did
not start.
- Updated documentation to explain when a shared cache is required,
including worker execution and reconnects through another server
process.
- Added tests for existing and absent cached logs, step numbering and
warnings, and offset replay scenarios.

## Validation
Test, build, and CI results: not available.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: NikAiyer <nick.aiyer@gmail.com>
2026-11-09 17:23:50 -08:00
dane-ai-mastra[bot]
f21e0742ce chore: version packages 2026-11-07 04:18:11 +00:00
mastra-platform[bot]
d6a5f98df3
fix: index observe chunks for resume (COR-1506) (#29520)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9

Linear:
[COR-1506](https://linear.app/kepler-crm/issue/COR-1506/observe-offset-cant-be-used-to-resume-it-counts-log-events-but-clients)

## Problem

`observe({ runId, offset })` takes `offset` as a position in the run
log. Clients can't compute that position:
- Some log entries never reach the client. These include
iteration-complete records after every step and ownership markers.
- Updates on the wire carried no index.

A client that resumes at "number of updates received" lands too early
and gets repeated updates. The drift grows by one per completed step, so
every tool-using run that reconnects after its first step is affected.
On a 3-step tool loop, 16 of 28 cut points repeated 1–2 updates on both
durable and evented engines.

## Fix

- **Index on updates:** every update now carries `index`, the run-log
position it came from. This covers:
  - `/observe` on every agent
  - `/stream` on durable and evented agents
- core `stream()`/`observe()` chunks and client-js
`observe()`/`stream()`
- **Resuming:** pass `index + 1` of the last received update that has an
index as `offset`. Gaps in the numbering are expected; they're log
entries that send nothing.
- **Unindexed updates:** the synthesized `abort` update carries no
index. Its log position is on the `finish` sent beside it. Client-built
structured-output `object` chunks have no index either. An
`object-result` logged by a structuring-model run carries one. Unindexed
updates don't move the resume point. A client that reconnects between
`abort` and `finish` gets the `abort` again.
- **`offset` is unchanged:** it still means a run-log position.
- Core no longer strips `index` in
`DurableAgent.__subscribeToRunStream`. The stream adapter and the
server's `toAgentStreamChunks` copy the envelope's index onto each chunk
and drop any index a forwarded sub-agent chunk carried. `index` is
optional on `BaseChunkType` and `DataChunkType`.
- **Also fixed:** on plain agents, `/observe` (HTTP and core
`agent.observe()`) with an offset past the end of a finished run used to
wait for the idle timeout or hang. It now closes immediately. It reads
the log from just before the offset and only falls back to a full read
when that window is empty. The check runs in the background, so it never
delays returning the stream or its cancel handle. A trailing `error`
chunk doesn't count as the end of the run.

## Migration

`index` is a new field and `offset` keeps its meaning:
- Callers that pass a log position need no change.
- Callers that replay from `offset: 0` need no change. They get the same
updates in the same order, now with `index` set.
- Only callers that passed a count of updates received need to change.
They were already getting repeats after step boundaries; switch them to
`index + 1` of the last indexed update.

## Evidence

| Check | Result |
| --- | --- |
| Red/green | With `main`'s core and server, 15 tests on numbering and
`/stream`–`/observe` matching fail; with this change all pass. |
| Cut points | Every cut point of a 3-step tool loop on durable and
evented engines resumes from the first `/stream` connection at last
index + 1. Covers the HTTP route and client-js `stream()` → `observe()`.
|
| `/stream` vs `/observe` match | Frames are identical JSON, including
`index`. Replay from 0 equals the first `/stream`. |
| End of run | Resume after finish or past the end returns the rest and
closes, including far past the end (offset 100). |
| Adapters | Hono 98/98, Fastify 98/98, Express 90 passed + 8 skipped
(see bug 1 below). The shared suite is now wired into Express and
Fastify. |
| Redis restart | Real-Redis split-deploy test (in
`pubsub/redis-streams/src/observe-cross-process.test.ts`) passes on both
engines. The client resumes on a freshly booted API process at last
index + 1 with nothing repeated or missing. Evented: the worker executes
the run and the API is killed with SIGKILL. Durable runs execute in the
API that started them, so that API stays up; the client drops its
connection and resumes on a second API. |
| Suspend | After tool approval, indexes keep increasing. Observers
parked mid-suspend and every cut across the suspend resume correctly on
both engines. |
| Unit suites | server `agents.test.ts` 289, core `src/stream` +
`src/agent/streams` 481 |
| Static | tsc clean on core, server, client-js, the test suite, the
Hono/Express/Fastify adapters and redis-streams. oxlint 0 warnings. |

## Scope

Plain (non-durable) agents' `/stream` updates aren't numbered yet. The
caller receives each update before the cache assigns its number. That's
tracked in COR-1510. Plain agents' `/observe` is covered here.

## Existing bugs found (not fixed here)

1. **Express untilIdle 404:** after a client drops an untilIdle stream
on a durable or evented agent, Express answers `/observe` with 404 "Run
not found". It fails the same way on `main`. Those 8 cases are skipped
on Express only, via `skipUntilIdleDisconnectReplay`.
2. **Finished durable run, other API:** a finished durable run returns
404 on `/observe` from any API process except the one that executed it
(`authorizeDurableRun`, `handlers/agents.ts`). Resuming while the run is
live works from any API.
3. **Plain replay mismatch:** on plain agents, replayed
`step-finish`/`finish` payloads differ from the live ones (for example
`timestamp` and `processorRetryCount`). Related to COR-1510.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

When a connection drops, clients need to know which updates they already
received. This change adds positions to many updates so clients can
resume from the right place. It also closes observations of finished
runs when the requested position is past the end.

## Summary

- Added optional `index` values to chunks from durable and evented agent
streams, `observe()` streams, and corresponding HTTP routes. Clients can
resume by passing the last indexed chunk’s `index + 1`.
- Preserved run-log indexes through core stream adapters and server
frame conversion. Sub-agent data chunks do not carry a child run’s index
into the parent stream.
- Some chunks remain unindexed, including synthesized `abort` chunks and
client-built structured-output `object` chunks. Failed cache writes and
non-atomic custom-cache indexing can affect replay.
- Updated API schemas and documentation to explain index gaps, replay
limits, and resumption.
- Added past-end handling for finished plain-agent observations in core
and HTTP `/observe`. History checks do not delay stream return or
prevent cancellation.
- Added tests for indexed replay, past-end closure, cancellation during
history reads, sub-agent index isolation, and observation across API
restarts.

**Validation:** Current-head CI, build, and test results are not
available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: NikAiyer <nick.aiyer@gmail.com>
2026-11-06 11:40:46 -08:00
mastra-platform[bot]
75435bc2cd
fix(redis-streams): retry publishes on transient connection errors (#29512)
## Summary

The Redis Streams pubsub now retries a publish that fails with a
transient connection error, instead of failing the delivery and letting
the step re-run.

`#publishRemote` already retried while Redis answered `LOADING` after a
restart. Any other connection failure (socket closed unexpectedly,
client offline, connection or socket timeout, `ECONNRESET`, `EPIPE`,
`ETIMEDOUT`, `ECONNREFUSED`) was rethrown immediately. A lost
`workflow.step.end` publish was nacked and redelivered, so the step ran
again. These errors now retry under the same 120 s budget (250 ms
between attempts) until Redis accepts the write, the pubsub closes, or
the budget runs out.

This addresses the transport side of #29504.

## Limits

- **Not exactly-once.** A worker that dies between finishing a step and
publishing `step.end` still gets the `step.run` redelivered, and that
step runs again. At-least-once delivery is accepted for that window.
- **Duplicates after a lost reply.** If Redis wrote the event but the
reply was lost, the retry appends it again. Subscribers can receive a
second delivery with the same event id. A duplicate `workflow.step.run`
is dropped by the step fence; other events are already handled as
at-least-once.
- **Slower failure in a long outage.** A publish that starts while
connected and then loses the connection now fails after the 2-minute
budget (`PublishRetryBudgetExhaustedError`) if Redis doesn't come back,
where it previously failed immediately. The trade-off is that no event
is lost in a short outage.

## Changes

- `pubsub/redis-streams/src/index.ts`: `isTransientPublishError()`
widens the publish retry from `LOADING` to transient socket errors;
budget/delay constants renamed to `PUBLISH_RETRY_BUDGET_MS` /
`PUBLISH_RETRY_DELAY_MS`. Each attempt races a timer for what is left of
the budget, because node-redis parks commands issued while reconnecting
in its offline queue and never settles them; running out throws
`PublishRetryBudgetExhaustedError`, which is not retried.
`publishRetryBudgetMs` overrides the 120 s default.
- `pubsub/redis-streams/test-fixtures/harness.ts`:
`makeXaddDropProxy()`, a TCP proxy that drops the client connection on
the first XADD either before Redis sees it or after Redis applies it,
optionally closing its listener after the drop so Redis looks down for
good.
- `pubsub/redis-streams/src/publish-transient-retry.test.ts`: both drop
modes deliver the event; all copies share one event id. A third case
closes the proxy listener after the drop and asserts publish rejects
with `PublishRetryBudgetExhaustedError` at a 2 s budget.
- Changeset: `@mastra/redis-streams` patch.

## Tests

- `publish-transient-retry.test.ts`: 2/2 pass. With the check reverted
to `isLoadingError`, both fail with `Socket closed unexpectedly`.
- `pnpm --filter @mastra/redis-streams run check:test` passes; full
`@mastra/redis-streams` test suite run against Redis 6381 / Postgres
5436.
- No changes to `@mastra/core`; `pnpm build:core` succeeds.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

If Redis briefly loses a connection while receiving an event, the
publisher now tries again instead of failing immediately. If Redis saved
the event but its reply was lost, the retry can deliver a duplicate with
the same event ID.

## Changes

- Redis Streams publishing retries `LOADING` errors and recognized
transient connection errors, including closed sockets, `ECONNRESET`,
`EPIPE`, and timeouts.
- Retries stop for non-transient errors, when the pub/sub instance is
closed, or after 120 seconds. Attempts are spaced 250 ms apart.
- Added a test proxy and tests for connection drops before and after
Redis writes an `XADD`.
- Added a patch changeset for `@mastra/redis-streams`.

When Redis writes an event but the reply is lost, the retry can append
the event again. Both copies have the same event ID.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: NikAiyer <nick.aiyer@gmail.com>
2026-11-05 13:57:59 -08:00
mastra-platform[bot]
2f93851c37
fix(core): make Agent.durable configure resumable streams only (#29503)
Co-authored-by: NikAiyer <nick.aiyer@gmail.com>
2026-11-05 11:32:10 -08:00
mastra-platform[bot]
f94beac862
fix(core): unwedge restarted nested evented loops (#29431)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9

## Problem

Setup: a split deploy (API with `MASTRA_WORKERS=false` plus dedicated
workers over Redis) with `recovery.durableAgents: 'auto'`. When the API
and the worker die together mid-tool and both restart, boot recovery
claims the run under a new generation and the interrupted tool runs
again. After that, the run stops making progress. It sits in
`durable-agentic-loop:running` forever while the API keeps renewing its
lease. Because the run never ends, it also keeps its thread lease, so
any later run on the same thread is rejected with
`AgentThreadLeaseConflictError`. Mastra Doctor hits this every time,
since its thread is the deployId and every re-diagnosis lands on the
stuck thread. The `DurableExecutionFenceError` and `This workflow run
was not active` messages in the ticket are side effects of this stuck
run.

## Root cause

Recovery behaves correctly: it claims a new generation and re-drives the
run. The run gets stuck in the evented step guard in
`WorkflowEventProcessor.endWorkflow`.

When a restarted nested loop body finishes, `endWorkflow` calls
`#processLoop` for the parent loop. The `!parentWorkflow.restart`
condition there, introduced by 11c83d5333 (#28054), skipped the outer
loop leaf's stored `startedBy` on restart. #28054 lists
restart/redelivery races as out of scope, so this exclusion was not a
deliberate restart contract. As a result:

1. The completion guard never ran, so the outer loop leaf stayed
`running` under the event id from before the crash.
2. The loop's next `workflow.step.run` carried a new event id, so
`assertWorkflowStepGuard` raised a `stale` conflict.
3. `process()` acknowledges `stale` conflicts without retrying or
reporting an error, so the step was dropped and the loop never advanced.

## Fix

Remove `restart` from that exclusion. A restarted child re-drives the
stored occurrence, so it now completes under that occurrence's
`startedBy`, the same way a normal redelivery does. Time-travel and
resume still skip the guard as before. The code change is 3 lines in
`packages/core/src/workflows/evented/workflow-event-processor/index.ts`,
plus a patch changeset for `@mastra/core`.

## Tests

**Core unit test:**
`packages/core/src/workflows/evented/restart-nested-loop-guard.test.ts`
- Runs on an in-memory store that reports `supportsWorkflowStepGuards()
=== true` and enforces `assertWorkflowStepGuard` the way `@mastra/pg`
does.
- A `dountil` over a nested workflow crashes mid-iteration on host A,
then host B restarts it over the same storage.
- The test asserts that the loop finishes and the leaf is not left
`running`.
- With the `index.ts` line reverted it fails: the leaf stays `running`
under the pre-crash `startedBy` and the run hangs. With the fix it
passes.

**End-to-end test:**
`pubsub/redis-streams/src/durable-full-restart.test.ts` (fixture
`test-fixtures/durable-full-restart.entry.mjs`)
- **Setup:** real Redis and Postgres, separate API and worker processes,
`recovery.durableAgents: 'auto'`, and the same restart hook the deployer
runs at boot.
- **Scenario:** kill both processes mid-tool, then restart both.
- **Each case asserts that:**
  - the run reaches `workflow.end`
  - every tool call completes
  - the run owner lease and the thread lease are both released
- a follow-up run on the same thread is admitted with no
`AgentThreadLeaseConflictError`

| Case | main | this PR |
| --- | --- | --- |
| core: restart of nested loop body (guarded store) | ❌ hangs, leaf
`running` | ✅ |
| SIGKILL both mid-tool, x1 | ❌ run never settles (owner `live:true`,
snapshot `running`) | ✅ |
| SIGTERM both mid-tool, x1 | ✅ (graceful drain doesn't take the restart
path) | ✅ |
| SIGKILL both mid-tool, x2 (killed again after recovery, so the
snapshot must survive a second restart) | not run (x1 already fails) | ✅
|

Re-run on current `main` with the fix:
- core `vitest run src/workflows/evented`: 44 files, 682 passed / 5
skipped
- `durable-full-restart.test.ts`: 3/3 pass
- `tsc --noEmit` on core: clean
- redis-streams `tsc --noEmit` and `tsc --project tsconfig.test.json`
(the fixture typecheck that CI's Worker takeover job runs): clean
- `oxlint` on touched files: clean

### Combined with COR-1502

COR-1502 is a parallel fix on
`user/session-6ca2c63a-0a42-4fed-8de8-25f33217e319` (554226de) for Redis
`LOADING` errors during lease extension. I merged it locally with this
branch:
- the merge had no conflicts
- `durable-full-restart.test.ts`: all 3 cases pass on the merged code
- `evented-redis-loading.test.ts`: passes


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

If the API and worker restart during a run, a nested loop could get
stuck and keep its thread busy. This change preserves the loop’s stored
event information during restart so the run can finish and another run
can use the same thread.

## Changes

- Pass the nested loop leaf’s stored `startedBy` value when the parent
workflow restarts. Time-travel and resumed workflows still skip this
guard.
- Add a core regression test for restarting an evented workflow with
step guards enabled.
- Add a Redis Streams and Postgres integration test for API and worker
restarts. It covers `SIGKILL`, `SIGTERM`, and a second crash after
recovery. It checks run and tool completion, lease release, and a
follow-up run on the same thread.
- Add the integration fixture, include the test in the durable-streams
CI job, and add a patch changeset for `@mastra/core`.

## Tests

The diff adds regression and integration tests. Test-run and type-check
results are not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: review <review@local>
Co-authored-by: NikAiyer <nick.aiyer@gmail.com>
2026-11-04 18:59:50 -08:00
mastra-platform[bot]
a0068a9ad1
feat(core): resumable streams for every Agent (#28424)
> Opened by @user_01KNPFT2RPETZTEP15J3AEBA08

Originally stacked on
[#28416](https://github.com/mastra-ai/mastra/pull/28416), which has
merged. This PR now targets `main`.

## What

Any `Agent` can now have resumable streams, not only durable agents. A
run is published on `agent.stream.<runId>` with a replay cache, so:

- a client can reconnect after a disconnect,
- another reader can follow the run with `agent.observe(runId)`,
- any reader can stop it (`observe().abort()`,
`agent.abortRunStream(runId)`, or the abort-run route).

Durable execution (surviving process restarts) stays with `DurableAgent`
/ `EventedAgent` / `InngestAgent`, which keep publishing through the
same stream layer.

### Config

New `streams?: boolean | { pubsub, cache, shouldCache, cleanupTimeoutMs
}` on `Agent`.

- Omitted: on when the agent is registered on `Mastra`, off for
standalone agents.
- `false`: off. `{ cache: false }`: live only, no replay.
- With `durable`, `streams` keys override `durable` keys. `durable:
true` + `streams: false` keeps publishing, drops the replay cache, and
warns.

### Bus selection
- Registered agents (plain and `DurableAgent` on any engine) publish on
`mastra.pubsub`, so other processes sharing the PubSub and run-record
cache can observe and abort.
- Precedence: `streams.pubsub` > `durable.pubsub` > `mastra.pubsub`. The
evented engine still forces `mastra.pubsub` over an agent-level pubsub
and warns when they differ.
- Unregistered agents keep a private in-process `EventEmitterPubSub`.
- The replay cache wraps the shared bus once (existing `CachingPubSub`
reuse check), so nothing is double-published.

### Behaviour change

With streams on, closing the HTTP stream detaches the reader instead of
aborting the run. To stop a run explicitly, use `POST
/agents/:agentId/runs/:runId/abort`, which now works for plain agents
too (ownership-checked, including across servers sharing the PubSub and
run-record cache), or the thread abort route. `streams: false` keeps the
old behaviour.

## Commits

1. `8c8975ef` loop publishes chunks through a `streamPublisher`
(step-start, finish, CHUNK, SUSPENDED on close, ERROR if the stream ends
without a finish).
2. `f6d7dabf` `Agent.observe` reads the agent's bus;
`hasResumableStreams()` added. DurableAgent opts its inner agent out so
nothing is published twice.
3. `8b33a9d7` abort requests go over the control topic, so any reader on
the bus can stop a run.
4. `92a5a320` server: stream routes gate on `hasResumableStreams`,
observe reads a plain agent's bus, abort-run supports plain runs;
cleanup timers centralised on `AgentStreamBus.scheduleCleanup`.
5. `ff5456df` "Resumable streams" docs page, `Agent` reference entry,
changeset (core minor, server patch).
6. `24936904` registered `DurableAgent` on the default engine publishes
on `mastra.pubsub`. Before, if the bus was read before registration, it
stayed on the private bus and other instances never saw its runs.

## Rebase onto #28416 / main
- `loop/workflows/stream.ts` conflicted with #28157, which added a
`finalization` parameter to `workflowLoopStream`. Kept main's signature
and added `streamPublisher` alongside; snapshot finalization and
publishing both run.
- No textual conflict, but #28531 (follow-up to #28514) made stream
output a proxy, so `fullStream` overrides must call the base getter with
`this`. `Agent.observe` had the old pattern and failed with "Cannot read
private member #options"; main's fix is applied there (folded into
`f6d7dabf`). Reverting it fails both observe tests in
`agent-streams.test.ts`.

## Proof

Head `d2bd22d4`:
- Core `tsc -p tsconfig.build.json`, oxlint, oxfmt clean. `vitest run
src/agent/durable src/agent/streams
src/agent/__tests__/thread-stream-restart.test.ts` plus the
observe-approval-resume, observe-lifecycle and observe-reattach aimock
scenarios: 197 files, 1985 passed, 28 skipped, exit 0.
- Server `tsc` clean; `vitest run src/server/handlers`: 67 files, 2476
passed, 2 skipped, 1 todo.
- Inngest `tsc` clean; `test:unit` 103 passed, 2 failed. The same 2
tests ("stream/generate terminates when the opening event and error
event cannot be published") fail identically on `main` (`df76f896`,
rebuilt core).
- Docs: `remark --frail` and `oxfmt-mdx --check` clean.
- Observation spans suspension: new shared aimock scenario
`observe-approval-resume` on normal, durable, evented and fs engines. An
observer attached before and one attached during the approval wait both
receive the approved answer and `finish`; a fresh replay after
completion holds both segments; reconnecting after the suspension index
repeats nothing; the tool runs once; `onSuspended` fires once; a
cancelled reader leaves the run going.
- Request context: an unscoped Studio caller (auth mapper, Studio FGA,
no server FGA) can observe and abort; other tenants and unscoped
non-Studio callers still get 403; observe resolves the agent with the
request context.
- Shared ownership: each plain run writes a record
(`agent.run.<agentId>.<runId>`: agent, run, resource, thread,
running/suspended/finished) to the replay cache, or `mastra.serverCache`
when replay caching is off, before its first event. Suspended and
finished are written after the terminal event, the record is deleted
with the topic, and it has the run log's 24h TTL. Routes return 404 for
a missing, expired or evicted record and 503 when the cache can't be
read. Abort returns `aborted: true` only when the executing process
acknowledges the stop.
- Two real HTTP servers
(`pubsub/redis-streams/src/plain-run-cross-server.test.ts`: LibSQL file,
Redis Streams PubSub, RedisServerCache, SimpleAuth, aimock): owner
observes on B through suspend, approve on A, and finish; 403/401/404 on
both servers; abort through B confirmed by A; a parked run's abort is
`false`; expired and cleaned-up runs 404; ownership with `cache: false`.
Run with the durable/evented controls (`run-abort-cross-server`,
`observe-cross-process`): 20/20 in 3 consecutive runs. On the
pre-ownership commit the plain suite fails 8/8 (remote observe/abort
404).
- Mutation checks (each restored afterwards): re-adding `closeOnSuspend`
to plain observe fails the normal and fs scenarios; re-adding the HTTP
close on `suspended` fails the server test; dropping the request context
fails the Studio observe/abort and agent lookup tests; removing the
initial, suspended or delete record writes fails the record lifecycle
tests; removing the finished guard, the 503 handling or the
acknowledgement in the abort route fails their tests; acking an abort
that reached an ended run fails the new publisher test.
- No double publishing: the parity, durable-shared-bus and agent-streams
tests still pass (DurableAgent's inner agent keeps `streams: false`; the
replay cache wraps the shared bus once).

## Separate main-merge verification

Conflict resolution is isolated in
[50a9ef89](50a9ef8947),
after candidate `d2bd22d4`; no feature commits were rewritten.

- Merged-result focused core regression suites: 6 files, 37 tests
passed, covering parity, shared buses/run records, remote abort and
approval/resume observation.
- Complete merged-result server-handler suite: 68 files, 2,478 passed, 2
skipped, 1 todo. A separate real-TCP Hono/real-Memory authorization
matrix also passed 13/13, including the formerly failing Studio-only FGA
observe and abort contracts (without expected-failure markers).
- Type-aware oxlint and formatting passed on all 5 conflict-resolved
files.
- Full core/server typechecks are **not clean in the verification
checkout**. Swapping each package to unchanged `main` source in the same
installed dependency environment reproduces the same diagnostics (core
byte-identical; server identical after normalizing source-line
positions). These include AI SDK schema compatibility errors and missing
agent-builder artifacts; no additional diagnostics were introduced by
the conflict resolution.
- A broader core run passed 2,020 tests but exited nonzero: 5 suites
initially lacked the built internal LLM recorder; after building it, all
5 suites passed (24 tests). The crash-recovery worker still exhausted
its JavaScript heap after 8 of 10 tests. The broad run also reported the
baseline type errors above. This is **not** a clean full-core result.
- Follow-up `ec0bc59b` clears control and acknowledgement topics with
the finished run log; `35debe7a` and `920a094a` correct documentation.
Independent verification below includes the cleanup production change.

## Independent adversarial review

Verdict: **approve within the requested scope**, with no actionable
supported-flow defect found. No formal GitHub approval was posted. The
reviewed production head is `ec0bc59b32`;
the only subsequent change in `920a094a` is a checked docs sentence
distinguishing expired ownership from missing replay history.

Independent reviewer ran **343 passing tests**: 38 focused core, 280
server handlers, 8 real two-process TCP/Redis/LibSQL/AIMock cases, 16
public-API processor-safety probes, and 1 actual Redis cleanup probe.

- Redaction/tripwire bypass hypothesis did not reproduce on normal,
durable, evented or filesystem engines, with either immediate or
deferred initiating-stream consumption. Serialized observer frames
excluded the raw sensitive value; redaction replay matched the masked
caller output, and tripwires exposed no raw text. No processor redesign
was made.
- Actual Redis cleanup was tested after an acknowledged cross-node
abort: physical run, control, acknowledgement, counter/list and
ownership keys were first observed, then all ten run-associated keys
disappeared after configured cleanup. This verifies more than the added
clearTopic spy assertion.
- The reviewer used source aliases for the exact production sources in a
disposable checkout, including both child HTTP processes. They did not
independently validate the full build, Inngest, browser Studio Stop,
natural 24-hour expiry or load costs.

## Unverified

- Studio Stop button for thread-less plain runs: not exercised in a
browser.
- Cost: every registered agent publishes each chunk and writes an
ownership record. Not benchmarked.
- Natural 24h expiry: tested by deleting the record, not by waiting.
Merged main now honors the requested minimum TTL for indexed in-memory
logs. Entry-cap eviction can still shorten replay; docs were corrected
in `35debe7a`.
- Abort of a run whose executing process died: returns `aborted: false`
after the ack timeout; the run's record stays until its TTL.

## Assumptions

- InngestAgent keeps its own pubsub/cache resolution. Importing a new
core export would fail `check-core-imports` until core is released, so
no Inngest change or peer-range bump here (and no inngest changeset).
Follow-up after release.
- thread-stream-restart's "leaves nothing" assertion now ignores the
agent stream topic, since run events are kept for `cleanupTimeoutMs` by
design.

## Decisions (confirmed by Tony)

1. Registered `DurableAgent`s on the default engine use `mastra.pubsub`,
same as plain agents; unregistered ones keep a private bus.
2. Streams are on by default for agents registered on `Mastra` and off
for standalone agents; docs explain the cost and how to turn streams
off.
3. With streams on, HTTP disconnect detaches instead of aborting.

## Open question

The Superagent security scan flags (3): a client can start expensive
runs and disconnect, and nothing limits active or concurrent detached
runs. Options: accept and document, cap detached runs, or keep
abort-on-disconnect for HTTP-started runs.

Risk: medium-high — every registered agent now publishes, HTTP
disconnect no longer aborts plain runs, and observe/abort now authorize
plain runs from a shared cache record.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

If someone disconnects while an agent is working, the agent can keep
running and save its updates so the reader can reconnect. Authorized
readers can also check on or stop a run; this does not add crash
recovery to plain agents.

## What changed

- Added configurable resumable streams for plain `Agent` runs. Streams
default to on for agents registered with `Mastra` and off for standalone
agents.
- Added `observe()` replay and live updates, including observation from
an event offset. Added abort support through `observe().abort()`,
`abortRunStream(runId)`, and server routes.
- When streams are enabled, an HTTP disconnect detaches the reader
instead of aborting the run. For plain agents, `streams: false` disables
publishing. Durable agents continue to publish; `streams: false`
disables their replay cache.
- Added run ownership records and authorization checks. Shared PubSub
and cache support observation and abort across servers. Unknown or
unverified runs return 404; cache-read failures return 503.
- Added run-ID claim checks. A run ID already owned by another resource
is rejected with `AGENT_RUN_ID_ALREADY_CLAIMED` and maps to HTTP 409.
Concurrent claims across processes may still overlap because the cache
has no compare-and-set operation.
- Added cleanup timers that do not keep the process alive. Cache TTL and
eviction can also limit event retention.
- Updated step bookkeeping to use the run’s original owner when
execution replaces its abort signal.
- Added documentation, tests, and a Redis/LibSQL cross-server
integration suite.

## Verification and open questions

The supplied context reports that the merged-result server-handler suite
passed 2,478 tests, with 2 skipped and 1 todo. It also reports real
two-server Redis/LibSQL tests, but provides no exact result count. Full
core and server typechecks were not clean; the reported diagnostics were
reproduced against unchanged `main`. A broader core run stopped after 8
of 10 tests when the crash-recovery worker exhausted its heap. Two
Inngest unit-test failures were also reported on `main`.

The author flagged that detached runs can continue without limits on
active or concurrent runs. Browser testing of Studio Stop for
thread-less plain runs, performance costs, natural 24-hour expiry, and
abort behavior after the executing process dies remain unverified. An
abort request cannot be confirmed if the process running the run is
unavailable.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: NikAiyer <nick.aiyer@gmail.com>
Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: review <review@local>
2026-11-04 17:29:16 -08:00
dane-ai-mastra[bot]
675caf0613
chore: version packages (alpha) (#29336)
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases

> The changelog information of each package has been omitted from this
message, as the content exceeds the size limit.

## @mastra/client-js@1.60.0-alpha.0


## @mastra/react@1.15.0-alpha.0


## @mastra/factory@0.28.0-alpha.0


## @mastra/code-sdk@1.19.0-alpha.0


## mastra@1.39.0-alpha.0


## @mastra/core@1.83.0-alpha.0


## @mastra/mcp@2.9.0-alpha.0


## @mastra/mcp-apps-components@0.1.0-alpha.0


## @mastra/playground-ui@69.0.0-alpha.0


## @mastra/server@1.83.0-alpha.0


## @mastra/nestjs@0.3.0-alpha.0


## @mastra/clickhouse@1.30.0-alpha.0


## @mastra/duckdb@1.19.0-alpha.0


## @mastra/libsql@1.30.0-alpha.0


## @mastra/pg@1.36.0-alpha.0


## @mastra/temporal@0.6.0-alpha.0


## @mastra/auth-better-auth@1.1.11-alpha.0


## @mastra/deployer-cloud@1.83.0-alpha.0


## @mastra/deployer-cloudflare@1.2.42-alpha.0


## @mastra/deployer-netlify@1.2.42-alpha.0


## @mastra/deployer-sandbox@0.3.27-alpha.0


## @mastra/deployer-vercel@1.2.42-alpha.0


## @mastra/longmemeval@1.1.42-alpha.0


## @mastra/opencode@0.1.42-alpha.0


## create-factory@0.4.3-alpha.0


## mastracode@0.49.1-alpha.0


## @mastra/arize@1.3.30-alpha.0


## @mastra/arthur@0.4.30-alpha.0


## @mastra/braintrust@1.3.27-alpha.0


## @mastra/datadog@1.4.22-alpha.0


## @mastra/deepeval@0.1.24-alpha.0


## @mastra/laminar@1.3.32-alpha.0


## @mastra/langfuse@1.6.6-alpha.0


## @mastra/langsmith@1.4.3-alpha.0


## @mastra/observability@1.23.2-alpha.0


## @mastra/otel-bridge@1.5.22-alpha.0


## @mastra/otel-exporter@1.5.2-alpha.0


## @mastra/posthog@1.4.7-alpha.0


## @mastra/sentry@1.2.32-alpha.0


## @mastra/agent-builder@1.2.1-alpha.0


## @mastra/connect@1.3.2-alpha.0


## create-mastra@1.39.0-alpha.0


## @mastra/deployer@1.83.0-alpha.0


## @mastra/editor@0.17.1-alpha.0


## @mastra/mcp-docs-server@1.3.13-alpha.0


## @mastra/memory@1.41.1-alpha.0


## @mastra/rag@2.7.3-alpha.0


## @mastra/redis-streams@0.8.1-alpha.0


## @mastra/elysia@0.1.23-alpha.0


## @mastra/express@1.5.27-alpha.0


## @mastra/fastify@1.5.27-alpha.0


## @mastra/hono@1.7.25-alpha.0


## @mastra/koa@1.7.27-alpha.0


## @mastra/next@0.2.41-alpha.0


## @mastra/tanstack-start@0.2.41-alpha.0


## @mastra/spanner@1.11.2-alpha.0


## @mastra/turso@0.1.21-alpha.0


## @mastra/inngest@1.11.2-alpha.0


## @mastra/gcs@0.3.6-alpha.0


## @internal/playground@1.39.0-alpha.0

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-11-04 14:40:27 -08:00
mastra-platform[bot]
49eaabb19b
fix(core,redis-streams): recover evented runs when Redis is LOADING mid-tool (COR-1502) (#29430)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9

## Summary

On the evented engine, if Redis restarts while a tool step is finishing,
it spends a while (often 30–60s) reloading its dataset. During that time
it answers every command with `LOADING Redis is loading the dataset in
memory`. When that happens, the run stays `running` forever and no
answer is saved. This PR makes the run wait out the restart and finish
normally: the answer is saved and each tool runs exactly once.

Fixes COR-1502.

## Root cause: three stacked problems

1. **Publishing never retried.** `RedisStreamsPubSub.publish()` sent the
tool result and `step.end` once. During `LOADING` that failed, so the
step was redelivered and its tools ran a second time.
2. **The thread lease expired during the window.** Redis kept counting
down the lease while it reloaded. On redelivery, Postgres still renewed
the run ownership, but `renewDurableThreadLease` found no lease key and
returned `false`. The execution fence then refused every redelivery
("could not verify that this execution still owns the run"). The event
was dropped after 5 deliveries and the run stayed `running`.
3. **The lease retake gave up when the step went idle.** `#retakeLease`
in `thread-stream-runtime` stopped trying once the tool step finished.
The run was aborted as lease-lost (`reason: 'abort'`, empty answer)
before the final model turn.

## Fix

- **`@mastra/redis-streams`:**
- `publish` retries `LOADING` errors every 250ms for up to 2 minutes,
and stops if the client closes. This covers both plain `XADD` and
`MULTI`/`EXEC` publishes.
- It implements the new `isUnavailableToAll(error)`, which returns true
for `LOADING`.
- **`@mastra/core` `events/pubsub.ts`:** adds an optional
`LeaseProvider.isUnavailableToAll?(error)`. A backend returns true when
it is refusing every client, so nobody else can be granted the lease
meanwhile. Any other error, including a connection failure, must return
`false`.
- **`@mastra/core` execution fence:** this applies only when Postgres
run ownership has just renewed for a generation-fenced claim.
- If the thread lease can't be renewed because the key is gone, the
fence takes it back with `SET NX`. It never takes a lease that another
owner holds.
- If the backend is refusing every client, the fence relies on the
Postgres run ownership instead.
  - Any other lease error still fences the execution.
- **`@mastra/core` thread-stream runtime:** while the backend refuses
every client, `#retakeLease` keeps asking, within the existing
`AGENT_LEASE_LOST_TOOL_GRACE_MS`. Generic errors keep the existing
strict lease-loss policy, which stays safe for network partitions.

## Tests

- New test `pubsub/redis-streams/src/evented-redis-loading.test.ts`. It
runs separate API and worker processes against real Redis and Postgres.
They connect through a TCP proxy (`makeLoadingProxy` in
`test-fixtures/harness.ts`) that simulates a restart: it drops every
connection, then answers `-LOADING` for 20s while two tools finish.
During the window, `MULTI`/`EXEC` behave as in real Redis (`EXEC` →
`EXECABORT`).
- **Red on main:** the run never reached a terminal state, and the
worker looped on the fence error.
- **Green with the fix (~30s):** the run completes, the answer is saved,
each tool runs exactly once in `tool_executions`, and run ownership is
released.
- Related suites pass:
  - core: execution-fence, thread-lease, and thread-stream suites.
- redis-streams: `pubsub`, `agent-thread-cross-process`,
`durable-worker-takeover`, `fail-publish-after-exhaustion`,
`local-redelivery-evented`.
- `tsc` (core, redis-streams, redis-streams test config) and `oxlint`
are clean on the touched files.
- Validated together with the COR-1497 branch
(`user/session-90e601e9-…`, `7c3d66f9`): the local merge had no
conflicts, and `durable-full-restart` (3 cases) plus this LOADING test
all pass.

## Out of scope / follow-up

On success, the workflow engine still publishes `workflow.step.end`
before it saves the step result. Any publish failure other than
`LOADING` still leaves a window where a redelivered `step.run` finds no
saved result and runs its tools again. Saving the result before
publishing would close that window. It is a broader engine change, so I
left it for a separate ticket.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

When Redis restarts, it can temporarily refuse requests while it reloads
data. This change lets evented runs wait for Redis and recover their
thread lease, so they can finish without repeating completed tools.

## What changed

- `@mastra/redis-streams` retries stream publishing when Redis returns
`LOADING`, up to two minutes. Other errors still fail immediately.
- `LeaseProvider` can optionally implement `isUnavailableToAll(error)`
to identify errors that mean the backend is refusing every client.
- After Postgres run ownership is renewed, the execution fence can
reclaim a missing thread lease. If the backend reports that it is
refusing all clients, the fence relies on run ownership. Other lease
errors still prevent execution.
- Thread-lease retakes can continue during backend-wide unavailability
within the existing grace period.
- Added an integration test for a Redis loading window during tool
execution. The test checks that the run completes, saves its answer,
executes each tool once, and releases run ownership.

## Tests

The PR description reports that the integration test passes against real
Redis and Postgres. It also reports passing related core and Redis
Streams suites, TypeScript checks, and `oxlint` on touched files. These
results were not independently verified here.

## Known limitation

The PR description notes that a workflow step may still rerun tools on
redelivery if publishing its result fails for an error other than
`LOADING`, because the result is published before it is saved.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-04 13:07:20 -08:00
dane-ai-mastra[bot]
6e8dfcb83f chore: version - exit prerelease mode 2026-11-03 23:45:58 +00:00
mastra-platform[bot]
ab4d89cd38
test(redis-streams): run durable worker-takeover suite against real Redis and Postgres (#28952)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9

Part 3 of 3, split from #28676. Refs COR-1490. Parts 1 and 2 (#28947,
#28951) are merged into `main`. This PR is based on `main`.

## What this does
The API-loss and worker-takeover guarantees from #28951 only hold with a
shared store that supports run fencing and a transport that works across
processes. In-memory suites can't test that. This PR:
- adds `.github/workflows/test-durable-streams.yml`, which runs the
two-process takeover suite against Redis (`:6381`) and Postgres
(`:5436`) service containers. The checkout uses `persist-credentials:
false`.
- calls it from `test-suite.yml` as the `durable-streams` job. The job
reports its own check but does not gate `Merge Test Reports`, because
its flake rate hasn't been measured yet.
- runs it when `prebuild.yml` sees changes to the evented agent and
workflow code, `packages/core/src/mastra/`, `packages/memory/`,
`pubsub/redis-streams/`, `stores/pg/`, or the workflow files
- extends `pubsub/redis-streams/src/durable-worker-takeover.test.ts` and
adds the `durable-api-loss.entry.mjs` fixture. The scenarios: API
SIGTERM/SIGKILL with remote workers, a thread follow-up after API loss,
a remote abort, worker shutdown, single-process Redis shutdown, and the
original caller completing with and without a survivor that is already
running
- adds `tsconfig.test.json` and a `check:test` script, which type-check
the tests and fixtures against core's `dist`

No changeset: this PR only touches CI and tests.

## Verification
- Reviewer run at `bed900442c` against Redis and Postgres: the
worker-takeover suite passes 11/11. Reverting #28951's production
changes makes 7 of 11 fail.
- `check:test` exits 0 with the single config and covers all 12 package
files, including `agent-thread-worker.entry.ts`.

Risk: low. CI and tests only.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## ELI5

This PR adds a CI check that runs worker-takeover tests with real Redis
and Postgres. The tests check whether durable work can continue or
recover when the API or a worker stops.

## Summary

- Adds a reusable GitHub Actions workflow with Redis 8 and
pgvector/Postgres 16 services.
- Updates `prebuild.yml` to detect changes that affect durable streams
and pass that result to `test-suite.yml`.
- Adds a `durable-streams` job that runs when the relevant code changes.
- Adds takeover tests for API `SIGTERM` and `SIGKILL`, follow-up
messages, remote aborts, worker shutdown, Redis shutdown, and caller
completion.
- Adds a fixture that records tool execution state in Postgres and uses
Redis Streams for pub/sub.
- Adds `check:test` and TypeScript test configuration for typechecking
the fixtures separately from the package build.

The author reports that `check:test` passed and the takeover suite
passed 11/11 against Postgres and Redis 8. The author also reports that
reverting `packages/core/src` to its state before `#28951` causes the
API `SIGTERM` takeover test to fail. The PR author states that no
changeset is required.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: NikAiyer <nick.aiyer@gmail.com>
Co-authored-by: Nik Aiyer <nik@mastra.ai>
Co-authored-by: review <review@local>
Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
2026-11-03 11:41:25 -08:00
Dane - AI (Mastra)
3bdb4ec5f4
chore(deps): update build tools (major) (#29026)
> [!NOTE]
> This PR body was truncated due to platform limits.

This PR contains the following updates:

| Package | Type | Update | Change | Pending |
|---|---|---|---|---|
| [@vitest/coverage-v8](https://vitest.dev/guide/coverage)
([source](https://redirect.github.com/vitest-dev/vitest/tree/HEAD/packages/coverage-v8))
| pnpm.catalog.default | major | [`4.1.12` →
`5.0.5`](https://renovatebot.com/diffs/npm/@vitest%2fcoverage-v8/4.1.12/5.0.5)
| `5.1.0` |
| [@vitest/mocker@>=2.1.0
<4.1.12](https://redirect.github.com/vitest-dev/vitest/tree/main/packages/mocker)
([source](https://redirect.github.com/vitest-dev/vitest/tree/HEAD/packages/mocker))
| pnpm-workspace.overrides | major | [`4.1.12` →
`5.0.5`](https://renovatebot.com/diffs/npm/@vitest%2fmocker@>=2.1.0
<4.1.12/4.1.12/5.0.5) | `5.1.0` |
| [@vitest/ui](https://vitest.dev/guide/ui)
([source](https://redirect.github.com/vitest-dev/vitest/tree/HEAD/packages/ui))
| pnpm.catalog.default | major | [`4.1.12` →
`5.0.5`](https://renovatebot.com/diffs/npm/@vitest%2fui/4.1.12/5.0.5) |
`5.1.0` |
| [dotenv](https://redirect.github.com/motdotla/dotenv) |
devDependencies | major | [`^17.4.2` →
`^18.0.9`](https://renovatebot.com/diffs/npm/dotenv/17.4.2/18.0.9) | |
| [dotenv](https://redirect.github.com/motdotla/dotenv) |
devDependencies | major | [`17.4.2` →
`18.0.9`](https://renovatebot.com/diffs/npm/dotenv/17.4.2/18.0.9) | |
| [vitest](https://vitest.dev)
([source](https://redirect.github.com/vitest-dev/vitest/tree/HEAD/packages/vitest))
| devDependencies | major | [`^4.1.12` →
`^5.0.5`](https://renovatebot.com/diffs/npm/vitest/4.1.12/5.0.5) |
`5.1.0` |
| [vitest](https://vitest.dev)
([source](https://redirect.github.com/vitest-dev/vitest/tree/HEAD/packages/vitest))
| pnpm.catalog.default | major | [`4.1.12` →
`5.0.5`](https://renovatebot.com/diffs/npm/vitest/4.1.12/5.0.5) |
`5.1.0` |
| [vitest](https://vitest.dev)
([source](https://redirect.github.com/vitest-dev/vitest/tree/HEAD/packages/vitest))
| devDependencies | major | [`4.1.12` →
`5.0.5`](https://renovatebot.com/diffs/npm/vitest/4.1.12/5.0.5) |
`5.1.0` |
| [vitest@>=2.1.0 <4.1.12](https://vitest.dev)
([source](https://redirect.github.com/vitest-dev/vitest/tree/HEAD/packages/vitest))
| pnpm-workspace.overrides | major | [`4.1.12` →
`5.0.5`](https://renovatebot.com/diffs/npm/vitest@>=2.1.0
<4.1.12/4.1.12/5.0.5) | `5.1.0` |

---

> [!WARNING]
> Some dependencies could not be looked up. Check the [Dependency
Dashboard](../issues/9505) for more information.

---

### Release Notes

<details>
<summary>vitest-dev/vitest (@&#8203;vitest/coverage-v8)</summary>

###
[`v5.0.5`](https://redirect.github.com/vitest-dev/vitest/releases/tag/v5.0.5)

[Compare
Source](https://redirect.github.com/vitest-dev/vitest/compare/v5.0.4...v5.0.5)

#####    🐞 Bug Fixes

- Keep vite default `fs.deny` with api token file  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **Claude Opus 5.5** in
[#&#8203;11578](https://redirect.github.com/vitest-dev/vitest/issues/11578)
[<samp>(28dc1)</samp>](28dc1edfb)
- Handle quoted file names in `--changed`  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11583](https://redirect.github.com/vitest-dev/vitest/issues/11583)
[<samp>(14921)</samp>](14921e962)
- Add `data` to the `import.meta.hot` stub  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11590](https://redirect.github.com/vitest-dev/vitest/issues/11590)
[<samp>(3a828)</samp>](3a8289ac3)
- Set `skip` in `skipIf`, ignore `undefined` options  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11591](https://redirect.github.com/vitest-dev/vitest/issues/11591)
[<samp>(d523a)</samp>](d523a15f2)
- Skip builtin loading in async leak detector  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11598](https://redirect.github.com/vitest-dev/vitest/issues/11598)
[<samp>(fe86a)</samp>](fe86a64c6)
- Keep the import frame in native loader errors  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11599](https://redirect.github.com/vitest-dev/vitest/issues/11599)
[<samp>(9697b)</samp>](9697b3894)
- Run hook cleanups when a later hook fails  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11605](https://redirect.github.com/vitest-dev/vitest/issues/11605)
[<samp>(6c5a4)</samp>](6c5a44720)
- Release mocks of finished test files  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11607](https://redirect.github.com/vitest-dev/vitest/issues/11607)
[<samp>(e9c3d)</samp>](e9c3d72f0)
- Abort all running tests on cancel  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11606](https://redirect.github.com/vitest-dev/vitest/issues/11606)
[<samp>(6621a)</samp>](6621a72d6)
- **browser**:
- Ignore synthetic pagehide events  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11582](https://redirect.github.com/vitest-dev/vitest/issues/11582)
[<samp>(ca081)</samp>](ca0813584)
- Run plugin buildStart on server creation  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11593](https://redirect.github.com/vitest-dev/vitest/issues/11593)
[<samp>(ea611)</samp>](ea611e44c)
- Drop late connections of closed sessions  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11600](https://redirect.github.com/vitest-dev/vitest/issues/11600)
[<samp>(f161e)</samp>](f161e6df2)
- Wait 30s for the browser to close  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11601](https://redirect.github.com/vitest-dev/vitest/issues/11601)
[<samp>(6498b)</samp>](6498b666a)
- Run global setup once with instances  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11596](https://redirect.github.com/vitest-dev/vitest/issues/11596)
[<samp>(8dd67)</samp>](8dd67c262)
- **reporter**:
- End reason `&#&#8203;39;failed&#&#8203;39;` on worker crash  -  by
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;11568](https://redirect.github.com/vitest-dev/vitest/issues/11568)
[<samp>(6ae8e)</samp>](6ae8e89ff)

#####     [View changes on
GitHub](https://redirect.github.com/vitest-dev/vitest/compare/v5.0.4...v5.0.5)

###
[`v5.0.4`](https://redirect.github.com/vitest-dev/vitest/releases/tag/v5.0.4)

[Compare
Source](https://redirect.github.com/vitest-dev/vitest/compare/v5.0.3...v5.0.4)

#####    🚀 Features

- **experimental**:
- Tag cache directories with CACHEDIR.TAG  -  by
[@&#8203;lifeiscontent](https://redirect.github.com/lifeiscontent) in
[#&#8203;11461](https://redirect.github.com/vitest-dev/vitest/issues/11461)
[<samp>(a8ea3)</samp>](a8ea3c8fd)
- Support `test.ui.theme` option  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11499](https://redirect.github.com/vitest-dev/vitest/issues/11499)
[<samp>(1da66)</samp>](1da66405d)
- Highlight selector multiple matches in trace view  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **Claude Opus 5.5** in
[#&#8203;11489](https://redirect.github.com/vitest-dev/vitest/issues/11489)
[<samp>(5abba)</samp>](5abba1e1b)
- Print how many tests were affected and a total when running
`--changed`  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11424](https://redirect.github.com/vitest-dev/vitest/issues/11424)
[<samp>(a2336)</samp>](a23368585)
- Add an option to cache dependencies instead of statically parsing them
 -  by [@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11515](https://redirect.github.com/vitest-dev/vitest/issues/11515)
[<samp>(c0601)</samp>](c06012ac8)
- Add fit and zoom controls to trace viewer  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **Claude Opus 5.5** in
[#&#8203;11490](https://redirect.github.com/vitest-dev/vitest/issues/11490)
[<samp>(f5dd9)</samp>](f5dd99ed0)

#####    🐞 Bug Fixes

- Fail the run on git and location filter errors  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11430](https://redirect.github.com/vitest-dev/vitest/issues/11430)
[<samp>(30d03)</samp>](30d0304d2)
- Enable fsModuleCache by default  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11435](https://redirect.github.com/vitest-dev/vitest/issues/11435)
[<samp>(ad208)</samp>](ad208cc29)
- Track project files and mocks with --changed  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11432](https://redirect.github.com/vitest-dev/vitest/issues/11432)
[<samp>(b0f2b)</samp>](b0f2b3bb9)
- Refactor caching, fix `:` in project and change the shape of `results`
cache  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11453](https://redirect.github.com/vitest-dev/vitest/issues/11453)
[<samp>(9673c)</samp>](9673c49a4)
- Keep full `$` string values when `taskTitleValueFormatTruncate` is 0
 -  by [@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa),
**Hiroshi Ogawa** and **Claude Opus 5.5** in
[#&#8203;11485](https://redirect.github.com/vitest-dev/vitest/issues/11485)
[<samp>(4f7e1)</samp>](4f7e1799b)
- Resolve node builtins in client environments  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11417](https://redirect.github.com/vitest-dev/vitest/issues/11417)
[<samp>(56b31)</samp>](56b319ac6)
- Fix module graph builtin externals after
[#&#8203;11421](https://redirect.github.com/vitest-dev/vitest/issues/11421)
 -  by [@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa),
**Hiroshi Ogawa** and **Claude Opus 5.5** in
[#&#8203;11491](https://redirect.github.com/vitest-dev/vitest/issues/11491)
and
[#&#8203;11421](https://redirect.github.com/vitest-dev/vitest/issues/11421)
[<samp>(008e1)</samp>](008e184cb)
- Wrap module mocks during in-source testing  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11505](https://redirect.github.com/vitest-dev/vitest/issues/11505)
[<samp>(36d19)</samp>](36d19cb00)
- Handle line endings in snapshot  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11522](https://redirect.github.com/vitest-dev/vitest/issues/11522)
[<samp>(4c468)</samp>](4c468aa40)
- Respect root in projects  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11521](https://redirect.github.com/vitest-dev/vitest/issues/11521)
[<samp>(26642)</samp>](26642f793)
- Resolve mocks in directories with spaces  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11539](https://redirect.github.com/vitest-dev/vitest/issues/11539)
[<samp>(caaef)</samp>](caaef9416)
- **api**:
- Publish the api token atomically  -  by
[@&#8203;umairnawaz333](https://redirect.github.com/umairnawaz333) in
[#&#8203;11323](https://redirect.github.com/vitest-dev/vitest/issues/11323)
[<samp>(6f990)</samp>](6f990fef1)
- **browser**:
- Close the mock route interception race  -  by
[@&#8203;cplieger](https://redirect.github.com/cplieger) and
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;8339](https://redirect.github.com/vitest-dev/vitest/issues/8339)
and
[#&#8203;11083](https://redirect.github.com/vitest-dev/vitest/issues/11083)
[<samp>(d6620)</samp>](d6620fae6)
- Allow toHaveTextContent without arguments  -  by
[@&#8203;nomadical](https://redirect.github.com/nomadical) in
[#&#8203;11425](https://redirect.github.com/vitest-dev/vitest/issues/11425)
[<samp>(8ebcf)</samp>](8ebcf5f8a)
- Apply API options without browser config  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10852](https://redirect.github.com/vitest-dev/vitest/issues/10852)
[<samp>(ad055)</samp>](ad055837b)
- Preserve undefined command arguments  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) and
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;11497](https://redirect.github.com/vitest-dev/vitest/issues/11497)
[<samp>(1576d)</samp>](1576d3d89)
- Wait for setup mocks before importing a test file  -  by
[@&#8203;kasperpeulen](https://redirect.github.com/kasperpeulen) in
[#&#8203;11520](https://redirect.github.com/vitest-dev/vitest/issues/11520)
[<samp>(b61d1)</samp>](b61d1ced3)
- Avoid extra scroll outside the iframe  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11523](https://redirect.github.com/vitest-dev/vitest/issues/11523)
[<samp>(662e8)</samp>](662e8a363)
- Run browser instances as workers  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11525](https://redirect.github.com/vitest-dev/vitest/issues/11525)
[<samp>(bf04b)</samp>](bf04b9ccc)
- ⚠️ This can degrade performance if you have a custom `maxWorkers`
setting. Previously, it was applied incorrectly and could lead to Vitest
using too many resources. For example, a lower number would still open
too many browsers. Fixing it means Vitest opens less instances by
default now.
- Open one cdp session per tester  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11542](https://redirect.github.com/vitest-dev/vitest/issues/11542)
[<samp>(69e63)</samp>](69e6369f6)
- Clean up unmocked and pending mocks  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11537](https://redirect.github.com/vitest-dev/vitest/issues/11537)
[<samp>(5c955)</samp>](5c9552173)
- **coverage**:
- V8 browser memory leak between test files  -  by
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;11466](https://redirect.github.com/vitest-dev/vitest/issues/11466)
[<samp>(2297d)</samp>](2297d93a8)
- `allowExternal` fails to include uncovered files  -  by
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;11513](https://redirect.github.com/vitest-dev/vitest/issues/11513)
[<samp>(4ef00)</samp>](4ef000a93)
- Filenames containing brackets incorrectly showing as uncovered  -  by
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;11531](https://redirect.github.com/vitest-dev/vitest/issues/11531)
[<samp>(0fa80)</samp>](0fa80f53c)
- Istanbul coverage wrong when module transform warmups  -  by
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;11529](https://redirect.github.com/vitest-dev/vitest/issues/11529)
[<samp>(9f61d)</samp>](9f61dcca2)
- **deps**:
- Update all non-major dependencies  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11167](https://redirect.github.com/vitest-dev/vitest/issues/11167)
[<samp>(c74e0)</samp>](c74e099a9)
- **expect**:
- Fix `Map` and `Set` deep equality in some assertions  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **Claude Opus 5.5** in
[#&#8203;11402](https://redirect.github.com/vitest-dev/vitest/issues/11402)
[<samp>(b4e79)</samp>](b4e79fd7a)
- Check value in chai-style returned  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11500](https://redirect.github.com/vitest-dev/vitest/issues/11500)
[<samp>(1aca0)</samp>](1aca02a21)
- **mocker**:
- Fix native loader mocking edge cases  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11540](https://redirect.github.com/vitest-dev/vitest/issues/11540)
[<samp>(cd423)</samp>](cd423ba04)
- **pretty-format**:
- Support boxed symbols  -  by
[@&#8203;lehni](https://redirect.github.com/lehni) in
[#&#8203;11445](https://redirect.github.com/vitest-dev/vitest/issues/11445)
and
[#&#8203;11446](https://redirect.github.com/vitest-dev/vitest/issues/11446)
[<samp>(eba44)</samp>](eba44926e)
- **reporters**:
- Junit to strip ANSI sequences from all XML attributes  -  by
[@&#8203;meganemura](https://redirect.github.com/meganemura) and
**Claude Opus 5.5** in
[#&#8203;11407](https://redirect.github.com/vitest-dev/vitest/issues/11407)
[<samp>(e1a73)</samp>](e1a731470)
- **runner**:
- Don't cache failed fixture setups  -  by
[@&#8203;hamed-bavar](https://redirect.github.com/hamed-bavar) in
[#&#8203;11237](https://redirect.github.com/vitest-dev/vitest/issues/11237)
and
[#&#8203;11238](https://redirect.github.com/vitest-dev/vitest/issues/11238)
[<samp>(4ee66)</samp>](4ee66c021)
- **typecheck**:
- Use typescript pool in toTestSpecification  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11451](https://redirect.github.com/vitest-dev/vitest/issues/11451)
[<samp>(5f273)</samp>](5f2739e74)
- **ui**:
- Remove duplicate browser-open call opening two tabs  -  by
[@&#8203;feastyr69](https://redirect.github.com/feastyr69) in
[#&#8203;11356](https://redirect.github.com/vitest-dev/vitest/issues/11356)
and
[#&#8203;11358](https://redirect.github.com/vitest-dev/vitest/issues/11358)
[<samp>(d7e13)</samp>](d7e1315c1)
- Simplify explorer filtering + fix nested tests when search matches a
suite  -  by [@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa),
**Hiroshi Ogawa**, **Codex** and **Claude Opus 5.5** in
[#&#8203;11262](https://redirect.github.com/vitest-dev/vitest/issues/11262)
[<samp>(a627b)</samp>](a627be498)
- Store module graph once per environment  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **Claude Opus 5.5** in
[#&#8203;11421](https://redirect.github.com/vitest-dev/vitest/issues/11421)
[<samp>(af903)</samp>](af903c65e)
- Mount index.html handler on the ui base path  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **Claude Opus 5.5** in
[#&#8203;11517](https://redirect.github.com/vitest-dev/vitest/issues/11517)
[<samp>(1bc24)</samp>](1bc2495c7)
- Center split pane resize handle  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **Claude Opus 5.5** in
[#&#8203;11506](https://redirect.github.com/vitest-dev/vitest/issues/11506)
[<samp>(e658b)</samp>](e658b9b6a)
- **vm**:
- Bind `this` to `module.exports`  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11504](https://redirect.github.com/vitest-dev/vitest/issues/11504)
[<samp>(b618a)</samp>](b618afd5c)

#####    🏎 Performance

- Reuse fsModuleCache when filtering by source  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11428](https://redirect.github.com/vitest-dev/vitest/issues/11428)
[<samp>(7ebd9)</samp>](7ebd95fdd)
- **spy**: Reset only mocks that were called or reconfigured  -  by
[@&#8203;johan-gorter](https://redirect.github.com/johan-gorter) and
**Claude Opus 5.5** in
[#&#8203;11197](https://redirect.github.com/vitest-dev/vitest/issues/11197)
[<samp>(28ef9)</samp>](28ef93df2)

#####     [View changes on
GitHub](https://redirect.github.com/vitest-dev/vitest/compare/v5.0.3...v5.0.4)

###
[`v5.0.3`](https://redirect.github.com/vitest-dev/vitest/releases/tag/v5.0.3)

[Compare
Source](https://redirect.github.com/vitest-dev/vitest/compare/v5.0.2...v5.0.3)

#####    🐞 Bug Fixes

- Isolate `result.status` between `repeats` runs  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **Codex (GPT-6)** in
[#&#8203;11218](https://redirect.github.com/vitest-dev/vitest/issues/11218)
[<samp>(5dbeb)</samp>](5dbebe9e3)
- Don't print an interceptor warning in browser mode  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11377](https://redirect.github.com/vitest-dev/vitest/issues/11377)
[<samp>(15cc0)</samp>](15cc006aa)
- Don't retry when `test.fails` expectedly failed  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **Codex (GPT-6)** in
[#&#8203;11219](https://redirect.github.com/vitest-dev/vitest/issues/11219)
[<samp>(b2458)</samp>](b24585f08)
- Scope cache key generators to projects  -  by
[@&#8203;ecoyoung](https://redirect.github.com/ecoyoung) in
[#&#8203;11281](https://redirect.github.com/vitest-dev/vitest/issues/11281)
and
[#&#8203;11301](https://redirect.github.com/vitest-dev/vitest/issues/11301)
[<samp>(92ba7)</samp>](92ba7fc1d)
- **browser**:
- Delay server `listen` until tests start running  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11366](https://redirect.github.com/vitest-dev/vitest/issues/11366)
[<samp>(7d8ed)</samp>](7d8ed3e9b)
- Check mock path boundaries  -  by
[@&#8203;saryn17](https://redirect.github.com/saryn17), **Ryosei Sato**
and [@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11361](https://redirect.github.com/vitest-dev/vitest/issues/11361)
and
[#&#8203;11362](https://redirect.github.com/vitest-dev/vitest/issues/11362)
[<samp>(1c388)</samp>](1c3888bce)
- Keep config of browser-consumed environments  -  by
[@&#8203;kasperpeulen](https://redirect.github.com/kasperpeulen) and
**Claude** in
[#&#8203;11378](https://redirect.github.com/vitest-dev/vitest/issues/11378)
[<samp>(aafc0)</samp>](aafc0996f)
- Ignore page crash while cancelling  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11386](https://redirect.github.com/vitest-dev/vitest/issues/11386)
[<samp>(7c367)</samp>](7c36748fa)
- `toMatchScreenshot` uses wrong reference on retried tests  -  by
[@&#8203;macarie](https://redirect.github.com/macarie) in
[#&#8203;11393](https://redirect.github.com/vitest-dev/vitest/issues/11393)
[<samp>(c22ab)</samp>](c22aba992)
- **cache**:
- Revalidate imports of cached modules  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11381](https://redirect.github.com/vitest-dev/vitest/issues/11381)
[<samp>(38f98)</samp>](38f98855f)
- **deps**:
- Pin `why-is-node-running` to `3.2.1` to avoid users running into
`ERR_PNPM_TRUST_DOWNGRADE`  -  by
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;11403](https://redirect.github.com/vitest-dev/vitest/issues/11403)
[<samp>(f6c9a)</samp>](f6c9a4977)
- **expect**:
- Pass current equality testers to `expect.extend` asymmetric matchers
 -  by [@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa),
**Hiroshi Ogawa** and **Claude** in
[#&#8203;11401](https://redirect.github.com/vitest-dev/vitest/issues/11401)
[<samp>(3e794)</samp>](3e794a96b)
- **jsdom**:
- Support Blob on jsdom 30.1  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11379](https://redirect.github.com/vitest-dev/vitest/issues/11379)
[<samp>(6c49b)</samp>](6c49b7197)
- **pool**:
- Preserve unique pool ids when `groupOrder` is set  -  by
[@&#8203;mtorp](https://redirect.github.com/mtorp) in
[#&#8203;11392](https://redirect.github.com/vitest-dev/vitest/issues/11392)
[<samp>(50312)</samp>](50312ebb4)
- **ui**:
- Split-pane handle overlapping iframe  -  by
[@&#8203;macarie](https://redirect.github.com/macarie) in
[#&#8203;11221](https://redirect.github.com/vitest-dev/vitest/issues/11221)
[<samp>(f91db)</samp>](f91db0dfd)
- **vitest**:
- Remove root temp dir on close  -  by
[@&#8203;abhinav-phi](https://redirect.github.com/abhinav-phi) in
[#&#8203;11248](https://redirect.github.com/vitest-dev/vitest/issues/11248)
[<samp>(7c711)</samp>](7c7119cf7)
- **vm**:
- Do not optimize deps from index.html  -  by
[@&#8203;ezefernandezyf](https://redirect.github.com/ezefernandezyf),
**Hiroshi Ogawa** and **Codex (GPT-6)** in
[#&#8203;11329](https://redirect.github.com/vitest-dev/vitest/issues/11329)
and
[#&#8203;11360](https://redirect.github.com/vitest-dev/vitest/issues/11360)
[<samp>(caf28)</samp>](caf2887de)
- Don't reuse scripts across vite environments  -  by
[@&#8203;MO2k4](https://redirect.github.com/MO2k4), **Martin Oehlert**
and **Claude** in
[#&#8203;11395](https://redirect.github.com/vitest-dev/vitest/issues/11395)
[<samp>(346d3)</samp>](346d3896b)

#####     [View changes on
GitHub](https://redirect.github.com/vitest-dev/vitest/compare/v5.0.2...v5.0.3)

###
[`v5.0.2`](https://redirect.github.com/vitest-dev/vitest/releases/tag/v5.0.2)

[Compare
Source](https://redirect.github.com/vitest-dev/vitest/compare/v5.0.1...v5.0.2)

#####    🐞 Bug Fixes

- Bind `process` in case global is overwritten  -  by
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;11343](https://redirect.github.com/vitest-dev/vitest/issues/11343)
[<samp>(0b792)</samp>](0b79231ad)
- **detect-async-leaks**:
- Ignore `process.stdio` handles  -  by
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;11333](https://redirect.github.com/vitest-dev/vitest/issues/11333)
[<samp>(0fd6b)</samp>](0fd6b9790)
- **expect**:
- Fix `toMatchObject` with asymmetric matchers  -  by
[@&#8203;ShreeBohara](https://redirect.github.com/ShreeBohara), **Claude
Opus 5**, [@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa),
**Hiroshi Ogawa** and **Codex (GPT-5)** in
[#&#8203;11100](https://redirect.github.com/vitest-dev/vitest/issues/11100)
[<samp>(42523)</samp>](42523289e)
- **jsdom**:
- Fix `Request` with `Blob` body on jsdom 28+  -  by
[@&#8203;harshit-d3v](https://redirect.github.com/harshit-d3v) in
[#&#8203;11295](https://redirect.github.com/vitest-dev/vitest/issues/11295)
[<samp>(d1c3e)</samp>](d1c3ecc93)
- **reporter**:
- `agent` to respect `--silent`  -  by
[@&#8203;Raj4478](https://redirect.github.com/Raj4478) and
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;11271](https://redirect.github.com/vitest-dev/vitest/issues/11271)
[<samp>(5b95e)</samp>](5b95efb6d)
- **reporters**:
- Handle concurrent `createReport` calls  -  by
[@&#8203;7rulnik](https://redirect.github.com/7rulnik) in
[#&#8203;11278](https://redirect.github.com/vitest-dev/vitest/issues/11278)
[<samp>(e8e55)</samp>](e8e556ff7)
- `hanging-process` to use ESM entrypoint  -  by
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;11316](https://redirect.github.com/vitest-dev/vitest/issues/11316)
[<samp>(4e91e)</samp>](4e91e5668)
- **spy**:
- Fix stack overflow when spying `Set.prototype.add`  -  by
[@&#8203;fengmk2](https://redirect.github.com/fengmk2) in
[#&#8203;11299](https://redirect.github.com/vitest-dev/vitest/issues/11299)
[<samp>(a0a93)</samp>](a0a939653)
- **ui**:
- Persist authentication cookie beyond current browser session  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **OpenCode (gpt-5.6-sol)** in
[#&#8203;11066](https://redirect.github.com/vitest-dev/vitest/issues/11066)
[<samp>(f8819)</samp>](f88195e16)

#####     [View changes on
GitHub](https://redirect.github.com/vitest-dev/vitest/compare/v5.0.1...v5.0.2)

###
[`v5.0.1`](https://redirect.github.com/vitest-dev/vitest/releases/tag/v5.0.1)

[Compare
Source](https://redirect.github.com/vitest-dev/vitest/compare/v5.0.0...v5.0.1)

#####    🚀 Features

- **ui**:
- Move trace attempts selector to viewer header  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **Codex** in
[#&#8203;11189](https://redirect.github.com/vitest-dev/vitest/issues/11189)
[<samp>(5dc4b)</samp>](5dc4b5c92)
- Add focused trace view layout mode  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa**, **OpenCode (gpt-5.6-sol)** and **Codex** in
[#&#8203;11190](https://redirect.github.com/vitest-dev/vitest/issues/11190)
[<samp>(376dc)</samp>](376dc3bc1)

#####    🐞 Bug Fixes

- Exit 1 when vitest list fails collection  -  by
[@&#8203;hamed-bavar](https://redirect.github.com/hamed-bavar) in
[#&#8203;11145](https://redirect.github.com/vitest-dev/vitest/issues/11145)
and
[#&#8203;11146](https://redirect.github.com/vitest-dev/vitest/issues/11146)
[<samp>(6108b)</samp>](6108b8197)
- Keep parse error details in static collection  -  by
[@&#8203;hamed-bavar](https://redirect.github.com/hamed-bavar) in
[#&#8203;11150](https://redirect.github.com/vitest-dev/vitest/issues/11150)
and
[#&#8203;11151](https://redirect.github.com/vitest-dev/vitest/issues/11151)
[<samp>(7c818)</samp>](7c818153a)
- Avoid recursive prototype in automocking  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11195](https://redirect.github.com/vitest-dev/vitest/issues/11195)
[<samp>(99fc5)</samp>](99fc52591)
- Prevent false Vitest import resolution  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11196](https://redirect.github.com/vitest-dev/vitest/issues/11196)
[<samp>(b426c)</samp>](b426c1976)
- Keep metadata file when clearing the cache  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11199](https://redirect.github.com/vitest-dev/vitest/issues/11199)
[<samp>(73614)</samp>](73614654a)
- Correct typos in error message and comments  -  by
[@&#8203;shinji00222](https://redirect.github.com/shinji00222) and
**Shinji** in
[#&#8203;11187](https://redirect.github.com/vitest-dev/vitest/issues/11187)
[<samp>(115c3)</samp>](115c3f6d2)
- Resolve ResolvedConfig exactOptionalPropertyTypes errors  -  by
[@&#8203;LukeAbby](https://redirect.github.com/LukeAbby) in
[#&#8203;11175](https://redirect.github.com/vitest-dev/vitest/issues/11175)
[<samp>(498fb)</samp>](498fbe922)
- Share the server on self-referencing `extends`  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11034](https://redirect.github.com/vitest-dev/vitest/issues/11034)
[<samp>(23dda)</samp>](23dda738c)
- Warn when deprecated `deps.optimizer.web` is used  -  by
[@&#8203;im10furry](https://redirect.github.com/im10furry) in
[#&#8203;11214](https://redirect.github.com/vitest-dev/vitest/issues/11214)
[<samp>(2ce29)</samp>](2ce29d5fa)
- **browser**:
- Avoid double quotes in `config.define`  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11198](https://redirect.github.com/vitest-dev/vitest/issues/11198)
[<samp>(972e2)</samp>](972e24bab)
- **doctor**:
- Measure vm pools for custom environments  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11212](https://redirect.github.com/vitest-dev/vitest/issues/11212)
[<samp>(91ab1)</samp>](91ab1588c)
- **expect**:
- Correct return value in `toMatchAriaSnapshot`  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11208](https://redirect.github.com/vitest-dev/vitest/issues/11208)
[<samp>(c119b)</samp>](c119be016)
- **fakeTimers**:
- Force `queueMicrotask` and `nextTick` in `toNotFake`  -  by
[@&#8203;kingmakeruix](https://redirect.github.com/kingmakeruix),
**kingmakeruix**, **Hiroshi Ogawa**, **Codex** and
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa) in
[#&#8203;11261](https://redirect.github.com/vitest-dev/vitest/issues/11261)
[<samp>(a47d7)</samp>](a47d7908f)
- **snapshot**:
- Report obsolete keys next to skipped tests  -  by
[@&#8203;hamed-bavar](https://redirect.github.com/hamed-bavar),
**Hiroshi Ogawa** and **OpenCode (gpt-5.6-sol)** in
[#&#8203;11157](https://redirect.github.com/vitest-dev/vitest/issues/11157)
and
[#&#8203;11158](https://redirect.github.com/vitest-dev/vitest/issues/11158)
[<samp>(17e2b)</samp>](17e2b22dd)
- **types**:
- Make public declarations self-contained  -  by
[@&#8203;ZoeySigel](https://redirect.github.com/ZoeySigel) in
[#&#8203;11141](https://redirect.github.com/vitest-dev/vitest/issues/11141)
[<samp>(45546)</samp>](455466c16)
- **ui**:
- Fix collapse/expand suite with file name search  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **Codex** in
[#&#8203;11260](https://redirect.github.com/vitest-dev/vitest/issues/11260)
[<samp>(0a712)</samp>](0a7122daa)
- Fix explorer file summary count  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa**, **OpenCode (gpt-5.6-sol)** and **Codex** in
[#&#8203;11125](https://redirect.github.com/vitest-dev/vitest/issues/11125)
[<samp>(05982)</samp>](05982297d)
- **utils**:
- Fix `deepMerge` to handle prototype  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **Codex** in
[#&#8203;11215](https://redirect.github.com/vitest-dev/vitest/issues/11215)
[<samp>(4944c)</samp>](4944cf498)

#####     [View changes on
GitHub](https://redirect.github.com/vitest-dev/vitest/compare/v5.0.0...v5.0.1)

###
[`v5.0.0`](https://redirect.github.com/vitest-dev/vitest/releases/tag/v5.0.0)

[Compare
Source](https://redirect.github.com/vitest-dev/vitest/compare/v4.1.12...v5.0.0)

Vitest 5 is officially out! This release focuses on performance and
brings a lot of new features while fixing long-standing bugs. See our
[blog post](https://vitest.dev/blog/vitest-5.html) for the official
announcement.

#####    🚨 Breaking Changes

- Replace `loupe.inspect` with pretty-format  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Claude Opus
5 (1M context)** and **OpenAI Codex** in
[#&#8203;9609](https://redirect.github.com/vitest-dev/vitest/issues/9609)
[<samp>(3f802)</samp>](3f802da4b)
- Remove quotes from string values in `test.for/each` title `$` variable
(take 2)  -  by [@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa)
in
[#&#8203;10170](https://redirect.github.com/vitest-dev/vitest/issues/10170)
[<samp>(04d37)</samp>](04d37e9d7)
- Default `attachmentsDir` from `.vitest-attachements/` to
`.vitest/attachments/`  -  by
[@&#8203;MdSadiqMd](https://redirect.github.com/MdSadiqMd) in
[#&#8203;10186](https://redirect.github.com/vitest-dev/vitest/issues/10186)
[<samp>(1ba73)</samp>](1ba7338c3)
- Remove `sequential` test/suite options in favor of `concurrent`  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa) and **OpenAI
Codex** in
[#&#8203;10198](https://redirect.github.com/vitest-dev/vitest/issues/10198)
[<samp>(9229f)</samp>](9229f2edc)
- Represent locator as an object instead of a string  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10212](https://redirect.github.com/vitest-dev/vitest/issues/10212)
[<samp>(80f07)</samp>](80f07edf6)
- Inline `expect` package  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10221](https://redirect.github.com/vitest-dev/vitest/issues/10221)
[<samp>(ad162)</samp>](ad16223e7)
- Remove deprecated entry points  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10222](https://redirect.github.com/vitest-dev/vitest/issues/10222)
[<samp>(994c6)</samp>](994c6ddb9)
- Require Node.js 22 and Vite 6.4  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10178](https://redirect.github.com/vitest-dev/vitest/issues/10178)
[<samp>(38762)</samp>](3876283e8)
- Fail `expect.poll` when function didn't resolve in time  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa) and **OpenAI
Codex** in
[#&#8203;10233](https://redirect.github.com/vitest-dev/vitest/issues/10233)
[<samp>(4df04)</samp>](4df048c11)
- Throw an error if hoistable methods are outside the top level scope
 -  by [@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10460](https://redirect.github.com/vitest-dev/vitest/issues/10460)
[<samp>(d0b4f)</samp>](d0b4fddcb)
- `toHaveTextContent` is strict, add `toMatchTextContent` as alternative
 -  by [@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10473](https://redirect.github.com/vitest-dev/vitest/issues/10473)
[<samp>(18f30)</samp>](18f303079)
- Don't lookup config file from ancestor directories  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **OpenAI
Codex** and **Hiroshi Ogawa** in
[#&#8203;10428](https://redirect.github.com/vitest-dev/vitest/issues/10428)
[<samp>(945d9)</samp>](945d9090e)
- Inline `@vitest/runner` package, do not publish it anymore  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10511](https://redirect.github.com/vitest-dev/vitest/issues/10511)
[<samp>(6d6e4)</samp>](6d6e46b1e)
- Allow mutating happy-dom/jsdom window object  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **OpenAI Codex** in
[#&#8203;10373](https://redirect.github.com/vitest-dev/vitest/issues/10373)
[<samp>(206e8)</samp>](206e8cff8)
- Expose `concurrencyId`/`workerId` on TestModule's diagnostics, make id
1-based  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10516](https://redirect.github.com/vitest-dev/vitest/issues/10516)
[<samp>(bdd98)</samp>](bdd985433)
- Add `screenshotDirectory` config to `browser.expect.toMatchScreenshot`
 -  by [@&#8203;macarie](https://redirect.github.com/macarie) in
[#&#8203;10592](https://redirect.github.com/vitest-dev/vitest/issues/10592)
[<samp>(a60de)</samp>](a60ded0fb)
- Update `@sinonjs/fake-timers` and support mocking `Temporal`  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **OpenCode (gpt-5.6-sol)** in
[#&#8203;10654](https://redirect.github.com/vitest-dev/vitest/issues/10654)
[<samp>(f8b15)</samp>](f8b1532fe)
- Remove webdriverio package  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10675](https://redirect.github.com/vitest-dev/vitest/issues/10675)
[<samp>(5fed6)</samp>](5fed68f72)
- Clear mocks by default before each test  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10613](https://redirect.github.com/vitest-dev/vitest/issues/10613)
[<samp>(0f646)</samp>](0f6463bf2)
- Don't emit localStorage warnings on Node 26, fail gracefully when
worker fails to start  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10293](https://redirect.github.com/vitest-dev/vitest/issues/10293)
[<samp>(334ed)</samp>](334edef92)
- Separate config resolution from the server creation  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10554](https://redirect.github.com/vitest-dev/vitest/issues/10554)
[<samp>(1c0ec)</samp>](1c0ec3444)
- Inline projects extend the root config by default  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10750](https://redirect.github.com/vitest-dev/vitest/issues/10750)
[<samp>(fec00)</samp>](fec001ad3)
- Enable mocking Temporal without fake timers  -  by
[@&#8203;fabon-f](https://redirect.github.com/fabon-f), **Hiroshi
Ogawa** and **OpenCode (gpt-5.6-sol)** in
[#&#8203;10757](https://redirect.github.com/vitest-dev/vitest/issues/10757)
[<samp>(ac2d4)</samp>](ac2d46b42)
- Support nested projects  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10846](https://redirect.github.com/vitest-dev/vitest/issues/10846)
[<samp>(ec367)</samp>](ec367cf2a)
- Use `>` as separator in `-t`, calculate `only` once  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10686](https://redirect.github.com/vitest-dev/vitest/issues/10686)
[<samp>(a0b20)</samp>](a0b20bc86)
- Fail the test when an asynchronous assertion is not awaited  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10868](https://redirect.github.com/vitest-dev/vitest/issues/10868)
[<samp>(86d4a)</samp>](86d4a9da9)
- Share the Vite server between inline projects  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10848](https://redirect.github.com/vitest-dev/vitest/issues/10848)
[<samp>(d87c9)</samp>](d87c96ee4)
- Parse files statically in vitest list by default  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11088](https://redirect.github.com/vitest-dev/vitest/issues/11088)
[<samp>(51e94)</samp>](51e949416)
- **benchmark**:
- Rewrite the public API  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10113](https://redirect.github.com/vitest-dev/vitest/issues/10113)
[<samp>(19f6e)</samp>](19f6e8947)
- **browser**:
- Iframe scale  -  by
[@&#8203;macarie](https://redirect.github.com/macarie) in
[#&#8203;9745](https://redirect.github.com/vitest-dev/vitest/issues/9745)
[<samp>(b6398)</samp>](b639852cc)
- Enable `locators.exact` by default  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10430](https://redirect.github.com/vitest-dev/vitest/issues/10430)
[<samp>(e2032)</samp>](e203202f9)
- Require `sessionId` for orchestrator html request  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **OpenAI Codex** in
[#&#8203;10522](https://redirect.github.com/vitest-dev/vitest/issues/10522)
[<samp>(79b7d)</samp>](79b7d8fcc)
- Save failure screenshots in `attachmentsDir`  -  by
[@&#8203;macarie](https://redirect.github.com/macarie) in
[#&#8203;10917](https://redirect.github.com/vitest-dev/vitest/issues/10917)
[<samp>(3b5bb)</samp>](3b5bbd8b4)
- **coverage**:
- `include/exclude` globs too eager  -  by
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;9818](https://redirect.github.com/vitest-dev/vitest/issues/9818)
[<samp>(edacb)</samp>](edacb0fd4)
- Allow `thresholds.perFile` to accept an object  -  by
[@&#8203;vladlenskiy](https://redirect.github.com/vladlenskiy) and
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;10190](https://redirect.github.com/vitest-dev/vitest/issues/10190)
[<samp>(13b78)</samp>](13b78d98b)
- **expect**:
- Fix `toThrow("")` behavior by reverting
[#&#8203;6710](https://redirect.github.com/vitest-dev/vitest/issues/6710)
 -  by [@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa) in
[#&#8203;9643](https://redirect.github.com/vitest-dev/vitest/issues/9643)
and
[#&#8203;6710](https://redirect.github.com/vitest-dev/vitest/issues/6710)
[<samp>(6c3e4)</samp>](6c3e4bdbf)
- **mocker**:
- Deserialize automock as automock  -  by
[@&#8203;nami8824](https://redirect.github.com/nami8824) in
[#&#8203;10192](https://redirect.github.com/vitest-dev/vitest/issues/10192)
[<samp>(2f892)</samp>](2f892712d)
- **reporters**:
- `blob` reporter and `--merge-reports` default to `.vitest/blob/`  - 
by [@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;10232](https://redirect.github.com/vitest-dev/vitest/issues/10232)
[<samp>(d22b0)</samp>](d22b029ae)
- Write json and junit reporter output files to `.vitest` by default  - 
by [@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa**, **OpenCode (gpt-5.6-sol)** and
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;10621](https://redirect.github.com/vitest-dev/vitest/issues/10621)
[<samp>(58577)</samp>](58577290a)
- **spy**:
- Preserve class mock prototype methods on instances  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10910](https://redirect.github.com/vitest-dev/vitest/issues/10910)
[<samp>(6093d)</samp>](6093d764a)
- **types**:
- Add better promise support in expects and matchers  -  by
[@&#8203;samchungy](https://redirect.github.com/samchungy) and
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;8266](https://redirect.github.com/vitest-dev/vitest/issues/8266)
[<samp>(41f55)</samp>](41f551bce)
- **ui**:
- Harden UI API access  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **OpenAI Codex** in
[#&#8203;10583](https://redirect.github.com/vitest-dev/vitest/issues/10583)
[<samp>(4c26d)</samp>](4c26d7675)
- Change html reporter default output to `.vitest`  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa) and **Hiroshi
Ogawa** in
[#&#8203;10620](https://redirect.github.com/vitest-dev/vitest/issues/10620)
[<samp>(29c36)</samp>](29c364d50)

#####    🚀 Features

- Add `createReport` and `.vitest` report directory convention  -  by
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;9993](https://redirect.github.com/vitest-dev/vitest/issues/9993)
[<samp>(72a6d)</samp>](72a6dc257)
- Expose default reporters through `configDefaults.reporters`  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa) and **Claude
Opus 5 (1M context)** in
[#&#8203;10219](https://redirect.github.com/vitest-dev/vitest/issues/10219)
[<samp>(083f6)</samp>](083f6bdd6)
- Support merge reports for non-sharded multi-environment runs (take 2)
 -  by [@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa),
**Claude Opus 5 (1M context)**, **OpenAI Codex** and
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;10031](https://redirect.github.com/vitest-dev/vitest/issues/10031)
[<samp>(e60b2)</samp>](e60b2f49e)
- Add `logger.formatError`  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa) and **OpenAI
Codex** in
[#&#8203;10268](https://redirect.github.com/vitest-dev/vitest/issues/10268)
[<samp>(2c5f3)</samp>](2c5f3ee2f)
- Support typescript build mode  -  by
[@&#8203;lishaduck](https://redirect.github.com/lishaduck) in
[#&#8203;9870](https://redirect.github.com/vitest-dev/vitest/issues/9870)
[<samp>(106da)</samp>](106da5896)
- Add toggable `injectCjsGlobals` option  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10709](https://redirect.github.com/vitest-dev/vitest/issues/10709)
[<samp>(82671)</samp>](826714ece)
- Promote fsModuleCache to a top-level option  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10734](https://redirect.github.com/vitest-dev/vitest/issues/10734)
[<samp>(78ed7)</samp>](78ed73afd)
- Support non-ascii characters in `for/each` title placeholders  -  by
[@&#8203;k-yle](https://redirect.github.com/k-yle) in
[#&#8203;10773](https://redirect.github.com/vitest-dev/vitest/issues/10773)
[<samp>(15e0a)</samp>](15e0af953)
- Report the duration breakdown as percentages  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) and
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;10820](https://redirect.github.com/vitest-dev/vitest/issues/10820)
[<samp>(c67d2)</samp>](c67d296f4)
- Promote parseSpecifications out of experimental  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11087](https://redirect.github.com/vitest-dev/vitest/issues/11087)
[<samp>(b628c)</samp>](b628cb680)
- Promote clearCache out of experimental  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11086](https://redirect.github.com/vitest-dev/vitest/issues/11086)
[<samp>(d4fe1)</samp>](d4fe1986f)
- **api**:
- Expose logs recorded during the test  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10277](https://redirect.github.com/vitest-dev/vitest/issues/10277)
[<samp>(cba20)</samp>](cba2036a1)
- **benchmark**:
- Add pluggable benchmark provider API  -  by
[@&#8203;GuillaumeLagrange](https://redirect.github.com/GuillaumeLagrange)
and [@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10799](https://redirect.github.com/vitest-dev/vitest/issues/10799)
[<samp>(86c70)</samp>](86c700ed8)
- **browser**:
- Export aria tree utils  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa) in
[#&#8203;10171](https://redirect.github.com/vitest-dev/vitest/issues/10171)
[<samp>(c3423)</samp>](c3423014c)
- Support dom snapshot trace view  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Claude Opus
5 (1M context)** and **OpenAI Codex** in
[#&#8203;10102](https://redirect.github.com/vitest-dev/vitest/issues/10102)
[<samp>(7eddd)</samp>](7eddd273d)
- Provide project reference in `ToMatchScreenshotResolvePath`  -  by
[@&#8203;macarie](https://redirect.github.com/macarie) and
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10138](https://redirect.github.com/vitest-dev/vitest/issues/10138)
[<samp>(16654)</samp>](166544e39)
- Show aria tree on locator element error  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa) and **OpenAI
Codex** in
[#&#8203;10257](https://redirect.github.com/vitest-dev/vitest/issues/10257)
[<samp>(04f04)</samp>](04f04cdd1)
- Support custom `kind` in `page.mark`  -  by
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in

> [!IMPORTANT]
> ✂ PR body was truncated to here.


</details>

---

### Configuration

📅 **Schedule**: (UTC)

- Branch creation
  - "before 6am every weekday"
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

👻 **Immortal**: This PR will be recreated if closed unmerged. Get
[config
help](https://redirect.github.com/renovatebot/renovate/discussions) if
that's undesired.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend Renovate
CLI](https://redirect.github.com/renovatebot/renovate).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4xOTguMiIsInVwZGF0ZWRJblZlciI6IjQ0LjIwNS4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJhdXRvbWF0aW9uIl19-->

---------

Co-authored-by: wardpeet <ward@coding-tech.com>
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-11-03 20:18:21 +01:00
Dane - AI (Mastra)
47a6cd82e5
chore(deps): update ai sdk (#29242)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-11-03 16:12:54 +01:00
dane-ai-mastra[bot]
30046969ad
chore: version packages (alpha) (#29146)
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases
## @mastra/client-js@1.59.0-alpha.1

### Minor Changes

- Added the `aggregateTimeToFirstMeasures` observability capability.
Trace and span aggregate requests for `timeToFirstToken.*` or
`timeToFirstChunk.*` measures now return `501` against stores that can't
compute them, and field discovery omits those measures for such stores.
([#29055](https://github.com/mastra-ai/mastra/pull/29055))

  ```typescript
const { capabilities } = await
mastraClient.getObservabilityCapabilities();

if (capabilities.traceAggregate &&
capabilities.aggregateTimeToFirstMeasures) {
    const { rows } = await mastraClient.aggregateTraces({
timeRange: { from: '2026-08-01T00:00:00.000Z', to:
'2026-08-02T00:00:00.000Z' },
      groupBy: ['entityName'],
      measures: ['timeToFirstChunk.p95', 'timeToFirstChunk.count'],
    });
  }
  ```

### Patch Changes

- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
## @mastra/factory@0.27.0-alpha.1

### Minor Changes

- The same Jira, Linear or incident.io item can now be filed in more
than one Factory project. Before, the first project to file an item held
it for the whole org, and other projects that ingested it got no card.
Each project now keeps its own card. Within a project the card is still
reused when the item is seen again, after the card finishes and when the
issue reopens. A Linear card is also found after the issue's identifier
changes, for example when the issue moves to another team.
([#29059](https://github.com/mastra-ai/mastra/pull/29059))

When intake routing sends an item to a different project, the new
project files its own card and the previous project's card stays where
it is.

  **Breaking changes**

- `claimKey` is removed from the `upsertLinkedWorkItem` decision type.
Rules that still return it, and decisions saved before this release,
keep working: the field is accepted and ignored.
- `WorkItemClaimConflictError`, `getByClaimKey`, `claimWorkItem`,
`releaseProjectClaims` and `WorkItemRow.claimKey` are removed.

  Drop `claimKey` from linked-item decisions your rules return:

  ```ts
  // Before
  return {
    type: 'upsertLinkedWorkItem',
    source: 'jira-issue',
    sourceKey: issue.id,
    claimKey: `jira:issue:${issue.id}` /* … */,
  };

  // After
return { type: 'upsertLinkedWorkItem', source: 'jira-issue', sourceKey:
issue.id /* … */ };
  ```

Remove calls to `getByClaimKey`, `claimWorkItem` and
`releaseProjectClaims`. To find a project's card for an item, use
`getByProjectSource`.

The `claim_key` column stays in the database. Values already stored
there are left in place and no longer affect anything.

### Patch Changes

- Fixed shell commands that exit with a failure code reading as
successful in the Factory transcript. The command now reads as failed
while the run streams, after a dropped stream reconnects, and after a
reload. Stopping a process with `kill_process` still reads as a success.
([#28963](https://github.com/mastra-ai/mastra/pull/28963))

A system reminder that loads project instructions (`AGENTS.md`,
`CLAUDE.md` or `CONTEXT.md`) now reads "Project instructions" and names
the file by its path in the working directory, such as
`packages/core/AGENTS.md`, instead of a bare "System reminder".

- A board tool rule that moves a work item back into its current stage
with `reenter: true` now runs that stage's entry rules, as documented.
Previously the entry rules were skipped, so no new agent run started.
([#29104](https://github.com/mastra-ai/mastra/pull/29104))

- Updated dependencies
[[`5735a50`](5735a50c6c),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/platform-workspace@1.11.1-alpha.0
  - @mastra/core@1.82.0-alpha.1
  - @mastra/code-sdk@1.18.0-alpha.1
## @mastra/core@1.82.0-alpha.1

### Minor Changes

- Added time to first token and time to first chunk measures to trace
and span aggregates, so dashboards can chart streaming latency next to
duration. ([#29055](https://github.com/mastra-ai/mastra/pull/29055))

The new measures are `timeToFirstToken.avg`, `.min`, `.max`, `.p50`,
`.p90`, `.p95`, `.p99`, and `.count`, plus the same set for
`timeToFirstChunk`. Traces and spans without a recorded value are
skipped instead of counted as 0, and `.count` reports how many had one.
Stores opt in through the `aggregate-time-to-first-measures` feature.

### Patch Changes

- Fixed durable agents (`durable: true`) ending the `model_generation`
trace span without `responseId` and `responseModel`. Durable runs now
record the provider's response id and model on the span, the same as
non-durable agents, so a trace can be linked back to the provider's own
response log. Fixes #29053.
([#29180](https://github.com/mastra-ai/mastra/pull/29180))

- Fixed metadata filters in the in-memory store returning records whose
nested metadata had different keys from the filter when one side
contained an `undefined` value.
([#29101](https://github.com/mastra-ai/mastra/pull/29101))

- Fixed directly resumed nested workflows in the evented engine so they
continue suspended parent runs.
([#25979](https://github.com/mastra-ai/mastra/pull/25979))
## @mastra/playground-ui@68.0.0-alpha.1

### Minor Changes

- Toolbar and filter bar controls now share one look: the same paddings,
icon gaps, chevron and spacing between controls.
([#28818](https://github.com/mastra-ai/mastra/pull/28818))

**Time range.** A filter bar's time range is now the
`DateTimeRangePicker` itself, so it looks and behaves the same on every
page: a calendar icon, the range and a chevron, with no separate "Time"
label. Pass `defaultPreset` to show a × that resets a custom range; it
replaces the chevron while a custom range is applied.

Added `FilterBar.TimeRange` with `FILTER_BAR_TIME_RANGE_FIELD`,
`FILTER_BAR_TIME_RANGE_ITEM` and `FILTER_BAR_TIME_RANGE_FIELD_ID`. They
replace `TraceTimeRangeChip` and the `TRACE_TIME_RANGE_*` names from
`domains/traces/components/trace-time-range-chip`, which are removed.

  ```tsx
  import {
    FilterBar,
    FILTER_BAR_TIME_RANGE_FIELD,
    FILTER_BAR_TIME_RANGE_FIELD_ID,
    FILTER_BAR_TIME_RANGE_ITEM,
  } from '@mastra/playground-ui/components/FilterBar';

  <FilterBar
    fields={[FILTER_BAR_TIME_RANGE_FIELD, ...fields]}
    operators={operators}
    value={[FILTER_BAR_TIME_RANGE_ITEM, ...items]}
// The time range item is synthetic: keep it out of the stored filters.
onValueChange={next => setItems(next.filter(item => item.fieldId !==
FILTER_BAR_TIME_RANGE_FIELD_ID))}
  >
    <FilterBar.Chips
      renderChip={item =>
        item.fieldId === FILTER_BAR_TIME_RANGE_FIELD_ID ? (
          <FilterBar.TimeRange
            preset={preset}
// A preset, including the × reset, drops the applied custom dates.
            onPresetChange={next => {
              setPreset(next);
              if (next !== 'custom') setRange({});
            }}
            dateFrom={range.from}
            dateTo={range.to}
            onDateRangeChange={(from, to) => setRange({ from, to })}
            defaultPreset="last-7d"
          />
        ) : (
          <FilterBar.Chip item={item} />
        )
      }
    />
    <FilterBar.Input />
  </FilterBar>;
  ```

`DateTimeRangePicker` gains `size`, `defaultPreset` and `ref` (the
trigger button).

  **Migrating removed APIs.**

- `TraceTimeRangeChip` → `FilterBar.TimeRange` (same props), and
`TRACE_TIME_RANGE_FIELD` / `_ITEM` / `_FIELD_ID` →
`FILTER_BAR_TIME_RANGE_FIELD` / `_ITEM` / `_FIELD_ID`, imported from
`@mastra/playground-ui/components/FilterBar`.

    ```tsx
    // Before
import { TraceTimeRangeChip, TRACE_TIME_RANGE_FIELD_ID } from
'@mastra/playground-ui/domains/traces/components/trace-time-range-chip';
item.fieldId === TRACE_TIME_RANGE_FIELD_ID ? <TraceTimeRangeChip
preset={preset} onPresetChange={setPreset} /> : …

    // After
import { FilterBar, FILTER_BAR_TIME_RANGE_FIELD_ID } from
'@mastra/playground-ui/components/FilterBar';
item.fieldId === FILTER_BAR_TIME_RANGE_FIELD_ID ? <FilterBar.TimeRange
preset={preset} onPresetChange={setPreset} /> : …
    ```

- `DateTimeRangePicker`'s `renderTrigger` → remove it: the trigger is
always the shared calendar button. Pass `size` for another rung and
`ref` to reach the trigger.
- `comboboxStyles.chevron` on a hand-built trigger → `<ButtonChevron
size={size} />` from `@mastra/playground-ui/components/Button`.

  **Other controls.**

- Added `ClearFiltersButton`, a round reset icon button with a tooltip,
used for every toolbar's "clear filters".
- Added `TriggerChevron` and `ButtonChevron`, the one chevron every
dropdown trigger (`Select`, `Combobox`, the date picker) shows.
  - `ComboboxTrigger` uses that chevron instead of the up-down one.
- Icons in every interactive control (buttons, fields, search inputs,
filter chips, menu and command rows) follow one rule: muted at rest,
brighter on hover. Two new theme tokens, `--control-icon-rest` and
`--control-icon-hover`, set those colours; menus retune them, and
icon-only controls always brighten. Search fields and combobox fields
now brighten their icons on hover like buttons do.
- Icons take their size from the control they sit in, as in a Button:
filter chips and search inputs no longer size their own icons.
- `ListSearch` defaults to the `sm` toolbar size, and `SearchInput`'s
icon sits at the same inset as a button's.
  - `ActionRow` and `FilterBar` space their controls 6px apart.

### Patch Changes

- Updated dependencies
[[`489d829`](489d8295c8),
[`0900c20`](0900c20592),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`e1e64f0`](e1e64f072c),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/memory@1.41.0-alpha.1
  - @mastra/client-js@1.59.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
  - @mastra/ai-sdk@1.11.3
  - @mastra/react@1.14.1-alpha.1
## @mastra/server@1.82.0-alpha.1

### Minor Changes

- Added the `aggregateTimeToFirstMeasures` observability capability.
Trace and span aggregate requests for `timeToFirstToken.*` or
`timeToFirstChunk.*` measures now return `501` against stores that can't
compute them, and field discovery omits those measures for such stores.
([#29055](https://github.com/mastra-ai/mastra/pull/29055))

  ```typescript
const { capabilities } = await
mastraClient.getObservabilityCapabilities();

if (capabilities.traceAggregate &&
capabilities.aggregateTimeToFirstMeasures) {
    const { rows } = await mastraClient.aggregateTraces({
timeRange: { from: '2026-08-01T00:00:00.000Z', to:
'2026-08-02T00:00:00.000Z' },
      groupBy: ['entityName'],
      measures: ['timeToFirstChunk.p95', 'timeToFirstChunk.count'],
    });
  }
  ```

### Patch Changes

- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
## @mastra/clickhouse@1.29.0-alpha.0

### Minor Changes

- Added support for `timeToFirstToken.*` and `timeToFirstChunk.*`
measures in trace and span aggregates. Rows without a recorded value are
skipped, never counted as 0.
([#29055](https://github.com/mastra-ai/mastra/pull/29055))

  ```typescript
  import { planSpanAggregate } from '@mastra/core/storage';

  const observability = await storage.getStore('observability');

if
(observability?.getFeatures().includes('aggregate-time-to-first-measures'))
{
    const { rows } = await observability.aggregateSpans(
      planSpanAggregate({
timeRange: { from: '2026-08-01T00:00:00.000Z', to:
'2026-08-02T00:00:00.000Z' },
where: { op: 'eq', left: { path: 'spanType' }, right: { literal:
'model_inference' } },
        measures: ['timeToFirstToken.p50', 'timeToFirstToken.p95'],
      }),
    );
  }
  ```

### Patch Changes

- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
## @mastra/duckdb@1.18.0-alpha.0

### Minor Changes

- Added support for `timeToFirstToken.*` and `timeToFirstChunk.*`
measures in trace and span aggregates. Rows without a recorded value are
skipped, never counted as 0.
([#29055](https://github.com/mastra-ai/mastra/pull/29055))

  ```typescript
  import { planSpanAggregate } from '@mastra/core/storage';

  const observability = await storage.getStore('observability');

if
(observability?.getFeatures().includes('aggregate-time-to-first-measures'))
{
    const { rows } = await observability.aggregateSpans(
      planSpanAggregate({
timeRange: { from: '2026-08-01T00:00:00.000Z', to:
'2026-08-02T00:00:00.000Z' },
where: { op: 'eq', left: { path: 'spanType' }, right: { literal:
'model_inference' } },
        measures: ['timeToFirstToken.p50', 'timeToFirstToken.p95'],
      }),
    );
  }
  ```

### Patch Changes

- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
## @mastra/pg@1.35.0-alpha.1

### Minor Changes

- Added support for `timeToFirstToken.*` and `timeToFirstChunk.*`
measures in trace and span aggregates. Rows without a recorded value are
skipped, never counted as 0.
([#29055](https://github.com/mastra-ai/mastra/pull/29055))

  ```typescript
  import { planSpanAggregate } from '@mastra/core/storage';

  const observability = await storage.getStore('observability');

if
(observability?.getFeatures().includes('aggregate-time-to-first-measures'))
{
    const { rows } = await observability.aggregateSpans(
      planSpanAggregate({
timeRange: { from: '2026-08-01T00:00:00.000Z', to:
'2026-08-02T00:00:00.000Z' },
where: { op: 'eq', left: { path: 'spanType' }, right: { literal:
'model_inference' } },
        measures: ['timeToFirstToken.p50', 'timeToFirstToken.p95'],
      }),
    );
  }
  ```

### Patch Changes

- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
## @mastra/react@1.14.1-alpha.1

### Patch Changes

- Updated dependencies
[[`0900c20`](0900c20592),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/client-js@1.59.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
## @mastra/deployer-cloud@1.82.0-alpha.1

### Patch Changes

- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
  - @mastra/deployer@1.82.0-alpha.1
## @mastra/longmemeval@1.1.41-alpha.1

### Patch Changes

- Updated dependencies
[[`489d829`](489d8295c8),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`e1e64f0`](e1e64f072c),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/memory@1.41.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
  - @mastra/libsql@1.29.0-alpha.0
## @mastra/opencode@0.1.41-alpha.1

### Patch Changes

- Updated dependencies
[[`489d829`](489d8295c8),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`e1e64f0`](e1e64f072c),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/memory@1.41.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
  - @mastra/libsql@1.29.0-alpha.0
## create-factory@0.4.2-alpha.1

### Patch Changes

- Updated dependencies:
  - mastra@1.38.0-alpha.1
## @mastra/code-sdk@1.18.0-alpha.1

### Patch Changes

- Updated dependencies
[[`489d829`](489d8295c8),
[`0900c20`](0900c20592),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`e1e64f0`](e1e64f072c),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/memory@1.41.0-alpha.1
  - @mastra/client-js@1.59.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
  - @mastra/pg@1.35.0-alpha.1
  - @mastra/duckdb@1.18.0-alpha.0
  - @mastra/libsql@1.29.0-alpha.0
## mastracode@0.49.0-alpha.1

### Patch Changes

- Updated dependencies
[[`489d829`](489d8295c8),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`e1e64f0`](e1e64f072c),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/memory@1.41.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
  - @mastra/pg@1.35.0-alpha.1
  - @mastra/duckdb@1.18.0-alpha.0
  - @mastra/code-sdk@1.18.0-alpha.1
  - @mastra/libsql@1.29.0-alpha.0
## @mastra/sentry@1.2.31-alpha.1

### Patch Changes

- dependencies updates:
([#29123](https://github.com/mastra-ai/mastra/pull/29123))
- Updated dependency [`@sentry/node@^11.10.0`
↗︎](https://www.npmjs.com/package/@sentry/node/v/11.10.0) (from
`^10.76.2`, in `dependencies`)
- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
## @mastra/agent-builder@1.1.33-alpha.1

### Patch Changes

- Updated dependencies
[[`489d829`](489d8295c8),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`e1e64f0`](e1e64f072c),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/memory@1.41.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
## mastra@1.38.0-alpha.1

### Patch Changes

- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
  - @mastra/deployer@1.82.0-alpha.1
## @mastra/deployer@1.82.0-alpha.1

### Patch Changes

- Updated dependencies
[[`0900c20`](0900c20592),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/server@1.82.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
## @mastra/editor@0.16.7-alpha.1

### Patch Changes

- Updated dependencies
[[`489d829`](489d8295c8),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`e1e64f0`](e1e64f072c),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/memory@1.41.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
## @mastra/mcp-docs-server@1.3.12-alpha.2

### Patch Changes

- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
## @mastra/memory@1.41.0-alpha.1

### Patch Changes

- dependencies updates:
([#29124](https://github.com/mastra-ai/mastra/pull/29124))
- Updated dependency [`diff@^9.0.0`
↗︎](https://www.npmjs.com/package/diff/v/9.0.0) (from `^8.0.4`, in
`dependencies`)

- Observational Memory now tells the Observer who sent each user signal.
In a thread shared by several people, messages sent with
`agent.sendMessage({ contents, attributes: { name: 'Alice' } }, ...)`
are labeled `User (name="Alice")` when observed, so observations can say
who said what instead of attributing everything to "User". Messages
without attributes keep the plain `User` label.
([#29176](https://github.com/mastra-ai/mastra/pull/29176))

- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
## @mastra/google-cloud-pubsub@1.1.7-alpha.0

### Patch Changes

- dependencies updates:
([#29113](https://github.com/mastra-ai/mastra/pull/29113))
- Updated dependency [`@google-cloud/pubsub@^6.3.1`
↗︎](https://www.npmjs.com/package/@google-cloud/pubsub/v/6.3.1) (from
`^5.3.1`, in `dependencies`)
- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
## @mastra/elysia@0.1.22-alpha.1

### Patch Changes

- Updated dependencies
[[`0900c20`](0900c20592),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/server@1.82.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
## @mastra/express@1.5.26-alpha.1

### Patch Changes

- Updated dependencies
[[`0900c20`](0900c20592),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/server@1.82.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
## @mastra/fastify@1.5.26-alpha.1

### Patch Changes

- Updated dependencies
[[`0900c20`](0900c20592),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/server@1.82.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
## @mastra/hono@1.7.24-alpha.1

### Patch Changes

- Updated dependencies
[[`0900c20`](0900c20592),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/server@1.82.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
## @mastra/koa@1.7.26-alpha.1

### Patch Changes

- Updated dependencies
[[`0900c20`](0900c20592),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/server@1.82.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
## @mastra/nestjs@0.2.41-alpha.1

### Patch Changes

- Updated dependencies
[[`0900c20`](0900c20592),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/server@1.82.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
## @mastra/next@0.2.40-alpha.1

### Patch Changes

- Updated dependencies
[[`0900c20`](0900c20592),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/server@1.82.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
  - @mastra/hono@1.7.24-alpha.1
## @mastra/tanstack-start@0.2.40-alpha.1

### Patch Changes

- Updated dependencies
[[`0900c20`](0900c20592),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/server@1.82.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
  - @mastra/hono@1.7.24-alpha.1
## @mastra/mysql@0.16.0-alpha.1

### Patch Changes

- Fixed experiment result writes putting plaintext back over a purged
dataset item. When a purge committed while a result was being written,
the writer could miss the purge marker and overwrite the redacted input,
output, ground truth, error and metadata.
([#29106](https://github.com/mastra-ai/mastra/pull/29106))

- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
## @mastra/inngest@1.11.1-alpha.0

### Patch Changes

- Fixed `InngestRun.watch()` crashing the process with an unhandled
rejection when the Inngest realtime subscription fails to connect (for
example `fetch failed` or a WebSocket error). The failure is now logged
as a warning and the run still completes through polling. Unwatching now
also closes the realtime connection instead of leaving it open.
([#29103](https://github.com/mastra-ai/mastra/pull/29103))

- Fixed Inngest durable agents ending the `model_generation` trace span
without `responseId` and `responseModel`. The span now carries the
provider's response id and model, the same as non-durable agents. Fixes
#29053. ([#29180](https://github.com/mastra-ai/mastra/pull/29180))

- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
## @mastra/temporal@0.5.0-alpha.1

### Patch Changes

- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
  - @mastra/deployer@1.82.0-alpha.1
## @mastra/e2b@0.16.1-alpha.0

### Patch Changes

- dependencies updates:
([#27993](https://github.com/mastra-ai/mastra/pull/27993))
- Updated dependency [`e2b@^2.62.0`
↗︎](https://www.npmjs.com/package/e2b/v/2.62.0) (from `^2.61.0`, in
`dependencies`)
- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
## @mastra/e2b-desktop@0.1.12-alpha.0

### Patch Changes

- dependencies updates:
([#27993](https://github.com/mastra-ai/mastra/pull/27993))
- Updated dependency [`e2b@^2.62.0`
↗︎](https://www.npmjs.com/package/e2b/v/2.62.0) (from `^2.61.0`, in
`dependencies`)
- Updated dependencies
[[`5735a50`](5735a50c6c),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/e2b@0.16.1-alpha.0
  - @mastra/core@1.82.0-alpha.1
## @mastra/platform-workspace@1.11.1-alpha.0

### Patch Changes

- dependencies updates:
([#27993](https://github.com/mastra-ai/mastra/pull/27993))
- Updated dependency [`e2b@^2.62.0`
↗︎](https://www.npmjs.com/package/e2b/v/2.62.0) (from `^2.61.0`, in
`dependencies`)
- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
## create-mastra@1.38.0-alpha.1


## @internal/playground@1.38.0-alpha.1

### Patch Changes

- Updated dependencies
[[`0900c20`](0900c20592),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`3314289`](33142896c1),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/client-js@1.59.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
  - @mastra/playground-ui@68.0.0-alpha.1
  - @mastra/react@1.14.1-alpha.1

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-11-02 16:03:13 +01:00
Dane - AI (Mastra)
7645fce513
chore(deps): update dependency @google-cloud/pubsub to v6 (#29113)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-11-01 23:03:34 +01:00
dane-ai-mastra[bot]
eed9cc214e chore: version packages 2026-11-01 04:20:58 +00:00
Abhi Aiyer
6ede0f10f6
fix(valkey-streams): trim saved thread-stream events (#29095)
> Opened by @abhiaiyer91

Valkey thread streams currently retain saved run events until their size
limit or idle TTL. Implement run-scoped trimming so persisted stream
parts can be removed without deleting other runs, pinned entries, or
retained terminal events. Closes #28974.

Before: `await pubsub.trimTopic(topic, { runId })` leaves all entries
behind. After: it removes that run's eligible entries; `producedBefore`
bounds removal to saved parts and `retainTerminalEvents` preserves
terminal markers. Cleanup now snapshots the current stream tail, so
ongoing publishers cannot extend the scan indefinitely or have their
later entries deleted by that scan.

## Proof
- Real-Valkey concurrency regression: pause the cleanup scan, publish
another target-run entry, resume, and assert only the in-flight entry
remains. This test failed before the tail-snapshot fix (`[]` instead of
`['in-flight']`) and passed after it.
- Existing real-Valkey integration covers more than one scan page,
other-run isolation, pinned entries, produced-time cutoff, and retained
terminal events; the targeted integration and lease suite passed (7/7).
- Full `@mastra/valkey-streams` tests passed (8 files, 42 tests);
package typecheck, lint (0 errors/warnings), formatting, diff check, and
ESM/CJS build passed.
- Earlier local Life Coach run with Postgres and Valkey emitted a finish
event and left 2 stream entries with the patched adapter, compared to
117 before.

## Unverified
- The Life Coach stream iterator remained open after finish, so that
isolated run was stopped by timeout; it is not a completed end-to-end
test. No sustained-production-throughput or multi-process soak test was
run.

## Risk: medium
- Trimming is best-effort and run-scoped, but removal is irreversible.
Real-Valkey tests cover the bounded concurrent-publish race and
preservation predicates; production scheduling and very large/live
streams have not been soak-tested.

Patch changeset included for `@mastra/valkey-streams`.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## ELI5

When a run is saved, Valkey can keep its stream entries around and use
extra storage. This change removes eligible entries for that run while
keeping entries that must remain.

## What changed

- Added `ValkeyStreamsPubSub.trimTopic` to delete entries for a
specified `runId`.
- When `producedBefore` is set, the method deletes only entries with a
numeric `data.producedAt` at or before the cutoff. It skips pinned
entries.
- When `retainTerminalEvents` is enabled, the method preserves
`run-registered`, `run-completed`, `run-failed`, and `run-aborted`
events, except those marked `suspended`.
- The scan stops at the stream tail captured before scanning starts.
Entries published during the scan are not included.
- Trimming failures are logged as warnings and do not propagate.
- Added integration tests for run-specific trimming, cutoff and
pinned-entry behavior, terminal-event retention, and entries published
during a scan.
- Added a patch changeset for `@mastra/valkey-streams`.

## Validation

The supplied report says six Valkey integration tests passed and the
package TypeScript check, lint, and build succeeded. It also reports a
local Postgres and Valkey test that left two stream entries after a
finish event, down from 117. The stream iterator remained open, so a
timeout stopped the isolated process; the Valkey regression suite
completed normally.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-31 14:29:32 -07:00
dane-ai-mastra[bot]
77076a1012 chore: version - exit prerelease mode 2026-10-30 23:02:25 +00:00
dane-ai-mastra[bot]
582ef17a2b
chore: version packages (alpha) (#28888)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-10-30 14:56:53 -07:00
mastra-platform[bot]
b0e86c50e2
fix(redis-streams): stop subscribe() hanging forever when Redis stops responding before connecting (#28941) 2026-10-30 11:06:52 -07:00
mastra-platform[bot]
39edceea4f
fix(core): open thread subscriptions with initial history when the lease read stalls or fails (#28924) 2026-10-30 10:39:35 -07:00
Ward Peeters
ba8b955b1a
fix(redis-streams,valkey-streams): expire idle streams by default and clear them on thread delete (#28968)
A production Factory deployment ran out of Redis memory. Every key was a
`mastra:topic:*` stream, and almost all of them (3,673 thread streams)
had no expiry. The deployment built `new RedisStreamsPubSub({ url })`
with defaults, and the default `streamIdleTtlMs` was `0`, meaning
streams never expire. Thread streams are open-ended and never get
`clearTopic()`, so they piled up forever. The Redis instance used
`volatile-lru`, which can't evict keys without a TTL, so writes just
started failing.

This PR makes `@mastra/redis-streams` and `@mastra/valkey-streams` give
every stream a sliding 7-day idle TTL by default. Writes refresh it, and
live subscribers keep receiving events after a stream expires because
the read loop recreates it. Streams are an event bus, not storage:
replay history lives in the caching layer and storage, so the backlog
only matters for grouped workers that are offline longer than the TTL.

```ts
// Before: streams never expire unless you opt in
new RedisStreamsPubSub({ url });

// After: 7-day idle TTL by default; opt out explicitly
new RedisStreamsPubSub({ url });
new RedisStreamsPubSub({ url, streamIdleTtlMs: 0 });
```

Both pubsubs now log one warning after they first connect if Redis has
`maxmemory` set with any policy other than `noeviction`. A TTL makes
streams, consumer groups and lease keys evictable under `volatile-*`,
which would silently drop events and leases (ElastiCache defaults to
`volatile-lru`). The check uses `INFO memory` because managed Redis
often blocks `CONFIG GET`.

`memory.deleteThread()` now also deletes the thread's event stream
through the registered Mastra pubsub. This runs fire-and-forget, so
neither deleting a thread nor `memory.settled()` waits on Redis during
an outage. It's skipped on older core versions. Agents configured with
their own pubsub still rely on the TTL.

The docs now require `noeviction`; the Helm guide used to say a
`volatile-*` policy was fine. They also add a runbook for reclaiming
streams created before the TTL: switch to `noeviction`, upgrade the
writers, then expire each stream relative to its last write. Tested
against real Redis/Valkey containers, including a real TTL expiry while
a subscriber is listening and a thread delete with Redis unreachable.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

Redis and Valkey can keep old event streams forever, which uses storage.
This change removes streams after seven days without writes by default
and cleans up a thread’s stream when the thread is deleted.

## What changed

- Redis and Valkey streams now use a sliding seven-day idle TTL by
default. Writes refresh the TTL; reads do not. Set `streamIdleTtlMs: 0`
to disable expiry.
- The read loop recreates an expired stream so a live subscriber can
continue receiving events.
- After connecting, each pubsub warns once when `maxmemory` is enabled
with a policy other than `noeviction`. The check uses `INFO memory`.
- Thread deletion starts best-effort stream cleanup through the
registered Mastra pubsub without waiting for Redis. Older core versions
skip this cleanup.
- Documentation and the local Docker configuration now recommend
`noeviction`. The documentation also explains how to expire existing
streams that lack a TTL.

## Tests

Tests cover TTL behavior, delivery after expiry for live subscribers,
eviction-policy warnings, and thread-stream cleanup. Test execution
results were not provided.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-30 11:51:23 +01:00
mastra-platform[bot]
52265cba91
Fix run abort route answering false on a server that does not own the run (#28522) 2026-10-29 14:11:05 -07:00
Dane - AI (Mastra)
cf92428a76
chore(deps): update docker images (#28011)
This PR contains the following updates:

| Package | Update | Change |
|---|---|---|
| amazon/aws-cli | minor | `2.36.44` → `2.39.1` |
| clickhouse/clickhouse-server | minor | `26.6.1.1193` → `26.10.3.7` |
| cr.weaviate.io/semitechnologies/weaviate | minor | `1.39.13` →
`1.40.3` |
|
[docker.elastic.co/elasticsearch/elasticsearch](https://www.elastic.co/products/elasticsearch)
([source](https://redirect.github.com/elastic/elasticsearch)) | minor |
`9.4.3` → `9.5.5` |
| edoburu/pgbouncer | minor | `v1.25.2-p0` → `v1.26.1-p0` |
|
[ghcr.io/shopify/toxiproxy](https://redirect.github.com/Shopify/toxiproxy)
| minor | `2.12.0` → `2.13.0` |
| inngest/inngest | minor | `v1.34.0` → `v1.49.0` |
| mcr.microsoft.com/azure-storage/azurite | minor | `3.35.0` → `3.37.0`
|
| [mongo](https://hub.docker.com/_/mongo)
([source](https://redirect.github.com/docker-library/mongo)) | patch |
`7.0.37` → `7.0.45` |
| pgvector/pgvector | patch | `0.8.4-pg16` → `0.8.7-pg16` |

---

> [!WARNING]
> Some dependencies could not be looked up. Check the [Dependency
Dashboard](../issues/9505) for more information.

---

### Release Notes

<details>
<summary>elastic/elasticsearch
(docker.elastic.co/elasticsearch/elasticsearch)</summary>

###
[`v9.5.5`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.5.5):
Elasticsearch 9.5.5

[Compare
Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.5.4...v9.5.5)

Downloads: <https://elastic.co/downloads/elasticsearch>
Release notes:
<https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.5.5-release-notes>

###
[`v9.5.4`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.5.4):
Elasticsearch 9.5.4

[Compare
Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.5.3...v9.5.4)

Downloads: <https://elastic.co/downloads/elasticsearch>
Release notes:
<https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.5.4-release-notes>

###
[`v9.5.3`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.5.3):
Elasticsearch 9.5.3

[Compare
Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.5.2...v9.5.3)

Downloads: <https://elastic.co/downloads/elasticsearch>
Release notes:
<https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.5.3-release-notes>

###
[`v9.5.2`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.5.2):
Elasticsearch 9.5.2

[Compare
Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.5.1...v9.5.2)

Downloads: <https://elastic.co/downloads/elasticsearch>
Release notes:
<https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.5.2-release-notes>

###
[`v9.5.1`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.5.1):
Elasticsearch 9.5.1

[Compare
Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.5.0...v9.5.1)

Downloads: <https://elastic.co/downloads/elasticsearch>
Release notes:
<https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.5.1-release-notes>

###
[`v9.5.0`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.5.0):
Elasticsearch 9.5.0

[Compare
Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.4.8...v9.5.0)

Downloads: <https://elastic.co/downloads/elasticsearch>
Release notes:
<https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.5.0-release-notes>

###
[`v9.4.8`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.4.8):
Elasticsearch 9.4.8

[Compare
Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.4.7...v9.4.8)

Downloads: <https://elastic.co/downloads/elasticsearch>
Release notes:
<https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.4.8-release-notes>

###
[`v9.4.7`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.4.7):
Elasticsearch 9.4.7

[Compare
Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.4.6...v9.4.7)

Downloads: <https://elastic.co/downloads/elasticsearch>
Release notes:
<https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.4.7-release-notes>

###
[`v9.4.6`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.4.6):
Elasticsearch 9.4.6

[Compare
Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.4.5...v9.4.6)

Downloads: <https://elastic.co/downloads/elasticsearch>
Release notes:
<https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.4.6-release-notes>

###
[`v9.4.5`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.4.5):
Elasticsearch 9.4.5

[Compare
Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.4.4...v9.4.5)

Downloads: <https://elastic.co/downloads/elasticsearch>
Release notes:
<https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.4.5-release-notes>

###
[`v9.4.4`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.4.4):
Elasticsearch 9.4.4

[Compare
Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.4.3...v9.4.4)

Downloads: <https://elastic.co/downloads/elasticsearch>
Release notes:
<https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.4.4-release-notes>

</details>

<details>
<summary>Shopify/toxiproxy (ghcr.io/shopify/toxiproxy)</summary>

###
[`v2.13.0`](https://redirect.github.com/Shopify/toxiproxy/blob/HEAD/CHANGELOG.md#2130---2026-10-14)

[Compare
Source](https://redirect.github.com/Shopify/toxiproxy/compare/v2.12.0...v2.13.0)

- Reject slicer toxics with `average_size` <= 0 or `size_variation`
outside `[0, average_size]`,
and stop the slicer from recursing forever on degenerate sizes
([#&#8203;774](https://redirect.github.com/Shopify/toxiproxy/issues/774),
[@&#8203;iyaz-shaikh](https://redirect.github.com/iyaz-shaikh))
- Add optional bearer token auth for the HTTP API via
`TOXIPROXY_AUTH_TOKEN`, supported by the Go
client and `toxiproxy-cli`, and warn when the API is unauthenticated on
a non-loopback address

([#&#8203;774](https://redirect.github.com/Shopify/toxiproxy/issues/774),
[@&#8203;iyaz-shaikh](https://redirect.github.com/iyaz-shaikh))
- Fix removing a toxic hanging forever, and freezing the whole HTTP API,
when the downstream
toxic or connection is already gone
([#&#8203;760](https://redirect.github.com/Shopify/toxiproxy/issues/760),
[@&#8203;matheusgb](https://redirect.github.com/matheusgb))
- Fix the bandwidth toxic hanging or panicking when `rate` <= 0
([#&#8203;764](https://redirect.github.com/Shopify/toxiproxy/issues/764),
[@&#8203;jaideeppyne](https://redirect.github.com/jaideeppyne))
- Update go version to 1.27
([#&#8203;777](https://redirect.github.com/Shopify/toxiproxy/issues/777),
[@&#8203;The-Black-Fox](https://redirect.github.com/The-Black-Fox))
- Update various go packages

</details>

---

### Configuration

📅 **Schedule**: (UTC)

- Branch creation
  - "before 6am every weekday"
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

👻 **Immortal**: This PR will be recreated if closed unmerged. Get
[config
help](https://redirect.github.com/renovatebot/renovate/discussions) if
that's undesired.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend Renovate
CLI](https://redirect.github.com/renovatebot/renovate).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4xNzUuMSIsInVwZGF0ZWRJblZlciI6IjQ0LjE5Mi4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJhdXRvbWF0aW9uIl19-->

---------

Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-29 15:53:50 +00:00
dane-ai-mastra[bot]
4f12792ed1
chore: version packages (alpha) (#28722)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-10-29 08:37:01 -07:00
Ward Peeters
1607791e91
fix(core): trim finished runs of memoryless agents from thread-stream topics (#28746)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-28 18:08:58 -07:00
Dane - AI (Mastra)
f2ad3844f3
chore(deps): update opentelemetry (#28581)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-28 16:48:54 -07:00
shreyas-mastra
6deb7c0a1e
chore(security): Vanta 2026-10-28 - qs 6.16.1, @modelcontextprotocol/node 2.1.1 floors (#28716)
Refs mastra-ai/platform#2931

Vanta remediation pass for 2026-10-28. It follows #28325, which has
merged.

## 2026-10-28 update
Today's live Vanta export added 3 advisory IDs on `mastra-ai/mastra`
(GitHub/Dependabot). This PR fixes the two that have a published fix and
leaves deepmerge, which has none.

| Package | From → To | Finding | Where | How |
|---|---|---|---|---|
| qs | 6.16.0 → 6.16.1 | CVE-2026-110816 / GHSA-6pq8-jhm2-2vcw (Medium
×4) | root `pnpm-lock.yaml`, 9 `e2e-tests/*` lockfiles,
`scripts/gh-issue-triage/package-lock.json` | Raised the existing floors
`qs@>=6.7.0 <6.16.0: 6.16.0` → `<6.16.1: 6.16.1` and `'qs@<6.7.0':
6.16.0` → `6.16.1`. Raised the npm override in gh-issue-triage. |
| @modelcontextprotocol/node | 2.0.0 → 2.1.1 | GHSA-f897-62hf-6gxq (High
×2; HTTP transports had no body-size or batch-length limit) | root
`pnpm-lock.yaml` (published `@mastra/mcp` 1.18.0, used as `mcp-legacy`),
`scripts/gh-issue-triage/package-lock.json` | New pnpm override
`'@modelcontextprotocol/node@>=2.0.0 <2.1.0': 2.1.1` and a new npm
override in gh-issue-triage. I used 2.1.1 rather than 2.1.2 because
2.1.2 needs a `@modelcontextprotocol/server` peer of `^2.4.0`, while
`@mastra/mcp` 1.18 resolves server 2.3.1. `packages/mcp` itself already
uses node 2.1.2. |

- Both versions are older than the 1-day `minimumReleaseAge`: qs 6.16.1
was published 10-14 and node 2.1.1 on 10-02. I did not use qs 6.16.2
(10-27), which is too new.
- No floors or overrides were removed; existing ones were only raised.
- `pnpm install --frozen-lockfile --lockfile-only` passes at the root.
Each nested e2e workspace was regenerated with `pnpm install
--lockfile-only`.
- Added the empty changeset
`.changeset/vanta-2026-10-28-security-floors.md`. No published package
changes.
- After this change, osv-scanner over all 33 lockfiles reports only the
documented leftovers:
  - zod 3.25.76 (zod v3 compat line on purpose)
  - ai 4.3.19 / `@ai-sdk/provider-utils` 2.2.8 (vendored v4 line)
- `@azure/msal-node` 5.6.0 / 6.0.1 (fix 7.1.0 needs majors in identity
and teams)
  - `@opentelemetry/core` 1.30.1 (needs pubsub 6)
  - extract-zip 2.0.1 (no patched release)
- deepmerge 4.3.1 (CVE-2026-93753 / GHSA-gcqx-w48m-w2h4; no patched
release, 4.3.1 is the latest)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

This PR updates dependency rules and package versions to use newer
releases, including security fixes for `qs` and
`@modelcontextprotocol/node`. It also updates AI SDK dependencies and
does not include published package changes.

## Summary

- Raise the `qs` override target to `6.16.1` in the root and e2e
workspaces and in `scripts/gh-issue-triage`.
- Add an `@modelcontextprotocol/node` override at `2.1.1` in the root
workspace and `scripts/gh-issue-triage`.
- Update AI SDK dependency versions across package manifests and add a
changeset describing the provider-utils update.
- Add an empty changeset for the security-only dependency updates.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

## Update 2026-10-28 (afternoon)
- Merged origin/main (clean).
- Bumped the AI SDK v5 compat line from @ai-sdk/provider-utils 3.0.41 to
3.0.42 (patch, published 2026-10-21; it blocks `.internal` hostnames in
download URL validation and encodes provider-returned IDs in request
paths). This addresses the Aikido /mastra Highs AIKIDO-2026-10990 and
AIKIDO-2026-10981:
- ai 5.0.277 → 5.0.282, @ai-sdk/gateway 2.0.168 → 2.0.173 (vendored
ai_v5)
- anthropic 2.0.109 → 2.0.110, google 2.0.103 → 2.0.106, groq 2.0.58 →
2.0.59, openai 2.0.135 → 2.0.137, openai-compatible 1.0.58 → 1.0.59, xai
2.0.99 → 2.0.100
- Nested lockfiles updated: memory/integration-tests and
playground/e2e/kitchen-sink.
- No new overrides. Changeset:
`vanta-2026-10-28-ai-sdk-v5-provider-utils.md`.
2026-10-28 18:51:27 -04:00
dane-ai-mastra[bot]
6a191d46dc chore: version - exit prerelease mode 2026-10-28 13:15:52 +00:00
dane-ai-mastra[bot]
e6764d26da
chore: version packages (alpha) (#28625)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-10-27 15:40:38 -05:00
mastra-platform[bot]
a7cc444233
fix(core): expose recoverable run outcomes to thread subscribers (#28621)
Thread subscribers now receive an ordered `thread-run-status` frame for
success, failure, abort, or suspension without closing the subscription.
Initial history also reports `activeRunId`, so an idle snapshot no
longer has to stand in for a run outcome. Retained outcomes reach late
subscribers even after saved output has been trimmed, with execution
authority preserved to reject superseded attempts.

For example, a subscription opened after a run finishes now receives
`thread-history` followed by `thread-run-status` with `{ status:
"success" }`, rather than history alone. Recovery is bounded by PubSub
retention; upgrade both `@mastra/core` and `@mastra/redis-streams` for
recovery after trimming. Client callback types, documentation, and
package changesets are included.

Verified with core build and typecheck, 220 core tests across the
thread-stream suites and four execution engines, 91 client tests, 55
Redis integration tests, and three HTTP conformance cases covering
plain, durable, and evented agents. Real Redis regressions cover restart
after trimming and superseded completion/abort after registration
eviction. The live outcome scenario also passed 20 repeated four-engine
runs. Typed lint, formatting, MDX prose checks, and final adversarial
review passed.

Fixes COR-1500.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

A thread subscription can now tell clients when a run succeeds, fails,
stops, or pauses without ending the subscription. This helps clients
understand what happened after reconnecting, even when the run’s saved
output is no longer available.

## Summary

- Added opt-in `withRunStatus` support to core, server, and client
thread subscriptions. Enabled subscriptions can receive ordered
`thread-run-status` chunks for `success`, `failed`, `aborted`, or
`suspended` outcomes.
- Added `activeRunId` to initial thread history so clients can identify
the run that held the thread lease during history loading.
- Added terminal-event retention during output trimming. Recovery
depends on PubSub retention. Redis Streams users must upgrade
`@mastra/core` and `@mastra/redis-streams` together.
- Updated React `useChat` to request run statuses and apply outcomes to
the matching run. Added API types, documentation, and changesets.

## Validation

The PR objectives report passing core build and typecheck, core and
client tests, Redis integration tests, HTTP conformance cases, lint and
formatting checks, and documentation checks. They also report repeated
run-outcome checks and adversarial review. These results are
author-reported.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-27 15:15:42 -05:00
mastra-platform[bot]
e718b91a64
fix(core): stop observe offsets past the log end replaying the whole run (#28348)
> Opened by @roaminro

Refs #28324 (partial fix; the issue stays open, see Open question)

## What changed

On transports that keep old messages, such as Redis Streams,
`observe(runId, { offset })` replayed the **whole run** when the offset
was at or past the end of the run's log. A new subscriber receives the
stream's retained frame 0. `CachingPubSub`'s log-restart check compared
the counter with the subscriber's cursor, and with a past-end offset the
counter is always below the cursor. So frame 0 looked like a log restart
and everything replayed. This affected every Redis setup, durable agents
included.

`subscribeFromOffset` in `packages/core/src/events/caching-pubsub.ts`
now decides restarts by frame 0's identity:
- On attach it records the id of the log's frame 0. A frame 0 arriving
below the cursor is a restart only if the log's frame 0 id has changed.
A retained copy of the same frame is acknowledged and dropped.
- On a restart it delivers the new segment from the cache log, not from
the frame that signalled the restart, because that frame may be a
delayed copy of the old frame 0.
- If an id can't be read, it falls back to the counter. If the read
failed at attach, the counter is compared with the log's end at that
point, so an intact log isn't mistaken for a restart.

The cursor stays at the offset, so the documented contract holds: an
offset beyond the retained index replays nothing and skips live events
below it. If the retained log expired and the run restarts numbering at
0, the observer resyncs to the new segment.

**Known limitation (not fixed):** `createEventedAgent` on a
cross-process pubsub with the default in-memory server cache, which is
the case in the issue title. Its step executor publishes unnumbered
frames there, because each process would count them separately. These
frames can't be skipped by position:
- `CachingPubSub` can't see where a transport's replay starts.
- Redis Streams trims the start of its backlog with `MAXLEN`.
- Other transports replay nothing.

That configuration now logs a one-time warning. It names both possible
causes, a failed cache append or no shared server cache, and recommends
a shared cache such as `RedisServerCache`. Both observe docs pages
(`docs/harness/durable-agents.mdx`,
`reference/agents/durable-agent.mdx`) document the requirement.

## Tests

- `packages/core/src/events/caching-pubsub.test.ts`:
- A past-end offset delivers nothing, then delivers live frames once the
log reaches the offset.
  - An offset inside a short log resumes without resyncing.
  - A short log past the offset expires and a new segment resyncs.
- A stale copy of the old frame 0 doesn't stand in for the new segment.
  - A failed frame 0 id read falls back to the counter.
  - A restart is detected by id when the counter read fails.
  - A failed id read at attach doesn't replay an intact log.
  - A log that is empty at attach still delivers the next run.
  - Unnumbered frames produce exactly one warning.
- `pubsub/redis-streams/src/observe-offset.test.ts`: new file against
real Redis, with the model served by aimock.
- It covers four setups: durable or evented engine, with the default or
`RedisServerCache` server cache.
- Each setup runs three cases: resuming a live run, a past-end offset on
a finished run, and a mid-run offset on a finished run.
- The evented/default setup checks for the warning and that the
observation still ends on `finish`.

| Check | Result |
|---|---|
| `packages/core` `src/events` | ✅ 323/323 |
| core `tsc --noEmit` | ✅ clean |
| `pubsub/redis-streams` observe-offset, real Redis, rebuilt core | ✅
12/12 |
| oxlint, prettier, docs `format:mdx:check` + remark | ✅ clean |

## Proof

- I checked each new unit test against the previous commit or the code
before its fix, and it failed there. The empty-log test is the
exception: it passes before its fix by design. It guards against
limiting the counter check whenever no id was captured, and it fails if
that change is applied.
- With `caching-pubsub.ts` at base and core rebuilt, 6 of the 12 Redis
tests fail. These are the past-end case in the three numbered setups,
where the whole run replays from `start`, and all three evented/default
cases, where no warning is logged.
- NikAiyer independently reproduced the base failures and the fixed-head
results.

## Unverified

- Redis Cluster deployments, and transports other than Redis Streams and
the in-process pubsub.
- `durable-agent-tool-writer` and `durable-agent-client-tool-pending`
already fail on `main`, with or without this change.

## Risk: low

The change is limited to the restart check and cursor handling in
`CachingPubSub.subscribeFromOffset`, plus a warning. Offset 0 and the
default observe path (no offset) don't read the counter and behave as
before.

## Open question

#28324 stays open for evented agents on a cross-process pubsub without a
shared server cache. Should evented step frames be numbered without a
shared cache (or a shared cache be required), or is the documented
limitation enough?

**Merge note:** the first commit (`162c296a`) says `Fixes #28324`. When
squash-merging, change it to `Refs #28324` so the issue isn't closed.

---------

Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-27 13:44:02 -05:00
mastra-platform[bot]
5381b414b3
test(redis-streams): fix the Lambda-death signal-drain test that times out on main (#28533)
> Opened by @abhiaiyer91

## Summary

`drains signal-enqueued from losers that die immediately (Vercel
Lambda-death)` in
`pubsub/redis-streams/src/agent-thread-cross-process.test.ts` times out
on main. It never actually exercised the scenario in its name. This is a
test-only fix. Product code is unchanged, so there's no changeset.

Two fixture problems caused it:

1. **Each worker had its own agent id.** The fixture used
`${WORKER_ID}-agent` (`test-fixtures/agent-thread-worker.entry.ts:59` on
main). The losers therefore treated the winner's run as another agent's
run. The cross-agent guard
(`packages/core/src/agent/thread-stream-runtime.ts:6852-6854`,
`:6900-6905`) correctly returned `blocked / thread-blocked` and
published nothing, so the winner had nothing to drain. Lambdas are
replicas of one deployed agent, so they should share an id.
2. **Timing.** The winner's run was a fixed 1500ms, but a worker takes
about 2.3s to boot through tsx. The losers always sent after the lease
was already free, never "while the run is in flight".

Changes:
- The fixture accepts `AGENT_ID` (default is still `${WORKER_ID}-agent`,
so other tests are unaffected). The test sets one shared id.
- The fixture accepts `HOLD_SIG_IDS` plus a `finish-run` command. The
winner's sig-1 run stays open until both losers have exited, then the
test ends it explicitly.
- The test asserts the winner owns sig-1, and that both losers lost the
lease and started no runs. It then waits for sig-1/2/3 to finish on the
winner. I removed the fixed 300ms settle sleep, so the test waits on
product events only.

## Proof

- On main: fails with `waitFor timed out after 20000ms` waiting for 3
`run-finished`. Only sig-1 finishes on the winner.
- With the fix, against core built from main: **5/5 consecutive
passes**.
- Reverting only the shared `AGENT_ID` makes it fail again with the same
20s `waitFor` timeout.
- `agent-thread-cross-process.test.ts`: 10/10 pass. On main this file
failed because of this test.
- Full redis-streams suite, run per file:
- Fail both on main and on this branch: `cluster`,
`durable-worker-takeover`, `observe-cross-process`. On main this run
only checked `observe-cross-process`; the other two are already known to
fail on main.
- `agent-thread-publish-lag`: 1 test failed. Known to fail on main; I
didn't rerun it on main myself.
- `auth-e2e` and `fanout-sharing` failed once in the long serial run.
Both passed 2/2 when rerun alone on this branch, and both pass on main.
They don't load the changed fixture.
- oxlint: 0 warnings, 0 errors. prettier: no changes. `tsc --noEmit` in
redis-streams gives the same 14 errors on main and on this branch (a
`RedisStreamsPubSub` vs `PubSub` mismatch, unrelated).

## Unverified

- **A sender that exits without closing its pubsub can still drop the
forwarded signal (product gap, out of scope).** With the shared id, the
losers settle `deliver` (I logged `settled.action`). That means they
take the `ifActive: deliver` branch at `thread-stream-runtime.ts:7027`,
not the lease-loser hand-off at `:2589`, which settles `remote`. Both
branches publish `signal-enqueued` with the fire-and-forget `#publish`
(`:2690-2692`) and resolve `accepted` before the publish reaches Redis.
So my earlier claim about the pattern holds, and it applies to the path
this test actually runs. The test passes only because the fixture awaits
`pubsub.close()` before `process.exit`. `#publish` reaches
`RedisStreamsPubSub.publish` synchronously, which adds the publish to
`#pendingPublishes` (`pubsub/redis-streams/src/index.ts:431-434`) before
`accepted` resolves, and `close()` waits for those publishes through
`flush()` (`:1133`, `:894-899`). It isn't a case of the publish just
being fast locally. I checked this by temporarily skipping `close()` in
`send-and-exit`: **3/3 runs timed out** and sig-2/sig-3 never reached
the winner. So a Lambda that settles and exits (or is frozen or killed)
without closing or flushing its pubsub loses the signal. A fix would
await the publish before resolving `accepted` on the remote-owner
branches. That needs its own core PR. I didn't check whether the server
or serverless deployers close or flush the pubsub on shutdown.
- **A non-durable owner that dies mid-run loses queued signals (separate
product gap, out of scope).** In an earlier run of this test the losers
won the free lease. When the lease owner died, sig-2/sig-3 stayed
stranded after the lease expired, and a later signal woke a fresh run
without replaying them. Signals are only admitted by the process that
owns the runId (`thread-stream-runtime.ts:1444`). Dead-owner recovery
only covers durable runs (`:5574-5584`, `:5824+`). This needs its own
issue and design (orphaned-signal recovery, with duplicate-delivery
risk).
- Not checked on CI runners. The fix doesn't depend on boot timing
anymore, so I expect it to behave the same there.

## Risk

low. Only test and fixture files changed, and the new fixture options
are opt-in.

Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-26 19:48:31 -05:00
mastra-platform[bot]
7c20f008c8
fix(core): abort evented workflow steps running on other replicas on cancel (#28489)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9

Closes #28444

## Summary
With evented workflows and workers on several replicas,
`workflow.cancel` is handled by only one processor in the consumer
group. That processor aborted only the AbortControllers in its own
process. If the step was running on another replica, its `abortSignal`
never fired and the step kept running, even though the run itself
settled as `canceled`.

This PR broadcasts cancellation on one shared control topic,
`workflow.control`:
- **One subscription per processor.** A processor opens a single
ungrouped subscription to `workflow.control` when it runs its first step
and keeps it for its lifetime. Transports share one reader/subscription
per topic per process, so the cost does not grow with runs or steps (no
per-run topics, consumer groups or Pub/Sub subscriptions to clean up).
It reads from `latest`; a failed subscribe is logged and retried on the
next step.
- **Cancel publishes the abort before finalizing.**
`processWorkflowCancel` aborts its local controllers, publishes an abort
for the run, and only then calls `endWorkflow`. If the publish fails,
the error propagates and the event is redelivered before anything is
finalized.
- **Receivers only act on runs they are executing.** Each processor
counts in-flight steps per run (a nested step counts against its
ancestors too). An abort for a run with no in-flight step here is
ignored. Otherwise the run's own controller is marked cancel-handled
(the step unwinds without a second `workflow.cancel`), and nested runs
are aborted like the local `cancelRunAndChildren`, so each settles
canceled. Parent/child links come from the step event's `parentWorkflow`
chain, so this works even if the processor never saw the child's
`workflow.start`.
- **Cancels issued before a processor could hear them.** The control
subscription starts at `latest`, so an abort published while it is still
being opened is missed. When `#dispatch` takes a `workflow.step.run`, it
records, before its state load, whether the subscription was ready and
how many aborts this processor has ignored for runs with no step
retained. The step re-reads the persisted status of its run and its
ancestors when the subscription wasn't ready at dispatch, or when that
count changed before the step was retained, and aborts if any of them is
`canceled` (`EventedRun.cancel` persists that before publishing). Other
steps skip the read. The count can't evict anything; when it changes,
the overlapping steps re-check.
- **A canceled step never starts its body.** If the pre-step check finds
the step's own run canceled, the step returns without running. If it
finds only an ancestor canceled, the body is skipped with a `canceled`
result, and the existing post-step branch publishes this run's
`workflow.cancel`, so the nested run settles canceled.
- **No state left on replicas that never finalize the run.** When a
run's last in-flight step is released on a processor that didn't start
it, the controller and parent links recorded by the step path are
dropped.
- **Double delivery does nothing.** Aborting is idempotent; a
redelivered or self-echoed abort finds the controllers already aborted.
Tests assert exactly one `workflow.end` per run.

Assumption: a shared broadcast topic over per-run topics. Cancels are
rare and tiny, so every processor seeing every abort is cheaper than
per-run subscriptions, which leaked topics/subscriptions on Google Cloud
Pub/Sub and churned consumer groups on Redis Streams.

## Proof
- `packages/core/src/workflows/evented/remote-cancel.test.ts`, suite
"workers on both replicas": two Mastra instances, both running workers,
over EventEmitter and UnixSocket, cancels from the owner and the other
replica, flat and nested. Asserts the step's `abortSignal` fires, the
run settles `canceled`, the only control subscription is
`workflow.control` and there is at most one per processor across all
runs, no in-flight step refs remain; with grouped delivery also exactly
one `workflow.end` and at least one cancel handled on a different
replica than the step. A pinned-topology test (steps on one replica,
start/end on the other) asserts the step-only replica keeps no
controllers or parent links. Five pinned-topology "skips a step …" tests
cover the fallback: (a) the step replica's `workflow.control` subscribe
is delayed and the run is canceled during the delay; (b) the abort lands
after the step's dispatch load but before it is retained (also asserts a
plain step on a ready subscription does no re-check); (c) same as (b)
with 2000 aborts for other runs published in that window; (d) a step
dispatched while the subscription is pending resumes after other steps
have run on the ready subscription; (e) nested: the parent is canceled
while the child's step is held after its dispatch load. Each asserts the
step body never ran, the run settles `canceled` and exactly one
`workflow.end` is published per run; (e) also asserts the child snapshot
is `canceled`. 26/26 pass. Proofs: **with the abort publish disabled**,
the 4 EventEmitter both-replica cases fail on the step-abort assertion.
**With the pre-body skip disabled**, (a)–(d) fail. **On the 7845d2be
processor (1024-entry set)**, (c) fails. **With the count comparison
removed**, (b) and (c) fail. **On 673eaf4a's processor (no per-step
dispatch decision)**, 2 tests fail, including (d). **On 13b89c3a's
processor (own-run skip only)**, (e) fails: the child body blocks on an
abort event that never fires, so the child never settles.
- `pubsub/redis-streams/src/evented-remote-cancel.test.ts` against a
real `redis-server`: canceller owner/other × sequential/dountil/nested ×
workers on both replicas or owner only. 7/7 pass on 434d531a. **With
core rebuilt with the abort publish disabled**, all 5 both-replica cases
fail.
- Other `src/workflows/evented` test files on 434d531a: 42/42 files, 560
tests pass (5 skipped). `evented-workflow.test.ts`: 315 pass / 5 skipped
on 434d531a (6 GB heap; it runs out of memory at a 3 GB heap on `main`
too). Earlier on this branch: `src/loop` 319/319 files and server
`handlers/workflows.test.ts` pass.
-
`packages/core/src/workflows/evented/cancel-check-read-failure.test.ts`
(EventEmitter and UnixSocket) fails only the pre-step cancel re-check
read. Two failing reads: the step event is retried twice, then the step
runs once and the run succeeds. Every read failing: three deliveries,
the last not retried, the run settles `failed` with the storage error
and the step never runs (exhaustion goes through
`#publishPendingFailure` → `errorWorkflow` → `workflow.fail`). The test
keeps the control subscribe failing so every delivery re-checks. 4/4
pass (with remote-cancel, 30/30). **With the old `.catch(() =>
undefined)` restored**, all 4 fail.
- Snapshot reads for one 100-step sequential run (EventEmitter,
`loadWorkflowSnapshot` counted): **402 before this change (d0227436),
302 after**. `main` is 302. The only remaining re-check is on the first
step after the subscription opens.
- Core `tsc --noEmit` clean; oxlint and prettier clean on changed files.

## Unverified
- **Google Cloud Pub/Sub:** no emulator here. With the fixed topic, each
process creates one `workflow.control-<instanceId>` subscription (like
other ungrouped topics on that adapter); not exercised by a test.
- **UnixSocket nested cases:** UnixSocketPubSub delivers the ungrouped
push `workflows` subscription to both replicas, so both process every
event (also on `main`, out of scope). A duplicate child run can finish
while the first call still blocks, so nested cases run only with grouped
delivery, and exactly-once `workflow.end` isn't checked on UnixSocket.
The UnixSocket flat cases don't fail with the abort publish disabled, so
they don't independently prove the fix there.
- **Unrelated redis-streams failures:** agent-thread-cross-process,
agent-thread-publish-lag, cluster and durable-worker-takeover fail in
the full package run, identically with `main`'s processor. `tsc` on
redis-streams reports errors only in the untouched
`test-fixtures/agent-thread-worker.entry.ts`.

## Risk: medium
- The cancel re-check (one full snapshot read for the run and each
ancestor) runs only for steps dispatched before the subscription was
ready, or whose dispatch→retain window overlapped an ignored abort for
any run. It doesn't run on every step. Around a burst of cancels,
unrelated in-flight steps re-check too. A failed read fails the step
event so it is retried. A status-only read would be cheaper but needs a
storage-contract change, so it's left for a follow-up.
- If the `workflow.control` subscribe fails, the step still runs (with a
warning) and re-checks storage on each delivery. Remote abort of an
already-running step then degrades to "the step finishes", and the run
still settles `canceled`. If `mastra.pubsub` is swapped, the old control
subscription is not unsubscribed.
- **Finalization is not a barrier on remote steps.** The cancelling
processor publishes the abort before `endWorkflow` but doesn't wait for
remote processors to observe it, so a remote step can run briefly after
the run is marked `canceled`. Its late events are dropped by the
dispatch guard (status `canceled`). A real barrier would need a registry
of which processors hold the run's steps.
- `startedRuns` entries on processors that start a run but never
finalize it are not cleaned up, matching the existing pattern for
controllers.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

When a workflow step runs on another server, canceling the workflow must
also tell that server to stop the step. This change broadcasts
cancellation and checks saved workflow status when a server may have
missed the notice.

## Changes

- Processors subscribe to the shared `workflow.control` topic and
broadcast workflow cancellations.
- Processors abort active steps for the canceled run, including nested
workflow steps.
- Before processing a step, a processor checks persisted run status if
it may have missed the cancellation broadcast. Read failures trigger
retries.
- The cancel handler publishes the abort before loading state or
finalizing the run. If publishing fails, it does not continue those
operations.
- Processors track active step references and clean up controller and
parent-link state, including on replicas that run steps but do not start
or finalize the run.

## Tests and limitations

- The author reports that `remote-cancel.test.ts` passed 26 tests.
- The author reports that `cancel-check-read-failure.test.ts` passed 4
tests.
- The author reports that the Redis Streams test passed 7 cases against
a real Redis server.
- The author reports that core TypeScript, lint, and formatting checks
passed. These results were not independently verified.
- Google Cloud Pub/Sub was not tested.
- Cancellation finalization does not wait for remote processors to
observe the abort. A remote step may continue briefly after the run is
finalized as canceled.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-26 17:35:12 -05:00
mastra-platform[bot]
9ffb79b17e
fix(core): stop closeOnSuspend durable streams closing before their chunks arrive on Redis Streams (#28504) 2026-10-26 17:34:30 -05:00
dane-ai-mastra[bot]
2c646b53d0
chore: version packages (alpha) (#28399)
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases

> The changelog information of each package has been omitted from this
message, as the content exceeds the size limit.

## @mastra/factory@0.25.0-alpha.5


## @mastra/langsmith@1.4.0-alpha.2


## @mastra/connect@1.2.0-alpha.2


## @mastra/core@1.80.0-alpha.4


## @mastra/playground-ui@66.0.0-alpha.4


## @mastra/acp@0.4.4-alpha.0


## @mastra/claude@0.3.4-alpha.0


## @mastra/cursor@0.3.4-alpha.0


## @mastra/openai@1.1.4-alpha.0


## @mastra/auth-auth0@1.2.6-alpha.0


## @mastra/auth-better-auth@1.1.9-alpha.0


## @mastra/auth-clerk@1.3.2-alpha.0


## @mastra/auth-cloud@1.2.8-alpha.0


## @mastra/auth-firebase@1.1.5-alpha.0


## @mastra/auth-google@0.1.7-alpha.0


## @mastra/auth-neon@0.3.5-alpha.0


## @mastra/auth-okta@0.2.6-alpha.0


## @mastra/auth-studio@1.3.11-alpha.1


## @mastra/auth-supabase@1.1.6-alpha.0


## @mastra/auth-workos@1.6.8-alpha.0


## @mastra/agent-browser@0.5.5-alpha.0


## @mastra/browser-viewer@0.2.6-alpha.0


## @mastra/browser-firecrawl@0.2.5-alpha.1


## @mastra/stagehand@0.3.7-alpha.1


## @mastra/discord@1.3.2-alpha.0


## @mastra/slack@1.8.0-alpha.1


## @mastra/teams@0.1.3-alpha.0


## @mastra/telegram@0.2.4-alpha.0


## @mastra/client-js@1.57.0-alpha.4


## @mastra/react@1.13.0-alpha.4


## @mastra/isolated-vm@0.1.4-alpha.0


## @mastra/quickjs@0.1.5-alpha.0


## @mastra/deployer-cloud@1.80.0-alpha.4


## @mastra/deployer-cloudflare@1.2.39-alpha.1


## @mastra/deployer-netlify@1.2.39-alpha.1


## @mastra/deployer-sandbox@0.3.24-alpha.1


## @mastra/deployer-vercel@1.2.39-alpha.1


## @mastra/voyageai@0.4.3-alpha.0


## @mastra/longmemeval@1.1.39-alpha.4


## @mastra/livekit@0.5.2-alpha.0


## @mastra/opencode@0.1.39-alpha.4


## @mastra/parallel@0.1.4-alpha.0


## create-factory@0.3.1-alpha.4


## @mastra/code-sdk@1.15.1-alpha.5


## mastracode@0.48.1-alpha.4


## @mastra/arize@1.3.27-alpha.2


## @mastra/arthur@0.4.27-alpha.2


## @mastra/braintrust@1.3.24-alpha.2


## @mastra/datadog@1.4.19-alpha.2


## @mastra/deepeval@0.1.21-alpha.2


## @mastra/laminar@1.3.29-alpha.2


## @mastra/langfuse@1.6.3-alpha.2


## @mastra/observability@1.21.1-alpha.2


## @mastra/otel-bridge@1.5.19-alpha.2


## @mastra/otel-exporter@1.4.10-alpha.2


## @mastra/posthog@1.4.4-alpha.2


## @mastra/sentry@1.2.29-alpha.2


## @mastra/agent-builder@1.1.31-alpha.2


## @mastra/auth@1.1.5-alpha.0


## mastra@1.36.0-alpha.4


## @mastra/codemod@1.1.8-alpha.0


## create-mastra@1.36.0-alpha.4


## @mastra/deployer@1.80.0-alpha.4


## @mastra/editor@0.16.5-alpha.2


## @mastra/evals@1.10.9-alpha.0


## @mastra/fastembed@1.3.5-alpha.2


## @mastra/loggers@1.3.10-alpha.1


## @mastra/mcp@2.6.0-alpha.1


## @mastra/mcp-docs-server@1.3.10-alpha.7


## @mastra/mcp-registry-registry@1.1.7-alpha.0


## @mastra/memory@1.39.1-alpha.2


## @mastra/rag@2.7.1-alpha.0


## @mastra/schema-compat@1.3.17-alpha.0


## @mastra/server@1.80.0-alpha.4


## @mastra/google-cloud-pubsub@1.1.5-alpha.1


## @mastra/redis-streams@0.6.5-alpha.1


## @mastra/valkey-streams@0.5.7-alpha.1


## @mastra/elysia@0.1.20-alpha.4


## @mastra/express@1.5.24-alpha.4


## @mastra/fastify@1.5.24-alpha.4


## @mastra/hono@1.7.22-alpha.4


## @mastra/koa@1.7.24-alpha.4


## @mastra/nestjs@0.2.39-alpha.4


## @mastra/next@0.2.38-alpha.4


## @mastra/server-adapters-test-suite@0.1.4-alpha.2


## @mastra/tanstack-start@0.2.38-alpha.4


## @mastra/astra@1.1.3-alpha.0


## @mastra/azure-ai-search@0.1.2-alpha.1


## @mastra/chroma@1.1.5-alpha.0


## @mastra/clickhouse@1.27.0-alpha.1


## @mastra/cloudflare@1.7.6-alpha.0


## @mastra/cloudflare-d1@1.4.5-alpha.1


## @mastra/convex@1.7.6-alpha.0


## @mastra/couchbase@1.1.4-alpha.1


## @mastra/dsql@1.5.5-alpha.0


## @mastra/duckdb@1.16.0-alpha.1


## @mastra/dynamodb@1.4.5-alpha.1


## @mastra/elasticsearch@1.4.6-alpha.1


## @mastra/lance@1.5.2-alpha.1


## @mastra/libsql@1.27.1-alpha.0


## @mastra/mongodb@1.24.1-alpha.1


## @mastra/mssql@1.10.6-alpha.0


## @mastra/mysql@0.14.1-alpha.0


## @mastra/opensearch@1.1.4-alpha.1


## @mastra/oracledb@0.5.5-alpha.0


## @mastra/pg@1.33.0-alpha.2


## @mastra/pinecone@1.1.3-alpha.0


## @mastra/qdrant@1.1.6-alpha.0


## @mastra/redis@1.4.12-alpha.1


## @mastra/s3vectors@1.1.5-alpha.0


## @mastra/spanner@1.10.1-alpha.1


## @mastra/turbopuffer@1.2.3-alpha.0


## @mastra/turso@0.1.18-alpha.0


## @mastra/upstash@1.5.5-alpha.1


## @mastra/valkey@0.2.8-alpha.1


## @mastra/vectorize@1.1.4-alpha.0


## @mastra/weaviate@0.1.4-alpha.0


## @mastra/voice-aws-nova-sonic@0.2.6-alpha.0


## @mastra/voice-azure@0.12.3-alpha.0


## @mastra/voice-cloudflare@0.13.4-alpha.0


## @mastra/voice-deepgram@0.13.3-alpha.0


## @mastra/voice-elevenlabs@0.13.3-alpha.0


## @mastra/voice-gladia@0.13.3-alpha.0


## @mastra/voice-google@0.14.4-alpha.0


## @mastra/voice-google-gemini-live@0.14.18-alpha.0


## @mastra/voice-inworld@0.4.6-alpha.0


## @mastra/voice-mistral@0.1.3-alpha.0


## @mastra/voice-modelslab@0.2.3-alpha.0


## @mastra/voice-murf@0.13.3-alpha.0


## @mastra/voice-openai@0.13.4-alpha.0


## @mastra/voice-openai-realtime@0.14.8-alpha.0


## @mastra/voice-playai@0.13.3-alpha.0


## @mastra/voice-sarvam@1.1.4-alpha.0


## @mastra/voice-speechify@0.14.3-alpha.0


## @mastra/voice-xai-realtime@0.2.17-alpha.0


## @mastra/inngest@1.10.8-alpha.3


## @mastra/temporal@0.4.18-alpha.4


## @mastra/agentcore@0.5.3-alpha.0


## @mastra/agentfs@0.2.3-alpha.0


## @mastra/apple-container@0.5.2-alpha.0


## @mastra/archil@0.3.2-alpha.0


## @mastra/azure@0.3.3-alpha.1


## @mastra/blaxel@0.10.2-alpha.0


## @mastra/cloudflare-sandbox@0.5.4-alpha.0


## @mastra/daytona@0.11.4-alpha.0


## @mastra/docker@0.12.1-alpha.0


## @mastra/e2b@0.15.1-alpha.1


## @mastra/e2b-desktop@0.1.10-alpha.1


## @mastra/files-sdk@0.3.3-alpha.0


## @mastra/gcs@0.3.5-alpha.1


## @mastra/google-drive@0.2.4-alpha.0


## @mastra/mesa@0.3.3-alpha.0


## @mastra/modal@0.7.3-alpha.0


## @mastra/platform-workspace@1.10.0-alpha.1


## @mastra/railway@0.10.2-alpha.1


## @mastra/s3@0.6.5-alpha.1


## @mastra/sprites@0.1.2-alpha.0


## @mastra/vercel@1.6.2-alpha.0


## @internal/playground@1.36.0-alpha.4

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-10-26 13:49:15 -07:00
mastra-platform[bot]
f51a29e82b
fix(core): settle evented workflow runs cancelled from another instance (#28434)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9

Closes #28427

## Summary

On the evented workflow engine, cancelling a run from a different Mastra
instance than the one running it (for example, the server cancel route
landing on a non-owner replica) marked the run `canceled` in storage.
The owner's `run.start()` promise never resolved, though. The same hang
happened for runs started with `startAsync()` and then cancelled.

**Cause:** `EventedRun.cancel()` wrote `canceled` to storage and then
called `abortController.abort()`. It relied on an abort listener to
publish `workflow.cancel`, but only `start()` and `resume()` register
that listener. A run handle created any other way (another instance, or
`startAsync()`) published nothing. The owner's event processor then
dropped the run's later events through the canceled-status gate. As a
result, `workflow.end` was never emitted and `start()` stayed pending.

**Fix:**
- `EventedRun.cancel()` now publishes `workflow.cancel` directly through
a `publishCancel()` helper. The abort listener uses the same helper, and
it publishes at most once per run handle.
- `WorkflowEventProcessor` tracks the controllers it has already
cancelled (`cancelHandledControllers`, a WeakSet). A step that unwinds
on one of those controllers no longer publishes a second
`workflow.cancel`, so a cancelled run emits exactly one `workflow.end`.
This also removes a duplicate `workflow.cancel`/`workflow.end` pair that
already happened on main whenever the owner cancelled its own run.

## Proof

New tests:
- `packages/core/src/workflows/evented/remote-cancel.test.ts` (12
tests): two Mastra instances sharing storage cancel across instances
over `EventEmitterPubSub` and `UnixSocketPubSub`. The tests cover
dountil, foreach, sequential, sleep and parallel shapes. The owner's
`start()` settles as `canceled` and the step's abortSignal fires. Two
more tests cancel after `startAsync()` and after an owner-local cancel,
and each asserts exactly one `workflow.cancel` and one `workflow.end`.
- `packages/server/src/server/handlers/workflows.test.ts`:
`CANCEL_WORKFLOW_RUN_ROUTE` is called on a non-owner instance and the
owner's run settles as `canceled`.
- `pubsub/redis-streams/src/evented-remote-cancel.test.ts` (4 tests):
cross-instance cancel over real Redis Streams, with each replica on its
own connection. Shapes are sequential and dountil, with workers on the
owner only and on both replicas.

Results with the fix reverted:
- Full revert: all 12 core tests fail, the server test times out
(`expected 'timed out' to be 'canceled'`), and all 4 Redis tests time
out.
- Reverting only `workflow.ts`: 11 of the 12 core tests fail.
- Reverting only the processor guard: the 2 single-event tests fail
(they get 2 events instead of 1).

With the fix, everything passes:
- `pnpm build:core`
- `tsc --noEmit` on core
- eslint and oxlint (type-aware) on the changed files
- oxfmt
- core `src/workflows`: 102 files; `workflow.test.ts` (329 passed) and
`evented-workflow.test.ts` (315 passed) were rerun alone after a heap
OOM when three suites ran in parallel
- server `workflows.test.ts`: 92 passed
- Redis: 4 of 4 passed against a local redis-server on port 6381

## Overlap with #24915

#24915 also edits `EventedRun.cancel()`. It calls `setupAbortHandler()`
before `abort()`, guarded by `#abortHandlerRegistered`. That would also
make cancel publish from a non-owner handle, so the two fixes overlap
and will conflict on the same lines. Whichever PR merges second should
keep a single publish path (`publishCancel()` here). The processor-side
duplicate-end guard in this PR is independent of #24915.

## Unverified

- **Multi-replica step abort (pre-existing, not fixed here):** when
every replica runs workers, `workflow.cancel` reaches only one consumer
in the group. If that consumer isn't the replica running the step, the
step's abortSignal doesn't fire. `start()` still settles as `canceled`.
Main behaves the same, even for local cancels. The Redis and server
tests check step abort only when the owner is the sole worker. This
should be tracked as a separate issue.
- **Inngest and durable/evented agents:** I didn't run these. From
reading the code, Inngest settles through `getRunOutput` on any terminal
status, and durable agents use their own `agent.control.{runId}` abort
transport.

## Risk: low

The change is limited to the evented engine's cancel path. The publish
is guarded to run at most once per handle. The WeakSet guard is
per-process, so if two replicas process cancel for the same run at the
same moment, the run could still end twice. That is no worse than main.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

When one Mastra instance cancels a workflow running on another, the
workflow owner’s `run.start()` can now finish with a canceled result.
The change also prevents duplicate cancellation messages from creating
duplicate end events.

## What changed

- `EventedRun` publishes `workflow.cancel` from both explicit
cancellation and the abort handler through one helper. The helper
prevents duplicate publishes and allows a retry if publishing fails.
- `WorkflowEventProcessor` tracks controllers already handled by a
cancellation event. When a step unwinds on one of those controllers, it
does not publish another cancellation event.
- Added tests for remote cancellation, `startAsync()`, server
cancellation, Redis Streams, and retry after a failed cancellation
publish.

## Validation

The author reports passing core build, type-check, lint, formatting, and
workflow tests. The author also reports passing server workflow tests
and Redis Streams tests. These results are author-reported.

## Limitations

When workers run on multiple replicas, a cancellation event may reach a
processor other than the one running the active step. That step may not
receive the abort signal. This case remains unfixed and unverified.
Inngest and durable/evented agents were not tested.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-26 12:19:03 -05:00
Ward Peeters
ce5d4731ee
chore(deps): update dependencies across the monorepo (#28417)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-26 16:59:30 +01:00
NikAiyer
2ded35cb0c
fix(core,redis,valkey): stop duplicate observe frames and save evented suspensions before publishing (#27352)
Fixes COR-1474: `Agent.observe()` / `POST /agents/:id/observe` sent
durable-run frames more than once (a client saw 20–21 tool calls for a
run that executed 16), and reconnecting with an `offset` did not
reliably resume. The other half of the ticket, closing `observe()` for
finished or unknown runs, shipped in #27358.

The cause was two unshared numbering sources: the agent's
`CachingPubSub` numbered its own frames, while evented durable steps
published unnumbered to the transport and a follower re-cached them
under new numbers. Now the cache list is the run's log and the transport
is only the live tail. Each frame is appended with its index before it
is published, and observers track one cursor.

This PR also makes the evented engine save a tool suspension or approval
request to memory before publishing its chunk. That change is needed
here: the faster frame path made clients see the suspension before
memory had it (see Proof 6).

Scope notes: no breaking API changes. Cache keys keep `main`'s format.
Atomic appends on Redis Cluster are tracked in COR-1491.

## Proof

1. **Frames arrive once, in order, and resume from an offset.**
`caching-pubsub.test.ts`, `step-executor.test.ts`,
`evented-split-transport.test.ts` and the aimock `observe-frames` /
`observe-lifecycle` scenarios pass on durable and evented. The cursor,
gap fill and log-restart guard each fail with their fix removed.
2. **Across processes with real Redis.**
`pubsub/redis-streams/src/observe-cross-process.test.ts`
(`RedisStreamsPubSub` + `RedisServerCache`, two processes), 8/8 on
durable and evented: an unknown run closes, a finished run replays then
closes, a live run is followed to its finish, and an observer stays
attached across an approval suspend/resume. Each run's text and `finish`
arrive exactly once.
3. **No shared cache behind a cross-process transport.** The libsql
cross-process suite (API process plus dedicated worker, in-memory cache)
passes, including `xproc-input-tripwire`. That test failed before steps
fell back to publishing uncached in this setup, as on `main`.
4. **Rolling deploys.** Old and new `CachingPubSub` instances sharing
one transport and cache delivered every frame once and in order: a
reconnect after the upgrade, old and new live observers, and old and new
workers alternating steps on one run. This ran as a local probe test
that is not committed.
5. **Run-log TTL.**
- In-memory, Redis and Valkey unit tests show `listPushIndexed` applies
the requested TTL to the list and counter on each append. They also show
a requested TTL only raises `ttlSeconds` and `ttlSeconds: 0` stays
persistent.
   - `agent.stream.*` logs request 24h (`RUN_LOG_TTL_MS`).
   - These tests fail with the TTL math zeroed.
- The Redis no-`eval` and `CROSSSLOT` fallback (`INCR` + `RPUSH`, as on
`main`) keeps the requested TTL. That test fails with the fallback
reverted to the default TTL.
6. **Suspensions are in memory before clients see them (evented).**
- Timing probes, branch before this change: the client received
`tool-call-suspended` at +0 ms, during the publish call, and read memory
at +2 ms before the suspension was saved.
- Same probes on `main`: the client received it at +1 ms, after the
publish finished, and memory already had it.
- `packages/memory` `replacement-instance-suspension.issue-27971`
(evented) failed every run before the change and passes 3/3 after.
- The observational-memory suites (1,881 tests), the
durable/events/cache/evented core suites (2,774 tests) and the aimock
scenarios pass. The 4 `background-task-lifecycle` aimock failures are
pre-existing on `main` (bisected to #28162).
7. `pnpm --filter ./packages/core check`, lint and formatting clean.
Redis and Valkey `tsc --noEmit` clean.

## Risk

- **Evented routing.** On the evented engine, run frames, aborts and
leases now use `mastra.pubsub` / `mastra.serverCache`. An evented agent
with its own `pubsub` or `cache` on a different backend logs a warning
and no longer uses it for run frames. The default engine is unchanged.
- **Live-only without a shared cache.** `observe({ offset })` without a
shared cache only sees the live tail.
- **Late frames.** A frame the transport delivers late is filled in from
the log at its original position instead of being dropped. This includes
reasoning from a signal-preempted attempt, which clients already see
when it arrives on time. It stays out of memory and the replacement
request.
- **Suspension order.** On the evented engine the suspension chunk now
goes out after the memory save, so it arrives one memory write later. If
the save throws (authorization failure or a superseded attempt), the
chunk is not published. Before, the client saw a suspension it could not
resume, followed by an error.
- **Deprecations and custom caches.** `CachingPubSubOptions.source` /
`__setSource()` are deprecated no-ops. Custom `MastraServerCache`
subclasses keep the base, non-atomic `listPushIndexed` and ignore the
TTL.
- **Not exercised in this PR:**
- **Redis Cluster.** The `CROSSSLOT` fallback is only unit-tested with a
faked rejection. On a real Cluster it is not atomic: concurrent writers
can interleave, so a list position may not match its index. Tracked in
COR-1491.
- **Long suspensions.** A suspension longer than the 300s default TTL
was not run against a real Redis. The 24h TTL is covered by unit tests
only.
- **Inngest.** No end-to-end Inngest run. Inngest gets the 24h TTL but
never deletes the run log when the run ends.
- **Split deploys.** The platform team's split-deploy tests have not yet
been rerun against this branch.

---------

Co-authored-by: review <review@local>
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-23 12:39:25 -05:00
mastra-platform[bot]
11c83d5333
fix(core): recover evented agent runs after worker death (#28054)
Fixes COR-1469. Postgres-backed evented agents now send distributed
worker orchestration through the shared broker, allowing Redis to
reclaim unfinished work after a worker dies while the original API
process remains alive. Other workflow locality and cross-process finish
delivery are preserved.

Postgres atomically checks step occurrence identifiers before writing
results. Interrupted occurrences can be taken over, completed loop
iterations reuse committed results and decisions, and stale deliveries
cannot overwrite newer results. Regression coverage includes
saved-result continuation, nested loop completion, retry publication,
state preservation, and legitimate next iterations.

Validated SIGTERM and SIGKILL with both a warm surviving worker and a
replacement worker: all four cases pass, including five repeated matrix
runs (20/20), with the original API caller completing without restart.
Core validation passed in two groups (344 and 519 tests); Postgres
workflow/run-fencing validation passed 42 tests earlier, and the final
focused rerun passed 22 tests. TypeScript, touched-file type-aware lint,
formatting, and whitespace checks pass. A Postgres query-plan assertion
failed once during the final broad rerun and passed on the focused
rerun.

The Redis suite is not fully green: after building missing fixture
dependencies, the seven affected files passed all 25 tests. The
remaining agent-thread Lambda-death signal-drain test times out at
agent-thread-cross-process.test.ts:389; removing all core changes and
rebuilding reproduces the identical failure on the unchanged baseline.
This limitation is not suppressed or changed by this PR.

Upgrade @mastra/core and @mastra/pg together and drain older runs first.
Interrupted external effects must be idempotent. Resume, individual
foreach item guards, time travel, restart/redelivery races, other
storage adapters, Redis persistence, and payload-size redesign remain
out of scope. Tony’s fencing/recovery sign-off is required before
merging and before platform#3224 lands.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

If a worker stops during an agent run, another worker can take over
while the original API process stays alive. The system protects work
that already finished, so recovery does not overwrite completed steps or
repeat completed loop iterations.

## What changed

- Routed eligible Postgres-backed evented workflow work through the
shared broker so another worker can reclaim it. Workflow finish events
still cross processes.
- Added workflow-step guards and Postgres checks to reject stale or
duplicate results.
- Reused saved step results, loop decisions, and state during recovery.
- Kept legacy step records without occurrence metadata on the existing
unguarded path until a guarded write stamps them.
- Added recovery tests, Redis Streams takeover fixtures, documentation,
and coordinated upgrade guidance.

## Validation

- SIGTERM and SIGKILL recovery passed with warm and replacement workers:
20/20, as reported.
- Core validation passed in groups of 344 and 519 tests. Postgres
workflow and run-fencing validation passed 42 tests earlier and 22 in
the focused rerun, as reported.
- TypeScript, touched-file type-aware lint, formatting, and whitespace
checks passed, as reported.
- A Postgres query-plan assertion failed in a broad rerun and passed in
a focused rerun.
- Redis validation remains partial. Seven affected files passed 25
tests.
- The timeout at `agent-thread-cross-process.test.ts:389` also occurred
on the unchanged baseline, as reported.

## Upgrade requirements and limitations

- Upgrade `@mastra/core` and `@mastra/pg` together.
- Drain runs started on older versions before upgrading. Older runs can
finish after an upgrade, but crash recovery is not guaranteed.
- Make interrupted external effects idempotent.
- Resume, foreach-item guards, time travel, restart/redelivery races,
other storage adapters, Redis persistence, and payload-size redesign are
out of scope.
- Tony’s fencing and recovery sign-off is required before merging and
before `platform#3224` lands.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: NikAiyer <nick.aiyer@gmail.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: Abhi Aiyer <abhiaiyer91@gmail.com>
2026-10-23 09:00:13 -05:00
dane-ai-mastra[bot]
dacb190944
chore: version packages (alpha) (#28009)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-10-23 13:57:18 +02:00
mastra-platform[bot]
e20bbd0ef4
fix(core): stop evented workflows/agents hanging when workflow.fail publish fails (#28218)
> Opened by @abhiaiyer91

## What

When an evented workflow event runs out of delivery attempts (3), or
fails in a way that must not be retried (for example a throwing
`stopWhen`, marked non-redeliverable since #28215),
`WorkflowEventProcessor.handle()` publishes `workflow.fail` so whoever
is waiting on `workflows-finish` gets an error. On `main` it marked the
event terminal **before** that publish and only logged if the publish
threw. Every later redelivery was then dropped as terminal, so in a live
process:

- `run.start()` on an evented workflow never resolves
- `agent.generate()` / `agent.stream()` on an evented agent never errors
or finishes
- the stored run stays `running` (only crash recovery would clean it up)

## Fix

`packages/core/src/workflows/evented/workflow-event-processor/index.ts`
now tracks "failure not yet published" separately from "done for good":

- If the `workflow.fail` publish throws, the processor returns `retry:
true` and remembers the error.
- On redelivery of that event it skips the handler and only retries the
publish, reusing the transport's normal redelivery. This also applies to
non-redeliverable failures, so a throwing `stopWhen` is still called
exactly once.
- The event becomes terminal once the publish lands, or after 3 publish
attempts.
- Publish attempts are folded into the delivery count. If the pending
entry is evicted from its bounded map, the processor still knows the
handler is spent and how many publishes it has tried, so it neither
reruns the handler nor resets the 3-attempt cap.
- The transport's `deliveryAttempt` sets a floor on that count. If a
redelivery lands on a processor that never saw the event, and the budget
is already spent, that processor only publishes `workflow.fail`.
- 3 handler deliveries plus at most 2 republish deliveries fit the
tightest transport cap (RedisStreamsPubSub, 5). UnixSocketPubSub allows
6.
- `workflow.fail` events still never republish `workflow.fail`.

There is no outbox, and nothing transport-specific.

**If all 3 publish attempts fail,** the processor logs the error and
stops retrying. That run is left `running` in storage and the live
caller still gets no result, the same as on `main`. Only
`recoverActiveRuns()` cleans it up on the next start. This is
intentional: without an outbox, a pubsub that keeps failing has no way
to reach the caller. The bounded give-up tests pin this behaviour
(exactly 3 attempts, then no more publishes, no extra model calls) on
EventEmitter, UnixSocket, Redis and the evented agent.

## Proof

Repro first, on `main`: a step fails past the retry budget (every
`workflow.step.end` publish throws) and the first N `workflow.fail`
publishes throw.

| Test | main (no fix) | with fix |
|---|---|---|
| `workflows/evented/fail-publish-after-exhaustion.test.ts`,
EventEmitterPubSub + UnixSocketPubSub (control, 1 throw, 2 throws,
bounded give-up) | 7 failed / 3 passed (controls pass): `start()` hangs
past 5 s, storage `running`, `workflow.fail` tried once | 8 passed |
|
`aimock/scenarios/evented-fail-publish-after-exhaustion.scenario.test.ts`
(evented agent; other engines skipped because they never use the event
processor): control, 1 throw, bounded give-up, **throwing `stopWhen` +
first `workflow.fail` publish throws** | 1 throw → `HANG`; give-up →
`workflow.fail` tried once, not 3; throwing `stopWhen` → `HANG` | 4
passed. The `stopWhen` case rejects with `stopWhen exploded`, with
`stopWhen` called once, 1 tool call and 1 model request. In the give-up
case the model request count equals the control run's |
| `workflow-event-processor/retry-budget.test.ts`: pending entry
evicted; non-redeliverable handler with its pending entry gone; pending
entry evicted before every redelivery; budget spent on another processor
(`deliveryAttempt: 4`) | each fails with its guard removed (handler
reruns or retries never stop) | 9 passed |
| `pubsub/redis-streams/src/fail-publish-after-exhaustion.test.ts` (real
Redis 6381; 0/1/2 throws, plus bounded give-up at 3) | 0/1/2 cases: 2
failed / 1 passed, `workflow.fail` tried once, hangs for 20 s (give-up
case added later in review, not run on `main`) | 4 passed |

What the caller sees with the fix, when the publish fails once or twice:
`start()` resolves `failed`, the stored snapshot is `failed`, and
`workflows-finish` carries `workflow.fail`.

Related suites, rebased on `main` with #28215, with the fix:
- `src/workflows/evented/**` and `src/events/**`: 59 files, 740 passed,
5 skipped. `evented-workflow.test.ts` crashes its vitest worker (OOM) in
the combined run. Run alone with a 6 GB heap: 315 passed, 5 skipped.
- aimock abort/error/stop-when/fail-publish scenarios on all engines: 32
files, 398 passed, 13 expected-fail, 60 skipped, 0 failed.
- `pubsub/redis-streams` `fail-publish-after-exhaustion.test.ts` +
`local-redelivery-evented.test.ts` against the rebuilt core: 6 passed.
- `tsc --noEmit` on core: clean. Prettier: clean. `pnpm build:core`:
succeeds.

Inngest never reaches the changed code:
- Runs are triggered with `inngest.send`
(`workflows/inngest/src/run.ts:253`, `:366`;
`durable-agent/create-inngest-agent.ts:837`).
- They execute inside the Inngest function (`workflow.ts:446`) through
`InngestExecutionEngine` (`workflow.ts:561`), which extends
`DefaultExecutionEngine` (`execution-engine.ts:61`).
- Nothing in `workflows/inngest/src` references `WorkflowEventProcessor`
or `handleWorkflowEvent`.

## Unverified

None.

## Risk: low

- The change only affects events that have used up their delivery budget
(or are non-redeliverable) **and** whose `workflow.fail` publish threw.
The normal path and the success path of the publish behave the same as
before.
- Worst case: 2 extra redeliveries of that one event, still inside every
transport's cap.
- Known limit, no worse than `main`: a non-redeliverable failure goes
terminal on attempt 1. If its `workflow.fail` publish throws and the
redelivery lands on a different processor, that processor sees
`deliveryAttempt: 2` and runs the handler once more. Ruling that out
needs shared per-event state, which this PR deliberately leaves out.

---------

Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-22 21:24:01 -04:00
mastra-platform[bot]
7d580884fb
fix(core): call a throwing stopWhen once on evented agents (#28215) 2026-10-22 20:52:16 -04:00
Dane - AI (Mastra)
40a522c142
chore(deps): update build tools (#27997) 2026-10-22 14:33:21 +02:00