0
Fork 0
mirror of https://github.com/mastra-ai/mastra.git synced 2026-11-17 08:44:46 +00:00
Commit graph

244 commits

Author SHA1 Message Date
NikAiyer
d929736435
fix(core): use one stream bus for durable agents (#30229)
Durable agents now use the wrapped Agent's stream bus instead of
constructing a second one. An Agent-level pubsub carries both run frames
and thread events, including after registration and forks; evented
execution still uses Mastra's shared transport. Standalone durable
agents remain observable, and plain-run records and abort checks stay
separate from durable execution.

Previously, observing across replicas required repeating the pubsub on
the durable wrapper. This now works with the Agent configuration alone:

```ts
const agent = createDurableAgent({
  agent: new Agent({ id: 'assistant', name: 'Assistant', instructions, model, pubsub }),
});
```

Inngest explicitly exposes its detached-streaming capability without
migrating its bus. The server retains the `isDurableAgentLike` fallback
in `hasResumableStreams` for compatibility with older Inngest wrappers:
their proxy can inherit `false` from a plain Agent or lack the method
entirely. Server and Inngest versions are not coupled, so removing the
fallback would silently turn HTTP disconnects into run cancellation on
supported mixed-version installs. This is a deliberate, user-approved
exception to the ticket's shortcut-removal bullet; no peer-dependency
bump is needed while the fallback remains. Regression coverage exercises
capability `true`, `false`, and missing; the latter two fail without the
fallback.

Server branch audit (`packages/server/src/server/handlers/agents.ts`):
`validateDurableToolCallAccess` (line 298) remains because approvals
need durable workflow snapshots; `authorizeRunAccess` (2676) remains
because durable ownership uses workflow rows/local and replay owners
rather than plain run records (COR-1542). The two abort-route branches
(2724 and 2742) retain durable cancellation support and its
missing/finished-run acknowledgement semantics, not separate transports.

The observe-route branch (3141) remains for generation-fenced
`__subscribeToRunStream` and the Inngest adapter's `pubsub` fallback;
consolidating observation is COR-1541, and migrating Inngest's bus is
COR-1539. The recover-route guard (3746) remains because only durable
execution has recoverable workflow snapshots. None of these remaining
branches requires a second stream bus on core DurableAgent.

The Mastra-fallback regression also runs through a fork created after
registration, both with and without resolving the bus before
registration. Another replica must receive replay plus live tail through
finish, and the terminal frame must be published on Mastra's transport.
The latest targeted rerun passed 33 tests, including cache-policy
warnings and plain-publisher isolation across both durable storage
variants; core typecheck, targeted lint, formatting and diff checks
passed.

Verified with 1,643 core durable/stream tests, 78 AIMock parity tests,
292 server handler tests (including the mixed-version capability cases),
119 Inngest tests, and six real Redis cross-server cases covering live
observation, thread events and HTTP Stop. Typechecks, lint and
formatting pass. The Agent-pubsub regression failed before the fix; a
built-package red/green demo also confirms the transport change. An
earlier approval-resume timing failure passed in isolation and on the
final full rerun.

Fixes COR-1540




## Transport precedence and upgrade scope

Default-engine durable run transport precedence is now: explicit
`createDurableAgent` pubsub → wrapped Agent pubsub → Mastra pubsub.
Evented runs retain Mastra's shared transport. Mastra-only
configurations are unchanged.

If the wrapped Agent already specifies a different transport from Mastra
and the wrapper has no override, this fix changes the effective
run-frame and remote Stop transport. To preserve routing across an
upgrade, explicitly pass the previous transport to `createDurableAgent`
or align the Agent and Mastra transports. The wrapper configures the
shared stream bus, so its override also affects resumable streams from
direct calls on the wrapped Agent. The core changeset includes this
upgrade note.

A local investigation used separately built pre-change main
(`049b31dd2c`) and this branch with real Redis in both owner/observer
version directions. Shared effective transport passed live completion,
replay, and remote Stop. Distinct selected transports lost live
completion and remote Stop acknowledgement; shared-cache replay still
worked. This is a configuration-dependent routing cutover, not a general
rolling-deploy incompatibility. The investigation-only second-checkout
harness and expected failures are not included in this PR; existing
same-version Redis coverage remains.

Permanent coverage now executes wrapper and fork runs with three
distinct wrapper/Agent/Mastra buses, verifies run frames only reach the
wrapper bus, and verifies replayed output. Dropping the wrapper override
makes this test fail on the intended publication assertion. Latest
focused verification: 38 tests passed; core typecheck, type-aware lint,
formatting, and diff checks passed.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

ELI5: Durable agents now send live updates through the same stream
channel as the normal agent they wrap. Before, they built a second
channel. Users can now set up cross-server viewing by configuring only
the wrapped agent.

## Summary

This PR changes durable agents to use the wrapped `Agent`'s stream bus.
They no longer build a second bus. Agent-level pubsub carries run frames
and thread events, including after registration and forks. Evented
execution continues to use Mastra's shared transport.

### Core changes (`packages/core`)

- `Agent` stores a fallback stream pubsub and a shared-run-log
capability callback.
- `__getStreamBus()` initializes a bus when wrapper stream configuration
exists or streams are enabled. It prefers wrapper configuration over
agent durable settings.
- `__getStreamBus()` uses the configured pubsub, or the agent pubsub. If
neither exists, it uses a reusable fallback `EventEmitterPubSub`.
- The shared-run-log check uses the wrapper callback when configured.
Otherwise, it calls `__usesSharedRunLog()`.
- `__ownsPlainStreamPublisher()` still selects the plain publisher. It
now gets its bus through `__getStreamBus()`.
- Plain publishing still requires enabled agent streams.
- `__configureDurableStreams(config, usesSharedRunLog)` overlays defined
wrapper values on the agent durable settings and any existing wrapper
settings. It stores the callback and resets the bus.
- Agent forks copy the durable stream configuration and share the
fallback pubsub.
- `DurableAgent` delegates stream-bus ownership and access to the
wrapped agent. Its cache, pubsub, cleanup scheduling, and replay-owner
operations use that bus.
- The `DurableAgent` constructor configures durable streams on the
wrapped agent. `__getStreamBus()` returns that bus.
- `DurableAgent` no longer copies a private bus configuration when
forked. It no longer adopts the inner pubsub. Mastra registration no
longer registers a separate bus.
- `DurableAgent` adds `__setLogger(logger: IMastraLogger)`. It forwards
logger updates to the wrapped agent.
- The `AgentStreamBus` constructor accepts an optional `defaultPubsub`.
The bus uses it when `config.pubsub` is absent, before the new
`EventEmitterPubSub` fallback.
- `AgentStreamBus.forkConfig` and `adoptInnerPubsub` are removed.
- Direct `stream` and `generate` calls on an agent with `durable: false`
stay unpublished. This holds after durable wrapping configures the
shared bus. The wrapper can still publish through that bus.
- Plain-run records and abort checks stay separate from durable
execution.

### Inngest changes (`workflows/inngest`)

- `InngestAgent` adds the public method `hasResumableStreams():
boolean`. It always returns `true`.
- The internal `Pick` type includes this method.
- Inngest exposes detached streaming. It does not migrate its bus.
- A new test checks that `createInngestAgent` reports resumable streams
when its wrapped `Agent` has `durable: false`.

### Server changes (`packages/server`)

- `hasResumableStreams` keeps the `isDurableAgentLike` fallback for
older Inngest wrappers in mixed-version installs.
- The observe-stream handler keeps the same fallback.
- The fallback applies when a wrapper reports `false` or does not report
the capability.
- Removing the fallback would cause the reported capability-`false` and
missing-method regression cases to fail.
- This PR does not require a peer-dependency bump while the fallback
remains.
- Other server branches handle durable tool-call access, run
authorization, abort, observation, and recovery. The PR states these
address separate durable-workflow behavior or deferred work. They do not
need a second stream bus on core `DurableAgent`.
- Comments in `hasResumableStreams` and the observe-stream handler were
revised. No executable behavior changed.
- The durable-wrapper stream-detachment test runs with
`hasResumableStreams` set to `true`, `false`, and absent. The stream
observer detaches in each case.

### Redis pubsub (`pubsub/redis-streams`)

- The cross-server abort test runs for the durable and evented engines
with the Mastra bus. It runs for the durable engine with the agent bus.
- Each child server receives the selected bus location.
- The successful-abort case subscribes the other server to the run's
thread. That server verifies a tool-call chunk. It then verifies
completion and a finish chunk after the abort.
- The remote abort response remains `{ aborted: true }`. The owner's run
ends aborted.
- The abort fixture adds `BUS_LOCATION=agent`. With this value, the
agent uses Redis Streams pubsub. The Mastra-level bus uses
`EventEmitterPubSub`.
- Without `BUS_LOCATION=agent`, the agent has no explicit pubsub. With
`ENGINE=evented`, the Mastra-level bus uses `CachingPubSub` over Redis
Streams. Otherwise, it uses Redis Streams directly.
- The `stream` command accepts optional `threadId` and `resourceId`. It
passes them as memory configuration when `threadId` is present.
- The fixture adds `observe` and `subscribe` commands. `observe` streams
run observations. `subscribe` streams thread subscription chunks.

### Tests

- The shared-bus tests add a standalone mode. They cover replay and
live-tail observation with and without forks. They also cover bus
resolution before registration.
- The tests verify finish-event publication.
- Replayed start and tool-call frames are asserted only for registered
replicas.
- A prior logger-warning assertion was removed.
- Thread-coordination tests abort across processes in every
`sharedFrames` case. They observe the aborted run's frames. They expect
an aborted finish reason and one model request.
- New tests check run-frame routing for plain, durable, and evented
agents. They cover configurations with both agent-level and Mastra
buses. They also check replay from an unregistered durable agent whose
wrapped agent has `durable: false`.
- The explicit durable-execution test expects the original agent and its
durable wrapper to share the same stream bus.
- A direct-execution test checks that `stream` and `generate` runs stay
unpublished when a `durable: false` agent is wrapped and registered.
Runs through the durable wrapper publish a finish event.

### Changesets

- Patch changesets cover `@mastra/core`, `@mastra/server`, and
`@mastra/inngest`.
- The `@mastra/core` changeset includes an upgrade note. Mastra-only
configurations keep their behavior. Users who need the previous
transport can keep it by configuring it explicitly.

### Verification

The PR description reports these results. This summary does not
independently verify them.

- 1,643 core durable and stream tests passed.
- 78 AIMock parity tests passed.
- 292 server handler tests passed.
- 119 Inngest tests passed.
- Six real Redis cross-server cases passed.
- A latest targeted rerun passed 33 tests. It covered cache-policy
warnings and plain-publisher isolation across both durable storage
variants.
- Typechecks, targeted lint, formatting, and diff checks passed.
- The Agent-pubsub regression test failed before the fix.
- An earlier approval-resume timing test failed. It passed in isolation
and on the final full rerun.

### Diff size

- 14 files changed.
- 341 lines added.
- 118 lines removed.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: review <review@local>
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-16 07:21:54 -08:00
dane-ai-mastra[bot]
d0f7dd3a80
chore: version packages (alpha) (#30420)
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases
## @mastra/factory@0.29.0-alpha.3

### Minor Changes

- Added opt-in health metrics to Factory, sent through `mastra.metrics`.
Pick the groups you want, since every record counts towards
observability usage:
([#30225](https://github.com/mastra-ai/mastra/pull/30225))

  ```ts
  new MastraFactory({
    // ...
metrics: { groups: ['process', 'postgres', 'redis', 'sandbox', 'intake']
}, // or 'all'
  });
  ```

Or set `FACTORY_METRICS=all` (or a list like `sandbox,intake`).
`FACTORY_METRICS_INTERVAL_MS` (default 60s, minimum 15s) and
`FACTORY_METRICS_MAX_PER_MIN` (default 500) tune how often records are
sent and cap their volume. Metrics are named `mastra_factory_*` and
labeled with `service` (`api`, `worker` or `all`) so split deployments
can be told apart.

### Patch Changes

- Fixed a pull request Factory opened for a card whose issue lives in
Jira, Linear or incident.io getting a card of its own on GitHub. The
pull request now attaches to that card and moves it to Review, as it
already does on GitLab, so the work is no longer tracked on two cards
while it is being reviewed. Fixes
[#30358](https://github.com/mastra-ai/mastra/issues/30358).
([#30418](https://github.com/mastra-ai/mastra/pull/30418))

- Deliver Factory notifications when their bound session no longer
exists, including concurrent notifications. Notifications for a role
that has been handed off remain suppressed.
([#30422](https://github.com/mastra-ai/mastra/pull/30422))

- Updated dependencies
[[`02dbde1`](02dbde13cc),
[`7f09a63`](7f09a63553),
[`5686fa1`](5686fa16ed)]:
  - @mastra/code-sdk@1.19.1-alpha.3
## @mastra/redis-streams@0.9.0-alpha.1

### Minor Changes

- Added `createMonitorClient()` and `streamKey()` to
`RedisStreamsPubSub`, so health checks can inspect Redis on their own
connection instead of queueing behind event traffic:
([#30225](https://github.com/mastra-ai/mastra/pull/30225))

  ```ts
const client = pubsub.createMonitorClient(); // unconnected; commands
fail fast while disconnected
  await client.connect();
const groups = await client.xInfoGroups(pubsub.streamKey('workflows'));
  await client.destroy();
  ```
## @mastra/code-sdk@1.19.1-alpha.3

### Patch Changes

- Fixed `mastracode` crashing on startup after a fresh install.
`@mastra/code-sdk` now requires `@mastra/core` 1.84 or newer, the first
version with the knowledge scope helpers it imports. Before, npm could
install `@mastra/core` 1.83, and `mastracode --help` failed with "does
not provide an export named 'encodeKnowledgeScopeId'".
([#30425](https://github.com/mastra-ai/mastra/pull/30425))

- `@mastra/code-sdk` now ships its own copy of the Knowledge scope id
encoder instead of importing `encodeKnowledgeScopeId` from
`@mastra/core`. If npm installs an older `@mastra/core`, `mastracode`
still starts instead of crashing with "does not provide an export named
'encodeKnowledgeScopeId'". A test keeps the copy's output identical to
core's. ([#30427](https://github.com/mastra-ai/mastra/pull/30427))

- **Kept a plugin running when its reload fails.** A reload that fails
for a plugin that was working (a bad update, a throwing resolver) now
keeps the plugin's last good tools, signal providers and version stamp,
and records the error as `reloadError` instead of removing the plugin.
Disabling, blocking or uninstalling it still removes everything, and the
next successful load replaces the kept one.
([#30412](https://github.com/mastra-ai/mastra/pull/30412))

**Added plugin error notifications.** `PluginManager.onPluginError`
reports each distinct reload failure once, along with signal provider
errors from running plugins. With no listener, errors still go to
`console.warn`.

  ```ts
const unsubscribe = pluginManager.onPluginError(({ pluginId, message })
=> {
    console.error(`[${pluginId}] ${message}`);
  });
  ```
## mastracode@0.50.1-alpha.3

### Patch Changes

- **Showed plugin failures in the session.** When a plugin fails to
reload, Mastra Code now shows an error saying it is still running the
previous version, and signal provider errors from plugins appear the
same way instead of only on the console.
([#30412](https://github.com/mastra-ai/mastra/pull/30412))

- Updated dependencies
[[`02dbde1`](02dbde13cc),
[`7f09a63`](7f09a63553),
[`5686fa1`](5686fa16ed)]:
  - @mastra/code-sdk@1.19.1-alpha.3

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-11-14 17:31:08 -08:00
mastra-platform[bot]
4dd77bac9f
feat(factory): opt-in health metrics by group (#30225)
> Opened by @roaminro

## What

Opt-in health metrics for Factory, sent through `mastra.metrics`, so a
deployment can tell at a glance which component to investigate first:
the Node process, Postgres, Redis, sandboxes and session start-up, or
intake.

Every record counts towards observability usage, so nothing is collected
unless a group is turned on:

```ts
new MastraFactory({ ..., metrics: { groups: ['sandbox', 'intake'] } }); // or groups: 'all'
```

Env overrides: `FACTORY_METRICS` (`all`, `0`, or a list),
`FACTORY_METRICS_INTERVAL_MS` (default 60s, min 15s),
`FACTORY_METRICS_MAX_PER_MIN` (default 500).

| Group | Metrics (`mastra_factory_*`) |
| --- | --- |
| process | event-loop delay and utilization, heap utilization and
memory against the heap limit, `process_cpu_cores` (all threads) against
`process_cpu_limit_cores`, major/incremental GC pauses,
`process_uptime_ms` |
| postgres | pool total/idle/waiting per pool (read after the check),
`SELECT 1` check per pool, `pg_stat_activity` sessions + longest
query/idle transaction, slowest `pg_stat_statements` logged every 10 min
by query ID |
| redis | `INFO` check, memory used/max and utilization, clients,
evictions, stream backlog (only with `RedisStreamsPubSub`; only the
check on a cluster) |
| sandbox | `session_ready_ms{kind=first\|resume,outcome}` per start
(failures included), `session_sandbox_wait_ms`, `sessions_starting` +
`session_start_oldest_age_ms` every interval, provider start time and
failures (`sandbox_provider`), setup phase and command timings,
recreate/reset/retire counts, active sandboxes |
| intake | GitHub/Linear ingest outcomes, reconcile replays (missed
webhooks), sweep failures, `intake_last_sweep_age_ms` per kind (counted
from when the sweep worker starts) |

## How

- Disabled groups are no-ops: no timers, checks or records.
- Counts and timings are aggregated in-process and sent once per
interval. A counter is one record per series. A timing is
`stat=avg|p95|max` on its `_ms` series plus a `<name>_count` series with
the number of observations, so a series ending in `_ms` is always
milliseconds. p95 uses reservoir sampling. Every duration is in ms.
- Quiet intervals still send 0 for known counters (configured intake
sources, sweep failures, sandbox start failures), so an empty chart
means "nothing happened", not "not reporting".
- A per-minute cap holds over the whole minute; event counts and timings
are sent before gauges, so the cap drops gauges first. Overflow is
counted in `mastra_factory_metrics_dropped_total`.
- Check durations are sent for `db_store`, `db_vector` and `redis` only;
the server-stats checks send failures only.
- Labels are closed sets only: no IDs, paths or repository names.
- Every record has `service=api|worker|all`, derived from
`MASTRA_WORKERS`. Server-wide stats (Postgres activity, Redis
memory/backlog) run only on `worker`/`all`, so split deployments don't
report them twice. Platform's built-in `serviceName` isn't set per
record: `metrics.emit()` has no override, so that needs a core or
Platform change.
- `mastra.metrics` and the logger are read on every flush, since
Platform registers its exporter after the app module loads. A failed
tick is logged, never thrown.
- `[factory:timing]` stderr lines keep main's rule (only when boot
diagnostics are on); metrics are recorded either way, gated by their own
group.
- `RedisStreamsPubSub` gains `createMonitorClient()` + `streamKey()`, so
checks use their own connection and never queue behind pub/sub traffic.
- Session sandbox retirement counts only real, successful retirements.

## Not included

Linear/GitLab issue-reconcile sweeps, Platform poller latency, Slack
time-to-first-reply (needs a hook in `packages/core` channels),
HTTP/storage timings.

## Test plan

- [x] `pnpm check`, `pnpm lint` in `mastracode/factory`; `tsc` in
`pubsub/redis-streams`
- [x] `src/metrics/metrics.test.ts`: config parsing, service mapping,
aggregation and `_count` series, cap and send order, zero counters,
disabled groups, api skips server-wide stats, session start timing
(joined callers, failures, starts in flight), check guards, sweep age
- [x] Existing tests for touched modules (workspace, github
rules/sandbox/reconcile, linear rules, sandbox, timing)
- [x] Snapshot on shipyard `beta` with `FACTORY_METRICS=all` (single
process, `service=all`)
- [ ] Split deployment reporting both `service=api` and `service=worker`

Docs: mastra-ai/softwarefactory-docs#18

---------

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: Roamin <97863888+roaminro@users.noreply.github.com>
2026-11-14 15:55:22 -05:00
Tyler Barnes
ca0af9fa6e
ci: skip memory benchmarks on PRs and run durable stream tests in parallel (#30416) 2026-11-14 11:54:40 -07:00
dane-ai-mastra[bot]
1c52fdd034
chore: version packages (alpha) (#30261)
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases

> The changelog information of each package has been omitted from this
message, as the content exceeds the size limit.

## @mastra/factory@0.29.0-alpha.1


## @mastra/core@1.84.0-alpha.1


## @mastra/playground-ui@69.1.0-alpha.1


## @mastra/client-js@1.60.1-alpha.1


## @mastra/react@1.15.1-alpha.1


## @mastra/deployer-cloud@1.84.0-alpha.1


## @mastra/longmemeval@1.1.43-alpha.1


## @mastra/opencode@0.1.43-alpha.1


## create-factory@0.4.4-alpha.1


## @mastra/code-sdk@1.19.1-alpha.1


## mastracode@0.50.1-alpha.1


## @mastra/arize@1.3.31-alpha.0


## @mastra/arthur@0.4.31-alpha.0


## @mastra/braintrust@1.3.28-alpha.0


## @mastra/datadog@1.4.23-alpha.0


## @mastra/deepeval@0.1.25-alpha.0


## @mastra/laminar@1.3.33-alpha.0


## @mastra/langfuse@1.6.7-alpha.0


## @mastra/langsmith@1.4.4-alpha.0


## @mastra/observability@1.24.1-alpha.0


## @mastra/otel-bridge@1.5.23-alpha.0


## @mastra/otel-exporter@1.5.3-alpha.0


## @mastra/posthog@1.4.8-alpha.0


## @mastra/sentry@1.2.33-alpha.0


## @mastra/agent-builder@1.2.2-alpha.0


## mastra@1.39.1-alpha.1


## @mastra/connect@1.4.1-alpha.0


## @mastra/deployer@1.84.0-alpha.1


## @mastra/editor@0.17.2-alpha.0


## @mastra/mcp-docs-server@1.3.14-alpha.1


## @mastra/memory@1.41.2-alpha.0


## @mastra/server@1.84.0-alpha.1


## @mastra/google-cloud-pubsub@1.1.8-alpha.0


## @mastra/redis-streams@0.8.2-alpha.0


## @mastra/valkey-streams@0.6.3-alpha.0


## @mastra/elysia@0.1.24-alpha.1


## @mastra/express@1.5.28-alpha.1


## @mastra/fastify@1.5.28-alpha.1


## @mastra/hono@1.7.26-alpha.1


## @mastra/koa@1.7.28-alpha.1


## @mastra/nestjs@0.3.1-alpha.1


## @mastra/next@0.2.42-alpha.1


## @mastra/tanstack-start@0.2.42-alpha.1


## @mastra/clickhouse@1.30.1-alpha.0


## @mastra/cloudflare@1.8.1-alpha.0


## @mastra/cloudflare-d1@1.5.1-alpha.0


## @mastra/convex@1.9.1-alpha.0


## @mastra/dsql@1.5.8-alpha.0


## @mastra/dynamodb@1.5.2-alpha.0


## @mastra/libsql@1.30.1-alpha.0


## @mastra/mongodb@1.27.1-alpha.0


## @mastra/mssql@1.11.2-alpha.0


## @mastra/mysql@0.17.1-alpha.0


## @mastra/oracledb@0.5.7-alpha.0


## @mastra/pg@1.36.1-alpha.0


## @mastra/redis@1.4.14-alpha.0


## @mastra/spanner@1.12.1-alpha.0


## @mastra/turso@0.1.22-alpha.0


## @mastra/upstash@1.6.2-alpha.0


## @mastra/valkey@0.2.10-alpha.0


## @mastra/temporal@0.6.1-alpha.1


## @mastra/daytona@0.11.6-alpha.0


## create-mastra@1.39.1-alpha.1


## @internal/playground@1.39.1-alpha.1

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-11-13 19:58:20 -08:00
Arthur Fiorette
4f16c08068
fix(redis-streams,valkey-streams): release leases while close() is flushing (#30387)
## Description

On Redis Streams and Valkey Streams, `releaseLease` now keeps working
while `close()` is flushing, until the writer connection is released.
Before, it returned without doing anything as soon as `close()` started.

### What changed

In `@mastra/redis-streams`, `releaseLease` checks the existing
`#writerReleased` flag instead of `#closed`. `close()` sets that flag
right before it disconnects the writer. `@mastra/valkey-streams` had no
such flag, so this adds one, sets it right before the writer's `quit()`
in `close()`, and checks it in `releaseLease`. The other lease methods
are unchanged.

### Why

A worker is shutting down while a step it handles for another process's
run is still executing, and the step takes longer than the drain
timeout. The step finishes while `close()` is flushing, and publishing
its result throws on the closing client. The handler then releases the
step's lease, which did nothing once `close()` had started, and nacks.
The nack still republished through the open writer. Another worker
received the copy, was refused the lease, treated the delivery as a step
a live worker was still running, and acked it. Nothing ran the step
again and the run stayed `running`.

With this change the release and the nack's republish go through the
same open writer, so the worker that gets the copy can take the lease
and run the step.

### Known limits

A release that races the writer's quit now rejects instead of returning.
The nack that follows fails too, so the entry stays pending and is
reclaimed after `reclaimIdleMs` (60 s by default), by which time the 30
s lease has expired.

Other lease releases in core, on durable agent claims and thread
streams, now really release during `close()` and can reject in that same
race. Their callers catch the error or turn it into a retry.

A release that fails for another reason while the republish succeeds
still drops the step.

Google Cloud Pub/Sub has no lease provider and is not affected.

### Tests

`pubsub/redis-streams/src/close-release-lease.test.ts` runs against a
real Redis with two `RedisStreamsPubSub` instances on one key prefix.
A's grouped handler takes a lease, then releases it and nacks while A's
`close()` is still in progress, held open by a fan-out reader. B
receives the second delivery and must be able to take the same lease.
Without the fix, B gets `{ acquired: false, owner: 'a' }`. The test also
checks that `close()` had not resolved when the release ran.

`pubsub/valkey-streams/src/close-drain.test.ts` gets a case with a
mocked client: a release called while `close()` waits on its flush sends
the release script, and one called after the writer quit sends nothing
and doesn't reconnect. Without the fix, the release sent during the
flush is missing.

## Related issue(s)

None

## Type of change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [ ] Test update

## Checklist

- [ ] I have linked the related issue(s) in the description above
- [ ] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

**ELI5:** When a worker shuts down, it can now still let go of its task
lock and retry the task. This lets another worker pick up the task
sooner instead of leaving it stuck.

## Summary

This change fixes a shutdown race in the Redis Streams and Valkey
Streams pub/sub adapters. Before this change, `releaseLease` stopped
work as soon as `close()` began. The writer stayed open during the
flush, so a lease release and the republish from a nack could not use
it. A nack could then republish a delivery while the step lease was
still held. Another worker could then drop the step.

### Changes

- **`pubsub/redis-streams/src/index.ts`**: `releaseLease` checks
`#writerReleased` instead of `#closed`. During `close()`, a lease
release can still run until the writer is marked released.
- **`pubsub/valkey-streams/src/index.ts`**: Adds a `#writerReleased`
flag. `releaseLease` stays available after `close()` marks the instance
closed. `close()` sets the flag after it flushes pending publishes and
before it quits the writer.
- **`pubsub/redis-streams/src/close-release-lease.test.ts`** (new, 89
lines): Uses two workers. Worker A releases the lease and nacks the
event during shutdown. The test expects redelivery on attempt 2 and
expects worker B to acquire the lease.
- **`pubsub/valkey-streams/src/close-drain.test.ts`** (46 lines added):
Uses a mocked client. It checks that `release` runs during a close
flush, before `xAdd` and `quit`. It also checks that a release requested
after the writer quits does not run again.
- **Changesets** (`.changeset/eleven-pears-wave.md`,
`.changeset/spicy-canyons-yawn.md`): Patch releases for
`@mastra/valkey-streams` and `@mastra/redis-streams`.

### Behavior notes

- If a release races the writer quit, the release rejects. The following
nack also fails. The entry stays pending for reclaim after
`reclaimIdleMs` (60 seconds by default). The 30-second lease expires
before that.
- Other core lease releases can now run during `close()` and reject in
the same race. Their callers must catch the error or retry.
- Google Cloud Pub/Sub is not changed.
- A release that fails for another reason can still drop the step if the
republish succeeds.

### Public API

No exported declarations change.

### Diff size

6 files changed, 153 lines added, 2 lines removed.

### Test and CI status

The author reports that both test files cover release during a close
flush and behavior after the writer quits. Results for the current head
are not available in the supplied evidence.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 19:34:46 -08:00
Arthur Fiorette
99ec3bd582
fix(core): leave evented workflow runs to a redelivering broker and restart only stopped runs (#30279)
## Description

This PR stops `restartAllActiveWorkflowRuns()` from restarting evented
runs that a redelivering broker (Redis Streams, Valkey Streams, Google
Cloud Pub/Sub) already recovers. It also restarts a default-engine run
only once no instance holds its run lease and a grace period (2 s by
default) has passed. Before, every booting instance restarted every
active evented run, so a step interrupted by a crash ran once per
booting pod (#28712). The broker already recovers those runs: Redis and
Valkey Streams reclaim an unacked message once it has been idle for
`reclaimIdleMs`, and Google Cloud Pub/Sub redelivers once the ack
deadline passes.

### Evented runs on a broker

`PubSub` gets a `redeliversUnacked` getter (default `false`). Redis and
Valkey Streams return `true` while the reclaim loop runs
(`reclaimIntervalMs > 0`), Google Cloud Pub/Sub returns `true`,
`CachingPubSub` forwards to its inner pubsub, and the in-process
`EventEmitterPubSub` and the Unix socket pubsub return `false`. When the
pubsub (read through `__rawBus()`) redelivers, recovery never lists,
reads or restarts an evented run. An evented workflow without
declarative schedules costs no listing at all. Evented runs on any other
pubsub (`EventEmitterPubSub`, Unix socket, or a third-party pubsub that
doesn't override the getter) are restarted at once by the explicit call,
as before.

### Scheduled runs

The `mastra.pubsub` proxy publishes every workflow event of a scheduled
run (`sched_wf_*`, plus the notification dispatcher's row) `localOnly`,
so the broker never holds it. Recovery still restarts those runs: an
evented workflow with declarative schedules gets one summary listing per
status, and only its process-local scheduled runs are kept. The proxy
and recovery share one predicate, `isProcessLocalScheduledRunId`.

### Default engine

One recovery rule: a `running` or `waiting` run is restarted when its
run lease isn't live and `recovery.workflowGraceMs` (new, 2,000 ms by
default for every pubsub) has passed since the later of the lease's last
expiry and the run's last write. The pass lists summaries
(`listWorkflowRuns({ status, perPage: false, summary: true })` per
status), reads leases in batches of 100 with `getRunOwnerships`,
re-reads a run's lease once its restart gets a slot, and runs every
restart on one instance-wide limiter (`recovery.workflowConcurrency`). A
run left live or in its grace period gets a per-run recheck timer, as on
main. Overlapping calls share one pass. `shutdown()` waits, within
`drainTimeout`, for a pass still reading the store and for the restarts
it handed out. Without run fencing every active run is restarted, as
before.

### Nested resume

A default-engine nested run resumed directly now holds its parent's run
lease while it runs and hands it to the parent's resume, so recovery
doesn't restart the parent meanwhile. On a store without
compare-and-set, where the parent stays `suspended`, the nested run
holds its own lease instead.

Also: `renewWorkflowLease` waits for each renewal to settle before
arming the next one (a self-rearming `setTimeout` instead of
`setInterval`), and `EventEmitterPubSub`'s JSDoc and the docs say it's
in-process only.

### Repro

Postgres + Valkey Streams with default reclaim settings (`reclaimIdleMs`
60 s, `reclaimIntervalMs` 30 s). Instance A starts a run whose step
records an execution and then hangs; A is killed with `SIGKILL`
mid-step. Then 2 or 4 fresh instances boot and all call
`restartAllActiveWorkflowRuns()` at once. "At boot" counts step
executions caused by those calls; "total" counts every execution after
the kill until the run succeeds; "rerun after" is seconds from the kill
to the first new execution. 3 trials per row, identical results within
each row except for timing.

| Build | Engine | Booting pods | Executions at boot | Executions total
| Rerun after |
|---|---|---|---|---|---|
| Base (this PR's base branch) | evented | 2 | 2 | 2 | 0.1 s |
| Base | evented | 4 | 4 | 4 | 0.1 s |
| Base | default | 2 | 0 | 1 | 15.1 to 16.6 s |
| Base | default | 4 | 0 | 1 | 15.0 to 16.9 s |
| This PR | evented | 2 | 0 | 1 | 60.1 to 60.2 s |
| This PR | evented | 4 | 0 | 1 | 60.1 to 60.2 s |
| This PR | default | 2 | 0 | 1 | 17.0 to 19.1 s |
| This PR | default | 4 | 0 | 1 | 17.2 to 21.0 s |

On this PR the evented rerun is the broker's reclaim: one execution on
one pod, about 60 s after the kill. The default-engine rerun comes once
A's lease (15 s TTL) lapsed plus the 2 s grace and up to 5 s of recheck
jitter.

### Benchmark

3,000 active default-engine runs that another instance holds, among
30,000 finished runs, two workflows. "Before" is a copy of main's sweep
run in the same process (full listing, then `createRun()` + `restart()`
per run, 5 at a time); "after" is `restartAllActiveWorkflowRuns()` on
this PR. Median of 5 calls.

| Store | Before | After | Listing payload before | Listing payload
after |
|---|---|---|---|---|
| Postgres | 1,773 ms | 30 ms | 1.12 MB | 0.60 MB |
| LibSQL | 627 ms | 44 ms | 1.12 MB | 0.60 MB |

The benchmark's snapshots are nearly empty, so the payload gap here is
small; with real step results it grows with snapshot size.

### Known limits

On a redelivering broker these evented runs stay `running` until someone
calls `restart()` on them. Main's boot restart recovered them; stack 2
takes them over.

- A sleeping evented run whose instance exits, after a crash or during a
normal deploy. The sleep acks its delivery and keeps a `setTimeout` in
the process that started it, so any exit of that process loses the
timer.
- A step whose message Redis or Valkey Streams trimmed. Both trim with
`MAXLEN ~ maxStreamLength` (10,000 by default), which also drops entries
that are still pending. A step that is in flight while that many newer
events are published loses its message if its instance then exits.
Deployments with long steps and a high event volume should raise
`maxStreamLength`.
- A message the broker dropped after `maxDeliveryAttempts` or at
`inFlightTimeoutMs`.
- A failure publish that gave up after its retries (the worker acks the
message).
- A run whose `running` snapshot was saved but whose start event never
reached the broker, because its process stopped between the two.
- A Redis or Valkey instance that restarted without persistence.
- A scheduled evented run that was in flight when a deploy stopped its
process and removed the workflow's schedule from code. Recovery doesn't
list it, so it stays `running` until `restart()`.

#28712 is fixed for evented runs on a redelivering broker. These are
still restarted by every booting instance, as before:

- scheduled evented runs (`sched_wf_*`), whose events never leave their
process
- evented runs on the in-process `EventEmitterPubSub` or the Unix socket
pubsub. Several instances sharing one storage with these pubsubs aren't
supported for evented workflows; the docs now say a multi-instance
evented deployment needs a broker.

With Redis or Valkey Streams, `reclaimIdleMs` must stay at 30 s or more:
a message reclaimed while the dead worker's 30 s step fence is still
held is dropped as a duplicate. The getter's JSDoc, the adapter docs and
the changeset state it.

Rolling deploys are out of scope.

Stack 3 of 4: storage projections and batch ownership reads, then the
finished-step check, then this PR, then the recovery loop with a leader.

## Related issue(s)

Fixes #28712

## Type of change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Code refactoring
- [x] Performance improvement
- [x] Test update

## Checklist

- [x] I have linked the related issue(s) in the description above
- [x] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

ELI5: When a server crashes in the middle of a workflow, Mastra can
restart that workflow. Before this change, some workflows ran their
steps again on every new server that started, so a step could run many
times. This change lets the message broker handle the restart for those
workflows, so each step runs only once. Other workflows now wait for a
short grace period before they restart.

## Summary

This pull request changes how workflow recovery restarts runs after a
crash or a boot. It targets duplicate step execution in two cases:

- **Evented runs on brokers that redeliver unacknowledged messages.**
Recovery skips these runs. The broker redelivers the unacknowledged step
events instead.
- **Default-engine runs.** Recovery restarts a run only after no live
lease remains and a grace period has passed.

### Behavior changes

- Google Cloud Pub/Sub, Redis Streams, and Valkey Streams report
`redeliversUnacked` as `true` when their reclaim or redelivery path is
active. Redis Streams and Valkey Streams report `true` only when
`reclaimIntervalMs` is above 0.
- `PubSub` adds a `redeliversUnacked` getter that defaults to `false`.
`CachingPubSub`, `EventEmitterPubSub`, and `UnixSocketPubSub` expose or
declare this capability.
- Recovery leaves evented runs to the broker when `redeliversUnacked` is
`true`. Scheduled evented runs restart on boot. Other evented runs
restart immediately when the broker does not redeliver.
- Default-engine recovery applies `recovery.workflowGraceMs`. The
default is 2,000 ms. Negative or non-finite values use the default. A
run restarts only after its lease is no longer live and the grace period
has elapsed since the later of its last update and lease expiry. Runs
that are not yet eligible are checked again later. If the storage
adapter does not support run fencing, active runs restart without the
grace rule.
- Recovery lists active run summaries, reads ownership in batches, and
rechecks ownership before each restart.
- Restarts use a shared concurrency limiter
(`createConcurrencyLimiter`). A restarted run holds its slot until it
finishes.
- Overlapping recovery passes are deduplicated. Shutdown waits for an
in-flight pass.
- Recovery skips nested runs whose active parent drives them.
- Lease renewal runs sequentially (`renewWorkflowLease`,
`repeatAfterSettled`). A new renewal starts only after the previous one
settles.
- Nested resume coordinates parent and child leases. A failed lease
claim rejects the resume and leaves the runs suspended.
- Run-claim operations return whether a status transition occurred. They
return `false` when storage is absent, running snapshots are not
persisted, or compare-and-set is not supported.

### Public API changes

- `MastraRecoveryConfig.workflowGraceMs?: number` (default 2,000 ms).
- `PubSub.redeliversUnacked` getter (default `false`).
- `redeliversUnacked` getters on `GoogleCloudPubSub`,
`RedisStreamsPubSub`, `ValkeyStreamsPubSub`, and `UnixSocketPubSub`.
- `ConcurrencyLimiter` interface and `createConcurrencyLimiter(limit)`
function in `run-with-concurrency.ts`.
- `renewWorkflowLease`, `repeatAfterSettled`, and
`WORKFLOW_RUN_LEASE_RENEW_INTERVAL_MS` in `run-ownership.ts`.
- `Run._resume(params)` accepts an optional `takeHeldOwnership`
callback.

### Documentation

- Crash-recovery and worker docs describe broker reassignment and
multi-instance limits.
- `EventEmitterPubSub` documents that delivery is process-local.
- Redis Streams and Valkey Streams docs recommend `reclaimIdleMs` of at
least 30 seconds. Reclaiming earlier can cause duplicate-dropped step
events.
- The recovery reference documents `workflowGraceMs`.

### Tests and test utilities

- New recovery tests cover exit cases (crash and shutdown), recovery
reads, rules, ownership during nested resumes, and lease renewal.
- Shared test helpers add `sharedBroker`, `killableProcess`,
`interceptStarts`, `persistActiveRun`, and `describeRecoveryExitCases`.
- Existing tests set `workflowGraceMs: 0` where they need immediate
recovery.
- Redis Streams and Valkey Streams tests check `redeliversUnacked`
against `reclaimIntervalMs`.
- LibSQL and Postgres recovery-pass suites run the shared exit cases.

### Scope and limits

- The change addresses issue `#28712` for evented runs on redelivering
brokers. Evented runs on in-process or Unix socket pubsubs still restart
at boot. The PR states that multi-instance evented deployments need a
broker.
- The PR lists unrecovered cases: sleeping evented runs, trimmed or
dropped messages, failed publishes, missing start events, non-persistent
Redis or Valkey restarts, and scheduled runs whose schedules were
removed. Rolling deploys are out of scope.
- Reported benchmark results (median recovery time on 3,000 active and
30,000 finished runs: 1,773 ms to 30 ms on Postgres, 627 ms to 44 ms on
LibSQL) and repro results are author-reported. They were not
independently verified.
- CI and test results: not available.
- Review findings and severity counts: not available.

### Size

- 41 files changed, 2,594 lines added, 209 lines removed (from the pull
request diff).

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 14:59:00 -03:00
dane-ai-mastra[bot]
d60ba71be0 chore: version - exit prerelease mode 2026-11-12 21:12:29 +00:00
dane-ai-mastra[bot]
055a8c3b82 chore: version packages 2026-11-12 04:17:40 +00:00
mastra-platform[bot]
743a02786e
feat(valkey): identify Mastra-created clients with a clientInfoTag (#30156) 2026-11-11 17:57:11 -08:00
Dane - AI (Mastra)
c014d187bb
chore(deps): update docker images (major) (#19788) 2026-11-11 17:08:49 -08:00
Arthur Fiorette
ce03237b34
fix(core): send only the response messages that changed on agent stream events (#29971)
## Description

Follow-up to #29965 (merged). After that PR, each plain Agent
`step-finish` on `agent.stream.<runId>` still carried the run's response
messages so far, so a long run stored every tool output once per later
step: the log still grew with the square of the step count, mostly from
large tool results.

The publisher now keeps a cursor of message fingerprints and sends only
the messages from the first one that changed:

```ts
// step 1: { from: 0, seq: 0, messages: [A1, T1] }
// step 2: { from: 2, seq: 1, messages: [A2, T2] }
// step 3: { from: 4, seq: 2, messages: [A3] }
_durableResponseMessagesDiff: { producer, seq, from, base, hash, messages }
```

"From the first changed index" (not "only new messages") because earlier
entries change: a step's assistant message grows across steps, and a
processor retry rolls the list back.

Readers rebuild the full list per step in the stream adapter, before
chunks reach `MastraModelOutput`, so
`onStepFinish(step).response.messages`, `output.steps` and `onFinish`
still see the whole list:

- `agent.observe(runId, { offset })` first folds the step-finish events
before the offset from the run's history (bounded wait, doesn't delay
the subscription).
- A diff only applies to the list it was made against (`base` must equal
the hash of the list the reader holds). One that can't be placed (a
missed or reordered delivery, a late copy) is never applied, and the
reader reports no messages for that step instead of a partial list.
- The publisher resends the whole list once the diffs since the last
whole list add up to its size, so whole lists cost at most about as much
as the diffs and the log stays linear. A failed publish or diff makes
the next one a whole list, so a reader that missed one recovers.
- On a bus that keeps no history of the run's topic (`cache: false`, or
a `CachingPubSub` whose `shouldCache` leaves the topic out), every
step-finish is a whole list, so an observer that attaches mid-run can
place it. An internal `PubSub.__replaysTopic(topic)` answers this per
topic, since `CachingPubSub.supportsOffsets` is true even for topics it
doesn't cache. Transports without offsets (Redis and Valkey Streams
without a cache) get whole lists too: an `observe()` from an offset
there follows the stream from its tail.
- Each diff carries its publisher's id: a run resumed after approval
publishes from a new publisher that starts with the whole list. Readers
keep the list of the current and previous publisher apart, so a late
event from the previous one doesn't break the next diff.
- The diff replaces `_durableResponseMessages`, so an older reader
during a rolling upgrade reports no messages rather than a partial list.
DurableAgent and EventedAgent still send whole lists, which the adapter
handles as before; they move to diffs in a follow-up.

## Related issue(s)

Part of COR-1532 (plain Agent; DurableAgent/EventedAgent and Inngest
follow in the next PRs of this stack). Follow-up to #29965 (production
Redis OOM from plain Agent run logs). Related: #27475.

## Type of change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [x] Performance improvement
- [x] Test update

## Checklist

- [x] I have linked the related issue(s) in the description above
- [ ] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR

Tests: `agent-stream-log.scenario.test.ts` (all engines) adds "an early
tool result is stored as often as the last one" (fails before this PR on
the plain Agent; durable engines pinned as the known gap), an observer
resuming mid-run at an offset (fails without the history fold), an
observer joining mid-run on a bus without history (fails without the
whole-list rule), and an approval suspend/resume replay compared step by
step. `response-messages.test.ts` covers the diff and apply rules
(edits, shrinking lists, duplicates, lost diffs, producer switches, a
late whole list from the previous producer, the size-based whole list).
The real-Redis test asserts the tool-result counts and that a 20-step
run stores less than 2.5x the bytes of a 10-step one (2.1x with this PR,
2.9x before it).



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

Each step used to save the full list of response messages again, even
when most messages had not changed. This change saves only the updates
and rebuilds the full list for observers, which can reduce storage
during long runs.

## Summary

- Plain Agent stream events now publish response-message diffs. The
publisher sends a full list first and periodically resends one as diffs
accumulate.
- Readers apply diffs when they match the current message list. If a
reader cannot rebuild the list, it reports no response messages for that
step rather than a partial list.
- When `agent.observe()` starts at a numeric offset, it replays earlier
events to rebuild response messages before delivering later events.
Observers continue to receive the full list for each step.
- DurableAgent and EventedAgent continue to publish full lists. Tests
add coverage for diff replay, offset replay, approval resumption,
tool-result repetition, and Redis stream storage growth.
- Older observers report no response messages for steps published with
diffs. The changeset recommends upgrading agent and observer processes
together.

Test results: not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-10 23:13:38 -03:00
Arthur Fiorette
749c09cf91
fix(core): publish lean agent stream events instead of the loop's chunks (#29965)
## Description

A registered Agent publishes every run on `agent.stream.<runId>` so
`agent.observe()` and reconnecting clients can replay it, and on Redis
that log is stored until the run is cleaned up. The publisher forwarded
the loop's in-process chunks as they were: every `step-start` carried
the provider request, and every `step-finish` carried the prompt again
(`metadata.request`, `messages.all`/`user`) plus every earlier step with
its own prompt (`output.steps`). The log grew roughly with the cube of
the step count; in production single runs of a plain Agent took 0.3 to
3.4 GB of Redis and filled it.

The publisher now builds `step-start`, `step-finish` and `finish` field
by field (`agent/streams/wire.ts`) with what readers use, which is what
DurableAgent already publishes:

```ts
// step-finish payload on the topic, before
{ stepResult, metadata: { ...response, request: { body: '<whole prompt>' } },
  output: { text, toolCalls, usage, steps: [/* every step, each with its prompt and response */] },
  messages: { all, user, nonUser } }

// after
{ stepResult, metadata: { id, timestamp, modelId, headers, providerMetadata, ... },
  output: { usage /*, steps: [{ tripwire }] when the step was rejected */ },
  _durableStepContent: [/* this step's content */],
  _durableResponseMessages: [/* response messages so far */] }
```

- The caller's own stream and callbacks are unchanged; only the
published copy is reduced.
- Observers of a plain Agent get better replays: each step's content was
`[]` before, and `onFinish` lost its text over JSON transports and threw
on in-memory ones (the finish steps carried AI SDK-shaped `toolCalls`
the reader can't read). Finish steps now carry `text`, and tool
calls/results in the record shape DurableAgent uses.
- A delegated agent's chunks inside `tool-output` /
`workflow-step-output` are slimmed by the plain publisher. Durable
engines still publish them as-is, since their callers read their own run
back from the topic; that case is pinned with `it.fails`.
- Step content is read from the run's message list when the step
finishes (one extra response-to-UI conversion per step, the same work
the caller's output does).

Still to come, stacked on this PR: each `step-finish` still sends the
response messages so far in full; follow-ups send only what changed, for
the plain Agent first and then DurableAgent/EventedAgent and Inngest
(superseding #27522).

## Related issue(s)

Fixes COR-1530. Production incident: shipyard Redis OOM from
`agent.stream.<runId>` keys of plain Agent runs (Factory). Related:
#27475 (the DurableAgent side of the per-step growth) and COR-1532
(per-step response messages, the follow-up stack), introduced with
#28424.

## Type of change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [x] Performance improvement
- [x] Test update

## Checklist

- [x] I have linked the related issue(s) in the description above
- [ ] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR

Tests: `agent-stream-log.scenario.test.ts` runs on every engine (prompt
copies on the log, delegated agent prompt, observer replay over
in-memory and JSON transports, a rejected and retried reply), and
`pubsub/redis-streams/src/agent-stream-size.test.ts` checks what real
Redis holds in both cache modes. The new plain-agent cases fail on main.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

Agent stream logs repeat old prompt and step data, which can make them
large. This change publishes smaller event records while keeping the
caller’s stream intact and giving observers step content they need.

## Summary

- The publisher creates reduced `step-start`, `step-finish`, and
`finish` events. It removes provider requests and repeated step data.
- For `step-finish`, the publisher reads the current step’s content from
the run’s message list. It omits content for retry steps and logs a
warning if it cannot read the content.
- The publisher reduces nested agent chunks inside `tool-output` and
`workflow-step-output`. DurableAgent and EventedAgent continue to
publish their delegated-agent steps in full.
- Observer replay gains step content and richer finish data, including
run text and tool calls, and step reasoning, sources, and files. The
event still includes response messages on each `step-finish`.
- Added scenarios cover prompt retention, delegated-agent prompts,
observer replay over in-memory and JSON transports, and rejected and
retried replies. A Redis integration test checks stored stream size and
observer replay with both cache modes.

## Validation

Completed CI, build, and test results: not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-10 19:51:50 -03:00
mastra-platform[bot]
d77893aed3
fix(core): resume evented agent observe() at the offset on Redis Streams with the default cache (#29798)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9

## What

Fixes #28324. On an evented agent (`createEventedAgent()`) using a
cross-process PubSub such as `RedisStreamsPubSub` with the default
in-memory server cache, `observe(runId, { offset })` ignored the offset.
A client reconnecting mid-run got earlier events again, and an offset
past the end of a finished run replayed nearly the whole run.

## Why it happened

Evented step frames went out unnumbered whenever an in-memory cache sat
behind a cross-process transport. The concern was that a worker in
another process would number the run from 0 in its own cache, and
observers would then drop its frames as duplicates. That skipped
numbering in the common case too, where every step runs in the process
that started the run.

## Change

- `CachingPubSub` gains two `@internal` options,
`extendExistingLogsOnly` and `onUnlogged`. With
`extendExistingLogsOnly`, a publish only appends to a topic whose log
this cache already holds. Any other publish goes out live and
unnumbered, and `onUnlogged` is called.
- `StepExecutor` now always wraps the step bus when a server cache
exists. If the cache is process-local and the bus crosses processes, it
turns on `extendExistingLogsOnly`. The caller writes the run's `start`
frame before any step runs, so steps in the caller's process extend that
log and get numbered. Steps on another worker find no log, stay
live-only as before, and log a one-time warning that recommends
`RedisServerCache`.
- `StepExecutor` now picks up the Mastra logger when it is constructed.
Workflow event processors create executors without registering them on
Mastra, so without this the warning was dropped.
- Docs (`durable-agent.mdx`) list the two setups that still need a
shared cache: steps running on a dedicated worker, and observers
reconnecting through another replica.

## Tests

- `pubsub/redis-streams/src/observe-offset.test.ts`: the evented +
default cache row now passes the exact-resume checks (resume at offset,
offset past end returns nothing, mid-run replay). The full matrix passes
12/12 against Redis 7.
- `step-executor.test.ts`:
  - When the caller holds the run's log, step frames are numbered.
- When a worker has no log, its frames stay unnumbered and one warning
reaches the logger configured on `Mastra`, using an executor that was
never registered on Mastra.
  - Both cases fail on main.
- `caching-pubsub.test.ts`: tests `extendExistingLogsOnly` both with and
without an existing log.
- `stores/libsql` cross-process suite, including `xproc-input-tripwire`:
48 passed, 0 failed.
- Core `tsc` is clean. oxlint and prettier pass on the changed source
files.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## ELI5
When an evented run starts, the process that started it can now number
its events even when events travel through Redis. That lets a client
resume from an offset without seeing old events again. Events from a
different worker remain unnumbered when only a local cache is available,
and Mastra warns about this limitation.

## Changes
- Added internal `CachingPubSub` options to append only to existing
cached logs and report events that cannot be numbered.
- Updated `StepExecutor` to use those options with a process-local cache
behind cross-process PubSub. It numbers events for runs whose log exists
locally and warns once when a worker publishes events for a run it did
not start.
- Updated documentation to explain when a shared cache is required,
including worker execution and reconnects through another server
process.
- Added tests for existing and absent cached logs, step numbering and
warnings, and offset replay scenarios.

## Validation
Test, build, and CI results: not available.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: NikAiyer <nick.aiyer@gmail.com>
2026-11-09 17:23:50 -08:00
dane-ai-mastra[bot]
f21e0742ce chore: version packages 2026-11-07 04:18:11 +00:00
mastra-platform[bot]
d6a5f98df3
fix: index observe chunks for resume (COR-1506) (#29520)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9

Linear:
[COR-1506](https://linear.app/kepler-crm/issue/COR-1506/observe-offset-cant-be-used-to-resume-it-counts-log-events-but-clients)

## Problem

`observe({ runId, offset })` takes `offset` as a position in the run
log. Clients can't compute that position:
- Some log entries never reach the client. These include
iteration-complete records after every step and ownership markers.
- Updates on the wire carried no index.

A client that resumes at "number of updates received" lands too early
and gets repeated updates. The drift grows by one per completed step, so
every tool-using run that reconnects after its first step is affected.
On a 3-step tool loop, 16 of 28 cut points repeated 1–2 updates on both
durable and evented engines.

## Fix

- **Index on updates:** every update now carries `index`, the run-log
position it came from. This covers:
  - `/observe` on every agent
  - `/stream` on durable and evented agents
- core `stream()`/`observe()` chunks and client-js
`observe()`/`stream()`
- **Resuming:** pass `index + 1` of the last received update that has an
index as `offset`. Gaps in the numbering are expected; they're log
entries that send nothing.
- **Unindexed updates:** the synthesized `abort` update carries no
index. Its log position is on the `finish` sent beside it. Client-built
structured-output `object` chunks have no index either. An
`object-result` logged by a structuring-model run carries one. Unindexed
updates don't move the resume point. A client that reconnects between
`abort` and `finish` gets the `abort` again.
- **`offset` is unchanged:** it still means a run-log position.
- Core no longer strips `index` in
`DurableAgent.__subscribeToRunStream`. The stream adapter and the
server's `toAgentStreamChunks` copy the envelope's index onto each chunk
and drop any index a forwarded sub-agent chunk carried. `index` is
optional on `BaseChunkType` and `DataChunkType`.
- **Also fixed:** on plain agents, `/observe` (HTTP and core
`agent.observe()`) with an offset past the end of a finished run used to
wait for the idle timeout or hang. It now closes immediately. It reads
the log from just before the offset and only falls back to a full read
when that window is empty. The check runs in the background, so it never
delays returning the stream or its cancel handle. A trailing `error`
chunk doesn't count as the end of the run.

## Migration

`index` is a new field and `offset` keeps its meaning:
- Callers that pass a log position need no change.
- Callers that replay from `offset: 0` need no change. They get the same
updates in the same order, now with `index` set.
- Only callers that passed a count of updates received need to change.
They were already getting repeats after step boundaries; switch them to
`index + 1` of the last indexed update.

## Evidence

| Check | Result |
| --- | --- |
| Red/green | With `main`'s core and server, 15 tests on numbering and
`/stream`–`/observe` matching fail; with this change all pass. |
| Cut points | Every cut point of a 3-step tool loop on durable and
evented engines resumes from the first `/stream` connection at last
index + 1. Covers the HTTP route and client-js `stream()` → `observe()`.
|
| `/stream` vs `/observe` match | Frames are identical JSON, including
`index`. Replay from 0 equals the first `/stream`. |
| End of run | Resume after finish or past the end returns the rest and
closes, including far past the end (offset 100). |
| Adapters | Hono 98/98, Fastify 98/98, Express 90 passed + 8 skipped
(see bug 1 below). The shared suite is now wired into Express and
Fastify. |
| Redis restart | Real-Redis split-deploy test (in
`pubsub/redis-streams/src/observe-cross-process.test.ts`) passes on both
engines. The client resumes on a freshly booted API process at last
index + 1 with nothing repeated or missing. Evented: the worker executes
the run and the API is killed with SIGKILL. Durable runs execute in the
API that started them, so that API stays up; the client drops its
connection and resumes on a second API. |
| Suspend | After tool approval, indexes keep increasing. Observers
parked mid-suspend and every cut across the suspend resume correctly on
both engines. |
| Unit suites | server `agents.test.ts` 289, core `src/stream` +
`src/agent/streams` 481 |
| Static | tsc clean on core, server, client-js, the test suite, the
Hono/Express/Fastify adapters and redis-streams. oxlint 0 warnings. |

## Scope

Plain (non-durable) agents' `/stream` updates aren't numbered yet. The
caller receives each update before the cache assigns its number. That's
tracked in COR-1510. Plain agents' `/observe` is covered here.

## Existing bugs found (not fixed here)

1. **Express untilIdle 404:** after a client drops an untilIdle stream
on a durable or evented agent, Express answers `/observe` with 404 "Run
not found". It fails the same way on `main`. Those 8 cases are skipped
on Express only, via `skipUntilIdleDisconnectReplay`.
2. **Finished durable run, other API:** a finished durable run returns
404 on `/observe` from any API process except the one that executed it
(`authorizeDurableRun`, `handlers/agents.ts`). Resuming while the run is
live works from any API.
3. **Plain replay mismatch:** on plain agents, replayed
`step-finish`/`finish` payloads differ from the live ones (for example
`timestamp` and `processorRetryCount`). Related to COR-1510.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

When a connection drops, clients need to know which updates they already
received. This change adds positions to many updates so clients can
resume from the right place. It also closes observations of finished
runs when the requested position is past the end.

## Summary

- Added optional `index` values to chunks from durable and evented agent
streams, `observe()` streams, and corresponding HTTP routes. Clients can
resume by passing the last indexed chunk’s `index + 1`.
- Preserved run-log indexes through core stream adapters and server
frame conversion. Sub-agent data chunks do not carry a child run’s index
into the parent stream.
- Some chunks remain unindexed, including synthesized `abort` chunks and
client-built structured-output `object` chunks. Failed cache writes and
non-atomic custom-cache indexing can affect replay.
- Updated API schemas and documentation to explain index gaps, replay
limits, and resumption.
- Added past-end handling for finished plain-agent observations in core
and HTTP `/observe`. History checks do not delay stream return or
prevent cancellation.
- Added tests for indexed replay, past-end closure, cancellation during
history reads, sub-agent index isolation, and observation across API
restarts.

**Validation:** Current-head CI, build, and test results are not
available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: NikAiyer <nick.aiyer@gmail.com>
2026-11-06 11:40:46 -08:00
mastra-platform[bot]
75435bc2cd
fix(redis-streams): retry publishes on transient connection errors (#29512)
## Summary

The Redis Streams pubsub now retries a publish that fails with a
transient connection error, instead of failing the delivery and letting
the step re-run.

`#publishRemote` already retried while Redis answered `LOADING` after a
restart. Any other connection failure (socket closed unexpectedly,
client offline, connection or socket timeout, `ECONNRESET`, `EPIPE`,
`ETIMEDOUT`, `ECONNREFUSED`) was rethrown immediately. A lost
`workflow.step.end` publish was nacked and redelivered, so the step ran
again. These errors now retry under the same 120 s budget (250 ms
between attempts) until Redis accepts the write, the pubsub closes, or
the budget runs out.

This addresses the transport side of #29504.

## Limits

- **Not exactly-once.** A worker that dies between finishing a step and
publishing `step.end` still gets the `step.run` redelivered, and that
step runs again. At-least-once delivery is accepted for that window.
- **Duplicates after a lost reply.** If Redis wrote the event but the
reply was lost, the retry appends it again. Subscribers can receive a
second delivery with the same event id. A duplicate `workflow.step.run`
is dropped by the step fence; other events are already handled as
at-least-once.
- **Slower failure in a long outage.** A publish that starts while
connected and then loses the connection now fails after the 2-minute
budget (`PublishRetryBudgetExhaustedError`) if Redis doesn't come back,
where it previously failed immediately. The trade-off is that no event
is lost in a short outage.

## Changes

- `pubsub/redis-streams/src/index.ts`: `isTransientPublishError()`
widens the publish retry from `LOADING` to transient socket errors;
budget/delay constants renamed to `PUBLISH_RETRY_BUDGET_MS` /
`PUBLISH_RETRY_DELAY_MS`. Each attempt races a timer for what is left of
the budget, because node-redis parks commands issued while reconnecting
in its offline queue and never settles them; running out throws
`PublishRetryBudgetExhaustedError`, which is not retried.
`publishRetryBudgetMs` overrides the 120 s default.
- `pubsub/redis-streams/test-fixtures/harness.ts`:
`makeXaddDropProxy()`, a TCP proxy that drops the client connection on
the first XADD either before Redis sees it or after Redis applies it,
optionally closing its listener after the drop so Redis looks down for
good.
- `pubsub/redis-streams/src/publish-transient-retry.test.ts`: both drop
modes deliver the event; all copies share one event id. A third case
closes the proxy listener after the drop and asserts publish rejects
with `PublishRetryBudgetExhaustedError` at a 2 s budget.
- Changeset: `@mastra/redis-streams` patch.

## Tests

- `publish-transient-retry.test.ts`: 2/2 pass. With the check reverted
to `isLoadingError`, both fail with `Socket closed unexpectedly`.
- `pnpm --filter @mastra/redis-streams run check:test` passes; full
`@mastra/redis-streams` test suite run against Redis 6381 / Postgres
5436.
- No changes to `@mastra/core`; `pnpm build:core` succeeds.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

If Redis briefly loses a connection while receiving an event, the
publisher now tries again instead of failing immediately. If Redis saved
the event but its reply was lost, the retry can deliver a duplicate with
the same event ID.

## Changes

- Redis Streams publishing retries `LOADING` errors and recognized
transient connection errors, including closed sockets, `ECONNRESET`,
`EPIPE`, and timeouts.
- Retries stop for non-transient errors, when the pub/sub instance is
closed, or after 120 seconds. Attempts are spaced 250 ms apart.
- Added a test proxy and tests for connection drops before and after
Redis writes an `XADD`.
- Added a patch changeset for `@mastra/redis-streams`.

When Redis writes an event but the reply is lost, the retry can append
the event again. Both copies have the same event ID.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: NikAiyer <nick.aiyer@gmail.com>
2026-11-05 13:57:59 -08:00
mastra-platform[bot]
2f93851c37
fix(core): make Agent.durable configure resumable streams only (#29503)
Co-authored-by: NikAiyer <nick.aiyer@gmail.com>
2026-11-05 11:32:10 -08:00
mastra-platform[bot]
f94beac862
fix(core): unwedge restarted nested evented loops (#29431)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9

## Problem

Setup: a split deploy (API with `MASTRA_WORKERS=false` plus dedicated
workers over Redis) with `recovery.durableAgents: 'auto'`. When the API
and the worker die together mid-tool and both restart, boot recovery
claims the run under a new generation and the interrupted tool runs
again. After that, the run stops making progress. It sits in
`durable-agentic-loop:running` forever while the API keeps renewing its
lease. Because the run never ends, it also keeps its thread lease, so
any later run on the same thread is rejected with
`AgentThreadLeaseConflictError`. Mastra Doctor hits this every time,
since its thread is the deployId and every re-diagnosis lands on the
stuck thread. The `DurableExecutionFenceError` and `This workflow run
was not active` messages in the ticket are side effects of this stuck
run.

## Root cause

Recovery behaves correctly: it claims a new generation and re-drives the
run. The run gets stuck in the evented step guard in
`WorkflowEventProcessor.endWorkflow`.

When a restarted nested loop body finishes, `endWorkflow` calls
`#processLoop` for the parent loop. The `!parentWorkflow.restart`
condition there, introduced by 11c83d5333 (#28054), skipped the outer
loop leaf's stored `startedBy` on restart. #28054 lists
restart/redelivery races as out of scope, so this exclusion was not a
deliberate restart contract. As a result:

1. The completion guard never ran, so the outer loop leaf stayed
`running` under the event id from before the crash.
2. The loop's next `workflow.step.run` carried a new event id, so
`assertWorkflowStepGuard` raised a `stale` conflict.
3. `process()` acknowledges `stale` conflicts without retrying or
reporting an error, so the step was dropped and the loop never advanced.

## Fix

Remove `restart` from that exclusion. A restarted child re-drives the
stored occurrence, so it now completes under that occurrence's
`startedBy`, the same way a normal redelivery does. Time-travel and
resume still skip the guard as before. The code change is 3 lines in
`packages/core/src/workflows/evented/workflow-event-processor/index.ts`,
plus a patch changeset for `@mastra/core`.

## Tests

**Core unit test:**
`packages/core/src/workflows/evented/restart-nested-loop-guard.test.ts`
- Runs on an in-memory store that reports `supportsWorkflowStepGuards()
=== true` and enforces `assertWorkflowStepGuard` the way `@mastra/pg`
does.
- A `dountil` over a nested workflow crashes mid-iteration on host A,
then host B restarts it over the same storage.
- The test asserts that the loop finishes and the leaf is not left
`running`.
- With the `index.ts` line reverted it fails: the leaf stays `running`
under the pre-crash `startedBy` and the run hangs. With the fix it
passes.

**End-to-end test:**
`pubsub/redis-streams/src/durable-full-restart.test.ts` (fixture
`test-fixtures/durable-full-restart.entry.mjs`)
- **Setup:** real Redis and Postgres, separate API and worker processes,
`recovery.durableAgents: 'auto'`, and the same restart hook the deployer
runs at boot.
- **Scenario:** kill both processes mid-tool, then restart both.
- **Each case asserts that:**
  - the run reaches `workflow.end`
  - every tool call completes
  - the run owner lease and the thread lease are both released
- a follow-up run on the same thread is admitted with no
`AgentThreadLeaseConflictError`

| Case | main | this PR |
| --- | --- | --- |
| core: restart of nested loop body (guarded store) | ❌ hangs, leaf
`running` | ✅ |
| SIGKILL both mid-tool, x1 | ❌ run never settles (owner `live:true`,
snapshot `running`) | ✅ |
| SIGTERM both mid-tool, x1 | ✅ (graceful drain doesn't take the restart
path) | ✅ |
| SIGKILL both mid-tool, x2 (killed again after recovery, so the
snapshot must survive a second restart) | not run (x1 already fails) | ✅
|

Re-run on current `main` with the fix:
- core `vitest run src/workflows/evented`: 44 files, 682 passed / 5
skipped
- `durable-full-restart.test.ts`: 3/3 pass
- `tsc --noEmit` on core: clean
- redis-streams `tsc --noEmit` and `tsc --project tsconfig.test.json`
(the fixture typecheck that CI's Worker takeover job runs): clean
- `oxlint` on touched files: clean

### Combined with COR-1502

COR-1502 is a parallel fix on
`user/session-6ca2c63a-0a42-4fed-8de8-25f33217e319` (554226de) for Redis
`LOADING` errors during lease extension. I merged it locally with this
branch:
- the merge had no conflicts
- `durable-full-restart.test.ts`: all 3 cases pass on the merged code
- `evented-redis-loading.test.ts`: passes


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

If the API and worker restart during a run, a nested loop could get
stuck and keep its thread busy. This change preserves the loop’s stored
event information during restart so the run can finish and another run
can use the same thread.

## Changes

- Pass the nested loop leaf’s stored `startedBy` value when the parent
workflow restarts. Time-travel and resumed workflows still skip this
guard.
- Add a core regression test for restarting an evented workflow with
step guards enabled.
- Add a Redis Streams and Postgres integration test for API and worker
restarts. It covers `SIGKILL`, `SIGTERM`, and a second crash after
recovery. It checks run and tool completion, lease release, and a
follow-up run on the same thread.
- Add the integration fixture, include the test in the durable-streams
CI job, and add a patch changeset for `@mastra/core`.

## Tests

The diff adds regression and integration tests. Test-run and type-check
results are not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: review <review@local>
Co-authored-by: NikAiyer <nick.aiyer@gmail.com>
2026-11-04 18:59:50 -08:00
mastra-platform[bot]
a0068a9ad1
feat(core): resumable streams for every Agent (#28424)
> Opened by @user_01KNPFT2RPETZTEP15J3AEBA08

Originally stacked on
[#28416](https://github.com/mastra-ai/mastra/pull/28416), which has
merged. This PR now targets `main`.

## What

Any `Agent` can now have resumable streams, not only durable agents. A
run is published on `agent.stream.<runId>` with a replay cache, so:

- a client can reconnect after a disconnect,
- another reader can follow the run with `agent.observe(runId)`,
- any reader can stop it (`observe().abort()`,
`agent.abortRunStream(runId)`, or the abort-run route).

Durable execution (surviving process restarts) stays with `DurableAgent`
/ `EventedAgent` / `InngestAgent`, which keep publishing through the
same stream layer.

### Config

New `streams?: boolean | { pubsub, cache, shouldCache, cleanupTimeoutMs
}` on `Agent`.

- Omitted: on when the agent is registered on `Mastra`, off for
standalone agents.
- `false`: off. `{ cache: false }`: live only, no replay.
- With `durable`, `streams` keys override `durable` keys. `durable:
true` + `streams: false` keeps publishing, drops the replay cache, and
warns.

### Bus selection
- Registered agents (plain and `DurableAgent` on any engine) publish on
`mastra.pubsub`, so other processes sharing the PubSub and run-record
cache can observe and abort.
- Precedence: `streams.pubsub` > `durable.pubsub` > `mastra.pubsub`. The
evented engine still forces `mastra.pubsub` over an agent-level pubsub
and warns when they differ.
- Unregistered agents keep a private in-process `EventEmitterPubSub`.
- The replay cache wraps the shared bus once (existing `CachingPubSub`
reuse check), so nothing is double-published.

### Behaviour change

With streams on, closing the HTTP stream detaches the reader instead of
aborting the run. To stop a run explicitly, use `POST
/agents/:agentId/runs/:runId/abort`, which now works for plain agents
too (ownership-checked, including across servers sharing the PubSub and
run-record cache), or the thread abort route. `streams: false` keeps the
old behaviour.

## Commits

1. `8c8975ef` loop publishes chunks through a `streamPublisher`
(step-start, finish, CHUNK, SUSPENDED on close, ERROR if the stream ends
without a finish).
2. `f6d7dabf` `Agent.observe` reads the agent's bus;
`hasResumableStreams()` added. DurableAgent opts its inner agent out so
nothing is published twice.
3. `8b33a9d7` abort requests go over the control topic, so any reader on
the bus can stop a run.
4. `92a5a320` server: stream routes gate on `hasResumableStreams`,
observe reads a plain agent's bus, abort-run supports plain runs;
cleanup timers centralised on `AgentStreamBus.scheduleCleanup`.
5. `ff5456df` "Resumable streams" docs page, `Agent` reference entry,
changeset (core minor, server patch).
6. `24936904` registered `DurableAgent` on the default engine publishes
on `mastra.pubsub`. Before, if the bus was read before registration, it
stayed on the private bus and other instances never saw its runs.

## Rebase onto #28416 / main
- `loop/workflows/stream.ts` conflicted with #28157, which added a
`finalization` parameter to `workflowLoopStream`. Kept main's signature
and added `streamPublisher` alongside; snapshot finalization and
publishing both run.
- No textual conflict, but #28531 (follow-up to #28514) made stream
output a proxy, so `fullStream` overrides must call the base getter with
`this`. `Agent.observe` had the old pattern and failed with "Cannot read
private member #options"; main's fix is applied there (folded into
`f6d7dabf`). Reverting it fails both observe tests in
`agent-streams.test.ts`.

## Proof

Head `d2bd22d4`:
- Core `tsc -p tsconfig.build.json`, oxlint, oxfmt clean. `vitest run
src/agent/durable src/agent/streams
src/agent/__tests__/thread-stream-restart.test.ts` plus the
observe-approval-resume, observe-lifecycle and observe-reattach aimock
scenarios: 197 files, 1985 passed, 28 skipped, exit 0.
- Server `tsc` clean; `vitest run src/server/handlers`: 67 files, 2476
passed, 2 skipped, 1 todo.
- Inngest `tsc` clean; `test:unit` 103 passed, 2 failed. The same 2
tests ("stream/generate terminates when the opening event and error
event cannot be published") fail identically on `main` (`df76f896`,
rebuilt core).
- Docs: `remark --frail` and `oxfmt-mdx --check` clean.
- Observation spans suspension: new shared aimock scenario
`observe-approval-resume` on normal, durable, evented and fs engines. An
observer attached before and one attached during the approval wait both
receive the approved answer and `finish`; a fresh replay after
completion holds both segments; reconnecting after the suspension index
repeats nothing; the tool runs once; `onSuspended` fires once; a
cancelled reader leaves the run going.
- Request context: an unscoped Studio caller (auth mapper, Studio FGA,
no server FGA) can observe and abort; other tenants and unscoped
non-Studio callers still get 403; observe resolves the agent with the
request context.
- Shared ownership: each plain run writes a record
(`agent.run.<agentId>.<runId>`: agent, run, resource, thread,
running/suspended/finished) to the replay cache, or `mastra.serverCache`
when replay caching is off, before its first event. Suspended and
finished are written after the terminal event, the record is deleted
with the topic, and it has the run log's 24h TTL. Routes return 404 for
a missing, expired or evicted record and 503 when the cache can't be
read. Abort returns `aborted: true` only when the executing process
acknowledges the stop.
- Two real HTTP servers
(`pubsub/redis-streams/src/plain-run-cross-server.test.ts`: LibSQL file,
Redis Streams PubSub, RedisServerCache, SimpleAuth, aimock): owner
observes on B through suspend, approve on A, and finish; 403/401/404 on
both servers; abort through B confirmed by A; a parked run's abort is
`false`; expired and cleaned-up runs 404; ownership with `cache: false`.
Run with the durable/evented controls (`run-abort-cross-server`,
`observe-cross-process`): 20/20 in 3 consecutive runs. On the
pre-ownership commit the plain suite fails 8/8 (remote observe/abort
404).
- Mutation checks (each restored afterwards): re-adding `closeOnSuspend`
to plain observe fails the normal and fs scenarios; re-adding the HTTP
close on `suspended` fails the server test; dropping the request context
fails the Studio observe/abort and agent lookup tests; removing the
initial, suspended or delete record writes fails the record lifecycle
tests; removing the finished guard, the 503 handling or the
acknowledgement in the abort route fails their tests; acking an abort
that reached an ended run fails the new publisher test.
- No double publishing: the parity, durable-shared-bus and agent-streams
tests still pass (DurableAgent's inner agent keeps `streams: false`; the
replay cache wraps the shared bus once).

## Separate main-merge verification

Conflict resolution is isolated in
[50a9ef89](50a9ef8947),
after candidate `d2bd22d4`; no feature commits were rewritten.

- Merged-result focused core regression suites: 6 files, 37 tests
passed, covering parity, shared buses/run records, remote abort and
approval/resume observation.
- Complete merged-result server-handler suite: 68 files, 2,478 passed, 2
skipped, 1 todo. A separate real-TCP Hono/real-Memory authorization
matrix also passed 13/13, including the formerly failing Studio-only FGA
observe and abort contracts (without expected-failure markers).
- Type-aware oxlint and formatting passed on all 5 conflict-resolved
files.
- Full core/server typechecks are **not clean in the verification
checkout**. Swapping each package to unchanged `main` source in the same
installed dependency environment reproduces the same diagnostics (core
byte-identical; server identical after normalizing source-line
positions). These include AI SDK schema compatibility errors and missing
agent-builder artifacts; no additional diagnostics were introduced by
the conflict resolution.
- A broader core run passed 2,020 tests but exited nonzero: 5 suites
initially lacked the built internal LLM recorder; after building it, all
5 suites passed (24 tests). The crash-recovery worker still exhausted
its JavaScript heap after 8 of 10 tests. The broad run also reported the
baseline type errors above. This is **not** a clean full-core result.
- Follow-up `ec0bc59b` clears control and acknowledgement topics with
the finished run log; `35debe7a` and `920a094a` correct documentation.
Independent verification below includes the cleanup production change.

## Independent adversarial review

Verdict: **approve within the requested scope**, with no actionable
supported-flow defect found. No formal GitHub approval was posted. The
reviewed production head is `ec0bc59b32`;
the only subsequent change in `920a094a` is a checked docs sentence
distinguishing expired ownership from missing replay history.

Independent reviewer ran **343 passing tests**: 38 focused core, 280
server handlers, 8 real two-process TCP/Redis/LibSQL/AIMock cases, 16
public-API processor-safety probes, and 1 actual Redis cleanup probe.

- Redaction/tripwire bypass hypothesis did not reproduce on normal,
durable, evented or filesystem engines, with either immediate or
deferred initiating-stream consumption. Serialized observer frames
excluded the raw sensitive value; redaction replay matched the masked
caller output, and tripwires exposed no raw text. No processor redesign
was made.
- Actual Redis cleanup was tested after an acknowledged cross-node
abort: physical run, control, acknowledgement, counter/list and
ownership keys were first observed, then all ten run-associated keys
disappeared after configured cleanup. This verifies more than the added
clearTopic spy assertion.
- The reviewer used source aliases for the exact production sources in a
disposable checkout, including both child HTTP processes. They did not
independently validate the full build, Inngest, browser Studio Stop,
natural 24-hour expiry or load costs.

## Unverified

- Studio Stop button for thread-less plain runs: not exercised in a
browser.
- Cost: every registered agent publishes each chunk and writes an
ownership record. Not benchmarked.
- Natural 24h expiry: tested by deleting the record, not by waiting.
Merged main now honors the requested minimum TTL for indexed in-memory
logs. Entry-cap eviction can still shorten replay; docs were corrected
in `35debe7a`.
- Abort of a run whose executing process died: returns `aborted: false`
after the ack timeout; the run's record stays until its TTL.

## Assumptions

- InngestAgent keeps its own pubsub/cache resolution. Importing a new
core export would fail `check-core-imports` until core is released, so
no Inngest change or peer-range bump here (and no inngest changeset).
Follow-up after release.
- thread-stream-restart's "leaves nothing" assertion now ignores the
agent stream topic, since run events are kept for `cleanupTimeoutMs` by
design.

## Decisions (confirmed by Tony)

1. Registered `DurableAgent`s on the default engine use `mastra.pubsub`,
same as plain agents; unregistered ones keep a private bus.
2. Streams are on by default for agents registered on `Mastra` and off
for standalone agents; docs explain the cost and how to turn streams
off.
3. With streams on, HTTP disconnect detaches instead of aborting.

## Open question

The Superagent security scan flags (3): a client can start expensive
runs and disconnect, and nothing limits active or concurrent detached
runs. Options: accept and document, cap detached runs, or keep
abort-on-disconnect for HTTP-started runs.

Risk: medium-high — every registered agent now publishes, HTTP
disconnect no longer aborts plain runs, and observe/abort now authorize
plain runs from a shared cache record.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

If someone disconnects while an agent is working, the agent can keep
running and save its updates so the reader can reconnect. Authorized
readers can also check on or stop a run; this does not add crash
recovery to plain agents.

## What changed

- Added configurable resumable streams for plain `Agent` runs. Streams
default to on for agents registered with `Mastra` and off for standalone
agents.
- Added `observe()` replay and live updates, including observation from
an event offset. Added abort support through `observe().abort()`,
`abortRunStream(runId)`, and server routes.
- When streams are enabled, an HTTP disconnect detaches the reader
instead of aborting the run. For plain agents, `streams: false` disables
publishing. Durable agents continue to publish; `streams: false`
disables their replay cache.
- Added run ownership records and authorization checks. Shared PubSub
and cache support observation and abort across servers. Unknown or
unverified runs return 404; cache-read failures return 503.
- Added run-ID claim checks. A run ID already owned by another resource
is rejected with `AGENT_RUN_ID_ALREADY_CLAIMED` and maps to HTTP 409.
Concurrent claims across processes may still overlap because the cache
has no compare-and-set operation.
- Added cleanup timers that do not keep the process alive. Cache TTL and
eviction can also limit event retention.
- Updated step bookkeeping to use the run’s original owner when
execution replaces its abort signal.
- Added documentation, tests, and a Redis/LibSQL cross-server
integration suite.

## Verification and open questions

The supplied context reports that the merged-result server-handler suite
passed 2,478 tests, with 2 skipped and 1 todo. It also reports real
two-server Redis/LibSQL tests, but provides no exact result count. Full
core and server typechecks were not clean; the reported diagnostics were
reproduced against unchanged `main`. A broader core run stopped after 8
of 10 tests when the crash-recovery worker exhausted its heap. Two
Inngest unit-test failures were also reported on `main`.

The author flagged that detached runs can continue without limits on
active or concurrent runs. Browser testing of Studio Stop for
thread-less plain runs, performance costs, natural 24-hour expiry, and
abort behavior after the executing process dies remain unverified. An
abort request cannot be confirmed if the process running the run is
unavailable.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: NikAiyer <nick.aiyer@gmail.com>
Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: review <review@local>
2026-11-04 17:29:16 -08:00
dane-ai-mastra[bot]
675caf0613
chore: version packages (alpha) (#29336)
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases

> The changelog information of each package has been omitted from this
message, as the content exceeds the size limit.

## @mastra/client-js@1.60.0-alpha.0


## @mastra/react@1.15.0-alpha.0


## @mastra/factory@0.28.0-alpha.0


## @mastra/code-sdk@1.19.0-alpha.0


## mastra@1.39.0-alpha.0


## @mastra/core@1.83.0-alpha.0


## @mastra/mcp@2.9.0-alpha.0


## @mastra/mcp-apps-components@0.1.0-alpha.0


## @mastra/playground-ui@69.0.0-alpha.0


## @mastra/server@1.83.0-alpha.0


## @mastra/nestjs@0.3.0-alpha.0


## @mastra/clickhouse@1.30.0-alpha.0


## @mastra/duckdb@1.19.0-alpha.0


## @mastra/libsql@1.30.0-alpha.0


## @mastra/pg@1.36.0-alpha.0


## @mastra/temporal@0.6.0-alpha.0


## @mastra/auth-better-auth@1.1.11-alpha.0


## @mastra/deployer-cloud@1.83.0-alpha.0


## @mastra/deployer-cloudflare@1.2.42-alpha.0


## @mastra/deployer-netlify@1.2.42-alpha.0


## @mastra/deployer-sandbox@0.3.27-alpha.0


## @mastra/deployer-vercel@1.2.42-alpha.0


## @mastra/longmemeval@1.1.42-alpha.0


## @mastra/opencode@0.1.42-alpha.0


## create-factory@0.4.3-alpha.0


## mastracode@0.49.1-alpha.0


## @mastra/arize@1.3.30-alpha.0


## @mastra/arthur@0.4.30-alpha.0


## @mastra/braintrust@1.3.27-alpha.0


## @mastra/datadog@1.4.22-alpha.0


## @mastra/deepeval@0.1.24-alpha.0


## @mastra/laminar@1.3.32-alpha.0


## @mastra/langfuse@1.6.6-alpha.0


## @mastra/langsmith@1.4.3-alpha.0


## @mastra/observability@1.23.2-alpha.0


## @mastra/otel-bridge@1.5.22-alpha.0


## @mastra/otel-exporter@1.5.2-alpha.0


## @mastra/posthog@1.4.7-alpha.0


## @mastra/sentry@1.2.32-alpha.0


## @mastra/agent-builder@1.2.1-alpha.0


## @mastra/connect@1.3.2-alpha.0


## create-mastra@1.39.0-alpha.0


## @mastra/deployer@1.83.0-alpha.0


## @mastra/editor@0.17.1-alpha.0


## @mastra/mcp-docs-server@1.3.13-alpha.0


## @mastra/memory@1.41.1-alpha.0


## @mastra/rag@2.7.3-alpha.0


## @mastra/redis-streams@0.8.1-alpha.0


## @mastra/elysia@0.1.23-alpha.0


## @mastra/express@1.5.27-alpha.0


## @mastra/fastify@1.5.27-alpha.0


## @mastra/hono@1.7.25-alpha.0


## @mastra/koa@1.7.27-alpha.0


## @mastra/next@0.2.41-alpha.0


## @mastra/tanstack-start@0.2.41-alpha.0


## @mastra/spanner@1.11.2-alpha.0


## @mastra/turso@0.1.21-alpha.0


## @mastra/inngest@1.11.2-alpha.0


## @mastra/gcs@0.3.6-alpha.0


## @internal/playground@1.39.0-alpha.0

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-11-04 14:40:27 -08:00
mastra-platform[bot]
49eaabb19b
fix(core,redis-streams): recover evented runs when Redis is LOADING mid-tool (COR-1502) (#29430)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9

## Summary

On the evented engine, if Redis restarts while a tool step is finishing,
it spends a while (often 30–60s) reloading its dataset. During that time
it answers every command with `LOADING Redis is loading the dataset in
memory`. When that happens, the run stays `running` forever and no
answer is saved. This PR makes the run wait out the restart and finish
normally: the answer is saved and each tool runs exactly once.

Fixes COR-1502.

## Root cause: three stacked problems

1. **Publishing never retried.** `RedisStreamsPubSub.publish()` sent the
tool result and `step.end` once. During `LOADING` that failed, so the
step was redelivered and its tools ran a second time.
2. **The thread lease expired during the window.** Redis kept counting
down the lease while it reloaded. On redelivery, Postgres still renewed
the run ownership, but `renewDurableThreadLease` found no lease key and
returned `false`. The execution fence then refused every redelivery
("could not verify that this execution still owns the run"). The event
was dropped after 5 deliveries and the run stayed `running`.
3. **The lease retake gave up when the step went idle.** `#retakeLease`
in `thread-stream-runtime` stopped trying once the tool step finished.
The run was aborted as lease-lost (`reason: 'abort'`, empty answer)
before the final model turn.

## Fix

- **`@mastra/redis-streams`:**
- `publish` retries `LOADING` errors every 250ms for up to 2 minutes,
and stops if the client closes. This covers both plain `XADD` and
`MULTI`/`EXEC` publishes.
- It implements the new `isUnavailableToAll(error)`, which returns true
for `LOADING`.
- **`@mastra/core` `events/pubsub.ts`:** adds an optional
`LeaseProvider.isUnavailableToAll?(error)`. A backend returns true when
it is refusing every client, so nobody else can be granted the lease
meanwhile. Any other error, including a connection failure, must return
`false`.
- **`@mastra/core` execution fence:** this applies only when Postgres
run ownership has just renewed for a generation-fenced claim.
- If the thread lease can't be renewed because the key is gone, the
fence takes it back with `SET NX`. It never takes a lease that another
owner holds.
- If the backend is refusing every client, the fence relies on the
Postgres run ownership instead.
  - Any other lease error still fences the execution.
- **`@mastra/core` thread-stream runtime:** while the backend refuses
every client, `#retakeLease` keeps asking, within the existing
`AGENT_LEASE_LOST_TOOL_GRACE_MS`. Generic errors keep the existing
strict lease-loss policy, which stays safe for network partitions.

## Tests

- New test `pubsub/redis-streams/src/evented-redis-loading.test.ts`. It
runs separate API and worker processes against real Redis and Postgres.
They connect through a TCP proxy (`makeLoadingProxy` in
`test-fixtures/harness.ts`) that simulates a restart: it drops every
connection, then answers `-LOADING` for 20s while two tools finish.
During the window, `MULTI`/`EXEC` behave as in real Redis (`EXEC` →
`EXECABORT`).
- **Red on main:** the run never reached a terminal state, and the
worker looped on the fence error.
- **Green with the fix (~30s):** the run completes, the answer is saved,
each tool runs exactly once in `tool_executions`, and run ownership is
released.
- Related suites pass:
  - core: execution-fence, thread-lease, and thread-stream suites.
- redis-streams: `pubsub`, `agent-thread-cross-process`,
`durable-worker-takeover`, `fail-publish-after-exhaustion`,
`local-redelivery-evented`.
- `tsc` (core, redis-streams, redis-streams test config) and `oxlint`
are clean on the touched files.
- Validated together with the COR-1497 branch
(`user/session-90e601e9-…`, `7c3d66f9`): the local merge had no
conflicts, and `durable-full-restart` (3 cases) plus this LOADING test
all pass.

## Out of scope / follow-up

On success, the workflow engine still publishes `workflow.step.end`
before it saves the step result. Any publish failure other than
`LOADING` still leaves a window where a redelivered `step.run` finds no
saved result and runs its tools again. Saving the result before
publishing would close that window. It is a broader engine change, so I
left it for a separate ticket.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

When Redis restarts, it can temporarily refuse requests while it reloads
data. This change lets evented runs wait for Redis and recover their
thread lease, so they can finish without repeating completed tools.

## What changed

- `@mastra/redis-streams` retries stream publishing when Redis returns
`LOADING`, up to two minutes. Other errors still fail immediately.
- `LeaseProvider` can optionally implement `isUnavailableToAll(error)`
to identify errors that mean the backend is refusing every client.
- After Postgres run ownership is renewed, the execution fence can
reclaim a missing thread lease. If the backend reports that it is
refusing all clients, the fence relies on run ownership. Other lease
errors still prevent execution.
- Thread-lease retakes can continue during backend-wide unavailability
within the existing grace period.
- Added an integration test for a Redis loading window during tool
execution. The test checks that the run completes, saves its answer,
executes each tool once, and releases run ownership.

## Tests

The PR description reports that the integration test passes against real
Redis and Postgres. It also reports passing related core and Redis
Streams suites, TypeScript checks, and `oxlint` on touched files. These
results were not independently verified here.

## Known limitation

The PR description notes that a workflow step may still rerun tools on
redelivery if publishing its result fails for an error other than
`LOADING`, because the result is published before it is saved.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-04 13:07:20 -08:00
dane-ai-mastra[bot]
6e8dfcb83f chore: version - exit prerelease mode 2026-11-03 23:45:58 +00:00
mastra-platform[bot]
ab4d89cd38
test(redis-streams): run durable worker-takeover suite against real Redis and Postgres (#28952)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9

Part 3 of 3, split from #28676. Refs COR-1490. Parts 1 and 2 (#28947,
#28951) are merged into `main`. This PR is based on `main`.

## What this does
The API-loss and worker-takeover guarantees from #28951 only hold with a
shared store that supports run fencing and a transport that works across
processes. In-memory suites can't test that. This PR:
- adds `.github/workflows/test-durable-streams.yml`, which runs the
two-process takeover suite against Redis (`:6381`) and Postgres
(`:5436`) service containers. The checkout uses `persist-credentials:
false`.
- calls it from `test-suite.yml` as the `durable-streams` job. The job
reports its own check but does not gate `Merge Test Reports`, because
its flake rate hasn't been measured yet.
- runs it when `prebuild.yml` sees changes to the evented agent and
workflow code, `packages/core/src/mastra/`, `packages/memory/`,
`pubsub/redis-streams/`, `stores/pg/`, or the workflow files
- extends `pubsub/redis-streams/src/durable-worker-takeover.test.ts` and
adds the `durable-api-loss.entry.mjs` fixture. The scenarios: API
SIGTERM/SIGKILL with remote workers, a thread follow-up after API loss,
a remote abort, worker shutdown, single-process Redis shutdown, and the
original caller completing with and without a survivor that is already
running
- adds `tsconfig.test.json` and a `check:test` script, which type-check
the tests and fixtures against core's `dist`

No changeset: this PR only touches CI and tests.

## Verification
- Reviewer run at `bed900442c` against Redis and Postgres: the
worker-takeover suite passes 11/11. Reverting #28951's production
changes makes 7 of 11 fail.
- `check:test` exits 0 with the single config and covers all 12 package
files, including `agent-thread-worker.entry.ts`.

Risk: low. CI and tests only.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## ELI5

This PR adds a CI check that runs worker-takeover tests with real Redis
and Postgres. The tests check whether durable work can continue or
recover when the API or a worker stops.

## Summary

- Adds a reusable GitHub Actions workflow with Redis 8 and
pgvector/Postgres 16 services.
- Updates `prebuild.yml` to detect changes that affect durable streams
and pass that result to `test-suite.yml`.
- Adds a `durable-streams` job that runs when the relevant code changes.
- Adds takeover tests for API `SIGTERM` and `SIGKILL`, follow-up
messages, remote aborts, worker shutdown, Redis shutdown, and caller
completion.
- Adds a fixture that records tool execution state in Postgres and uses
Redis Streams for pub/sub.
- Adds `check:test` and TypeScript test configuration for typechecking
the fixtures separately from the package build.

The author reports that `check:test` passed and the takeover suite
passed 11/11 against Postgres and Redis 8. The author also reports that
reverting `packages/core/src` to its state before `#28951` causes the
API `SIGTERM` takeover test to fail. The PR author states that no
changeset is required.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: NikAiyer <nick.aiyer@gmail.com>
Co-authored-by: Nik Aiyer <nik@mastra.ai>
Co-authored-by: review <review@local>
Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
2026-11-03 11:41:25 -08:00
Dane - AI (Mastra)
3bdb4ec5f4
chore(deps): update build tools (major) (#29026)
> [!NOTE]
> This PR body was truncated due to platform limits.

This PR contains the following updates:

| Package | Type | Update | Change | Pending |
|---|---|---|---|---|
| [@vitest/coverage-v8](https://vitest.dev/guide/coverage)
([source](https://redirect.github.com/vitest-dev/vitest/tree/HEAD/packages/coverage-v8))
| pnpm.catalog.default | major | [`4.1.12` →
`5.0.5`](https://renovatebot.com/diffs/npm/@vitest%2fcoverage-v8/4.1.12/5.0.5)
| `5.1.0` |
| [@vitest/mocker@>=2.1.0
<4.1.12](https://redirect.github.com/vitest-dev/vitest/tree/main/packages/mocker)
([source](https://redirect.github.com/vitest-dev/vitest/tree/HEAD/packages/mocker))
| pnpm-workspace.overrides | major | [`4.1.12` →
`5.0.5`](https://renovatebot.com/diffs/npm/@vitest%2fmocker@>=2.1.0
<4.1.12/4.1.12/5.0.5) | `5.1.0` |
| [@vitest/ui](https://vitest.dev/guide/ui)
([source](https://redirect.github.com/vitest-dev/vitest/tree/HEAD/packages/ui))
| pnpm.catalog.default | major | [`4.1.12` →
`5.0.5`](https://renovatebot.com/diffs/npm/@vitest%2fui/4.1.12/5.0.5) |
`5.1.0` |
| [dotenv](https://redirect.github.com/motdotla/dotenv) |
devDependencies | major | [`^17.4.2` →
`^18.0.9`](https://renovatebot.com/diffs/npm/dotenv/17.4.2/18.0.9) | |
| [dotenv](https://redirect.github.com/motdotla/dotenv) |
devDependencies | major | [`17.4.2` →
`18.0.9`](https://renovatebot.com/diffs/npm/dotenv/17.4.2/18.0.9) | |
| [vitest](https://vitest.dev)
([source](https://redirect.github.com/vitest-dev/vitest/tree/HEAD/packages/vitest))
| devDependencies | major | [`^4.1.12` →
`^5.0.5`](https://renovatebot.com/diffs/npm/vitest/4.1.12/5.0.5) |
`5.1.0` |
| [vitest](https://vitest.dev)
([source](https://redirect.github.com/vitest-dev/vitest/tree/HEAD/packages/vitest))
| pnpm.catalog.default | major | [`4.1.12` →
`5.0.5`](https://renovatebot.com/diffs/npm/vitest/4.1.12/5.0.5) |
`5.1.0` |
| [vitest](https://vitest.dev)
([source](https://redirect.github.com/vitest-dev/vitest/tree/HEAD/packages/vitest))
| devDependencies | major | [`4.1.12` →
`5.0.5`](https://renovatebot.com/diffs/npm/vitest/4.1.12/5.0.5) |
`5.1.0` |
| [vitest@>=2.1.0 <4.1.12](https://vitest.dev)
([source](https://redirect.github.com/vitest-dev/vitest/tree/HEAD/packages/vitest))
| pnpm-workspace.overrides | major | [`4.1.12` →
`5.0.5`](https://renovatebot.com/diffs/npm/vitest@>=2.1.0
<4.1.12/4.1.12/5.0.5) | `5.1.0` |

---

> [!WARNING]
> Some dependencies could not be looked up. Check the [Dependency
Dashboard](../issues/9505) for more information.

---

### Release Notes

<details>
<summary>vitest-dev/vitest (@&#8203;vitest/coverage-v8)</summary>

###
[`v5.0.5`](https://redirect.github.com/vitest-dev/vitest/releases/tag/v5.0.5)

[Compare
Source](https://redirect.github.com/vitest-dev/vitest/compare/v5.0.4...v5.0.5)

#####    🐞 Bug Fixes

- Keep vite default `fs.deny` with api token file  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **Claude Opus 5.5** in
[#&#8203;11578](https://redirect.github.com/vitest-dev/vitest/issues/11578)
[<samp>(28dc1)</samp>](28dc1edfb)
- Handle quoted file names in `--changed`  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11583](https://redirect.github.com/vitest-dev/vitest/issues/11583)
[<samp>(14921)</samp>](14921e962)
- Add `data` to the `import.meta.hot` stub  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11590](https://redirect.github.com/vitest-dev/vitest/issues/11590)
[<samp>(3a828)</samp>](3a8289ac3)
- Set `skip` in `skipIf`, ignore `undefined` options  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11591](https://redirect.github.com/vitest-dev/vitest/issues/11591)
[<samp>(d523a)</samp>](d523a15f2)
- Skip builtin loading in async leak detector  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11598](https://redirect.github.com/vitest-dev/vitest/issues/11598)
[<samp>(fe86a)</samp>](fe86a64c6)
- Keep the import frame in native loader errors  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11599](https://redirect.github.com/vitest-dev/vitest/issues/11599)
[<samp>(9697b)</samp>](9697b3894)
- Run hook cleanups when a later hook fails  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11605](https://redirect.github.com/vitest-dev/vitest/issues/11605)
[<samp>(6c5a4)</samp>](6c5a44720)
- Release mocks of finished test files  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11607](https://redirect.github.com/vitest-dev/vitest/issues/11607)
[<samp>(e9c3d)</samp>](e9c3d72f0)
- Abort all running tests on cancel  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11606](https://redirect.github.com/vitest-dev/vitest/issues/11606)
[<samp>(6621a)</samp>](6621a72d6)
- **browser**:
- Ignore synthetic pagehide events  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11582](https://redirect.github.com/vitest-dev/vitest/issues/11582)
[<samp>(ca081)</samp>](ca0813584)
- Run plugin buildStart on server creation  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11593](https://redirect.github.com/vitest-dev/vitest/issues/11593)
[<samp>(ea611)</samp>](ea611e44c)
- Drop late connections of closed sessions  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11600](https://redirect.github.com/vitest-dev/vitest/issues/11600)
[<samp>(f161e)</samp>](f161e6df2)
- Wait 30s for the browser to close  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11601](https://redirect.github.com/vitest-dev/vitest/issues/11601)
[<samp>(6498b)</samp>](6498b666a)
- Run global setup once with instances  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11596](https://redirect.github.com/vitest-dev/vitest/issues/11596)
[<samp>(8dd67)</samp>](8dd67c262)
- **reporter**:
- End reason `&#&#8203;39;failed&#&#8203;39;` on worker crash  -  by
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;11568](https://redirect.github.com/vitest-dev/vitest/issues/11568)
[<samp>(6ae8e)</samp>](6ae8e89ff)

#####     [View changes on
GitHub](https://redirect.github.com/vitest-dev/vitest/compare/v5.0.4...v5.0.5)

###
[`v5.0.4`](https://redirect.github.com/vitest-dev/vitest/releases/tag/v5.0.4)

[Compare
Source](https://redirect.github.com/vitest-dev/vitest/compare/v5.0.3...v5.0.4)

#####    🚀 Features

- **experimental**:
- Tag cache directories with CACHEDIR.TAG  -  by
[@&#8203;lifeiscontent](https://redirect.github.com/lifeiscontent) in
[#&#8203;11461](https://redirect.github.com/vitest-dev/vitest/issues/11461)
[<samp>(a8ea3)</samp>](a8ea3c8fd)
- Support `test.ui.theme` option  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11499](https://redirect.github.com/vitest-dev/vitest/issues/11499)
[<samp>(1da66)</samp>](1da66405d)
- Highlight selector multiple matches in trace view  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **Claude Opus 5.5** in
[#&#8203;11489](https://redirect.github.com/vitest-dev/vitest/issues/11489)
[<samp>(5abba)</samp>](5abba1e1b)
- Print how many tests were affected and a total when running
`--changed`  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11424](https://redirect.github.com/vitest-dev/vitest/issues/11424)
[<samp>(a2336)</samp>](a23368585)
- Add an option to cache dependencies instead of statically parsing them
 -  by [@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11515](https://redirect.github.com/vitest-dev/vitest/issues/11515)
[<samp>(c0601)</samp>](c06012ac8)
- Add fit and zoom controls to trace viewer  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **Claude Opus 5.5** in
[#&#8203;11490](https://redirect.github.com/vitest-dev/vitest/issues/11490)
[<samp>(f5dd9)</samp>](f5dd99ed0)

#####    🐞 Bug Fixes

- Fail the run on git and location filter errors  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11430](https://redirect.github.com/vitest-dev/vitest/issues/11430)
[<samp>(30d03)</samp>](30d0304d2)
- Enable fsModuleCache by default  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11435](https://redirect.github.com/vitest-dev/vitest/issues/11435)
[<samp>(ad208)</samp>](ad208cc29)
- Track project files and mocks with --changed  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11432](https://redirect.github.com/vitest-dev/vitest/issues/11432)
[<samp>(b0f2b)</samp>](b0f2b3bb9)
- Refactor caching, fix `:` in project and change the shape of `results`
cache  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11453](https://redirect.github.com/vitest-dev/vitest/issues/11453)
[<samp>(9673c)</samp>](9673c49a4)
- Keep full `$` string values when `taskTitleValueFormatTruncate` is 0
 -  by [@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa),
**Hiroshi Ogawa** and **Claude Opus 5.5** in
[#&#8203;11485](https://redirect.github.com/vitest-dev/vitest/issues/11485)
[<samp>(4f7e1)</samp>](4f7e1799b)
- Resolve node builtins in client environments  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11417](https://redirect.github.com/vitest-dev/vitest/issues/11417)
[<samp>(56b31)</samp>](56b319ac6)
- Fix module graph builtin externals after
[#&#8203;11421](https://redirect.github.com/vitest-dev/vitest/issues/11421)
 -  by [@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa),
**Hiroshi Ogawa** and **Claude Opus 5.5** in
[#&#8203;11491](https://redirect.github.com/vitest-dev/vitest/issues/11491)
and
[#&#8203;11421](https://redirect.github.com/vitest-dev/vitest/issues/11421)
[<samp>(008e1)</samp>](008e184cb)
- Wrap module mocks during in-source testing  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11505](https://redirect.github.com/vitest-dev/vitest/issues/11505)
[<samp>(36d19)</samp>](36d19cb00)
- Handle line endings in snapshot  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11522](https://redirect.github.com/vitest-dev/vitest/issues/11522)
[<samp>(4c468)</samp>](4c468aa40)
- Respect root in projects  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11521](https://redirect.github.com/vitest-dev/vitest/issues/11521)
[<samp>(26642)</samp>](26642f793)
- Resolve mocks in directories with spaces  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11539](https://redirect.github.com/vitest-dev/vitest/issues/11539)
[<samp>(caaef)</samp>](caaef9416)
- **api**:
- Publish the api token atomically  -  by
[@&#8203;umairnawaz333](https://redirect.github.com/umairnawaz333) in
[#&#8203;11323](https://redirect.github.com/vitest-dev/vitest/issues/11323)
[<samp>(6f990)</samp>](6f990fef1)
- **browser**:
- Close the mock route interception race  -  by
[@&#8203;cplieger](https://redirect.github.com/cplieger) and
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;8339](https://redirect.github.com/vitest-dev/vitest/issues/8339)
and
[#&#8203;11083](https://redirect.github.com/vitest-dev/vitest/issues/11083)
[<samp>(d6620)</samp>](d6620fae6)
- Allow toHaveTextContent without arguments  -  by
[@&#8203;nomadical](https://redirect.github.com/nomadical) in
[#&#8203;11425](https://redirect.github.com/vitest-dev/vitest/issues/11425)
[<samp>(8ebcf)</samp>](8ebcf5f8a)
- Apply API options without browser config  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10852](https://redirect.github.com/vitest-dev/vitest/issues/10852)
[<samp>(ad055)</samp>](ad055837b)
- Preserve undefined command arguments  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) and
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;11497](https://redirect.github.com/vitest-dev/vitest/issues/11497)
[<samp>(1576d)</samp>](1576d3d89)
- Wait for setup mocks before importing a test file  -  by
[@&#8203;kasperpeulen](https://redirect.github.com/kasperpeulen) in
[#&#8203;11520](https://redirect.github.com/vitest-dev/vitest/issues/11520)
[<samp>(b61d1)</samp>](b61d1ced3)
- Avoid extra scroll outside the iframe  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11523](https://redirect.github.com/vitest-dev/vitest/issues/11523)
[<samp>(662e8)</samp>](662e8a363)
- Run browser instances as workers  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11525](https://redirect.github.com/vitest-dev/vitest/issues/11525)
[<samp>(bf04b)</samp>](bf04b9ccc)
- ⚠️ This can degrade performance if you have a custom `maxWorkers`
setting. Previously, it was applied incorrectly and could lead to Vitest
using too many resources. For example, a lower number would still open
too many browsers. Fixing it means Vitest opens less instances by
default now.
- Open one cdp session per tester  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11542](https://redirect.github.com/vitest-dev/vitest/issues/11542)
[<samp>(69e63)</samp>](69e6369f6)
- Clean up unmocked and pending mocks  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11537](https://redirect.github.com/vitest-dev/vitest/issues/11537)
[<samp>(5c955)</samp>](5c9552173)
- **coverage**:
- V8 browser memory leak between test files  -  by
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;11466](https://redirect.github.com/vitest-dev/vitest/issues/11466)
[<samp>(2297d)</samp>](2297d93a8)
- `allowExternal` fails to include uncovered files  -  by
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;11513](https://redirect.github.com/vitest-dev/vitest/issues/11513)
[<samp>(4ef00)</samp>](4ef000a93)
- Filenames containing brackets incorrectly showing as uncovered  -  by
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;11531](https://redirect.github.com/vitest-dev/vitest/issues/11531)
[<samp>(0fa80)</samp>](0fa80f53c)
- Istanbul coverage wrong when module transform warmups  -  by
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;11529](https://redirect.github.com/vitest-dev/vitest/issues/11529)
[<samp>(9f61d)</samp>](9f61dcca2)
- **deps**:
- Update all non-major dependencies  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11167](https://redirect.github.com/vitest-dev/vitest/issues/11167)
[<samp>(c74e0)</samp>](c74e099a9)
- **expect**:
- Fix `Map` and `Set` deep equality in some assertions  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **Claude Opus 5.5** in
[#&#8203;11402](https://redirect.github.com/vitest-dev/vitest/issues/11402)
[<samp>(b4e79)</samp>](b4e79fd7a)
- Check value in chai-style returned  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11500](https://redirect.github.com/vitest-dev/vitest/issues/11500)
[<samp>(1aca0)</samp>](1aca02a21)
- **mocker**:
- Fix native loader mocking edge cases  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11540](https://redirect.github.com/vitest-dev/vitest/issues/11540)
[<samp>(cd423)</samp>](cd423ba04)
- **pretty-format**:
- Support boxed symbols  -  by
[@&#8203;lehni](https://redirect.github.com/lehni) in
[#&#8203;11445](https://redirect.github.com/vitest-dev/vitest/issues/11445)
and
[#&#8203;11446](https://redirect.github.com/vitest-dev/vitest/issues/11446)
[<samp>(eba44)</samp>](eba44926e)
- **reporters**:
- Junit to strip ANSI sequences from all XML attributes  -  by
[@&#8203;meganemura](https://redirect.github.com/meganemura) and
**Claude Opus 5.5** in
[#&#8203;11407](https://redirect.github.com/vitest-dev/vitest/issues/11407)
[<samp>(e1a73)</samp>](e1a731470)
- **runner**:
- Don't cache failed fixture setups  -  by
[@&#8203;hamed-bavar](https://redirect.github.com/hamed-bavar) in
[#&#8203;11237](https://redirect.github.com/vitest-dev/vitest/issues/11237)
and
[#&#8203;11238](https://redirect.github.com/vitest-dev/vitest/issues/11238)
[<samp>(4ee66)</samp>](4ee66c021)
- **typecheck**:
- Use typescript pool in toTestSpecification  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11451](https://redirect.github.com/vitest-dev/vitest/issues/11451)
[<samp>(5f273)</samp>](5f2739e74)
- **ui**:
- Remove duplicate browser-open call opening two tabs  -  by
[@&#8203;feastyr69](https://redirect.github.com/feastyr69) in
[#&#8203;11356](https://redirect.github.com/vitest-dev/vitest/issues/11356)
and
[#&#8203;11358](https://redirect.github.com/vitest-dev/vitest/issues/11358)
[<samp>(d7e13)</samp>](d7e1315c1)
- Simplify explorer filtering + fix nested tests when search matches a
suite  -  by [@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa),
**Hiroshi Ogawa**, **Codex** and **Claude Opus 5.5** in
[#&#8203;11262](https://redirect.github.com/vitest-dev/vitest/issues/11262)
[<samp>(a627b)</samp>](a627be498)
- Store module graph once per environment  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **Claude Opus 5.5** in
[#&#8203;11421](https://redirect.github.com/vitest-dev/vitest/issues/11421)
[<samp>(af903)</samp>](af903c65e)
- Mount index.html handler on the ui base path  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **Claude Opus 5.5** in
[#&#8203;11517](https://redirect.github.com/vitest-dev/vitest/issues/11517)
[<samp>(1bc24)</samp>](1bc2495c7)
- Center split pane resize handle  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **Claude Opus 5.5** in
[#&#8203;11506](https://redirect.github.com/vitest-dev/vitest/issues/11506)
[<samp>(e658b)</samp>](e658b9b6a)
- **vm**:
- Bind `this` to `module.exports`  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11504](https://redirect.github.com/vitest-dev/vitest/issues/11504)
[<samp>(b618a)</samp>](b618afd5c)

#####    🏎 Performance

- Reuse fsModuleCache when filtering by source  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11428](https://redirect.github.com/vitest-dev/vitest/issues/11428)
[<samp>(7ebd9)</samp>](7ebd95fdd)
- **spy**: Reset only mocks that were called or reconfigured  -  by
[@&#8203;johan-gorter](https://redirect.github.com/johan-gorter) and
**Claude Opus 5.5** in
[#&#8203;11197](https://redirect.github.com/vitest-dev/vitest/issues/11197)
[<samp>(28ef9)</samp>](28ef93df2)

#####     [View changes on
GitHub](https://redirect.github.com/vitest-dev/vitest/compare/v5.0.3...v5.0.4)

###
[`v5.0.3`](https://redirect.github.com/vitest-dev/vitest/releases/tag/v5.0.3)

[Compare
Source](https://redirect.github.com/vitest-dev/vitest/compare/v5.0.2...v5.0.3)

#####    🐞 Bug Fixes

- Isolate `result.status` between `repeats` runs  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **Codex (GPT-6)** in
[#&#8203;11218](https://redirect.github.com/vitest-dev/vitest/issues/11218)
[<samp>(5dbeb)</samp>](5dbebe9e3)
- Don't print an interceptor warning in browser mode  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11377](https://redirect.github.com/vitest-dev/vitest/issues/11377)
[<samp>(15cc0)</samp>](15cc006aa)
- Don't retry when `test.fails` expectedly failed  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **Codex (GPT-6)** in
[#&#8203;11219](https://redirect.github.com/vitest-dev/vitest/issues/11219)
[<samp>(b2458)</samp>](b24585f08)
- Scope cache key generators to projects  -  by
[@&#8203;ecoyoung](https://redirect.github.com/ecoyoung) in
[#&#8203;11281](https://redirect.github.com/vitest-dev/vitest/issues/11281)
and
[#&#8203;11301](https://redirect.github.com/vitest-dev/vitest/issues/11301)
[<samp>(92ba7)</samp>](92ba7fc1d)
- **browser**:
- Delay server `listen` until tests start running  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11366](https://redirect.github.com/vitest-dev/vitest/issues/11366)
[<samp>(7d8ed)</samp>](7d8ed3e9b)
- Check mock path boundaries  -  by
[@&#8203;saryn17](https://redirect.github.com/saryn17), **Ryosei Sato**
and [@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11361](https://redirect.github.com/vitest-dev/vitest/issues/11361)
and
[#&#8203;11362](https://redirect.github.com/vitest-dev/vitest/issues/11362)
[<samp>(1c388)</samp>](1c3888bce)
- Keep config of browser-consumed environments  -  by
[@&#8203;kasperpeulen](https://redirect.github.com/kasperpeulen) and
**Claude** in
[#&#8203;11378](https://redirect.github.com/vitest-dev/vitest/issues/11378)
[<samp>(aafc0)</samp>](aafc0996f)
- Ignore page crash while cancelling  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11386](https://redirect.github.com/vitest-dev/vitest/issues/11386)
[<samp>(7c367)</samp>](7c36748fa)
- `toMatchScreenshot` uses wrong reference on retried tests  -  by
[@&#8203;macarie](https://redirect.github.com/macarie) in
[#&#8203;11393](https://redirect.github.com/vitest-dev/vitest/issues/11393)
[<samp>(c22ab)</samp>](c22aba992)
- **cache**:
- Revalidate imports of cached modules  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11381](https://redirect.github.com/vitest-dev/vitest/issues/11381)
[<samp>(38f98)</samp>](38f98855f)
- **deps**:
- Pin `why-is-node-running` to `3.2.1` to avoid users running into
`ERR_PNPM_TRUST_DOWNGRADE`  -  by
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;11403](https://redirect.github.com/vitest-dev/vitest/issues/11403)
[<samp>(f6c9a)</samp>](f6c9a4977)
- **expect**:
- Pass current equality testers to `expect.extend` asymmetric matchers
 -  by [@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa),
**Hiroshi Ogawa** and **Claude** in
[#&#8203;11401](https://redirect.github.com/vitest-dev/vitest/issues/11401)
[<samp>(3e794)</samp>](3e794a96b)
- **jsdom**:
- Support Blob on jsdom 30.1  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11379](https://redirect.github.com/vitest-dev/vitest/issues/11379)
[<samp>(6c49b)</samp>](6c49b7197)
- **pool**:
- Preserve unique pool ids when `groupOrder` is set  -  by
[@&#8203;mtorp](https://redirect.github.com/mtorp) in
[#&#8203;11392](https://redirect.github.com/vitest-dev/vitest/issues/11392)
[<samp>(50312)</samp>](50312ebb4)
- **ui**:
- Split-pane handle overlapping iframe  -  by
[@&#8203;macarie](https://redirect.github.com/macarie) in
[#&#8203;11221](https://redirect.github.com/vitest-dev/vitest/issues/11221)
[<samp>(f91db)</samp>](f91db0dfd)
- **vitest**:
- Remove root temp dir on close  -  by
[@&#8203;abhinav-phi](https://redirect.github.com/abhinav-phi) in
[#&#8203;11248](https://redirect.github.com/vitest-dev/vitest/issues/11248)
[<samp>(7c711)</samp>](7c7119cf7)
- **vm**:
- Do not optimize deps from index.html  -  by
[@&#8203;ezefernandezyf](https://redirect.github.com/ezefernandezyf),
**Hiroshi Ogawa** and **Codex (GPT-6)** in
[#&#8203;11329](https://redirect.github.com/vitest-dev/vitest/issues/11329)
and
[#&#8203;11360](https://redirect.github.com/vitest-dev/vitest/issues/11360)
[<samp>(caf28)</samp>](caf2887de)
- Don't reuse scripts across vite environments  -  by
[@&#8203;MO2k4](https://redirect.github.com/MO2k4), **Martin Oehlert**
and **Claude** in
[#&#8203;11395](https://redirect.github.com/vitest-dev/vitest/issues/11395)
[<samp>(346d3)</samp>](346d3896b)

#####     [View changes on
GitHub](https://redirect.github.com/vitest-dev/vitest/compare/v5.0.2...v5.0.3)

###
[`v5.0.2`](https://redirect.github.com/vitest-dev/vitest/releases/tag/v5.0.2)

[Compare
Source](https://redirect.github.com/vitest-dev/vitest/compare/v5.0.1...v5.0.2)

#####    🐞 Bug Fixes

- Bind `process` in case global is overwritten  -  by
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;11343](https://redirect.github.com/vitest-dev/vitest/issues/11343)
[<samp>(0b792)</samp>](0b79231ad)
- **detect-async-leaks**:
- Ignore `process.stdio` handles  -  by
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;11333](https://redirect.github.com/vitest-dev/vitest/issues/11333)
[<samp>(0fd6b)</samp>](0fd6b9790)
- **expect**:
- Fix `toMatchObject` with asymmetric matchers  -  by
[@&#8203;ShreeBohara](https://redirect.github.com/ShreeBohara), **Claude
Opus 5**, [@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa),
**Hiroshi Ogawa** and **Codex (GPT-5)** in
[#&#8203;11100](https://redirect.github.com/vitest-dev/vitest/issues/11100)
[<samp>(42523)</samp>](42523289e)
- **jsdom**:
- Fix `Request` with `Blob` body on jsdom 28+  -  by
[@&#8203;harshit-d3v](https://redirect.github.com/harshit-d3v) in
[#&#8203;11295](https://redirect.github.com/vitest-dev/vitest/issues/11295)
[<samp>(d1c3e)</samp>](d1c3ecc93)
- **reporter**:
- `agent` to respect `--silent`  -  by
[@&#8203;Raj4478](https://redirect.github.com/Raj4478) and
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;11271](https://redirect.github.com/vitest-dev/vitest/issues/11271)
[<samp>(5b95e)</samp>](5b95efb6d)
- **reporters**:
- Handle concurrent `createReport` calls  -  by
[@&#8203;7rulnik](https://redirect.github.com/7rulnik) in
[#&#8203;11278](https://redirect.github.com/vitest-dev/vitest/issues/11278)
[<samp>(e8e55)</samp>](e8e556ff7)
- `hanging-process` to use ESM entrypoint  -  by
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;11316](https://redirect.github.com/vitest-dev/vitest/issues/11316)
[<samp>(4e91e)</samp>](4e91e5668)
- **spy**:
- Fix stack overflow when spying `Set.prototype.add`  -  by
[@&#8203;fengmk2](https://redirect.github.com/fengmk2) in
[#&#8203;11299](https://redirect.github.com/vitest-dev/vitest/issues/11299)
[<samp>(a0a93)</samp>](a0a939653)
- **ui**:
- Persist authentication cookie beyond current browser session  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **OpenCode (gpt-5.6-sol)** in
[#&#8203;11066](https://redirect.github.com/vitest-dev/vitest/issues/11066)
[<samp>(f8819)</samp>](f88195e16)

#####     [View changes on
GitHub](https://redirect.github.com/vitest-dev/vitest/compare/v5.0.1...v5.0.2)

###
[`v5.0.1`](https://redirect.github.com/vitest-dev/vitest/releases/tag/v5.0.1)

[Compare
Source](https://redirect.github.com/vitest-dev/vitest/compare/v5.0.0...v5.0.1)

#####    🚀 Features

- **ui**:
- Move trace attempts selector to viewer header  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **Codex** in
[#&#8203;11189](https://redirect.github.com/vitest-dev/vitest/issues/11189)
[<samp>(5dc4b)</samp>](5dc4b5c92)
- Add focused trace view layout mode  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa**, **OpenCode (gpt-5.6-sol)** and **Codex** in
[#&#8203;11190](https://redirect.github.com/vitest-dev/vitest/issues/11190)
[<samp>(376dc)</samp>](376dc3bc1)

#####    🐞 Bug Fixes

- Exit 1 when vitest list fails collection  -  by
[@&#8203;hamed-bavar](https://redirect.github.com/hamed-bavar) in
[#&#8203;11145](https://redirect.github.com/vitest-dev/vitest/issues/11145)
and
[#&#8203;11146](https://redirect.github.com/vitest-dev/vitest/issues/11146)
[<samp>(6108b)</samp>](6108b8197)
- Keep parse error details in static collection  -  by
[@&#8203;hamed-bavar](https://redirect.github.com/hamed-bavar) in
[#&#8203;11150](https://redirect.github.com/vitest-dev/vitest/issues/11150)
and
[#&#8203;11151](https://redirect.github.com/vitest-dev/vitest/issues/11151)
[<samp>(7c818)</samp>](7c818153a)
- Avoid recursive prototype in automocking  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11195](https://redirect.github.com/vitest-dev/vitest/issues/11195)
[<samp>(99fc5)</samp>](99fc52591)
- Prevent false Vitest import resolution  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11196](https://redirect.github.com/vitest-dev/vitest/issues/11196)
[<samp>(b426c)</samp>](b426c1976)
- Keep metadata file when clearing the cache  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11199](https://redirect.github.com/vitest-dev/vitest/issues/11199)
[<samp>(73614)</samp>](73614654a)
- Correct typos in error message and comments  -  by
[@&#8203;shinji00222](https://redirect.github.com/shinji00222) and
**Shinji** in
[#&#8203;11187](https://redirect.github.com/vitest-dev/vitest/issues/11187)
[<samp>(115c3)</samp>](115c3f6d2)
- Resolve ResolvedConfig exactOptionalPropertyTypes errors  -  by
[@&#8203;LukeAbby](https://redirect.github.com/LukeAbby) in
[#&#8203;11175](https://redirect.github.com/vitest-dev/vitest/issues/11175)
[<samp>(498fb)</samp>](498fbe922)
- Share the server on self-referencing `extends`  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11034](https://redirect.github.com/vitest-dev/vitest/issues/11034)
[<samp>(23dda)</samp>](23dda738c)
- Warn when deprecated `deps.optimizer.web` is used  -  by
[@&#8203;im10furry](https://redirect.github.com/im10furry) in
[#&#8203;11214](https://redirect.github.com/vitest-dev/vitest/issues/11214)
[<samp>(2ce29)</samp>](2ce29d5fa)
- **browser**:
- Avoid double quotes in `config.define`  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11198](https://redirect.github.com/vitest-dev/vitest/issues/11198)
[<samp>(972e2)</samp>](972e24bab)
- **doctor**:
- Measure vm pools for custom environments  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11212](https://redirect.github.com/vitest-dev/vitest/issues/11212)
[<samp>(91ab1)</samp>](91ab1588c)
- **expect**:
- Correct return value in `toMatchAriaSnapshot`  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11208](https://redirect.github.com/vitest-dev/vitest/issues/11208)
[<samp>(c119b)</samp>](c119be016)
- **fakeTimers**:
- Force `queueMicrotask` and `nextTick` in `toNotFake`  -  by
[@&#8203;kingmakeruix](https://redirect.github.com/kingmakeruix),
**kingmakeruix**, **Hiroshi Ogawa**, **Codex** and
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa) in
[#&#8203;11261](https://redirect.github.com/vitest-dev/vitest/issues/11261)
[<samp>(a47d7)</samp>](a47d7908f)
- **snapshot**:
- Report obsolete keys next to skipped tests  -  by
[@&#8203;hamed-bavar](https://redirect.github.com/hamed-bavar),
**Hiroshi Ogawa** and **OpenCode (gpt-5.6-sol)** in
[#&#8203;11157](https://redirect.github.com/vitest-dev/vitest/issues/11157)
and
[#&#8203;11158](https://redirect.github.com/vitest-dev/vitest/issues/11158)
[<samp>(17e2b)</samp>](17e2b22dd)
- **types**:
- Make public declarations self-contained  -  by
[@&#8203;ZoeySigel](https://redirect.github.com/ZoeySigel) in
[#&#8203;11141](https://redirect.github.com/vitest-dev/vitest/issues/11141)
[<samp>(45546)</samp>](455466c16)
- **ui**:
- Fix collapse/expand suite with file name search  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **Codex** in
[#&#8203;11260](https://redirect.github.com/vitest-dev/vitest/issues/11260)
[<samp>(0a712)</samp>](0a7122daa)
- Fix explorer file summary count  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa**, **OpenCode (gpt-5.6-sol)** and **Codex** in
[#&#8203;11125](https://redirect.github.com/vitest-dev/vitest/issues/11125)
[<samp>(05982)</samp>](05982297d)
- **utils**:
- Fix `deepMerge` to handle prototype  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **Codex** in
[#&#8203;11215](https://redirect.github.com/vitest-dev/vitest/issues/11215)
[<samp>(4944c)</samp>](4944cf498)

#####     [View changes on
GitHub](https://redirect.github.com/vitest-dev/vitest/compare/v5.0.0...v5.0.1)

###
[`v5.0.0`](https://redirect.github.com/vitest-dev/vitest/releases/tag/v5.0.0)

[Compare
Source](https://redirect.github.com/vitest-dev/vitest/compare/v4.1.12...v5.0.0)

Vitest 5 is officially out! This release focuses on performance and
brings a lot of new features while fixing long-standing bugs. See our
[blog post](https://vitest.dev/blog/vitest-5.html) for the official
announcement.

#####    🚨 Breaking Changes

- Replace `loupe.inspect` with pretty-format  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Claude Opus
5 (1M context)** and **OpenAI Codex** in
[#&#8203;9609](https://redirect.github.com/vitest-dev/vitest/issues/9609)
[<samp>(3f802)</samp>](3f802da4b)
- Remove quotes from string values in `test.for/each` title `$` variable
(take 2)  -  by [@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa)
in
[#&#8203;10170](https://redirect.github.com/vitest-dev/vitest/issues/10170)
[<samp>(04d37)</samp>](04d37e9d7)
- Default `attachmentsDir` from `.vitest-attachements/` to
`.vitest/attachments/`  -  by
[@&#8203;MdSadiqMd](https://redirect.github.com/MdSadiqMd) in
[#&#8203;10186](https://redirect.github.com/vitest-dev/vitest/issues/10186)
[<samp>(1ba73)</samp>](1ba7338c3)
- Remove `sequential` test/suite options in favor of `concurrent`  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa) and **OpenAI
Codex** in
[#&#8203;10198](https://redirect.github.com/vitest-dev/vitest/issues/10198)
[<samp>(9229f)</samp>](9229f2edc)
- Represent locator as an object instead of a string  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10212](https://redirect.github.com/vitest-dev/vitest/issues/10212)
[<samp>(80f07)</samp>](80f07edf6)
- Inline `expect` package  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10221](https://redirect.github.com/vitest-dev/vitest/issues/10221)
[<samp>(ad162)</samp>](ad16223e7)
- Remove deprecated entry points  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10222](https://redirect.github.com/vitest-dev/vitest/issues/10222)
[<samp>(994c6)</samp>](994c6ddb9)
- Require Node.js 22 and Vite 6.4  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10178](https://redirect.github.com/vitest-dev/vitest/issues/10178)
[<samp>(38762)</samp>](3876283e8)
- Fail `expect.poll` when function didn't resolve in time  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa) and **OpenAI
Codex** in
[#&#8203;10233](https://redirect.github.com/vitest-dev/vitest/issues/10233)
[<samp>(4df04)</samp>](4df048c11)
- Throw an error if hoistable methods are outside the top level scope
 -  by [@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10460](https://redirect.github.com/vitest-dev/vitest/issues/10460)
[<samp>(d0b4f)</samp>](d0b4fddcb)
- `toHaveTextContent` is strict, add `toMatchTextContent` as alternative
 -  by [@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10473](https://redirect.github.com/vitest-dev/vitest/issues/10473)
[<samp>(18f30)</samp>](18f303079)
- Don't lookup config file from ancestor directories  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **OpenAI
Codex** and **Hiroshi Ogawa** in
[#&#8203;10428](https://redirect.github.com/vitest-dev/vitest/issues/10428)
[<samp>(945d9)</samp>](945d9090e)
- Inline `@vitest/runner` package, do not publish it anymore  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10511](https://redirect.github.com/vitest-dev/vitest/issues/10511)
[<samp>(6d6e4)</samp>](6d6e46b1e)
- Allow mutating happy-dom/jsdom window object  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **OpenAI Codex** in
[#&#8203;10373](https://redirect.github.com/vitest-dev/vitest/issues/10373)
[<samp>(206e8)</samp>](206e8cff8)
- Expose `concurrencyId`/`workerId` on TestModule's diagnostics, make id
1-based  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10516](https://redirect.github.com/vitest-dev/vitest/issues/10516)
[<samp>(bdd98)</samp>](bdd985433)
- Add `screenshotDirectory` config to `browser.expect.toMatchScreenshot`
 -  by [@&#8203;macarie](https://redirect.github.com/macarie) in
[#&#8203;10592](https://redirect.github.com/vitest-dev/vitest/issues/10592)
[<samp>(a60de)</samp>](a60ded0fb)
- Update `@sinonjs/fake-timers` and support mocking `Temporal`  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **OpenCode (gpt-5.6-sol)** in
[#&#8203;10654](https://redirect.github.com/vitest-dev/vitest/issues/10654)
[<samp>(f8b15)</samp>](f8b1532fe)
- Remove webdriverio package  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10675](https://redirect.github.com/vitest-dev/vitest/issues/10675)
[<samp>(5fed6)</samp>](5fed68f72)
- Clear mocks by default before each test  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10613](https://redirect.github.com/vitest-dev/vitest/issues/10613)
[<samp>(0f646)</samp>](0f6463bf2)
- Don't emit localStorage warnings on Node 26, fail gracefully when
worker fails to start  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10293](https://redirect.github.com/vitest-dev/vitest/issues/10293)
[<samp>(334ed)</samp>](334edef92)
- Separate config resolution from the server creation  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10554](https://redirect.github.com/vitest-dev/vitest/issues/10554)
[<samp>(1c0ec)</samp>](1c0ec3444)
- Inline projects extend the root config by default  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10750](https://redirect.github.com/vitest-dev/vitest/issues/10750)
[<samp>(fec00)</samp>](fec001ad3)
- Enable mocking Temporal without fake timers  -  by
[@&#8203;fabon-f](https://redirect.github.com/fabon-f), **Hiroshi
Ogawa** and **OpenCode (gpt-5.6-sol)** in
[#&#8203;10757](https://redirect.github.com/vitest-dev/vitest/issues/10757)
[<samp>(ac2d4)</samp>](ac2d46b42)
- Support nested projects  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10846](https://redirect.github.com/vitest-dev/vitest/issues/10846)
[<samp>(ec367)</samp>](ec367cf2a)
- Use `>` as separator in `-t`, calculate `only` once  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10686](https://redirect.github.com/vitest-dev/vitest/issues/10686)
[<samp>(a0b20)</samp>](a0b20bc86)
- Fail the test when an asynchronous assertion is not awaited  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10868](https://redirect.github.com/vitest-dev/vitest/issues/10868)
[<samp>(86d4a)</samp>](86d4a9da9)
- Share the Vite server between inline projects  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10848](https://redirect.github.com/vitest-dev/vitest/issues/10848)
[<samp>(d87c9)</samp>](d87c96ee4)
- Parse files statically in vitest list by default  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11088](https://redirect.github.com/vitest-dev/vitest/issues/11088)
[<samp>(51e94)</samp>](51e949416)
- **benchmark**:
- Rewrite the public API  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10113](https://redirect.github.com/vitest-dev/vitest/issues/10113)
[<samp>(19f6e)</samp>](19f6e8947)
- **browser**:
- Iframe scale  -  by
[@&#8203;macarie](https://redirect.github.com/macarie) in
[#&#8203;9745](https://redirect.github.com/vitest-dev/vitest/issues/9745)
[<samp>(b6398)</samp>](b639852cc)
- Enable `locators.exact` by default  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10430](https://redirect.github.com/vitest-dev/vitest/issues/10430)
[<samp>(e2032)</samp>](e203202f9)
- Require `sessionId` for orchestrator html request  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **OpenAI Codex** in
[#&#8203;10522](https://redirect.github.com/vitest-dev/vitest/issues/10522)
[<samp>(79b7d)</samp>](79b7d8fcc)
- Save failure screenshots in `attachmentsDir`  -  by
[@&#8203;macarie](https://redirect.github.com/macarie) in
[#&#8203;10917](https://redirect.github.com/vitest-dev/vitest/issues/10917)
[<samp>(3b5bb)</samp>](3b5bbd8b4)
- **coverage**:
- `include/exclude` globs too eager  -  by
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;9818](https://redirect.github.com/vitest-dev/vitest/issues/9818)
[<samp>(edacb)</samp>](edacb0fd4)
- Allow `thresholds.perFile` to accept an object  -  by
[@&#8203;vladlenskiy](https://redirect.github.com/vladlenskiy) and
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;10190](https://redirect.github.com/vitest-dev/vitest/issues/10190)
[<samp>(13b78)</samp>](13b78d98b)
- **expect**:
- Fix `toThrow("")` behavior by reverting
[#&#8203;6710](https://redirect.github.com/vitest-dev/vitest/issues/6710)
 -  by [@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa) in
[#&#8203;9643](https://redirect.github.com/vitest-dev/vitest/issues/9643)
and
[#&#8203;6710](https://redirect.github.com/vitest-dev/vitest/issues/6710)
[<samp>(6c3e4)</samp>](6c3e4bdbf)
- **mocker**:
- Deserialize automock as automock  -  by
[@&#8203;nami8824](https://redirect.github.com/nami8824) in
[#&#8203;10192](https://redirect.github.com/vitest-dev/vitest/issues/10192)
[<samp>(2f892)</samp>](2f892712d)
- **reporters**:
- `blob` reporter and `--merge-reports` default to `.vitest/blob/`  - 
by [@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;10232](https://redirect.github.com/vitest-dev/vitest/issues/10232)
[<samp>(d22b0)</samp>](d22b029ae)
- Write json and junit reporter output files to `.vitest` by default  - 
by [@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa**, **OpenCode (gpt-5.6-sol)** and
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;10621](https://redirect.github.com/vitest-dev/vitest/issues/10621)
[<samp>(58577)</samp>](58577290a)
- **spy**:
- Preserve class mock prototype methods on instances  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10910](https://redirect.github.com/vitest-dev/vitest/issues/10910)
[<samp>(6093d)</samp>](6093d764a)
- **types**:
- Add better promise support in expects and matchers  -  by
[@&#8203;samchungy](https://redirect.github.com/samchungy) and
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;8266](https://redirect.github.com/vitest-dev/vitest/issues/8266)
[<samp>(41f55)</samp>](41f551bce)
- **ui**:
- Harden UI API access  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi
Ogawa** and **OpenAI Codex** in
[#&#8203;10583](https://redirect.github.com/vitest-dev/vitest/issues/10583)
[<samp>(4c26d)</samp>](4c26d7675)
- Change html reporter default output to `.vitest`  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa) and **Hiroshi
Ogawa** in
[#&#8203;10620](https://redirect.github.com/vitest-dev/vitest/issues/10620)
[<samp>(29c36)</samp>](29c364d50)

#####    🚀 Features

- Add `createReport` and `.vitest` report directory convention  -  by
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;9993](https://redirect.github.com/vitest-dev/vitest/issues/9993)
[<samp>(72a6d)</samp>](72a6dc257)
- Expose default reporters through `configDefaults.reporters`  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa) and **Claude
Opus 5 (1M context)** in
[#&#8203;10219](https://redirect.github.com/vitest-dev/vitest/issues/10219)
[<samp>(083f6)</samp>](083f6bdd6)
- Support merge reports for non-sharded multi-environment runs (take 2)
 -  by [@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa),
**Claude Opus 5 (1M context)**, **OpenAI Codex** and
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;10031](https://redirect.github.com/vitest-dev/vitest/issues/10031)
[<samp>(e60b2)</samp>](e60b2f49e)
- Add `logger.formatError`  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa) and **OpenAI
Codex** in
[#&#8203;10268](https://redirect.github.com/vitest-dev/vitest/issues/10268)
[<samp>(2c5f3)</samp>](2c5f3ee2f)
- Support typescript build mode  -  by
[@&#8203;lishaduck](https://redirect.github.com/lishaduck) in
[#&#8203;9870](https://redirect.github.com/vitest-dev/vitest/issues/9870)
[<samp>(106da)</samp>](106da5896)
- Add toggable `injectCjsGlobals` option  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10709](https://redirect.github.com/vitest-dev/vitest/issues/10709)
[<samp>(82671)</samp>](826714ece)
- Promote fsModuleCache to a top-level option  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10734](https://redirect.github.com/vitest-dev/vitest/issues/10734)
[<samp>(78ed7)</samp>](78ed73afd)
- Support non-ascii characters in `for/each` title placeholders  -  by
[@&#8203;k-yle](https://redirect.github.com/k-yle) in
[#&#8203;10773](https://redirect.github.com/vitest-dev/vitest/issues/10773)
[<samp>(15e0a)</samp>](15e0af953)
- Report the duration breakdown as percentages  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) and
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in
[#&#8203;10820](https://redirect.github.com/vitest-dev/vitest/issues/10820)
[<samp>(c67d2)</samp>](c67d296f4)
- Promote parseSpecifications out of experimental  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11087](https://redirect.github.com/vitest-dev/vitest/issues/11087)
[<samp>(b628c)</samp>](b628cb680)
- Promote clearCache out of experimental  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;11086](https://redirect.github.com/vitest-dev/vitest/issues/11086)
[<samp>(d4fe1)</samp>](d4fe1986f)
- **api**:
- Expose logs recorded during the test  -  by
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10277](https://redirect.github.com/vitest-dev/vitest/issues/10277)
[<samp>(cba20)</samp>](cba2036a1)
- **benchmark**:
- Add pluggable benchmark provider API  -  by
[@&#8203;GuillaumeLagrange](https://redirect.github.com/GuillaumeLagrange)
and [@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10799](https://redirect.github.com/vitest-dev/vitest/issues/10799)
[<samp>(86c70)</samp>](86c700ed8)
- **browser**:
- Export aria tree utils  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa) in
[#&#8203;10171](https://redirect.github.com/vitest-dev/vitest/issues/10171)
[<samp>(c3423)</samp>](c3423014c)
- Support dom snapshot trace view  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa), **Claude Opus
5 (1M context)** and **OpenAI Codex** in
[#&#8203;10102](https://redirect.github.com/vitest-dev/vitest/issues/10102)
[<samp>(7eddd)</samp>](7eddd273d)
- Provide project reference in `ToMatchScreenshotResolvePath`  -  by
[@&#8203;macarie](https://redirect.github.com/macarie) and
[@&#8203;sheremet-va](https://redirect.github.com/sheremet-va) in
[#&#8203;10138](https://redirect.github.com/vitest-dev/vitest/issues/10138)
[<samp>(16654)</samp>](166544e39)
- Show aria tree on locator element error  -  by
[@&#8203;hi-ogawa](https://redirect.github.com/hi-ogawa) and **OpenAI
Codex** in
[#&#8203;10257](https://redirect.github.com/vitest-dev/vitest/issues/10257)
[<samp>(04f04)</samp>](04f04cdd1)
- Support custom `kind` in `page.mark`  -  by
[@&#8203;AriPerkkio](https://redirect.github.com/AriPerkkio) in

> [!IMPORTANT]
> ✂ PR body was truncated to here.


</details>

---

### Configuration

📅 **Schedule**: (UTC)

- Branch creation
  - "before 6am every weekday"
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

👻 **Immortal**: This PR will be recreated if closed unmerged. Get
[config
help](https://redirect.github.com/renovatebot/renovate/discussions) if
that's undesired.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend Renovate
CLI](https://redirect.github.com/renovatebot/renovate).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4xOTguMiIsInVwZGF0ZWRJblZlciI6IjQ0LjIwNS4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJhdXRvbWF0aW9uIl19-->

---------

Co-authored-by: wardpeet <ward@coding-tech.com>
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-11-03 20:18:21 +01:00
Dane - AI (Mastra)
47a6cd82e5
chore(deps): update ai sdk (#29242)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-11-03 16:12:54 +01:00
dane-ai-mastra[bot]
30046969ad
chore: version packages (alpha) (#29146)
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases
## @mastra/client-js@1.59.0-alpha.1

### Minor Changes

- Added the `aggregateTimeToFirstMeasures` observability capability.
Trace and span aggregate requests for `timeToFirstToken.*` or
`timeToFirstChunk.*` measures now return `501` against stores that can't
compute them, and field discovery omits those measures for such stores.
([#29055](https://github.com/mastra-ai/mastra/pull/29055))

  ```typescript
const { capabilities } = await
mastraClient.getObservabilityCapabilities();

if (capabilities.traceAggregate &&
capabilities.aggregateTimeToFirstMeasures) {
    const { rows } = await mastraClient.aggregateTraces({
timeRange: { from: '2026-08-01T00:00:00.000Z', to:
'2026-08-02T00:00:00.000Z' },
      groupBy: ['entityName'],
      measures: ['timeToFirstChunk.p95', 'timeToFirstChunk.count'],
    });
  }
  ```

### Patch Changes

- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
## @mastra/factory@0.27.0-alpha.1

### Minor Changes

- The same Jira, Linear or incident.io item can now be filed in more
than one Factory project. Before, the first project to file an item held
it for the whole org, and other projects that ingested it got no card.
Each project now keeps its own card. Within a project the card is still
reused when the item is seen again, after the card finishes and when the
issue reopens. A Linear card is also found after the issue's identifier
changes, for example when the issue moves to another team.
([#29059](https://github.com/mastra-ai/mastra/pull/29059))

When intake routing sends an item to a different project, the new
project files its own card and the previous project's card stays where
it is.

  **Breaking changes**

- `claimKey` is removed from the `upsertLinkedWorkItem` decision type.
Rules that still return it, and decisions saved before this release,
keep working: the field is accepted and ignored.
- `WorkItemClaimConflictError`, `getByClaimKey`, `claimWorkItem`,
`releaseProjectClaims` and `WorkItemRow.claimKey` are removed.

  Drop `claimKey` from linked-item decisions your rules return:

  ```ts
  // Before
  return {
    type: 'upsertLinkedWorkItem',
    source: 'jira-issue',
    sourceKey: issue.id,
    claimKey: `jira:issue:${issue.id}` /* … */,
  };

  // After
return { type: 'upsertLinkedWorkItem', source: 'jira-issue', sourceKey:
issue.id /* … */ };
  ```

Remove calls to `getByClaimKey`, `claimWorkItem` and
`releaseProjectClaims`. To find a project's card for an item, use
`getByProjectSource`.

The `claim_key` column stays in the database. Values already stored
there are left in place and no longer affect anything.

### Patch Changes

- Fixed shell commands that exit with a failure code reading as
successful in the Factory transcript. The command now reads as failed
while the run streams, after a dropped stream reconnects, and after a
reload. Stopping a process with `kill_process` still reads as a success.
([#28963](https://github.com/mastra-ai/mastra/pull/28963))

A system reminder that loads project instructions (`AGENTS.md`,
`CLAUDE.md` or `CONTEXT.md`) now reads "Project instructions" and names
the file by its path in the working directory, such as
`packages/core/AGENTS.md`, instead of a bare "System reminder".

- A board tool rule that moves a work item back into its current stage
with `reenter: true` now runs that stage's entry rules, as documented.
Previously the entry rules were skipped, so no new agent run started.
([#29104](https://github.com/mastra-ai/mastra/pull/29104))

- Updated dependencies
[[`5735a50`](5735a50c6c),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/platform-workspace@1.11.1-alpha.0
  - @mastra/core@1.82.0-alpha.1
  - @mastra/code-sdk@1.18.0-alpha.1
## @mastra/core@1.82.0-alpha.1

### Minor Changes

- Added time to first token and time to first chunk measures to trace
and span aggregates, so dashboards can chart streaming latency next to
duration. ([#29055](https://github.com/mastra-ai/mastra/pull/29055))

The new measures are `timeToFirstToken.avg`, `.min`, `.max`, `.p50`,
`.p90`, `.p95`, `.p99`, and `.count`, plus the same set for
`timeToFirstChunk`. Traces and spans without a recorded value are
skipped instead of counted as 0, and `.count` reports how many had one.
Stores opt in through the `aggregate-time-to-first-measures` feature.

### Patch Changes

- Fixed durable agents (`durable: true`) ending the `model_generation`
trace span without `responseId` and `responseModel`. Durable runs now
record the provider's response id and model on the span, the same as
non-durable agents, so a trace can be linked back to the provider's own
response log. Fixes #29053.
([#29180](https://github.com/mastra-ai/mastra/pull/29180))

- Fixed metadata filters in the in-memory store returning records whose
nested metadata had different keys from the filter when one side
contained an `undefined` value.
([#29101](https://github.com/mastra-ai/mastra/pull/29101))

- Fixed directly resumed nested workflows in the evented engine so they
continue suspended parent runs.
([#25979](https://github.com/mastra-ai/mastra/pull/25979))
## @mastra/playground-ui@68.0.0-alpha.1

### Minor Changes

- Toolbar and filter bar controls now share one look: the same paddings,
icon gaps, chevron and spacing between controls.
([#28818](https://github.com/mastra-ai/mastra/pull/28818))

**Time range.** A filter bar's time range is now the
`DateTimeRangePicker` itself, so it looks and behaves the same on every
page: a calendar icon, the range and a chevron, with no separate "Time"
label. Pass `defaultPreset` to show a × that resets a custom range; it
replaces the chevron while a custom range is applied.

Added `FilterBar.TimeRange` with `FILTER_BAR_TIME_RANGE_FIELD`,
`FILTER_BAR_TIME_RANGE_ITEM` and `FILTER_BAR_TIME_RANGE_FIELD_ID`. They
replace `TraceTimeRangeChip` and the `TRACE_TIME_RANGE_*` names from
`domains/traces/components/trace-time-range-chip`, which are removed.

  ```tsx
  import {
    FilterBar,
    FILTER_BAR_TIME_RANGE_FIELD,
    FILTER_BAR_TIME_RANGE_FIELD_ID,
    FILTER_BAR_TIME_RANGE_ITEM,
  } from '@mastra/playground-ui/components/FilterBar';

  <FilterBar
    fields={[FILTER_BAR_TIME_RANGE_FIELD, ...fields]}
    operators={operators}
    value={[FILTER_BAR_TIME_RANGE_ITEM, ...items]}
// The time range item is synthetic: keep it out of the stored filters.
onValueChange={next => setItems(next.filter(item => item.fieldId !==
FILTER_BAR_TIME_RANGE_FIELD_ID))}
  >
    <FilterBar.Chips
      renderChip={item =>
        item.fieldId === FILTER_BAR_TIME_RANGE_FIELD_ID ? (
          <FilterBar.TimeRange
            preset={preset}
// A preset, including the × reset, drops the applied custom dates.
            onPresetChange={next => {
              setPreset(next);
              if (next !== 'custom') setRange({});
            }}
            dateFrom={range.from}
            dateTo={range.to}
            onDateRangeChange={(from, to) => setRange({ from, to })}
            defaultPreset="last-7d"
          />
        ) : (
          <FilterBar.Chip item={item} />
        )
      }
    />
    <FilterBar.Input />
  </FilterBar>;
  ```

`DateTimeRangePicker` gains `size`, `defaultPreset` and `ref` (the
trigger button).

  **Migrating removed APIs.**

- `TraceTimeRangeChip` → `FilterBar.TimeRange` (same props), and
`TRACE_TIME_RANGE_FIELD` / `_ITEM` / `_FIELD_ID` →
`FILTER_BAR_TIME_RANGE_FIELD` / `_ITEM` / `_FIELD_ID`, imported from
`@mastra/playground-ui/components/FilterBar`.

    ```tsx
    // Before
import { TraceTimeRangeChip, TRACE_TIME_RANGE_FIELD_ID } from
'@mastra/playground-ui/domains/traces/components/trace-time-range-chip';
item.fieldId === TRACE_TIME_RANGE_FIELD_ID ? <TraceTimeRangeChip
preset={preset} onPresetChange={setPreset} /> : …

    // After
import { FilterBar, FILTER_BAR_TIME_RANGE_FIELD_ID } from
'@mastra/playground-ui/components/FilterBar';
item.fieldId === FILTER_BAR_TIME_RANGE_FIELD_ID ? <FilterBar.TimeRange
preset={preset} onPresetChange={setPreset} /> : …
    ```

- `DateTimeRangePicker`'s `renderTrigger` → remove it: the trigger is
always the shared calendar button. Pass `size` for another rung and
`ref` to reach the trigger.
- `comboboxStyles.chevron` on a hand-built trigger → `<ButtonChevron
size={size} />` from `@mastra/playground-ui/components/Button`.

  **Other controls.**

- Added `ClearFiltersButton`, a round reset icon button with a tooltip,
used for every toolbar's "clear filters".
- Added `TriggerChevron` and `ButtonChevron`, the one chevron every
dropdown trigger (`Select`, `Combobox`, the date picker) shows.
  - `ComboboxTrigger` uses that chevron instead of the up-down one.
- Icons in every interactive control (buttons, fields, search inputs,
filter chips, menu and command rows) follow one rule: muted at rest,
brighter on hover. Two new theme tokens, `--control-icon-rest` and
`--control-icon-hover`, set those colours; menus retune them, and
icon-only controls always brighten. Search fields and combobox fields
now brighten their icons on hover like buttons do.
- Icons take their size from the control they sit in, as in a Button:
filter chips and search inputs no longer size their own icons.
- `ListSearch` defaults to the `sm` toolbar size, and `SearchInput`'s
icon sits at the same inset as a button's.
  - `ActionRow` and `FilterBar` space their controls 6px apart.

### Patch Changes

- Updated dependencies
[[`489d829`](489d8295c8),
[`0900c20`](0900c20592),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`e1e64f0`](e1e64f072c),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/memory@1.41.0-alpha.1
  - @mastra/client-js@1.59.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
  - @mastra/ai-sdk@1.11.3
  - @mastra/react@1.14.1-alpha.1
## @mastra/server@1.82.0-alpha.1

### Minor Changes

- Added the `aggregateTimeToFirstMeasures` observability capability.
Trace and span aggregate requests for `timeToFirstToken.*` or
`timeToFirstChunk.*` measures now return `501` against stores that can't
compute them, and field discovery omits those measures for such stores.
([#29055](https://github.com/mastra-ai/mastra/pull/29055))

  ```typescript
const { capabilities } = await
mastraClient.getObservabilityCapabilities();

if (capabilities.traceAggregate &&
capabilities.aggregateTimeToFirstMeasures) {
    const { rows } = await mastraClient.aggregateTraces({
timeRange: { from: '2026-08-01T00:00:00.000Z', to:
'2026-08-02T00:00:00.000Z' },
      groupBy: ['entityName'],
      measures: ['timeToFirstChunk.p95', 'timeToFirstChunk.count'],
    });
  }
  ```

### Patch Changes

- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
## @mastra/clickhouse@1.29.0-alpha.0

### Minor Changes

- Added support for `timeToFirstToken.*` and `timeToFirstChunk.*`
measures in trace and span aggregates. Rows without a recorded value are
skipped, never counted as 0.
([#29055](https://github.com/mastra-ai/mastra/pull/29055))

  ```typescript
  import { planSpanAggregate } from '@mastra/core/storage';

  const observability = await storage.getStore('observability');

if
(observability?.getFeatures().includes('aggregate-time-to-first-measures'))
{
    const { rows } = await observability.aggregateSpans(
      planSpanAggregate({
timeRange: { from: '2026-08-01T00:00:00.000Z', to:
'2026-08-02T00:00:00.000Z' },
where: { op: 'eq', left: { path: 'spanType' }, right: { literal:
'model_inference' } },
        measures: ['timeToFirstToken.p50', 'timeToFirstToken.p95'],
      }),
    );
  }
  ```

### Patch Changes

- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
## @mastra/duckdb@1.18.0-alpha.0

### Minor Changes

- Added support for `timeToFirstToken.*` and `timeToFirstChunk.*`
measures in trace and span aggregates. Rows without a recorded value are
skipped, never counted as 0.
([#29055](https://github.com/mastra-ai/mastra/pull/29055))

  ```typescript
  import { planSpanAggregate } from '@mastra/core/storage';

  const observability = await storage.getStore('observability');

if
(observability?.getFeatures().includes('aggregate-time-to-first-measures'))
{
    const { rows } = await observability.aggregateSpans(
      planSpanAggregate({
timeRange: { from: '2026-08-01T00:00:00.000Z', to:
'2026-08-02T00:00:00.000Z' },
where: { op: 'eq', left: { path: 'spanType' }, right: { literal:
'model_inference' } },
        measures: ['timeToFirstToken.p50', 'timeToFirstToken.p95'],
      }),
    );
  }
  ```

### Patch Changes

- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
## @mastra/pg@1.35.0-alpha.1

### Minor Changes

- Added support for `timeToFirstToken.*` and `timeToFirstChunk.*`
measures in trace and span aggregates. Rows without a recorded value are
skipped, never counted as 0.
([#29055](https://github.com/mastra-ai/mastra/pull/29055))

  ```typescript
  import { planSpanAggregate } from '@mastra/core/storage';

  const observability = await storage.getStore('observability');

if
(observability?.getFeatures().includes('aggregate-time-to-first-measures'))
{
    const { rows } = await observability.aggregateSpans(
      planSpanAggregate({
timeRange: { from: '2026-08-01T00:00:00.000Z', to:
'2026-08-02T00:00:00.000Z' },
where: { op: 'eq', left: { path: 'spanType' }, right: { literal:
'model_inference' } },
        measures: ['timeToFirstToken.p50', 'timeToFirstToken.p95'],
      }),
    );
  }
  ```

### Patch Changes

- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
## @mastra/react@1.14.1-alpha.1

### Patch Changes

- Updated dependencies
[[`0900c20`](0900c20592),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/client-js@1.59.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
## @mastra/deployer-cloud@1.82.0-alpha.1

### Patch Changes

- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
  - @mastra/deployer@1.82.0-alpha.1
## @mastra/longmemeval@1.1.41-alpha.1

### Patch Changes

- Updated dependencies
[[`489d829`](489d8295c8),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`e1e64f0`](e1e64f072c),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/memory@1.41.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
  - @mastra/libsql@1.29.0-alpha.0
## @mastra/opencode@0.1.41-alpha.1

### Patch Changes

- Updated dependencies
[[`489d829`](489d8295c8),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`e1e64f0`](e1e64f072c),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/memory@1.41.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
  - @mastra/libsql@1.29.0-alpha.0
## create-factory@0.4.2-alpha.1

### Patch Changes

- Updated dependencies:
  - mastra@1.38.0-alpha.1
## @mastra/code-sdk@1.18.0-alpha.1

### Patch Changes

- Updated dependencies
[[`489d829`](489d8295c8),
[`0900c20`](0900c20592),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`e1e64f0`](e1e64f072c),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/memory@1.41.0-alpha.1
  - @mastra/client-js@1.59.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
  - @mastra/pg@1.35.0-alpha.1
  - @mastra/duckdb@1.18.0-alpha.0
  - @mastra/libsql@1.29.0-alpha.0
## mastracode@0.49.0-alpha.1

### Patch Changes

- Updated dependencies
[[`489d829`](489d8295c8),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`e1e64f0`](e1e64f072c),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/memory@1.41.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
  - @mastra/pg@1.35.0-alpha.1
  - @mastra/duckdb@1.18.0-alpha.0
  - @mastra/code-sdk@1.18.0-alpha.1
  - @mastra/libsql@1.29.0-alpha.0
## @mastra/sentry@1.2.31-alpha.1

### Patch Changes

- dependencies updates:
([#29123](https://github.com/mastra-ai/mastra/pull/29123))
- Updated dependency [`@sentry/node@^11.10.0`
↗︎](https://www.npmjs.com/package/@sentry/node/v/11.10.0) (from
`^10.76.2`, in `dependencies`)
- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
## @mastra/agent-builder@1.1.33-alpha.1

### Patch Changes

- Updated dependencies
[[`489d829`](489d8295c8),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`e1e64f0`](e1e64f072c),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/memory@1.41.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
## mastra@1.38.0-alpha.1

### Patch Changes

- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
  - @mastra/deployer@1.82.0-alpha.1
## @mastra/deployer@1.82.0-alpha.1

### Patch Changes

- Updated dependencies
[[`0900c20`](0900c20592),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/server@1.82.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
## @mastra/editor@0.16.7-alpha.1

### Patch Changes

- Updated dependencies
[[`489d829`](489d8295c8),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`e1e64f0`](e1e64f072c),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/memory@1.41.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
## @mastra/mcp-docs-server@1.3.12-alpha.2

### Patch Changes

- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
## @mastra/memory@1.41.0-alpha.1

### Patch Changes

- dependencies updates:
([#29124](https://github.com/mastra-ai/mastra/pull/29124))
- Updated dependency [`diff@^9.0.0`
↗︎](https://www.npmjs.com/package/diff/v/9.0.0) (from `^8.0.4`, in
`dependencies`)

- Observational Memory now tells the Observer who sent each user signal.
In a thread shared by several people, messages sent with
`agent.sendMessage({ contents, attributes: { name: 'Alice' } }, ...)`
are labeled `User (name="Alice")` when observed, so observations can say
who said what instead of attributing everything to "User". Messages
without attributes keep the plain `User` label.
([#29176](https://github.com/mastra-ai/mastra/pull/29176))

- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
## @mastra/google-cloud-pubsub@1.1.7-alpha.0

### Patch Changes

- dependencies updates:
([#29113](https://github.com/mastra-ai/mastra/pull/29113))
- Updated dependency [`@google-cloud/pubsub@^6.3.1`
↗︎](https://www.npmjs.com/package/@google-cloud/pubsub/v/6.3.1) (from
`^5.3.1`, in `dependencies`)
- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
## @mastra/elysia@0.1.22-alpha.1

### Patch Changes

- Updated dependencies
[[`0900c20`](0900c20592),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/server@1.82.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
## @mastra/express@1.5.26-alpha.1

### Patch Changes

- Updated dependencies
[[`0900c20`](0900c20592),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/server@1.82.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
## @mastra/fastify@1.5.26-alpha.1

### Patch Changes

- Updated dependencies
[[`0900c20`](0900c20592),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/server@1.82.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
## @mastra/hono@1.7.24-alpha.1

### Patch Changes

- Updated dependencies
[[`0900c20`](0900c20592),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/server@1.82.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
## @mastra/koa@1.7.26-alpha.1

### Patch Changes

- Updated dependencies
[[`0900c20`](0900c20592),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/server@1.82.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
## @mastra/nestjs@0.2.41-alpha.1

### Patch Changes

- Updated dependencies
[[`0900c20`](0900c20592),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/server@1.82.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
## @mastra/next@0.2.40-alpha.1

### Patch Changes

- Updated dependencies
[[`0900c20`](0900c20592),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/server@1.82.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
  - @mastra/hono@1.7.24-alpha.1
## @mastra/tanstack-start@0.2.40-alpha.1

### Patch Changes

- Updated dependencies
[[`0900c20`](0900c20592),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/server@1.82.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
  - @mastra/hono@1.7.24-alpha.1
## @mastra/mysql@0.16.0-alpha.1

### Patch Changes

- Fixed experiment result writes putting plaintext back over a purged
dataset item. When a purge committed while a result was being written,
the writer could miss the purge marker and overwrite the redacted input,
output, ground truth, error and metadata.
([#29106](https://github.com/mastra-ai/mastra/pull/29106))

- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
## @mastra/inngest@1.11.1-alpha.0

### Patch Changes

- Fixed `InngestRun.watch()` crashing the process with an unhandled
rejection when the Inngest realtime subscription fails to connect (for
example `fetch failed` or a WebSocket error). The failure is now logged
as a warning and the run still completes through polling. Unwatching now
also closes the realtime connection instead of leaving it open.
([#29103](https://github.com/mastra-ai/mastra/pull/29103))

- Fixed Inngest durable agents ending the `model_generation` trace span
without `responseId` and `responseModel`. The span now carries the
provider's response id and model, the same as non-durable agents. Fixes
#29053. ([#29180](https://github.com/mastra-ai/mastra/pull/29180))

- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
## @mastra/temporal@0.5.0-alpha.1

### Patch Changes

- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
  - @mastra/deployer@1.82.0-alpha.1
## @mastra/e2b@0.16.1-alpha.0

### Patch Changes

- dependencies updates:
([#27993](https://github.com/mastra-ai/mastra/pull/27993))
- Updated dependency [`e2b@^2.62.0`
↗︎](https://www.npmjs.com/package/e2b/v/2.62.0) (from `^2.61.0`, in
`dependencies`)
- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
## @mastra/e2b-desktop@0.1.12-alpha.0

### Patch Changes

- dependencies updates:
([#27993](https://github.com/mastra-ai/mastra/pull/27993))
- Updated dependency [`e2b@^2.62.0`
↗︎](https://www.npmjs.com/package/e2b/v/2.62.0) (from `^2.61.0`, in
`dependencies`)
- Updated dependencies
[[`5735a50`](5735a50c6c),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/e2b@0.16.1-alpha.0
  - @mastra/core@1.82.0-alpha.1
## @mastra/platform-workspace@1.11.1-alpha.0

### Patch Changes

- dependencies updates:
([#27993](https://github.com/mastra-ai/mastra/pull/27993))
- Updated dependency [`e2b@^2.62.0`
↗︎](https://www.npmjs.com/package/e2b/v/2.62.0) (from `^2.61.0`, in
`dependencies`)
- Updated dependencies
[[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/core@1.82.0-alpha.1
## create-mastra@1.38.0-alpha.1


## @internal/playground@1.38.0-alpha.1

### Patch Changes

- Updated dependencies
[[`0900c20`](0900c20592),
[`8751dd7`](8751dd7f56),
[`682e8f3`](682e8f3e44),
[`3314289`](33142896c1),
[`0900c20`](0900c20592),
[`ebf8a9a`](ebf8a9a137)]:
  - @mastra/client-js@1.59.0-alpha.1
  - @mastra/core@1.82.0-alpha.1
  - @mastra/playground-ui@68.0.0-alpha.1
  - @mastra/react@1.14.1-alpha.1

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-11-02 16:03:13 +01:00
Dane - AI (Mastra)
7645fce513
chore(deps): update dependency @google-cloud/pubsub to v6 (#29113)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-11-01 23:03:34 +01:00
dane-ai-mastra[bot]
eed9cc214e chore: version packages 2026-11-01 04:20:58 +00:00
Abhi Aiyer
6ede0f10f6
fix(valkey-streams): trim saved thread-stream events (#29095)
> Opened by @abhiaiyer91

Valkey thread streams currently retain saved run events until their size
limit or idle TTL. Implement run-scoped trimming so persisted stream
parts can be removed without deleting other runs, pinned entries, or
retained terminal events. Closes #28974.

Before: `await pubsub.trimTopic(topic, { runId })` leaves all entries
behind. After: it removes that run's eligible entries; `producedBefore`
bounds removal to saved parts and `retainTerminalEvents` preserves
terminal markers. Cleanup now snapshots the current stream tail, so
ongoing publishers cannot extend the scan indefinitely or have their
later entries deleted by that scan.

## Proof
- Real-Valkey concurrency regression: pause the cleanup scan, publish
another target-run entry, resume, and assert only the in-flight entry
remains. This test failed before the tail-snapshot fix (`[]` instead of
`['in-flight']`) and passed after it.
- Existing real-Valkey integration covers more than one scan page,
other-run isolation, pinned entries, produced-time cutoff, and retained
terminal events; the targeted integration and lease suite passed (7/7).
- Full `@mastra/valkey-streams` tests passed (8 files, 42 tests);
package typecheck, lint (0 errors/warnings), formatting, diff check, and
ESM/CJS build passed.
- Earlier local Life Coach run with Postgres and Valkey emitted a finish
event and left 2 stream entries with the patched adapter, compared to
117 before.

## Unverified
- The Life Coach stream iterator remained open after finish, so that
isolated run was stopped by timeout; it is not a completed end-to-end
test. No sustained-production-throughput or multi-process soak test was
run.

## Risk: medium
- Trimming is best-effort and run-scoped, but removal is irreversible.
Real-Valkey tests cover the bounded concurrent-publish race and
preservation predicates; production scheduling and very large/live
streams have not been soak-tested.

Patch changeset included for `@mastra/valkey-streams`.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## ELI5

When a run is saved, Valkey can keep its stream entries around and use
extra storage. This change removes eligible entries for that run while
keeping entries that must remain.

## What changed

- Added `ValkeyStreamsPubSub.trimTopic` to delete entries for a
specified `runId`.
- When `producedBefore` is set, the method deletes only entries with a
numeric `data.producedAt` at or before the cutoff. It skips pinned
entries.
- When `retainTerminalEvents` is enabled, the method preserves
`run-registered`, `run-completed`, `run-failed`, and `run-aborted`
events, except those marked `suspended`.
- The scan stops at the stream tail captured before scanning starts.
Entries published during the scan are not included.
- Trimming failures are logged as warnings and do not propagate.
- Added integration tests for run-specific trimming, cutoff and
pinned-entry behavior, terminal-event retention, and entries published
during a scan.
- Added a patch changeset for `@mastra/valkey-streams`.

## Validation

The supplied report says six Valkey integration tests passed and the
package TypeScript check, lint, and build succeeded. It also reports a
local Postgres and Valkey test that left two stream entries after a
finish event, down from 117. The stream iterator remained open, so a
timeout stopped the isolated process; the Valkey regression suite
completed normally.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-31 14:29:32 -07:00
dane-ai-mastra[bot]
77076a1012 chore: version - exit prerelease mode 2026-10-30 23:02:25 +00:00
dane-ai-mastra[bot]
582ef17a2b
chore: version packages (alpha) (#28888)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-10-30 14:56:53 -07:00
mastra-platform[bot]
b0e86c50e2
fix(redis-streams): stop subscribe() hanging forever when Redis stops responding before connecting (#28941) 2026-10-30 11:06:52 -07:00
mastra-platform[bot]
39edceea4f
fix(core): open thread subscriptions with initial history when the lease read stalls or fails (#28924) 2026-10-30 10:39:35 -07:00
Ward Peeters
ba8b955b1a
fix(redis-streams,valkey-streams): expire idle streams by default and clear them on thread delete (#28968)
A production Factory deployment ran out of Redis memory. Every key was a
`mastra:topic:*` stream, and almost all of them (3,673 thread streams)
had no expiry. The deployment built `new RedisStreamsPubSub({ url })`
with defaults, and the default `streamIdleTtlMs` was `0`, meaning
streams never expire. Thread streams are open-ended and never get
`clearTopic()`, so they piled up forever. The Redis instance used
`volatile-lru`, which can't evict keys without a TTL, so writes just
started failing.

This PR makes `@mastra/redis-streams` and `@mastra/valkey-streams` give
every stream a sliding 7-day idle TTL by default. Writes refresh it, and
live subscribers keep receiving events after a stream expires because
the read loop recreates it. Streams are an event bus, not storage:
replay history lives in the caching layer and storage, so the backlog
only matters for grouped workers that are offline longer than the TTL.

```ts
// Before: streams never expire unless you opt in
new RedisStreamsPubSub({ url });

// After: 7-day idle TTL by default; opt out explicitly
new RedisStreamsPubSub({ url });
new RedisStreamsPubSub({ url, streamIdleTtlMs: 0 });
```

Both pubsubs now log one warning after they first connect if Redis has
`maxmemory` set with any policy other than `noeviction`. A TTL makes
streams, consumer groups and lease keys evictable under `volatile-*`,
which would silently drop events and leases (ElastiCache defaults to
`volatile-lru`). The check uses `INFO memory` because managed Redis
often blocks `CONFIG GET`.

`memory.deleteThread()` now also deletes the thread's event stream
through the registered Mastra pubsub. This runs fire-and-forget, so
neither deleting a thread nor `memory.settled()` waits on Redis during
an outage. It's skipped on older core versions. Agents configured with
their own pubsub still rely on the TTL.

The docs now require `noeviction`; the Helm guide used to say a
`volatile-*` policy was fine. They also add a runbook for reclaiming
streams created before the TTL: switch to `noeviction`, upgrade the
writers, then expire each stream relative to its last write. Tested
against real Redis/Valkey containers, including a real TTL expiry while
a subscriber is listening and a thread delete with Redis unreachable.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

Redis and Valkey can keep old event streams forever, which uses storage.
This change removes streams after seven days without writes by default
and cleans up a thread’s stream when the thread is deleted.

## What changed

- Redis and Valkey streams now use a sliding seven-day idle TTL by
default. Writes refresh the TTL; reads do not. Set `streamIdleTtlMs: 0`
to disable expiry.
- The read loop recreates an expired stream so a live subscriber can
continue receiving events.
- After connecting, each pubsub warns once when `maxmemory` is enabled
with a policy other than `noeviction`. The check uses `INFO memory`.
- Thread deletion starts best-effort stream cleanup through the
registered Mastra pubsub without waiting for Redis. Older core versions
skip this cleanup.
- Documentation and the local Docker configuration now recommend
`noeviction`. The documentation also explains how to expire existing
streams that lack a TTL.

## Tests

Tests cover TTL behavior, delivery after expiry for live subscribers,
eviction-policy warnings, and thread-stream cleanup. Test execution
results were not provided.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-30 11:51:23 +01:00
mastra-platform[bot]
52265cba91
Fix run abort route answering false on a server that does not own the run (#28522) 2026-10-29 14:11:05 -07:00
Dane - AI (Mastra)
cf92428a76
chore(deps): update docker images (#28011)
This PR contains the following updates:

| Package | Update | Change |
|---|---|---|
| amazon/aws-cli | minor | `2.36.44` → `2.39.1` |
| clickhouse/clickhouse-server | minor | `26.6.1.1193` → `26.10.3.7` |
| cr.weaviate.io/semitechnologies/weaviate | minor | `1.39.13` →
`1.40.3` |
|
[docker.elastic.co/elasticsearch/elasticsearch](https://www.elastic.co/products/elasticsearch)
([source](https://redirect.github.com/elastic/elasticsearch)) | minor |
`9.4.3` → `9.5.5` |
| edoburu/pgbouncer | minor | `v1.25.2-p0` → `v1.26.1-p0` |
|
[ghcr.io/shopify/toxiproxy](https://redirect.github.com/Shopify/toxiproxy)
| minor | `2.12.0` → `2.13.0` |
| inngest/inngest | minor | `v1.34.0` → `v1.49.0` |
| mcr.microsoft.com/azure-storage/azurite | minor | `3.35.0` → `3.37.0`
|
| [mongo](https://hub.docker.com/_/mongo)
([source](https://redirect.github.com/docker-library/mongo)) | patch |
`7.0.37` → `7.0.45` |
| pgvector/pgvector | patch | `0.8.4-pg16` → `0.8.7-pg16` |

---

> [!WARNING]
> Some dependencies could not be looked up. Check the [Dependency
Dashboard](../issues/9505) for more information.

---

### Release Notes

<details>
<summary>elastic/elasticsearch
(docker.elastic.co/elasticsearch/elasticsearch)</summary>

###
[`v9.5.5`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.5.5):
Elasticsearch 9.5.5

[Compare
Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.5.4...v9.5.5)

Downloads: <https://elastic.co/downloads/elasticsearch>
Release notes:
<https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.5.5-release-notes>

###
[`v9.5.4`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.5.4):
Elasticsearch 9.5.4

[Compare
Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.5.3...v9.5.4)

Downloads: <https://elastic.co/downloads/elasticsearch>
Release notes:
<https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.5.4-release-notes>

###
[`v9.5.3`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.5.3):
Elasticsearch 9.5.3

[Compare
Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.5.2...v9.5.3)

Downloads: <https://elastic.co/downloads/elasticsearch>
Release notes:
<https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.5.3-release-notes>

###
[`v9.5.2`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.5.2):
Elasticsearch 9.5.2

[Compare
Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.5.1...v9.5.2)

Downloads: <https://elastic.co/downloads/elasticsearch>
Release notes:
<https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.5.2-release-notes>

###
[`v9.5.1`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.5.1):
Elasticsearch 9.5.1

[Compare
Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.5.0...v9.5.1)

Downloads: <https://elastic.co/downloads/elasticsearch>
Release notes:
<https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.5.1-release-notes>

###
[`v9.5.0`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.5.0):
Elasticsearch 9.5.0

[Compare
Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.4.8...v9.5.0)

Downloads: <https://elastic.co/downloads/elasticsearch>
Release notes:
<https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.5.0-release-notes>

###
[`v9.4.8`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.4.8):
Elasticsearch 9.4.8

[Compare
Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.4.7...v9.4.8)

Downloads: <https://elastic.co/downloads/elasticsearch>
Release notes:
<https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.4.8-release-notes>

###
[`v9.4.7`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.4.7):
Elasticsearch 9.4.7

[Compare
Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.4.6...v9.4.7)

Downloads: <https://elastic.co/downloads/elasticsearch>
Release notes:
<https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.4.7-release-notes>

###
[`v9.4.6`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.4.6):
Elasticsearch 9.4.6

[Compare
Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.4.5...v9.4.6)

Downloads: <https://elastic.co/downloads/elasticsearch>
Release notes:
<https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.4.6-release-notes>

###
[`v9.4.5`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.4.5):
Elasticsearch 9.4.5

[Compare
Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.4.4...v9.4.5)

Downloads: <https://elastic.co/downloads/elasticsearch>
Release notes:
<https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.4.5-release-notes>

###
[`v9.4.4`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.4.4):
Elasticsearch 9.4.4

[Compare
Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.4.3...v9.4.4)

Downloads: <https://elastic.co/downloads/elasticsearch>
Release notes:
<https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.4.4-release-notes>

</details>

<details>
<summary>Shopify/toxiproxy (ghcr.io/shopify/toxiproxy)</summary>

###
[`v2.13.0`](https://redirect.github.com/Shopify/toxiproxy/blob/HEAD/CHANGELOG.md#2130---2026-10-14)

[Compare
Source](https://redirect.github.com/Shopify/toxiproxy/compare/v2.12.0...v2.13.0)

- Reject slicer toxics with `average_size` <= 0 or `size_variation`
outside `[0, average_size]`,
and stop the slicer from recursing forever on degenerate sizes
([#&#8203;774](https://redirect.github.com/Shopify/toxiproxy/issues/774),
[@&#8203;iyaz-shaikh](https://redirect.github.com/iyaz-shaikh))
- Add optional bearer token auth for the HTTP API via
`TOXIPROXY_AUTH_TOKEN`, supported by the Go
client and `toxiproxy-cli`, and warn when the API is unauthenticated on
a non-loopback address

([#&#8203;774](https://redirect.github.com/Shopify/toxiproxy/issues/774),
[@&#8203;iyaz-shaikh](https://redirect.github.com/iyaz-shaikh))
- Fix removing a toxic hanging forever, and freezing the whole HTTP API,
when the downstream
toxic or connection is already gone
([#&#8203;760](https://redirect.github.com/Shopify/toxiproxy/issues/760),
[@&#8203;matheusgb](https://redirect.github.com/matheusgb))
- Fix the bandwidth toxic hanging or panicking when `rate` <= 0
([#&#8203;764](https://redirect.github.com/Shopify/toxiproxy/issues/764),
[@&#8203;jaideeppyne](https://redirect.github.com/jaideeppyne))
- Update go version to 1.27
([#&#8203;777](https://redirect.github.com/Shopify/toxiproxy/issues/777),
[@&#8203;The-Black-Fox](https://redirect.github.com/The-Black-Fox))
- Update various go packages

</details>

---

### Configuration

📅 **Schedule**: (UTC)

- Branch creation
  - "before 6am every weekday"
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

👻 **Immortal**: This PR will be recreated if closed unmerged. Get
[config
help](https://redirect.github.com/renovatebot/renovate/discussions) if
that's undesired.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend Renovate
CLI](https://redirect.github.com/renovatebot/renovate).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4xNzUuMSIsInVwZGF0ZWRJblZlciI6IjQ0LjE5Mi4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJhdXRvbWF0aW9uIl19-->

---------

Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-29 15:53:50 +00:00
dane-ai-mastra[bot]
4f12792ed1
chore: version packages (alpha) (#28722)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-10-29 08:37:01 -07:00
Ward Peeters
1607791e91
fix(core): trim finished runs of memoryless agents from thread-stream topics (#28746)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-28 18:08:58 -07:00
Dane - AI (Mastra)
f2ad3844f3
chore(deps): update opentelemetry (#28581)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-28 16:48:54 -07:00
shreyas-mastra
6deb7c0a1e
chore(security): Vanta 2026-10-28 - qs 6.16.1, @modelcontextprotocol/node 2.1.1 floors (#28716)
Refs mastra-ai/platform#2931

Vanta remediation pass for 2026-10-28. It follows #28325, which has
merged.

## 2026-10-28 update
Today's live Vanta export added 3 advisory IDs on `mastra-ai/mastra`
(GitHub/Dependabot). This PR fixes the two that have a published fix and
leaves deepmerge, which has none.

| Package | From → To | Finding | Where | How |
|---|---|---|---|---|
| qs | 6.16.0 → 6.16.1 | CVE-2026-110816 / GHSA-6pq8-jhm2-2vcw (Medium
×4) | root `pnpm-lock.yaml`, 9 `e2e-tests/*` lockfiles,
`scripts/gh-issue-triage/package-lock.json` | Raised the existing floors
`qs@>=6.7.0 <6.16.0: 6.16.0` → `<6.16.1: 6.16.1` and `'qs@<6.7.0':
6.16.0` → `6.16.1`. Raised the npm override in gh-issue-triage. |
| @modelcontextprotocol/node | 2.0.0 → 2.1.1 | GHSA-f897-62hf-6gxq (High
×2; HTTP transports had no body-size or batch-length limit) | root
`pnpm-lock.yaml` (published `@mastra/mcp` 1.18.0, used as `mcp-legacy`),
`scripts/gh-issue-triage/package-lock.json` | New pnpm override
`'@modelcontextprotocol/node@>=2.0.0 <2.1.0': 2.1.1` and a new npm
override in gh-issue-triage. I used 2.1.1 rather than 2.1.2 because
2.1.2 needs a `@modelcontextprotocol/server` peer of `^2.4.0`, while
`@mastra/mcp` 1.18 resolves server 2.3.1. `packages/mcp` itself already
uses node 2.1.2. |

- Both versions are older than the 1-day `minimumReleaseAge`: qs 6.16.1
was published 10-14 and node 2.1.1 on 10-02. I did not use qs 6.16.2
(10-27), which is too new.
- No floors or overrides were removed; existing ones were only raised.
- `pnpm install --frozen-lockfile --lockfile-only` passes at the root.
Each nested e2e workspace was regenerated with `pnpm install
--lockfile-only`.
- Added the empty changeset
`.changeset/vanta-2026-10-28-security-floors.md`. No published package
changes.
- After this change, osv-scanner over all 33 lockfiles reports only the
documented leftovers:
  - zod 3.25.76 (zod v3 compat line on purpose)
  - ai 4.3.19 / `@ai-sdk/provider-utils` 2.2.8 (vendored v4 line)
- `@azure/msal-node` 5.6.0 / 6.0.1 (fix 7.1.0 needs majors in identity
and teams)
  - `@opentelemetry/core` 1.30.1 (needs pubsub 6)
  - extract-zip 2.0.1 (no patched release)
- deepmerge 4.3.1 (CVE-2026-93753 / GHSA-gcqx-w48m-w2h4; no patched
release, 4.3.1 is the latest)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

This PR updates dependency rules and package versions to use newer
releases, including security fixes for `qs` and
`@modelcontextprotocol/node`. It also updates AI SDK dependencies and
does not include published package changes.

## Summary

- Raise the `qs` override target to `6.16.1` in the root and e2e
workspaces and in `scripts/gh-issue-triage`.
- Add an `@modelcontextprotocol/node` override at `2.1.1` in the root
workspace and `scripts/gh-issue-triage`.
- Update AI SDK dependency versions across package manifests and add a
changeset describing the provider-utils update.
- Add an empty changeset for the security-only dependency updates.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

## Update 2026-10-28 (afternoon)
- Merged origin/main (clean).
- Bumped the AI SDK v5 compat line from @ai-sdk/provider-utils 3.0.41 to
3.0.42 (patch, published 2026-10-21; it blocks `.internal` hostnames in
download URL validation and encodes provider-returned IDs in request
paths). This addresses the Aikido /mastra Highs AIKIDO-2026-10990 and
AIKIDO-2026-10981:
- ai 5.0.277 → 5.0.282, @ai-sdk/gateway 2.0.168 → 2.0.173 (vendored
ai_v5)
- anthropic 2.0.109 → 2.0.110, google 2.0.103 → 2.0.106, groq 2.0.58 →
2.0.59, openai 2.0.135 → 2.0.137, openai-compatible 1.0.58 → 1.0.59, xai
2.0.99 → 2.0.100
- Nested lockfiles updated: memory/integration-tests and
playground/e2e/kitchen-sink.
- No new overrides. Changeset:
`vanta-2026-10-28-ai-sdk-v5-provider-utils.md`.
2026-10-28 18:51:27 -04:00
dane-ai-mastra[bot]
6a191d46dc chore: version - exit prerelease mode 2026-10-28 13:15:52 +00:00
dane-ai-mastra[bot]
e6764d26da
chore: version packages (alpha) (#28625)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-10-27 15:40:38 -05:00
mastra-platform[bot]
a7cc444233
fix(core): expose recoverable run outcomes to thread subscribers (#28621)
Thread subscribers now receive an ordered `thread-run-status` frame for
success, failure, abort, or suspension without closing the subscription.
Initial history also reports `activeRunId`, so an idle snapshot no
longer has to stand in for a run outcome. Retained outcomes reach late
subscribers even after saved output has been trimmed, with execution
authority preserved to reject superseded attempts.

For example, a subscription opened after a run finishes now receives
`thread-history` followed by `thread-run-status` with `{ status:
"success" }`, rather than history alone. Recovery is bounded by PubSub
retention; upgrade both `@mastra/core` and `@mastra/redis-streams` for
recovery after trimming. Client callback types, documentation, and
package changesets are included.

Verified with core build and typecheck, 220 core tests across the
thread-stream suites and four execution engines, 91 client tests, 55
Redis integration tests, and three HTTP conformance cases covering
plain, durable, and evented agents. Real Redis regressions cover restart
after trimming and superseded completion/abort after registration
eviction. The live outcome scenario also passed 20 repeated four-engine
runs. Typed lint, formatting, MDX prose checks, and final adversarial
review passed.

Fixes COR-1500.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

A thread subscription can now tell clients when a run succeeds, fails,
stops, or pauses without ending the subscription. This helps clients
understand what happened after reconnecting, even when the run’s saved
output is no longer available.

## Summary

- Added opt-in `withRunStatus` support to core, server, and client
thread subscriptions. Enabled subscriptions can receive ordered
`thread-run-status` chunks for `success`, `failed`, `aborted`, or
`suspended` outcomes.
- Added `activeRunId` to initial thread history so clients can identify
the run that held the thread lease during history loading.
- Added terminal-event retention during output trimming. Recovery
depends on PubSub retention. Redis Streams users must upgrade
`@mastra/core` and `@mastra/redis-streams` together.
- Updated React `useChat` to request run statuses and apply outcomes to
the matching run. Added API types, documentation, and changesets.

## Validation

The PR objectives report passing core build and typecheck, core and
client tests, Redis integration tests, HTTP conformance cases, lint and
formatting checks, and documentation checks. They also report repeated
run-outcome checks and adversarial review. These results are
author-reported.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-27 15:15:42 -05:00
mastra-platform[bot]
e718b91a64
fix(core): stop observe offsets past the log end replaying the whole run (#28348)
> Opened by @roaminro

Refs #28324 (partial fix; the issue stays open, see Open question)

## What changed

On transports that keep old messages, such as Redis Streams,
`observe(runId, { offset })` replayed the **whole run** when the offset
was at or past the end of the run's log. A new subscriber receives the
stream's retained frame 0. `CachingPubSub`'s log-restart check compared
the counter with the subscriber's cursor, and with a past-end offset the
counter is always below the cursor. So frame 0 looked like a log restart
and everything replayed. This affected every Redis setup, durable agents
included.

`subscribeFromOffset` in `packages/core/src/events/caching-pubsub.ts`
now decides restarts by frame 0's identity:
- On attach it records the id of the log's frame 0. A frame 0 arriving
below the cursor is a restart only if the log's frame 0 id has changed.
A retained copy of the same frame is acknowledged and dropped.
- On a restart it delivers the new segment from the cache log, not from
the frame that signalled the restart, because that frame may be a
delayed copy of the old frame 0.
- If an id can't be read, it falls back to the counter. If the read
failed at attach, the counter is compared with the log's end at that
point, so an intact log isn't mistaken for a restart.

The cursor stays at the offset, so the documented contract holds: an
offset beyond the retained index replays nothing and skips live events
below it. If the retained log expired and the run restarts numbering at
0, the observer resyncs to the new segment.

**Known limitation (not fixed):** `createEventedAgent` on a
cross-process pubsub with the default in-memory server cache, which is
the case in the issue title. Its step executor publishes unnumbered
frames there, because each process would count them separately. These
frames can't be skipped by position:
- `CachingPubSub` can't see where a transport's replay starts.
- Redis Streams trims the start of its backlog with `MAXLEN`.
- Other transports replay nothing.

That configuration now logs a one-time warning. It names both possible
causes, a failed cache append or no shared server cache, and recommends
a shared cache such as `RedisServerCache`. Both observe docs pages
(`docs/harness/durable-agents.mdx`,
`reference/agents/durable-agent.mdx`) document the requirement.

## Tests

- `packages/core/src/events/caching-pubsub.test.ts`:
- A past-end offset delivers nothing, then delivers live frames once the
log reaches the offset.
  - An offset inside a short log resumes without resyncing.
  - A short log past the offset expires and a new segment resyncs.
- A stale copy of the old frame 0 doesn't stand in for the new segment.
  - A failed frame 0 id read falls back to the counter.
  - A restart is detected by id when the counter read fails.
  - A failed id read at attach doesn't replay an intact log.
  - A log that is empty at attach still delivers the next run.
  - Unnumbered frames produce exactly one warning.
- `pubsub/redis-streams/src/observe-offset.test.ts`: new file against
real Redis, with the model served by aimock.
- It covers four setups: durable or evented engine, with the default or
`RedisServerCache` server cache.
- Each setup runs three cases: resuming a live run, a past-end offset on
a finished run, and a mid-run offset on a finished run.
- The evented/default setup checks for the warning and that the
observation still ends on `finish`.

| Check | Result |
|---|---|
| `packages/core` `src/events` | ✅ 323/323 |
| core `tsc --noEmit` | ✅ clean |
| `pubsub/redis-streams` observe-offset, real Redis, rebuilt core | ✅
12/12 |
| oxlint, prettier, docs `format:mdx:check` + remark | ✅ clean |

## Proof

- I checked each new unit test against the previous commit or the code
before its fix, and it failed there. The empty-log test is the
exception: it passes before its fix by design. It guards against
limiting the counter check whenever no id was captured, and it fails if
that change is applied.
- With `caching-pubsub.ts` at base and core rebuilt, 6 of the 12 Redis
tests fail. These are the past-end case in the three numbered setups,
where the whole run replays from `start`, and all three evented/default
cases, where no warning is logged.
- NikAiyer independently reproduced the base failures and the fixed-head
results.

## Unverified

- Redis Cluster deployments, and transports other than Redis Streams and
the in-process pubsub.
- `durable-agent-tool-writer` and `durable-agent-client-tool-pending`
already fail on `main`, with or without this change.

## Risk: low

The change is limited to the restart check and cursor handling in
`CachingPubSub.subscribeFromOffset`, plus a warning. Offset 0 and the
default observe path (no offset) don't read the counter and behave as
before.

## Open question

#28324 stays open for evented agents on a cross-process pubsub without a
shared server cache. Should evented step frames be numbered without a
shared cache (or a shared cache be required), or is the documented
limitation enough?

**Merge note:** the first commit (`162c296a`) says `Fixes #28324`. When
squash-merging, change it to `Refs #28324` so the issue isn't closed.

---------

Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-27 13:44:02 -05:00
mastra-platform[bot]
5381b414b3
test(redis-streams): fix the Lambda-death signal-drain test that times out on main (#28533)
> Opened by @abhiaiyer91

## Summary

`drains signal-enqueued from losers that die immediately (Vercel
Lambda-death)` in
`pubsub/redis-streams/src/agent-thread-cross-process.test.ts` times out
on main. It never actually exercised the scenario in its name. This is a
test-only fix. Product code is unchanged, so there's no changeset.

Two fixture problems caused it:

1. **Each worker had its own agent id.** The fixture used
`${WORKER_ID}-agent` (`test-fixtures/agent-thread-worker.entry.ts:59` on
main). The losers therefore treated the winner's run as another agent's
run. The cross-agent guard
(`packages/core/src/agent/thread-stream-runtime.ts:6852-6854`,
`:6900-6905`) correctly returned `blocked / thread-blocked` and
published nothing, so the winner had nothing to drain. Lambdas are
replicas of one deployed agent, so they should share an id.
2. **Timing.** The winner's run was a fixed 1500ms, but a worker takes
about 2.3s to boot through tsx. The losers always sent after the lease
was already free, never "while the run is in flight".

Changes:
- The fixture accepts `AGENT_ID` (default is still `${WORKER_ID}-agent`,
so other tests are unaffected). The test sets one shared id.
- The fixture accepts `HOLD_SIG_IDS` plus a `finish-run` command. The
winner's sig-1 run stays open until both losers have exited, then the
test ends it explicitly.
- The test asserts the winner owns sig-1, and that both losers lost the
lease and started no runs. It then waits for sig-1/2/3 to finish on the
winner. I removed the fixed 300ms settle sleep, so the test waits on
product events only.

## Proof

- On main: fails with `waitFor timed out after 20000ms` waiting for 3
`run-finished`. Only sig-1 finishes on the winner.
- With the fix, against core built from main: **5/5 consecutive
passes**.
- Reverting only the shared `AGENT_ID` makes it fail again with the same
20s `waitFor` timeout.
- `agent-thread-cross-process.test.ts`: 10/10 pass. On main this file
failed because of this test.
- Full redis-streams suite, run per file:
- Fail both on main and on this branch: `cluster`,
`durable-worker-takeover`, `observe-cross-process`. On main this run
only checked `observe-cross-process`; the other two are already known to
fail on main.
- `agent-thread-publish-lag`: 1 test failed. Known to fail on main; I
didn't rerun it on main myself.
- `auth-e2e` and `fanout-sharing` failed once in the long serial run.
Both passed 2/2 when rerun alone on this branch, and both pass on main.
They don't load the changed fixture.
- oxlint: 0 warnings, 0 errors. prettier: no changes. `tsc --noEmit` in
redis-streams gives the same 14 errors on main and on this branch (a
`RedisStreamsPubSub` vs `PubSub` mismatch, unrelated).

## Unverified

- **A sender that exits without closing its pubsub can still drop the
forwarded signal (product gap, out of scope).** With the shared id, the
losers settle `deliver` (I logged `settled.action`). That means they
take the `ifActive: deliver` branch at `thread-stream-runtime.ts:7027`,
not the lease-loser hand-off at `:2589`, which settles `remote`. Both
branches publish `signal-enqueued` with the fire-and-forget `#publish`
(`:2690-2692`) and resolve `accepted` before the publish reaches Redis.
So my earlier claim about the pattern holds, and it applies to the path
this test actually runs. The test passes only because the fixture awaits
`pubsub.close()` before `process.exit`. `#publish` reaches
`RedisStreamsPubSub.publish` synchronously, which adds the publish to
`#pendingPublishes` (`pubsub/redis-streams/src/index.ts:431-434`) before
`accepted` resolves, and `close()` waits for those publishes through
`flush()` (`:1133`, `:894-899`). It isn't a case of the publish just
being fast locally. I checked this by temporarily skipping `close()` in
`send-and-exit`: **3/3 runs timed out** and sig-2/sig-3 never reached
the winner. So a Lambda that settles and exits (or is frozen or killed)
without closing or flushing its pubsub loses the signal. A fix would
await the publish before resolving `accepted` on the remote-owner
branches. That needs its own core PR. I didn't check whether the server
or serverless deployers close or flush the pubsub on shutdown.
- **A non-durable owner that dies mid-run loses queued signals (separate
product gap, out of scope).** In an earlier run of this test the losers
won the free lease. When the lease owner died, sig-2/sig-3 stayed
stranded after the lease expired, and a later signal woke a fresh run
without replaying them. Signals are only admitted by the process that
owns the runId (`thread-stream-runtime.ts:1444`). Dead-owner recovery
only covers durable runs (`:5574-5584`, `:5824+`). This needs its own
issue and design (orphaned-signal recovery, with duplicate-delivery
risk).
- Not checked on CI runners. The fix doesn't depend on boot timing
anymore, so I expect it to behave the same there.

## Risk

low. Only test and fixture files changed, and the new fixture options
are opt-in.

Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-26 19:48:31 -05:00
mastra-platform[bot]
7c20f008c8
fix(core): abort evented workflow steps running on other replicas on cancel (#28489)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9

Closes #28444

## Summary
With evented workflows and workers on several replicas,
`workflow.cancel` is handled by only one processor in the consumer
group. That processor aborted only the AbortControllers in its own
process. If the step was running on another replica, its `abortSignal`
never fired and the step kept running, even though the run itself
settled as `canceled`.

This PR broadcasts cancellation on one shared control topic,
`workflow.control`:
- **One subscription per processor.** A processor opens a single
ungrouped subscription to `workflow.control` when it runs its first step
and keeps it for its lifetime. Transports share one reader/subscription
per topic per process, so the cost does not grow with runs or steps (no
per-run topics, consumer groups or Pub/Sub subscriptions to clean up).
It reads from `latest`; a failed subscribe is logged and retried on the
next step.
- **Cancel publishes the abort before finalizing.**
`processWorkflowCancel` aborts its local controllers, publishes an abort
for the run, and only then calls `endWorkflow`. If the publish fails,
the error propagates and the event is redelivered before anything is
finalized.
- **Receivers only act on runs they are executing.** Each processor
counts in-flight steps per run (a nested step counts against its
ancestors too). An abort for a run with no in-flight step here is
ignored. Otherwise the run's own controller is marked cancel-handled
(the step unwinds without a second `workflow.cancel`), and nested runs
are aborted like the local `cancelRunAndChildren`, so each settles
canceled. Parent/child links come from the step event's `parentWorkflow`
chain, so this works even if the processor never saw the child's
`workflow.start`.
- **Cancels issued before a processor could hear them.** The control
subscription starts at `latest`, so an abort published while it is still
being opened is missed. When `#dispatch` takes a `workflow.step.run`, it
records, before its state load, whether the subscription was ready and
how many aborts this processor has ignored for runs with no step
retained. The step re-reads the persisted status of its run and its
ancestors when the subscription wasn't ready at dispatch, or when that
count changed before the step was retained, and aborts if any of them is
`canceled` (`EventedRun.cancel` persists that before publishing). Other
steps skip the read. The count can't evict anything; when it changes,
the overlapping steps re-check.
- **A canceled step never starts its body.** If the pre-step check finds
the step's own run canceled, the step returns without running. If it
finds only an ancestor canceled, the body is skipped with a `canceled`
result, and the existing post-step branch publishes this run's
`workflow.cancel`, so the nested run settles canceled.
- **No state left on replicas that never finalize the run.** When a
run's last in-flight step is released on a processor that didn't start
it, the controller and parent links recorded by the step path are
dropped.
- **Double delivery does nothing.** Aborting is idempotent; a
redelivered or self-echoed abort finds the controllers already aborted.
Tests assert exactly one `workflow.end` per run.

Assumption: a shared broadcast topic over per-run topics. Cancels are
rare and tiny, so every processor seeing every abort is cheaper than
per-run subscriptions, which leaked topics/subscriptions on Google Cloud
Pub/Sub and churned consumer groups on Redis Streams.

## Proof
- `packages/core/src/workflows/evented/remote-cancel.test.ts`, suite
"workers on both replicas": two Mastra instances, both running workers,
over EventEmitter and UnixSocket, cancels from the owner and the other
replica, flat and nested. Asserts the step's `abortSignal` fires, the
run settles `canceled`, the only control subscription is
`workflow.control` and there is at most one per processor across all
runs, no in-flight step refs remain; with grouped delivery also exactly
one `workflow.end` and at least one cancel handled on a different
replica than the step. A pinned-topology test (steps on one replica,
start/end on the other) asserts the step-only replica keeps no
controllers or parent links. Five pinned-topology "skips a step …" tests
cover the fallback: (a) the step replica's `workflow.control` subscribe
is delayed and the run is canceled during the delay; (b) the abort lands
after the step's dispatch load but before it is retained (also asserts a
plain step on a ready subscription does no re-check); (c) same as (b)
with 2000 aborts for other runs published in that window; (d) a step
dispatched while the subscription is pending resumes after other steps
have run on the ready subscription; (e) nested: the parent is canceled
while the child's step is held after its dispatch load. Each asserts the
step body never ran, the run settles `canceled` and exactly one
`workflow.end` is published per run; (e) also asserts the child snapshot
is `canceled`. 26/26 pass. Proofs: **with the abort publish disabled**,
the 4 EventEmitter both-replica cases fail on the step-abort assertion.
**With the pre-body skip disabled**, (a)–(d) fail. **On the 7845d2be
processor (1024-entry set)**, (c) fails. **With the count comparison
removed**, (b) and (c) fail. **On 673eaf4a's processor (no per-step
dispatch decision)**, 2 tests fail, including (d). **On 13b89c3a's
processor (own-run skip only)**, (e) fails: the child body blocks on an
abort event that never fires, so the child never settles.
- `pubsub/redis-streams/src/evented-remote-cancel.test.ts` against a
real `redis-server`: canceller owner/other × sequential/dountil/nested ×
workers on both replicas or owner only. 7/7 pass on 434d531a. **With
core rebuilt with the abort publish disabled**, all 5 both-replica cases
fail.
- Other `src/workflows/evented` test files on 434d531a: 42/42 files, 560
tests pass (5 skipped). `evented-workflow.test.ts`: 315 pass / 5 skipped
on 434d531a (6 GB heap; it runs out of memory at a 3 GB heap on `main`
too). Earlier on this branch: `src/loop` 319/319 files and server
`handlers/workflows.test.ts` pass.
-
`packages/core/src/workflows/evented/cancel-check-read-failure.test.ts`
(EventEmitter and UnixSocket) fails only the pre-step cancel re-check
read. Two failing reads: the step event is retried twice, then the step
runs once and the run succeeds. Every read failing: three deliveries,
the last not retried, the run settles `failed` with the storage error
and the step never runs (exhaustion goes through
`#publishPendingFailure` → `errorWorkflow` → `workflow.fail`). The test
keeps the control subscribe failing so every delivery re-checks. 4/4
pass (with remote-cancel, 30/30). **With the old `.catch(() =>
undefined)` restored**, all 4 fail.
- Snapshot reads for one 100-step sequential run (EventEmitter,
`loadWorkflowSnapshot` counted): **402 before this change (d0227436),
302 after**. `main` is 302. The only remaining re-check is on the first
step after the subscription opens.
- Core `tsc --noEmit` clean; oxlint and prettier clean on changed files.

## Unverified
- **Google Cloud Pub/Sub:** no emulator here. With the fixed topic, each
process creates one `workflow.control-<instanceId>` subscription (like
other ungrouped topics on that adapter); not exercised by a test.
- **UnixSocket nested cases:** UnixSocketPubSub delivers the ungrouped
push `workflows` subscription to both replicas, so both process every
event (also on `main`, out of scope). A duplicate child run can finish
while the first call still blocks, so nested cases run only with grouped
delivery, and exactly-once `workflow.end` isn't checked on UnixSocket.
The UnixSocket flat cases don't fail with the abort publish disabled, so
they don't independently prove the fix there.
- **Unrelated redis-streams failures:** agent-thread-cross-process,
agent-thread-publish-lag, cluster and durable-worker-takeover fail in
the full package run, identically with `main`'s processor. `tsc` on
redis-streams reports errors only in the untouched
`test-fixtures/agent-thread-worker.entry.ts`.

## Risk: medium
- The cancel re-check (one full snapshot read for the run and each
ancestor) runs only for steps dispatched before the subscription was
ready, or whose dispatch→retain window overlapped an ignored abort for
any run. It doesn't run on every step. Around a burst of cancels,
unrelated in-flight steps re-check too. A failed read fails the step
event so it is retried. A status-only read would be cheaper but needs a
storage-contract change, so it's left for a follow-up.
- If the `workflow.control` subscribe fails, the step still runs (with a
warning) and re-checks storage on each delivery. Remote abort of an
already-running step then degrades to "the step finishes", and the run
still settles `canceled`. If `mastra.pubsub` is swapped, the old control
subscription is not unsubscribed.
- **Finalization is not a barrier on remote steps.** The cancelling
processor publishes the abort before `endWorkflow` but doesn't wait for
remote processors to observe it, so a remote step can run briefly after
the run is marked `canceled`. Its late events are dropped by the
dispatch guard (status `canceled`). A real barrier would need a registry
of which processors hold the run's steps.
- `startedRuns` entries on processors that start a run but never
finalize it are not cleaned up, matching the existing pattern for
controllers.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

When a workflow step runs on another server, canceling the workflow must
also tell that server to stop the step. This change broadcasts
cancellation and checks saved workflow status when a server may have
missed the notice.

## Changes

- Processors subscribe to the shared `workflow.control` topic and
broadcast workflow cancellations.
- Processors abort active steps for the canceled run, including nested
workflow steps.
- Before processing a step, a processor checks persisted run status if
it may have missed the cancellation broadcast. Read failures trigger
retries.
- The cancel handler publishes the abort before loading state or
finalizing the run. If publishing fails, it does not continue those
operations.
- Processors track active step references and clean up controller and
parent-link state, including on replicas that run steps but do not start
or finalize the run.

## Tests and limitations

- The author reports that `remote-cancel.test.ts` passed 26 tests.
- The author reports that `cancel-check-read-failure.test.ts` passed 4
tests.
- The author reports that the Redis Streams test passed 7 cases against
a real Redis server.
- The author reports that core TypeScript, lint, and formatting checks
passed. These results were not independently verified.
- Google Cloud Pub/Sub was not tested.
- Cancellation finalization does not wait for remote processors to
observe the abort. A remote step may continue briefly after the run is
finalized as canceled.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-26 17:35:12 -05:00
mastra-platform[bot]
9ffb79b17e
fix(core): stop closeOnSuspend durable streams closing before their chunks arrive on Redis Streams (#28504) 2026-10-26 17:34:30 -05:00
dane-ai-mastra[bot]
2c646b53d0
chore: version packages (alpha) (#28399)
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases

> The changelog information of each package has been omitted from this
message, as the content exceeds the size limit.

## @mastra/factory@0.25.0-alpha.5


## @mastra/langsmith@1.4.0-alpha.2


## @mastra/connect@1.2.0-alpha.2


## @mastra/core@1.80.0-alpha.4


## @mastra/playground-ui@66.0.0-alpha.4


## @mastra/acp@0.4.4-alpha.0


## @mastra/claude@0.3.4-alpha.0


## @mastra/cursor@0.3.4-alpha.0


## @mastra/openai@1.1.4-alpha.0


## @mastra/auth-auth0@1.2.6-alpha.0


## @mastra/auth-better-auth@1.1.9-alpha.0


## @mastra/auth-clerk@1.3.2-alpha.0


## @mastra/auth-cloud@1.2.8-alpha.0


## @mastra/auth-firebase@1.1.5-alpha.0


## @mastra/auth-google@0.1.7-alpha.0


## @mastra/auth-neon@0.3.5-alpha.0


## @mastra/auth-okta@0.2.6-alpha.0


## @mastra/auth-studio@1.3.11-alpha.1


## @mastra/auth-supabase@1.1.6-alpha.0


## @mastra/auth-workos@1.6.8-alpha.0


## @mastra/agent-browser@0.5.5-alpha.0


## @mastra/browser-viewer@0.2.6-alpha.0


## @mastra/browser-firecrawl@0.2.5-alpha.1


## @mastra/stagehand@0.3.7-alpha.1


## @mastra/discord@1.3.2-alpha.0


## @mastra/slack@1.8.0-alpha.1


## @mastra/teams@0.1.3-alpha.0


## @mastra/telegram@0.2.4-alpha.0


## @mastra/client-js@1.57.0-alpha.4


## @mastra/react@1.13.0-alpha.4


## @mastra/isolated-vm@0.1.4-alpha.0


## @mastra/quickjs@0.1.5-alpha.0


## @mastra/deployer-cloud@1.80.0-alpha.4


## @mastra/deployer-cloudflare@1.2.39-alpha.1


## @mastra/deployer-netlify@1.2.39-alpha.1


## @mastra/deployer-sandbox@0.3.24-alpha.1


## @mastra/deployer-vercel@1.2.39-alpha.1


## @mastra/voyageai@0.4.3-alpha.0


## @mastra/longmemeval@1.1.39-alpha.4


## @mastra/livekit@0.5.2-alpha.0


## @mastra/opencode@0.1.39-alpha.4


## @mastra/parallel@0.1.4-alpha.0


## create-factory@0.3.1-alpha.4


## @mastra/code-sdk@1.15.1-alpha.5


## mastracode@0.48.1-alpha.4


## @mastra/arize@1.3.27-alpha.2


## @mastra/arthur@0.4.27-alpha.2


## @mastra/braintrust@1.3.24-alpha.2


## @mastra/datadog@1.4.19-alpha.2


## @mastra/deepeval@0.1.21-alpha.2


## @mastra/laminar@1.3.29-alpha.2


## @mastra/langfuse@1.6.3-alpha.2


## @mastra/observability@1.21.1-alpha.2


## @mastra/otel-bridge@1.5.19-alpha.2


## @mastra/otel-exporter@1.4.10-alpha.2


## @mastra/posthog@1.4.4-alpha.2


## @mastra/sentry@1.2.29-alpha.2


## @mastra/agent-builder@1.1.31-alpha.2


## @mastra/auth@1.1.5-alpha.0


## mastra@1.36.0-alpha.4


## @mastra/codemod@1.1.8-alpha.0


## create-mastra@1.36.0-alpha.4


## @mastra/deployer@1.80.0-alpha.4


## @mastra/editor@0.16.5-alpha.2


## @mastra/evals@1.10.9-alpha.0


## @mastra/fastembed@1.3.5-alpha.2


## @mastra/loggers@1.3.10-alpha.1


## @mastra/mcp@2.6.0-alpha.1


## @mastra/mcp-docs-server@1.3.10-alpha.7


## @mastra/mcp-registry-registry@1.1.7-alpha.0


## @mastra/memory@1.39.1-alpha.2


## @mastra/rag@2.7.1-alpha.0


## @mastra/schema-compat@1.3.17-alpha.0


## @mastra/server@1.80.0-alpha.4


## @mastra/google-cloud-pubsub@1.1.5-alpha.1


## @mastra/redis-streams@0.6.5-alpha.1


## @mastra/valkey-streams@0.5.7-alpha.1


## @mastra/elysia@0.1.20-alpha.4


## @mastra/express@1.5.24-alpha.4


## @mastra/fastify@1.5.24-alpha.4


## @mastra/hono@1.7.22-alpha.4


## @mastra/koa@1.7.24-alpha.4


## @mastra/nestjs@0.2.39-alpha.4


## @mastra/next@0.2.38-alpha.4


## @mastra/server-adapters-test-suite@0.1.4-alpha.2


## @mastra/tanstack-start@0.2.38-alpha.4


## @mastra/astra@1.1.3-alpha.0


## @mastra/azure-ai-search@0.1.2-alpha.1


## @mastra/chroma@1.1.5-alpha.0


## @mastra/clickhouse@1.27.0-alpha.1


## @mastra/cloudflare@1.7.6-alpha.0


## @mastra/cloudflare-d1@1.4.5-alpha.1


## @mastra/convex@1.7.6-alpha.0


## @mastra/couchbase@1.1.4-alpha.1


## @mastra/dsql@1.5.5-alpha.0


## @mastra/duckdb@1.16.0-alpha.1


## @mastra/dynamodb@1.4.5-alpha.1


## @mastra/elasticsearch@1.4.6-alpha.1


## @mastra/lance@1.5.2-alpha.1


## @mastra/libsql@1.27.1-alpha.0


## @mastra/mongodb@1.24.1-alpha.1


## @mastra/mssql@1.10.6-alpha.0


## @mastra/mysql@0.14.1-alpha.0


## @mastra/opensearch@1.1.4-alpha.1


## @mastra/oracledb@0.5.5-alpha.0


## @mastra/pg@1.33.0-alpha.2


## @mastra/pinecone@1.1.3-alpha.0


## @mastra/qdrant@1.1.6-alpha.0


## @mastra/redis@1.4.12-alpha.1


## @mastra/s3vectors@1.1.5-alpha.0


## @mastra/spanner@1.10.1-alpha.1


## @mastra/turbopuffer@1.2.3-alpha.0


## @mastra/turso@0.1.18-alpha.0


## @mastra/upstash@1.5.5-alpha.1


## @mastra/valkey@0.2.8-alpha.1


## @mastra/vectorize@1.1.4-alpha.0


## @mastra/weaviate@0.1.4-alpha.0


## @mastra/voice-aws-nova-sonic@0.2.6-alpha.0


## @mastra/voice-azure@0.12.3-alpha.0


## @mastra/voice-cloudflare@0.13.4-alpha.0


## @mastra/voice-deepgram@0.13.3-alpha.0


## @mastra/voice-elevenlabs@0.13.3-alpha.0


## @mastra/voice-gladia@0.13.3-alpha.0


## @mastra/voice-google@0.14.4-alpha.0


## @mastra/voice-google-gemini-live@0.14.18-alpha.0


## @mastra/voice-inworld@0.4.6-alpha.0


## @mastra/voice-mistral@0.1.3-alpha.0


## @mastra/voice-modelslab@0.2.3-alpha.0


## @mastra/voice-murf@0.13.3-alpha.0


## @mastra/voice-openai@0.13.4-alpha.0


## @mastra/voice-openai-realtime@0.14.8-alpha.0


## @mastra/voice-playai@0.13.3-alpha.0


## @mastra/voice-sarvam@1.1.4-alpha.0


## @mastra/voice-speechify@0.14.3-alpha.0


## @mastra/voice-xai-realtime@0.2.17-alpha.0


## @mastra/inngest@1.10.8-alpha.3


## @mastra/temporal@0.4.18-alpha.4


## @mastra/agentcore@0.5.3-alpha.0


## @mastra/agentfs@0.2.3-alpha.0


## @mastra/apple-container@0.5.2-alpha.0


## @mastra/archil@0.3.2-alpha.0


## @mastra/azure@0.3.3-alpha.1


## @mastra/blaxel@0.10.2-alpha.0


## @mastra/cloudflare-sandbox@0.5.4-alpha.0


## @mastra/daytona@0.11.4-alpha.0


## @mastra/docker@0.12.1-alpha.0


## @mastra/e2b@0.15.1-alpha.1


## @mastra/e2b-desktop@0.1.10-alpha.1


## @mastra/files-sdk@0.3.3-alpha.0


## @mastra/gcs@0.3.5-alpha.1


## @mastra/google-drive@0.2.4-alpha.0


## @mastra/mesa@0.3.3-alpha.0


## @mastra/modal@0.7.3-alpha.0


## @mastra/platform-workspace@1.10.0-alpha.1


## @mastra/railway@0.10.2-alpha.1


## @mastra/s3@0.6.5-alpha.1


## @mastra/sprites@0.1.2-alpha.0


## @mastra/vercel@1.6.2-alpha.0


## @internal/playground@1.36.0-alpha.4

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-10-26 13:49:15 -07:00
mastra-platform[bot]
f51a29e82b
fix(core): settle evented workflow runs cancelled from another instance (#28434)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9

Closes #28427

## Summary

On the evented workflow engine, cancelling a run from a different Mastra
instance than the one running it (for example, the server cancel route
landing on a non-owner replica) marked the run `canceled` in storage.
The owner's `run.start()` promise never resolved, though. The same hang
happened for runs started with `startAsync()` and then cancelled.

**Cause:** `EventedRun.cancel()` wrote `canceled` to storage and then
called `abortController.abort()`. It relied on an abort listener to
publish `workflow.cancel`, but only `start()` and `resume()` register
that listener. A run handle created any other way (another instance, or
`startAsync()`) published nothing. The owner's event processor then
dropped the run's later events through the canceled-status gate. As a
result, `workflow.end` was never emitted and `start()` stayed pending.

**Fix:**
- `EventedRun.cancel()` now publishes `workflow.cancel` directly through
a `publishCancel()` helper. The abort listener uses the same helper, and
it publishes at most once per run handle.
- `WorkflowEventProcessor` tracks the controllers it has already
cancelled (`cancelHandledControllers`, a WeakSet). A step that unwinds
on one of those controllers no longer publishes a second
`workflow.cancel`, so a cancelled run emits exactly one `workflow.end`.
This also removes a duplicate `workflow.cancel`/`workflow.end` pair that
already happened on main whenever the owner cancelled its own run.

## Proof

New tests:
- `packages/core/src/workflows/evented/remote-cancel.test.ts` (12
tests): two Mastra instances sharing storage cancel across instances
over `EventEmitterPubSub` and `UnixSocketPubSub`. The tests cover
dountil, foreach, sequential, sleep and parallel shapes. The owner's
`start()` settles as `canceled` and the step's abortSignal fires. Two
more tests cancel after `startAsync()` and after an owner-local cancel,
and each asserts exactly one `workflow.cancel` and one `workflow.end`.
- `packages/server/src/server/handlers/workflows.test.ts`:
`CANCEL_WORKFLOW_RUN_ROUTE` is called on a non-owner instance and the
owner's run settles as `canceled`.
- `pubsub/redis-streams/src/evented-remote-cancel.test.ts` (4 tests):
cross-instance cancel over real Redis Streams, with each replica on its
own connection. Shapes are sequential and dountil, with workers on the
owner only and on both replicas.

Results with the fix reverted:
- Full revert: all 12 core tests fail, the server test times out
(`expected 'timed out' to be 'canceled'`), and all 4 Redis tests time
out.
- Reverting only `workflow.ts`: 11 of the 12 core tests fail.
- Reverting only the processor guard: the 2 single-event tests fail
(they get 2 events instead of 1).

With the fix, everything passes:
- `pnpm build:core`
- `tsc --noEmit` on core
- eslint and oxlint (type-aware) on the changed files
- oxfmt
- core `src/workflows`: 102 files; `workflow.test.ts` (329 passed) and
`evented-workflow.test.ts` (315 passed) were rerun alone after a heap
OOM when three suites ran in parallel
- server `workflows.test.ts`: 92 passed
- Redis: 4 of 4 passed against a local redis-server on port 6381

## Overlap with #24915

#24915 also edits `EventedRun.cancel()`. It calls `setupAbortHandler()`
before `abort()`, guarded by `#abortHandlerRegistered`. That would also
make cancel publish from a non-owner handle, so the two fixes overlap
and will conflict on the same lines. Whichever PR merges second should
keep a single publish path (`publishCancel()` here). The processor-side
duplicate-end guard in this PR is independent of #24915.

## Unverified

- **Multi-replica step abort (pre-existing, not fixed here):** when
every replica runs workers, `workflow.cancel` reaches only one consumer
in the group. If that consumer isn't the replica running the step, the
step's abortSignal doesn't fire. `start()` still settles as `canceled`.
Main behaves the same, even for local cancels. The Redis and server
tests check step abort only when the owner is the sole worker. This
should be tracked as a separate issue.
- **Inngest and durable/evented agents:** I didn't run these. From
reading the code, Inngest settles through `getRunOutput` on any terminal
status, and durable agents use their own `agent.control.{runId}` abort
transport.

## Risk: low

The change is limited to the evented engine's cancel path. The publish
is guarded to run at most once per handle. The WeakSet guard is
per-process, so if two replicas process cancel for the same run at the
same moment, the run could still end twice. That is no worse than main.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

When one Mastra instance cancels a workflow running on another, the
workflow owner’s `run.start()` can now finish with a canceled result.
The change also prevents duplicate cancellation messages from creating
duplicate end events.

## What changed

- `EventedRun` publishes `workflow.cancel` from both explicit
cancellation and the abort handler through one helper. The helper
prevents duplicate publishes and allows a retry if publishing fails.
- `WorkflowEventProcessor` tracks controllers already handled by a
cancellation event. When a step unwinds on one of those controllers, it
does not publish another cancellation event.
- Added tests for remote cancellation, `startAsync()`, server
cancellation, Redis Streams, and retry after a failed cancellation
publish.

## Validation

The author reports passing core build, type-check, lint, formatting, and
workflow tests. The author also reports passing server workflow tests
and Redis Streams tests. These results are author-reported.

## Limitations

When workers run on multiple replicas, a cancellation event may reach a
processor other than the one running the active step. That step may not
receive the abort signal. This case remains unfixed and unverified.
Inngest and durable/evented agents were not tested.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-26 12:19:03 -05:00