0
Fork 0
mirror of https://github.com/mastra-ai/mastra.git synced 2026-11-17 08:44:46 +00:00
Commit graph

5,186 commits

Author SHA1 Message Date
shreyas-mastra
68f6924bf7 Merge main into security/agent-browser-cli 2026-11-16 10:34:55 -05:00
Ward Peeters
da04116579
feat(factory): let a person's reply wake a card's session while automated runs hold every slot (#30307)
## What

When a trusted collaborator comments on or reviews a card's GitHub pull
request or issue, Factory wakes the card's session even while automated
runs hold every run slot. Before, those replies were
`sendMessage`/`signalActivity` decisions that counted against the single
automated cap (`dispatcher.maxInFlight`, default 10), so a reply could
sit behind multi-hour triage runs.

## How

- **Provenance.** GitHub rule actors gain `human: true` when the webhook
`sender.type` is `User`. Bots, and Factory's synthetic reconciler events
(no `sender.type`), get `human: false`.
- **Classification** (`isInteractiveDecision`, `rules/dispatcher.ts`): a
decision is a reply only if it's `sendMessage` or `signalActivity`,
would wake the session (`idleBehavior` isn't `persist`), has an empty
causal chain (no rule passed it along, e.g. not a transition's message),
and its actor is a GitHub human who is trusted and not Factory.
Everything else stays automated. GitLab, Linear, Jira and incident.io
activity stays automated because those sources give no reliable
bot/human signal.
- **Lane.** Each process keeps a reserve of reply slots
(`dispatcher.maxInteractiveInFlight` /
`MASTRACODE_DISPATCH_MAX_INTERACTIVE_IN_FLIGHT`, default `2`, `0`
allowed). Claim order per tick: pending manual starts → replies (reserve
first, then borrowing free automated slots) → automated runs. The claim
filter now receives the row's `actor` and `causalChain` as well as the
decision payload, so no schema change.
- **Honest bounds.** The caps limit dispatcher slot occupancy, not total
running agents: `signalActivity` frees its slot once the session accepts
it, and UI/Slack messages plus approval notices to following sessions go
to the session directly and never wait for a slot. The README says so.
- **Run-queue API (additive).** Entries gain `lane`; the response gains
`queuedByLane`, and `slots` gains `interactive: { inUse, cap }`.
Existing fields keep their meaning. `position` now counts within a lane.
The board's queue menu lists waiting replies first, with a reply-slot
footer.

## Testing

- Dispatcher (LibSQL + throwaway Postgres): with automated cap 1 held by
a hanging run, a trusted person's PR reply (`sendMessage`) and issue
comment (`signalActivity`) are claimed and delivered. Automated runs,
bot replies, untrusted replies, Factory-authored replies, actors with no
`human` flag, rule-passed messages and `persist` activity all stay
pending. Other cases: the reserve caps at its size, replies borrow a
free automated slot ahead of automated runs, and the default reserve is
2. Red proof: disabling the classification fails 3 of the 4 lane tests.
- GitHub rules: `issue_comment` and `pull_request_review` from a User →
`human: true`; from a Bot or without `sender.type` → `human: false`.
- Approval notice to a following session commits nothing to the run
queue.
- MSW: queue menu groups replies apart from automated runs and shows
both slot footers.
- Gates: factory `check`, oxlint and oxfmt clean. Factory suites on
LibSQL: 942 passed. factory-ui typecheck and lint clean; unit tests 696
passed. The full MSW suite has 21 failures, the same as on main
(AppStartup, OMSection, MemorySettings, NewSessionStarters,
ThreadPageTeammateSession, TranscriptMessageMeta). `mastracode/web`
index.test.ts has 2 failures that are also on main. Postgres runs of the
full dispatcher suite at `--maxWorkers=3` show the timing flakes main
already has; every test that failed passes on its own.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

**ELI5:** When a trusted person replies to a GitHub issue or pull
request, that reply gets its own reserved run slots. It no longer waits
behind routine automated runs.

## Overview

This change adds an interactive lane to the Factory decision dispatcher.
Trusted GitHub human replies can wake a card's session while automated
runs hold the automated slots. The run-queue API changes are additive.

## Classification

`isInteractiveDecision` in `mastracode/factory/src/rules/dispatcher.ts`
marks a decision as interactive only when all of these hold:

- The decision is `sendMessage` or `signalActivity`.
- The decision wakes the session.
- The causal chain is empty.
- The decision does not persist.
- The actor is a trusted GitHub human.
- The actor is not Factory-authored.

All other decisions use the automated lane. This includes bot actors,
untrusted actors, Factory events, and activity from GitLab, Linear,
Jira, and incident.io.

## GitHub actor changes

- `mastracode/factory/src/rules/types.ts` adds optional `human?:
boolean` to the GitHub variant of `FactoryRuleActor`.
- `mastracode/factory/src/integrations/github/rules.ts` sets `human` to
`true` only when the webhook sender type is exactly `User`.
- Trust lookup and its failure handling are unchanged. A lookup error
still marks the actor untrusted.

## Dispatcher behavior

- The default interactive reserve is 2 slots per process.
- `dispatcher.maxInteractiveInFlight` in
`mastracode/factory/src/factory.ts` and
`MASTRACODE_DISPATCH_MAX_INTERACTIVE_IN_FLIGHT` in
`mastracode/web/src/mastra/index.ts` set the reserve.
- A value of `0` disables the reserve. Invalid values use the default of
2.
- Each tick claims in this order: pending manual starts, then
interactive decisions, then automated deferred runs.
- Interactive decisions can use free automated slots.
- `runSlots()` returns overall `inUse` and `cap`, plus `interactive: {
inUse, cap }`.
- The caps limit dispatcher-held slots. They do not limit all running
agents.
- Pull-request activity releases its slot when the session accepts it.
- Direct session messages and approval notices do not wait for a slot.

## Storage

- `FactoryDeferredDecisionClaimInput.decisionFilter` receives a second
argument, `provenance`.
- The new `FactoryDecisionProvenance` type contains `actor` and
`causalChain`.
- No schema change is made. Invalid stored actor JSON parses to `null`.
Invalid causal-chain JSON parses to an empty array.

## Run-queue API (additive)

- `runQueueResponseSchema` in
`mastracode/factory/src/routes/contracts.ts` adds:
  - `slots.interactive: { inUse, cap }`
  - `queuedByLane: { automated, interactive }`
- Each queue entry has a `lane` field with the value `automated` or
`interactive`.
- `position` counts within a lane.
- The listing limit of 50 entries is unchanged. Waiting replies are
listed before automated runs.
- The `runSlots` types in `routes/surface.ts` and `routes/work-items.ts`
include the interactive pool.

## Board UI

- `RunQueueIndicator.tsx` lists waiting replies before automated runs.
- Each nonempty group has a label.
- The menu shows reply-slot usage next to overall slot usage.
- The tooltip fallback reads "Runs waiting for a run slot."
- `msw-server.ts` and `BoardPageRunQueue.msw.test.tsx` add lane fixtures
and a test for a waiting reply.

## Documentation and release notes

- `mastracode/factory/README.md`,
`docs/src/mastra-code/configuration.mdx`, and two changesets describe
the reserve, its default, the `0` behavior, the queue fields, and the
board ordering.
- The `mastra` changeset is a patch. The `@mastra/factory` changeset is
a minor release.

## Tests

Added or updated tests cover:

- GitHub actor `human` classification for issue comments and PR reviews,
for `User`, `Bot`, and missing sender types.
- Dispatcher reserve use, interactive cap overflow, borrowing of free
automated slots, and exclusions. Exclusions include automated, bot,
untrusted, Factory-authored, untyped, causally moved, and persisted
messages.
- The default interactive cap of 2.
- The polling-loop claim count, which changes from two to three per
tick.
- Factory config forwarding in
`mastracode/web/src/mastra/index.test.ts`, which now expects
`maxInteractiveInFlight: 0` when the environment variable is `0`.
- Clearing `MASTRACODE_DISPATCH_MAX_INTERACTIVE_IN_FLIGHT` in
sandbox-provider tests.
- No deferred decisions recorded after approval-notification delivery,
in `approval-answer-resume.test.ts`.

## Validation (author-reported)

The PR objectives report the following results. They were not checked
against completed CI checks on the current head.

- Dispatcher tests were run on LibSQL and throwaway Postgres. Full
Postgres dispatcher runs at `--maxWorkers=3` showed timing flakes. The
author reports that each failing test passes when run alone.
- Factory suites on LibSQL report 942 passing tests.
- Factory `check`, oxlint, and oxfmt are reported clean.
- factory-ui unit tests report 696 passing tests.
- The author reports 21 full MSW suite failures and 2 `mastracode/web`
`index.test.ts` failures. The author states these also occur on `main`.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-16 16:12:18 +01:00
Tyler Barnes
adc0a70bf8
fix(memory, couchbase): check semantic recall scope in Memory and warn on ignored Couchbase filters (#30446) 2026-11-15 14:29:24 -07:00
mastra-platform[bot]
c8a35ac624
fix(memory): keep Observational Memory local when the gateway has no memory API (#30414)
> Opened by @abhiaiyer91

> Opened by @factory-project:7d5175c3-3a20-42ac-aba9-756f183136f4

## Summary

Keep Observational Memory local for agents using
`mastra/<provider>/<model>` with gateway organization keys
(`msk-gateway-*` and `mskv2_*`), because those keys do not provide
gateway-managed memory. Gateway project keys (`msk_*`) continue to use
gateway-managed memory. Finish an open local turn before switching to
gateway-managed memory.

Credential ownership checks report local memory only for models that are
not on the Mastra gateway. A credential the gateway cannot resolve —
including one it reports as missing — is thrown instead of read as local
ownership: guessing local ownership can cause both the gateway and local
store to persist the same thread, because the model call resolves the
same credential again. The Observational Memory processor also
propagates ownership-check errors. Older models that cannot report
gateway memory ownership retain their previous behavior.

Docs explain the gateway-key distinction and the paired `@mastra/core`
and `@mastra/memory` upgrade. Project keys routed BYOK or with gateway
memory disabled remain outside this change's scope.

## Verification (head `67ca1ea9c4`)

- Core gateway-memory predicate tests: 11/11; memory gateway ownership
and persistence tests: 14/14.
- Full `@mastra/memory` suite: 119 files, 2,680 passed, 1 todo; core LLM
suite: 45 files, 550 passed, 2 skipped.
- Core and memory TypeScript checks passed.
- Reverting only the two behavior files makes the new regression tests
fail (after `a0a4d3f6`: core 2 failed/8 passed, memory 2 failed/11
passed; after `1981f9ca`: core 2 failed/9 passed, memory 1 failed/13
passed), confirming the tests detect the bug.

Fixes #30391

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

ELI5: Mastra has two kinds of gateway keys. One kind lets the gateway
save chat memory. The other kind does not. This change makes Mastra save
and recall chat messages locally when the gateway does not save them.

## Summary

This change fixes memory loss for agents that use a
`mastra/<provider>/<model>` model. Before this change, `@mastra/memory`
skipped local Observational Memory (OM) and message persistence for
every `mastra/` model. Messages were then not saved, and the agent could
not recall earlier turns.

The change decides gateway memory ownership by key type:

- Gateway project keys (`msk_*`) use gateway-managed memory.
- Gateway organization keys (`msk-gateway-*` and `mskv2_*`) use local
OM.
- Models that cannot report ownership keep their previous behavior.

Gateway-managed memory is not active when a project key routes to a BYOK
provider or when gateway memory is disabled. The docs describe this
distinction.

## Changes by area

**`@mastra/core`**
- `packages/core/src/llm/model/gateways/mastra.ts`: adds
`gatewayServesObservationalMemory(apiKey)`. It returns `false` for
missing, non-string, or empty keys and for `msk-gateway-` and `mskv2_`
keys. It returns `true` only for `msk_` keys.
- `packages/core/src/llm/model/gateways/index.ts`: re-exports
`gatewayServesObservationalMemory`.
- `packages/core/src/llm/model/router.ts`: adds
`ModelRouterLanguageModel.isGatewayMemoryManaged(): Promise<boolean>`.
It returns `false` for non-Mastra gateways. For Mastra gateways, it
resolves the route and auth, then checks the API key.
Credential-resolution errors propagate to the caller.

**`@mastra/memory`**
- `packages/memory/src/processors/observational-memory/processor.ts`:
uses `isGatewayMemoryManaged()` when the model provides it. If the model
does not provide it, the code falls back to the `mastra/` prefix check.
The result is stored per step.
- For gateway-managed input, the processor ends any open local turn and
skips local processing.
- For gateway-managed output, the processor ends any open local turn
before it returns.
- If ending a turn fails, the processor persists the unsaved messages
directly when possible. It then rethrows the original error.
- Context-overflow recovery remains skipped for gateway-managed memory.
- Ownership-check errors propagate. The processor does not treat them as
`false`.

**Documentation and changesets**
- `docs/src/content/en/docs/memory/observational-memory.mdx`: adds a
"Gateway models" section.
- `docs/src/content/en/models/gateways/mastra.mdx`: explains the key and
routing conditions for OM.
- `.changeset/tidy-geese-report.md`: minor bump for `@mastra/core`.
- `.changeset/wild-pumas-recall.md`: patch bump for `@mastra/memory`. It
tells users to upgrade `@mastra/core` and `@mastra/memory` together.
Older `@mastra/core` versions do not report ownership, so
`@mastra/memory` keeps its previous behavior for them.

**Tests**
- `packages/core/src/llm/model/router-gateway-memory.test.ts`: covers
project and organization keys, explicit-key precedence over the
environment key, models served by another gateway, credential lookup
errors, and lookups that fail once and then succeed.
- `packages/memory/src/gateway-memory-persistence.test.ts`: end-to-end
tests with real agent, router, memory, and in-memory storage.
Organization keys store messages and recall them. Project keys store no
messages locally. A failed ownership lookup rejects generation and
stores no assistant text.
-
`packages/memory/src/processors/observational-memory/__tests__/gateway-skip.test.ts`:
covers ownership results, legacy `gatewayId: 'mastra'` behavior,
rejected checks, and ownership changes during a run. It also covers
drain storage failures.

## Compatibility

- The change needs paired `@mastra/core` and `@mastra/memory` releases.
An older `@mastra/core` cannot report gateway memory ownership.
- Models that cannot report ownership keep the previous behavior.

## Testing

The following results are reported by the PR author. They were not
verified in this review, and no CI results were available.

- Core gateway-memory predicate tests: 10 of 10 passing.
- Memory gateway ownership and persistence tests: 13 of 13 passing.
- Full `@mastra/memory` suite: 119 files, 2,679 passed, 1 todo.
- Core LLM suite: 45 files, 549 passed, 2 skipped.
- Core and memory TypeScript checks: passing.
- The author reports that reverting the two behavior files makes the new
regression tests fail.

## Linked issue

Issue `#30391` reports that gateway-model messages are not saved or
recalled. The issue asks that messages be saved and recalled, or that
local processing be skipped only when gateway-side memory is active.
This change follows the second option for organization keys. It keeps
local processing for those keys. It skips local processing only for
project keys that the gateway manages.

The issue also reports a separate 422 error on the OpenAI-compatible URL
workaround. This change does not address that error.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform <284800079+mastra-platform@users.noreply.github.com>
2026-11-15 12:00:06 -08:00
mastra-platform[bot]
b666fdbf03
fix(client-js): stop retrying state-changing requests by default (#30434)
> Opened by @factory-project:7d5175c3-3a20-42ac-aba9-756f183136f4

The JavaScript client retried every HTTP method the same way, so a
`POST`, `PUT`, `PATCH`, or `DELETE` that hit a server error was silently
re-sent up to four times. When the server had already committed the work
before failing, that replay duplicated side effects — agent generations,
workflow starts, and similar writes.

`BaseResource.request()` now resolves retries by method: `GET`, `HEAD`,
and `OPTIONS` keep the default of three attempts, while every other
method is attempted once unless the caller opts in with an explicit
`retries` count on the client or the call. Requests that pass no method
keep the previous read behavior.

Four observability queries only read server state even though they
travel as `POST`, so `aggregateTraces()`, `aggregateSpans()`,
`queryTraceThreads()`, and `querySpans()` keep the read default. An
explicit `retries` on the client still wins.

**Approved** — head `a2e46c41b1` (branch
`factory/issue-30404-3ca17cc5ca80`, base `5b7f1302`).

### Checks on this head

| Check | Result | Tests |
| --- | --- | --- |
| `@mastra/client-js` tests + typecheck (`npx vitest run`) | ✅ Passed |
910 passed across 48 files; typecheck no errors |
| Type-aware lint (`npx oxlint --type-aware .`) | ✅ Passed | 0 warnings,
0 errors |
| New tests on base code | ✅ Catches the bug | 4 of 4 write cases fail
on base, pass on head |
| Independent reproduction through the public client API against a real
500 server | ✅ Passed | base repeats POST/PATCH/PUT/DELETE and
`agent.generate` 4× each; head sends each once, GET still 4×, opt-in
counts unchanged |
| Changeset package validation | ✅ Passed | `@mastra/client-js` patch
accepted |

### Follow-ups

- `queryTraces()` is the one read-style `POST` left on the
single-attempt default
(`client-sdks/client-js/src/resources/observability.ts:338`). The four
queries the review named keep the read default, so the `@mastra/react`
hooks that pin React Query `retry: false` on them
(`client-sdks/react/src/hooks/traces/use-trace-aggregate.ts:30`,
`use-span-aggregate.ts:29`, `use-query-spans.ts:64`,
`use-query-threads.ts:67`) still get the client retry.
- `client-sdks/react/src/hooks/traces/use-trace-aggregate.ts:17` and
`use-span-aggregate.ts:16` say "the client already retries server
errors", which holds again for these endpoints on head `5ba90bf1`, so
those comments need no change.
- `docs/src/content/en/reference/client-js/observability.mdx:523` still
recommends `{ retries: 0, anchorRetryTimeoutMs: 0 }` for PUT feedback
replacement; `retries: 0` is now the PUT default, so only the
`anchorRetryTimeoutMs: 0` half still matters.

### Adversarial check

Most damaging reading — that this silently strips the only retry layer
from shipped `@mastra/react` observability queries and breaks the
contract those hooks document — fails on the evidence: the loss is
limited to read-style POST endpoints and is the method-based gating the
issue asked for, those hooks live in a package the approved plan scoped
out, and self-review treats adjacent hardening as a follow-up rather
than a blocking finding.

### Follow-up commit

`5ba90bf15a` adds the fix for the review
finding above: the four read-only observability queries pass `retries:
this.readRetries()` (new `BaseResource` helper, explicit client
`retries` wins), 5 new tests fail on `a2e46c41` and pass here, and the
playground-ui metrics tests that the write-only default broke pass
again.

Fixes #30404

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

ELI5: The JavaScript client used to resend failed requests up to three
times, even for actions that change data, like creating or deleting
things. Now, only read requests are retried by default. Change requests
are sent once unless the developer asks for retries.

## Summary

This pull request changes the default retry behavior of
`BaseResource.request()` in `@mastra/client-js`. The change stops
implicit retries of state-changing requests after a server error. A
server error does not show whether the server applied the change before
it failed, so a retry can repeat a side effect.

## Behavior change

- `GET`, `HEAD`, and `OPTIONS` keep the default of three retries.
- `POST`, `PUT`, `PATCH`, `DELETE`, and any other unrecognized method
send one attempt by default.
- Requests without a method use `GET`, so they keep the read-style
default of three retries. The method is normalized to upper case before
the lookup.
- An explicit `retries` value on the request overrides the client-level
value. The client-level value overrides the method default.
- Existing validation of the retry count is unchanged. A non-negative
safe integer is still required.
- Four read-only observability queries use `POST`: `aggregateTraces`,
`aggregateSpans`, `queryTraceThreads`, and `querySpans`. These queries
keep the read-style default of three retries. They do not follow the
write default.

## Implementation

- `client-sdks/client-js/src/resources/base.ts`:
- Adds a default of three retries for read methods and a set of read
methods (`GET`, `HEAD`, `OPTIONS`).
- Resolves retries in this order: request-level value, then client-level
value, then the method default. Other methods default to zero.
- Adds a protected `readRetries()` helper. It returns the
client-configured retry count, or three when none is set.
- `client-sdks/client-js/src/resources/observability.ts`: The four
observability queries listed above pass `readRetries()` as their retry
count. `aggregateTraces` still forwards its optional abort signal. The
other three calls do not add a signal option.
- `client-sdks/client-js/src/types.ts`: Updates the JSDoc for
`ClientOptions.retries` and `RequestOptions.retries`. The docs state the
read-only default. They also state that a per-request count overrides
the client count and enables retries for writes. Declarations are
unchanged.
- `client-sdks/client-js/src/resources/base.test.ts`: Adds 52 lines of
tests. The tests cover:
- One attempt for `POST`, `PATCH`, `PUT`, and `DELETE` after a 500
response, with no retry count configured.
  - Three attempts for read methods.
- Explicit client-level and request-level retry counts that enable
retries for writes.
- `client-sdks/client-js/src/resources/observability.test.ts`: Adds
tests that the four observability queries make four fetch attempts after
a server error. Adds tests that `retries: 0` on `aggregateTraces` limits
the request to one attempt. Moves the thread-query and span-query
request fixtures into their `describe` scopes. Payloads are unchanged.
- `client-sdks/client-js/src/resources/agent.client-tools.test.ts`:
Removes a mocked continuation response from one streaming test. The test
expects no recursive request.
- `.changeset/witty-wolves-write-once.md`: Adds a patch changeset for
`@mastra/client-js`.
- `docs/src/content/en/docs/server/mastra-client.mdx` and
`docs/src/content/en/reference/client-js/mastra-client.mdx`: Document
the per-method default. The reference page changes the `retries` default
to `3 for read requests, 0 for write requests`. The server page notes
that the listed observability queries keep the read default.

## Diff scope

The diff covers 9 files, with 200 insertions and 53 deletions. Six files
are in `client-sdks/client-js`, with 190 insertions and 51 deletions.
The changeset and two docs files make up the rest.

## Verification

The PR author reports that `npx vitest run` passed 910 tests across 48
files, with no typecheck errors. The author also reports that type-aware
lint and changeset validation passed. These results come from the PR
description. This summary did not rerun them.

## Notes

- The PR description lists `@mastra/react` hooks that use observability
POST endpoints as a follow-up. Hooks that set `retry: false`, and two
hook comments, may describe retry behavior that no longer applies. This
summary did not verify those files.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-15 10:55:47 -08:00
mastra-platform[bot]
a48d697b35
feat(e2b): run commands as another OS user, and choose the owner of S3/GCS mounts (#30433)
> Opened by @factory-project:7d5175c3-3a20-42ac-aba9-756f183136f4

## Description

E2B sandboxes ran every agent command as the template's default OS user,
and S3/GCS FUSE mounts were always owned by that same user. That user
has to be privileged — mounting FUSE filesystems needs `sudo`, and the
staged s3fs credentials file (`/tmp/.passwd-s3fs-*`, mode 600) is only
readable by it — so an agent running shell commands as that user could
read bucket credentials or become root.

Two opt-in options fix that:

- **`E2BSandbox({ user })`** — every command the sandbox runs on the
agent's behalf runs as that OS user: `executeCommand`, background
processes, and the tree-kill script that takes their descendants down
(another user cannot signal them). Mounting still runs as the default
user, because it needs `sudo`.
- **`S3Filesystem({ owner })` / `GCSFilesystem({ owner })`** — `{ uid,
gid }` naming the owner of the files inside the FUSE mount, replacing
the `id -u && id -g` probe. This is what gives the command user read and
write access to the mount.

Both are additive and default to today's behavior. `owner` lives on the
filesystem options rather than the sandbox because the mount config is
produced by `filesystem.getMountConfig()`, which is what E2B's `mount()`
reads; `E2BSandboxOptions.user` reaches the provider through the shared
`CommandsApiProcessManager`, and the key is left off entirely when unset
so the SDK's own default user still applies. `owner` is validated as
non-negative integers before it is interpolated into a mount command.
The `user` must already exist in the sandbox template.

## Related issue(s)

Fixes #30349

## Type of change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [ ] Test update

## Checklist

- [x] I have linked the related issue(s) in the description above
- [x] I have made corresponding changes to the documentation (if
applicable)
- [ ] I have addressed all Coderabbit comments on this PR

## Test plan

All checks run on this head (`dec4e6ec`):

- `packages/_internals/workspace`: 160 tests, including a new test that
the spawn and the kill script both carry the configured user, and one
that the key stays unset without a user.
- `@mastra/e2b` unit suite: 367 passed, 9 skipped (`validateOwner`
cases; S3 and GCS mounts using an explicit owner skip the `id -u && id
-g` probe and emit `uid=`/`gid=` and `--uid=`/`--gid=`; mounts without
an owner still probe).
- `@mastra/s3` 150 passed, `@mastra/gcs` 91 passed, including
`getMountConfig()` carrying `owner` when set and omitting it when not.
- Typecheck (e2b, s3, gcs, platform-workspace) clean; lint 0 warnings /
0 errors and builds clean for e2b, s3 and gcs.
- Each new test fails against the base source without the change (10 in
e2b, 1 internal, 1 s3, 1 gcs).
- Integration coverage: `index.integration.test.ts` gained a case
asserting an explicit owner yields `1234:1234` from `stat -c %u:%g`. It
needs `E2B_API_KEY`, so it does not run in CI — a manual run on a real
sandbox is the remaining gap.

## Follow-ups (not part of this change)

1. File-API writes (`sandbox.files`, `FilesApiFilesystem`) and code-mode
staging under `/home/user` still run as the default user, so with `user`
set, files written through file tools are not owned by the command user.
2. `E2BProviderConfig.configSchema` does not list `user` (it already
omits `workingDirectory`, `reattach` and `lifecycle`), so the Studio
provider form does not offer it even though the runtime forwards it.
3. `commands.list()` scope relative to the configured user is unverified
without an API key; tracked handles still surface.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

**ELI5:** Agent commands in an E2B sandbox used to run as the same user
that mounts cloud storage. This PR lets you pick a separate,
less-privileged user for agent commands, and lets you set the file owner
for S3 and GCS mounts. Both options are off by default.

## Summary

This PR adds two opt-in E2B options, as described in issue `#30349`:

- **Command user.** `E2BSandbox({ user })` runs agent commands,
background processes, and descendant cleanup as the named OS user.
Mounting still uses the default user. The user must already exist in the
sandbox template.
- **Mount owner.** `S3Filesystem({ owner })` and `GCSFilesystem({ owner
})` accept `{ uid, gid }`. When set, the mount uses these IDs. When
unset, the existing `id -u && id -g` probe runs.

### Changes by area

- **Workspace process manager**
(`packages/_internals/workspace/src/commands-api-process-manager.ts`):
`CommandsApi.run` and `CommandsApiProcessManagerOptions` accept an
optional `user`. Spawned commands and descendant cleanup in `kill()` use
it.
- **E2B provider** (`workspaces/e2b/src/sandbox/`):
  - `index.ts` adds `user?: string` to `E2BSandboxOptions`.
  - `process-manager.ts` forwards `user` to the base class.
- `mounts/types.ts` adds `MountOwner` and `validateOwner`. Validation
requires non-negative integers for `uid` and `gid`.
- `mounts/s3.ts` and `mounts/gcs.ts` use a supplied owner. Otherwise,
they probe the default user.
  - `index.ts` re-exports `MountOwner`.
- **S3 and GCS filesystems** (`workspaces/s3/src/filesystem/index.ts`,
`workspaces/gcs/src/filesystem/index.ts`): add `MountOwner`, an optional
`owner` option, and a `readonly owner` property. `getMountConfig()`
copies `uid` and `gid` into a plain object only when an owner is set.
This copy also handles owners whose fields are prototype getters. Both
packages re-export `MountOwner`.
- **Docs and changeset:** document both options. The changeset sets
minor releases for `@mastra/e2b`, `@mastra/s3`, and `@mastra/gcs`.

### Tests

- Workspace tests check that a configured user reaches spawned commands
and their kill commands. They also check that an unset user is passed as
`undefined`.
- E2B unit tests check owner validation and mount behavior with and
without an owner.
- S3 and GCS tests check `getMountConfig()` with and without an owner.
They also check that later changes to the input owner do not change the
config.
- An E2B integration test checks that an explicit owner of `1234` mounts
with ownership `1234:1234`. This test requires `E2B_API_KEY` and was not
run in CI.

The author reports passing workspace, E2B, S3, and GCS tests. The author
also reports clean typecheck, lint, and build results. These results are
not tied to a completed CI run on the current head. A manual run against
a real E2B sandbox is outstanding.

### Out of scope (per PR description)

- File-API writes and code-mode staging still use the default user.
- The Studio configuration schema does not expose `user`.
- The scope of `commands.list()` relative to the configured user is not
verified.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-15 08:51:28 -08:00
mastra-platform[bot]
82c98f9f1c
fix(rag): keep long JSON string content when chunking JSON documents (#30407)
> Opened by @factory-project:7d5175c3-3a20-42ac-aba9-756f183136f4

JSON chunking (`MDocument.chunk({ strategy: 'json' })`) silently
discarded most of a long string value whenever the field's serialized
path was longer than 13 characters, or the value contained characters
that JSON escapes. The splitter budgeted a fixed 20-character overhead
regardless of the real field name, and then dropped every piece that
still exceeded `maxSize`. The result was silent data loss with no error,
no warning, and no config that avoided it — a 3000-character value under
a 16-character key kept 120 characters at `maxSize` 500, and a
14-character key at `maxSize` 21 produced zero chunks.

The fix sizes each piece from the value it is about to emit: it measures
the serialized envelope of the candidate piece and its full field path,
cuts only at code-point boundaries so astral characters are never split,
and never discards a completed piece. When a field path alone leaves no
room for a single character, the remainder is emitted as one oversized
chunk instead of being dropped.

**Result:** Approved — `d8caff7436`

**Checks on this head**

| Check | Result | Tests |
| --- | --- | --- |
| New #30405 regression tests
(`packages/rag/src/document/document.test.ts`) | ✅ Passed | 10 passed |
| Full `packages/rag` suite | ✅ Passed | 471 passed across 24 files |
| `tsc --noEmit` (`packages/rag`) | ✅ Passed | |
| `oxlint --type-aware .` (`packages/rag`) | ✅ Passed | 0 warnings, 0
errors |
| `oxfmt --check` on the changed files | ✅ Passed | |
| New tests on base code (`5b7f1302`) | ✅ Catches the bug | 8 of 10 fail
without the fix |
| Independent round-trip probe (16 cases) | ✅ Passed | base: 9 fail →
head: 16 pass |

**Follow-ups**

- `minSize` remains accepted but ignored for JSON chunking; already
documented in `docs/src/content/en/reference/rag/chunk.mdx`.
- Long strings nested two or more levels deep are still emitted as one
oversized chunk rather than split; this boundary is unchanged by the fix
and already described in the `maxSize` docs.
- `ensureAscii` escaping runs after sizing, so an ASCII-escaped chunk
can exceed `maxSize`; unchanged and out of scope.
- `handlePrimitiveValue` tests "fits" against `{ [key]: value }`
(immediate key only) while emitting with the full path; this predates
the change and cannot lose or oversize data, since a chunk that fits
without the path also fits as emitted.

Adversarial check: I tried to falsify the fix with inputs the author did
not test — a 100k-character value (round-trips, 103 chunks, 11 ms),
astral characters (no chunk cut mid-pair), space-heavy values, nested
and array shapes, and the smallest allowed `maxSize` (21) — and the
round-trip invariant held on every case, while a probe written before
reading the diff reproduced the content loss on the base commit.

Fixes #30405

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

JSON chunking could drop parts of long text values. This change keeps
the full text and splits it into smaller pieces. Most pieces now fit the
size limit. A few cases still produce pieces that are larger than the
limit.

## Summary

This change fixes content loss in `MDocument.chunk({ strategy: 'json'
})` for `@mastra/rag`. The splitter reserved a fixed 20-character
overhead for each long string value. Field names or JSON escape
sequences can use more space than that overhead. The splitter then
discarded the pieces that exceeded `maxSize`.

## Changes

- `packages/rag/src/document/transformers/json.ts` (+51/-20):
- `RecursiveJsonTransformer` sizes each candidate piece by its
serialized chunk and full field path. It no longer uses the fixed
20-character overhead.
- Pieces split at code-point boundaries. The splitter prefers a
preceding space.
- If the field path leaves no room for one character, the remaining
string is emitted as one oversized chunk.
  - The filter that removed string chunks above `maxSize` was removed.
- Non-string primitive values that do not fit in a new chunk are
appended without a serialized-size check.
- `packages/rag/src/document/document.test.ts` (+108): Adds JSON
chunking tests. They cover long field names, escaped string values,
surrogate pairs, non-string primitives, and the no-room case. A
1,000,000-character string test checks reconstruction, a 500-character
chunk limit, and completion within 1,000 ms.
- `docs/src/content/en/reference/rag/chunk.mdx` (+1/-1): Documents that
long string values are split without discarding content. It states that
chunks may exceed `maxSize` when the field path leaves no room for one
character, when `ensureAscii` escaping expands a value (the default), or
when a long value is nested in an object.
- `.changeset/smooth-donkeys-split.md` (+5): Adds a patch changeset for
`@mastra/rag`.

## Linked issue

Issue `#30405` reports content loss with `maxSize: 90`, `minSize: 1`,
and `ensureAscii: false`. The issue expects complete reconstruction. It
also expects serialized chunk sizes within `maxSize`. It asks for a
configuration error when the field name leaves no room for a character.

This change does not implement the configuration error. When the field
path leaves no room, it emits the remaining content as one oversized
chunk.

## Limits

- Long strings nested two or more levels deep remain one oversized
chunk.
- `ensureAscii` escaping can still produce a chunk larger than
`maxSize`.
- When the field path leaves no room for one character, the remainder is
emitted as one oversized chunk.

## Verification

The author reports the following results on the PR head:
- 10 new regression tests pass. 8 of them fail on the base commit.
- The `packages/rag` suite passes: 471 tests across 24 files.
- `tsc --noEmit` and `oxlint --type-aware .` pass in `packages/rag`.
`oxfmt --check` passes on changed files.
- A 16-case round-trip probe passes on the PR head. The same probe fails
9 times on the base commit.

These results were not re-run for this summary. CI status on the current
head: not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-15 08:44:43 -08:00
mastra-platform[bot]
0170d700a4
feat(mcp): let MCPServer set maxRequestBodySize for HTTP requests (#30411)
> Opened by @factory-project:7d5175c3-3a20-42ac-aba9-756f183136f4

## Description

Every MCP HTTP request served by a Mastra `MCPServer` was capped at the
MCP SDK's fixed 4 MiB default. A client sending a larger `tools/call`
body received

```json
{"jsonrpc":"2.0","error":{"code":-32000,"message":"Payload Too Large: Request body must not exceed 4194304 bytes"},"id":null}
```

before the tool ever ran, with no way to raise the bound. That breaks
legitimate payloads — for example a Claude connector MCP App intake form
that submits a signed PDF and a photo of an ID in a single `tools/call`.

`MCPServer` now accepts an optional `maxRequestBodySize` (bytes) in its
config, next to `legacy`:

```typescript
const server = new MCPServer({
  name: 'intake',
  version: '1.0.0',
  maxRequestBodySize: 16 * 1024 * 1024,
  tools: { submitIntake },
});
```

The option is forwarded to both SDK layers that enforce the cap:
`createMcpHandler(...)` owns the handler's bound, and
`toNodeHandler(...)` owns an adapter-level bound —
`@modelcontextprotocol/node` buffers the Node request stream itself
before the handler sees it, and per its
`ToNodeHandlerOptions.maxRequestBodySize` docs "the adapter's bound
applies first". Passing it to only one layer leaves the other's 4 MiB
default in force.

Leaving the option unset keeps the SDK default of `4194304` (4 MiB), so
existing servers behave exactly as before, and an over-limit body is
still rejected with the same `413` before it is parsed. The option
applies to `startHTTP` only; `stdio` has no request bodies. Documented
in the `MCPServer` configuration table of
`docs/src/content/en/reference/tools/mcp-server.mdx`.

## Test plan

New `packages/mcp/src/server/server-http-body-size.test.ts` drives the
real `startHTTP` path through the shared harness:

- default server rejects a 5 MiB body with `413`, `-32000`, and the
default byte count in the message, without running the tool;
- `maxRequestBodySize: 8 MiB` accepts a 5 MiB body and the tool runs;
- `maxRequestBodySize: 1024` rejects a 2 KiB body with `413` naming the
configured limit;
- a body under the configured limit is accepted.

## Verification

| Category | Command | Result |
| --- | --- | --- |
| Package tests | `pnpm exec vitest run --project unit:packages/mcp
src/server` | ✅ 36 files, 669 passed, 8 skipped |
| Package tests | `pnpm exec vitest run --project unit:packages/mcp
src/client` | ✅ 34 files, 627 passed |
| Type check / build | `pnpm --filter @mastra/mcp build:lib` | ✅ Passed;
emits `maxRequestBodySize?: number` in `dist/server/server.d.ts` |
| Lint | `npx oxlint --type-aware <changed .ts>` | ✅ 0 warnings, 0
errors |
| Format | `npx oxfmt --check <changed .ts>` | ✅ Passed |
| New tests on base code | new tests against the pre-change `server.ts`
| ✅ 2 of 4 fail on base (`expected 413, received 200`), 4 of 4 pass on
head |
| Docs checks | `run-checks.sh --docs
docs/src/content/en/reference/tools/mcp-server.mdx` | ✅ format, remark,
validate pass |
| Public surface | scratch script importing the built `@mastra/mcp` | ✅
default → `413`, raised limit passes the size gate |
| Not run | Vale prose check | ⚠️ binary not installed, so prose was not
linted |

One full-suite run failed on `server-events.test.ts` (stdio events). The
same test also fails at the base commit under the same full-suite load
and passes on re-run and in isolation, so it is a pre-existing flake
rather than a regression; this change does not touch the `serveStdio`
path.

## Follow-ups (non-blocking)

- A non-positive or non-finite `maxRequestBodySize` is not validated at
construction; the SDK's `RangeError` surfaces on the first HTTP request
as a `500`. Constructor-time validation would fail faster.
- The built-in `/api/mcp` endpoint (`server.mcp` / `ServerMCPConfig`)
and adapter-level `MCPOptions` still expose no body-size option; this
change deliberately scopes to a user-instantiated `MCPServer`.

Fixes #30207


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

ELI5: MCP servers reject requests bigger than 4 MB. This PR lets a
developer set a bigger limit for one server, so large uploads can reach
the tool.

## Summary

This PR adds an optional `maxRequestBodySize` setting to
`MCPServerConfig`. `MCPServer` stores the value and passes it to both
`createMcpHandler(...)` and `toNodeHandler(...)`. When the setting is
unset, the SDK default of 4 MiB (4194304 bytes) applies.

## Changes

- `packages/mcp/src/server/server.ts`: Adds `maxRequestBodySize?:
number` to `MCPServerConfig`. Adds a private readonly field and assigns
it from config. Passes the value to both HTTP handler layers. Adds 12
lines.
- `packages/mcp/src/server/server-http-body-size.test.ts`: Adds 109
lines and 4 test cases:
- A body over the default 4 MiB limit returns `413`, and the tool does
not run.
- A body larger than 4 MiB succeeds when `maxRequestBodySize` raises the
limit.
- A body over a lowered limit returns `413`, and the message includes
the configured limit.
  - A body under the configured limit succeeds.
- `docs/src/content/en/reference/tools/mcp-server.mdx`: Documents the
option in the configuration table. Adds 7 lines.
- `.changeset/olive-pandas-carry.md`: Adds a minor release note for
`@mastra/mcp`. Adds 20 lines.

## Behavior

- The limit is in bytes.
- The documentation states that the value must be positive. The diff
does not validate this. The code stores any value it receives.
- An oversized body receives a `413` JSON-RPC error before it is parsed.
No tool runs.
- The limit applies only to bodies the server reads from the request
stream. When body-parsing middleware supplies the parsed body, the limit
does not apply. The documentation tells the user to set the limit on
that parser too.
- The setting does not apply to `stdio`.

## Scope

- The PR does not add constructor-time validation for non-positive or
non-finite values.
- The PR does not add body-size options to the built-in `/api/mcp`
endpoint or to adapter-level `MCPOptions`.

## Diff

4 files changed, 148 lines added, 0 removed.

## Verification

The PR author reports these results. This review did not run them.
- `vitest run --project unit:packages/mcp src/server`: 36 files, 669
passed, 8 skipped.
- `vitest run --project unit:packages/mcp src/client`: 34 files, 627
passed.
- `pnpm --filter @mastra/mcp build:lib`: passed.
- Vale prose check: not run. The binary was unavailable.
- One full-suite failure in `server-events.test.ts`. The author reports
the same failure on the base commit under full-suite load. The author
reports that it passes on rerun and in isolation.

## Linked issue

Issue `#30207` requests a per-server `maxRequestBodySize` on
`MCPServer`, passed to `createMcpHandler`. The PR adds the constructor
option and forwards it to both handler layers. The diff and test names
cover the requested behavior: the default is kept when unset, a `413` is
returned above the limit, and the setting applies per server.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
2026-11-15 08:41:25 -08:00
Arthur Fiorette
33eaa293e2
fix(core): give unnamed sleeps the same step id in every process (#30397)
## Description

A `.sleep()` or `.sleepUntil()` declared without an `id` now gets the
same step ID in every process: `sleep_<workflowId>_<n>`, where `n`
starts at the number of sleeps declared before this one and moves up
past any ID already in use.

### Why

Each process generated a random ID for an unnamed sleep when it built
the workflow, and that ID is the key of the sleep's result in the stored
run. A process that restarts or time travels a run stored by another
process looked the sleep up under its own ID, found nothing, and the
step after the sleep got no input. On the evented engine a restart
failed the run with an input validation error (the step received
`undefined`). Time travel on either engine ran the step with `{}`.

Scheduled workflows were affected too: the schedule's definition hash
covers the serialized step graph, which includes the sleep IDs, so every
instance computed a different hash and the instances that hadn't
registered the schedule last refused to fire it.

### What changed

`packages/core/src/workflows/workflow.ts` uses the rule unnamed `.map()`
steps have used since #21222: a deterministic ID from the workflow ID
and an ordinal, unless the workflow is already registered with a Mastra
instance that has a custom ID generator when the sleep is declared, in
which case that generator still produces the part after `sleep_`.
`sleep` and `sleepUntil` share one count, since both use the `sleep_`
prefix. An ID already taken by an explicit one is skipped. An explicit
`id` is unchanged.

```ts
// before: sleep_3f2c9e1a-... (different in every process)
// after:  sleep_my-workflow_0
createWorkflow({ id: 'my-workflow', ... }).then(a).sleep(5000).then(b).commit();
```

The `sleep` and `sleepUntil` reference pages now describe the default
ID.

### Known limits

- Adding or removing a sleep shifts the default IDs of later unnamed
sleeps. Pass `{ id }` to pin one.
- Runs stored before the upgrade keep their random sleep IDs and still
won't match after a restart or time travel, as before.
- A custom ID generator that returns random IDs still gives a different
ID per process.

### Tests

- New `packages/core/src/workflows/sleep-id-determinism.test.ts`: two
builds of the same workflow give the same IDs with one ordinal over
`sleep` and `sleepUntil` (without the fix: random `sleep_<uuid>` IDs);
an explicit ID that claims a default name is skipped (without the fix: a
random ID); a default-engine time travel past a sleep after the workflow
is rebuilt, on `InMemoryStore`, hands the next step its recorded input
(without the fix: the run ends `failed` with "Step input validation
failed", because the step got `{}`).
- `packages/core/src/workflows/evented/sleep-restart.test.ts`: the sleep
and `sleepUntil` restart cases now build the workflow once per runtime,
as a real restart does, and assert the step after the sleep receives its
input (without the fix: the run ends `failed` with
`WORKFLOW_STEP_INPUT_VALIDATION_FAILED`, input `undefined`).

## Related issue(s)

None

## Type of change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [ ] Test update

## Checklist

- [ ] I have linked the related issue(s) in the description above
- [x] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

**ELI5:** When a workflow waits, it gives that wait a name. Before this
change, the name was random each time the code ran. Another process
could not find the saved wait, so a restarted or time-traveled run lost
its data. Now each unnamed wait gets a name based on its position in the
workflow, so every process uses the same name.

## Summary

Unnamed `.sleep()` and `.sleepUntil()` steps now get a deterministic ID.
The ID has the form `sleep_<workflowId>_<n>`. Both sleep types share one
counter. The counter skips IDs that already exist in the workflow's
steps. Explicit `{ id }` values are unchanged.

## Changes

- **`packages/core/src/workflows/workflow.ts`:** Adds a private
`#defaultSleepId(stepType)` helper. `sleep()` and `sleepUntil()` call it
when no explicit ID is given.
- With no custom ID generator on the Mastra instance, `n` equals the
number of earlier `sleep` and `sleepUntil` entries in `stepFlow`. The
helper increments `n` while `sleep_<workflowId>_<n>` already exists in
`steps`.
- With a custom ID generator, the helper calls `generateId` with the
workflow ID and the step type. It then prefixes the result with
`sleep_`. The generator supplies the suffix.
  - The `crypto.randomUUID()` fallback is removed from both methods.
- **`.changeset/curly-pillows-argue.md`:** Adds a `@mastra/core` patch
changeset. It describes the fix and states these limits:
- Runs stored before this change keep their random IDs. They do not
match after a restart or time travel.
- Scheduled workflows with unnamed sleeps get the same definition hash
on each instance.
- Adding or removing a sleep changes the IDs of later unnamed sleeps.
Pass `{ id }` to keep an ID fixed.

## Documentation and tests

- The raw summary reports that the `sleep` and `sleepUntil` reference
pages describe the ID format, the counter, and the custom ID generator
behavior.
- The raw summary reports a new
`packages/core/src/workflows/sleep-id-determinism.test.ts`. It covers
stable IDs, the shared counter, explicit-ID collisions, and a
rebuild-and-time-travel case.
- The raw summary reports that `sleep-restart.test.ts` now creates a
separate workflow object for each runtime. It asserts that the
`afterSleep` step receives `{ value: 'persist me' }`.
- Verification status: the doc pages and test files were not inspected.
Test results are not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-14 22:44:11 -03:00
mastra-platform[bot]
dbbe40f04e
docs(observability): document Agnost AI OpenTelemetry exporter setup (#30408)
> Opened by @factory-project:7d5175c3-3a20-42ac-aba9-756f183136f4

## Summary

The OpenTelemetry exporter page lists platforms that work through the
generic `custom` provider but leaves out Agnost AI. This adds it: Agnost
AI joins the intro platform list and the HTTP/Protobuf installation
heading, and a new section documents the configuration.

Agnost AI ingests OTLP/HTTP protobuf traces at
`https://otel.agnost.ai/v1/traces` and identifies the organization with
an `X-Agnost-Org-ID` header read from `AGNOST_ORG_ID`. It stores traces
only, so the example sets `signals: { logs: false }` to skip log export,
which the exporter otherwise enables by default against a derived
`/v1/logs` endpoint.

Docs-only, one file, +27/−2. Mirrors the Sazabi section added in #28637
and sits next to it, before Dash0.

Fixes #30406

## Review

**Result:** Approved f16115e9cc

### Checks on this head

| Check | Result |
| - | - |
| Docs checks, whole tree (`pnpm check --no-vale`) | ✅ Passed: 0 errors,
0 warnings, 1008 files |
| Docs check on the changed file (frontmatter, sidebar, mdx, vale,
format, redirects) | ✅ Passed: 0 findings |
| Vale CI gate (`.vale-ci.ini`, `--minAlertLevel=error`) on the changed
file | ✅ Passed |
| Vale on the changed file with `integrations` added to the config glob,
compared against the base version | ✅ Passed: same 4 pre-existing
findings, none from the new section |
| Documented config booted on the real `OtelExporter` (`new` + `init` +
await setup) | ✅ Passed: endpoint `https://otel.agnost.ai/v1/traces`,
protocol `http/protobuf`, `X-Agnost-Org-ID` from `AGNOST_ORG_ID`, traces
set up, no log provider; a control without `signals` does set up log
export |
| Documented snippet typechecked against the package's public types
(`tsc --noEmit --strict`) | ✅ Passed |
| Vendor links resolve (`https://agnost.ai`,
`https://docs.agnost.ai/mastra`) | ✅ Passed: both 200 |
| `git diff --check` | ✅ Passed |

**Not run:** full Docusaurus site build. It is heavy and the MDX compile
check covers the edited file, which is the same gap the Sazabi precedent
(PR #28637) left open.

### Verification notes

- `signals` is a top-level sibling of `provider` in
`OtelExporterConfig`, not nested inside `provider.custom`. The section
places it correctly, and `signals: { logs: false }` is what actually
skips log export (`setupAllSignals` gates logs on `signals?.logs !==
false`). Without it the exporter derives `/v1/logs` from the trace
endpoint and sets up log export, exactly as the page's own Signals
section describes.
- `resolveCustomConfig` accepts `endpoint`, `protocol`, and `headers`
together and defaults `headers` to `{}`, so the `X-Agnost-Org-ID` header
reaches the OTLP exporter.
- `tracingOptions.metadata` is real and reaches spans: `getOrCreateSpan`
merges it into span metadata and `span-converter` emits each key as
`mastra.metadata.<key>`, so the grouping explanation matches the
implementation.
- Scope matches the approved plan: one MDX file, +27/−2, no source,
sidebar, redirect, or changeset changes. Placement after Sazabi and
before Dash0 groups it with the other `custom`-provider vendors.

### Follow-ups

- `.vale.ini` only globs `src/content/en/{docs,guides,reference}`, so
prose rules never run on `integrations/` pages. With `integrations` in
the glob this file carries 4 pre-existing error-level findings (line 16
Metacommentary, line 558 SemicolonUsage, line 768 Contractions twice);
none come from this change.
- `### Protocol selection guide` lists no `custom`-provider vendors
(Sazabi, telemetry.dev, MLflow, Latitude), so Agnost AI is left out too,
consistent with existing practice.

**Adversarial check:** tried to break the central claim by booting the
real exporter with the documented config and with a control that omits
`signals`, and by checking the header default path; the section is
accurate in both cases.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

ELI5: This change adds setup steps for Agnost AI to the OpenTelemetry
exporter docs. Users can now send Mastra traces to Agnost AI with a
short config.

Technical summary:

This PR updates one documentation file:
`docs/src/content/en/integrations/observability/opentelemetry.mdx`. It
makes no code changes.

Changes:
- Adds Agnost AI to the exporter platform list in the intro.
- Adds Agnost AI to the "For HTTP/Protobuf providers" installation
heading.
- Adds a new `#### Agnost AI` section after the Sazabi section. The
section covers:
- The `custom` provider with `endpoint:
'https://otel.agnost.ai/v1/traces'` and `protocol: 'http/protobuf'`.
  - An `X-Agnost-Org-ID` header read from `AGNOST_ORG_ID`.
  - `signals: { logs: false }`, because Agnost AI stores traces only.
- A note that Agnost AI reads `userId` and `conversationId` (or
`threadId`) from `tracingOptions.metadata` to group traces.
  - A link to Agnost AI's Mastra guide.

Diff size: 1 file changed, 27 lines added, 2 lines removed.

Verification: The diff confirms the changes listed above. The PR
objectives report a docs check (`pnpm check --no-vale`) and an MDX
compile check on the edited file. The full Docusaurus site build was not
run. Results from CI on the current head are not available.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-14 09:06:26 -08:00
Arthur Fiorette
951a18b085
feat(core): background process exit metadata (#30161)
Ignore the rest of the stack, just this one is open for review for now
because it will help me on workflow-chaos

---

## Description

`execute_command` runs a background process and returns its PID right
away, and `backgroundProcesses` callbacks let a host watch it. Until now
those callbacks only got the `pid` and `toolCallId`, so a host had no
way to tell which conversation started the process or why it stopped.

The callbacks now also get `agentId`, `threadId`, `resourceId` and
`runId` when they're available, and `onExit` gets `killedByAbort`, which
is `true` when the abort signal fired (the run's signal by default) and
killed the process. A host can send an exit notice to the right thread
and skip processes the abort killed:

```ts
const workspace = new Workspace({
  sandbox,
  tools: {
    [WORKSPACE_TOOLS.SANDBOX.EXECUTE_COMMAND]: {
      backgroundProcesses: {
        onExit: ({ pid, exitCode, threadId, killedByAbort }) => {
          if (!killedByAbort) notifyThread(threadId, `Process ${pid} exited with code ${exitCode}`);
        },
      },
    },
  },
})
```

`truncateOutput` and `DEFAULT_TAIL_LINES` are now exported from
`@mastra/core/workspace` so a host can trim the output in that notice
the same way `execute_command` does.

Tests run real `LocalSandbox` processes and cover an abort kill, a
normal exit, `kill_process`, an agent with no id, and a direct
`execute()` call that passes the ids at the top level.

## Related issue(s)

First PR of a stack for background shells. There's no tracking issue.
Next in the stack is #30176.

## Type of change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [ ] Test update

## Checklist

- [ ] I have linked the related issue(s) in the description above
- [x] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

**ELI5:** Background commands can now tell their callbacks which agent,
chat thread, user resource, and run started them. The exit callback also
says whether the run's abort signal stopped the command. Other code can
now reuse the output-trimming helper.

## Summary

This change adds origin metadata to `backgroundProcesses` callbacks for
background `execute_command` processes. It adds an exit flag that shows
abort-signal termination. It also exports two workspace helpers for host
code.

## Changes by file

- `packages/core/src/workspace/tools/types.ts`: Adds optional `agentId`,
`threadId`, `resourceId`, and `runId` to `BackgroundProcessMeta`. Adds
optional `killedByAbort` to `BackgroundProcessExitMeta`.
- `packages/core/src/workspace/tools/execute-command.ts`: Passes
`agentId`, `threadId`, `resourceId`, and `runId` to stdout, stderr, and
exit callbacks. Empty or missing IDs become `undefined`. Thread and
resource IDs fall back from `context.agent` to the top-level context.
- `packages/core/src/workspace/index.ts`: Exports `truncateOutput` and
`DEFAULT_TAIL_LINES` from `./tools`.
- `packages/core/src/workspace/tools/__tests__/sandbox-tools.test.ts`:
Adds tests for callback metadata, including a context with no agent ID.
Adds tests that expect `killedByAbort: true` when the spawning run's
abort kills the process. Adds tests that expect `false` for normal exits
and for `kill_process`. The tests use a Node-based long-running command
for cross-platform runs.
- `.changeset/six-mugs-lead.md`: Declares a minor `@mastra/core` release
that describes the new metadata and exports.
- `docs/src/content/en/docs/sandbox/overview.mdx`: Documents that
lifecycle callbacks receive `pid` and, when available, `toolCallId`,
`agentId`, `threadId`, `resourceId`, and `runId`. Documents
`killedByAbort` on `onExit`.
- `docs/src/content/en/reference/workspace/workspace-class.mdx`:
Documents stdout and stderr metadata. Documents exit metadata, including
`killedByAbort`.

## Public API changes

- `BackgroundProcessMeta` gains optional `agentId`, `threadId`,
`resourceId`, and `runId`.
- `BackgroundProcessExitMeta` gains optional `killedByAbort`.
- `@mastra/core/workspace` exports `truncateOutput` and
`DEFAULT_TAIL_LINES`.

## Review and verification status

- Review comments: The author requested CodeRabbit reviews three times.
The bot responses report completed reviews and contain no review
findings. Review severity counts: not available.
- CI and test results on the current head: not available.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-14 11:54:49 -03:00
Arthur Fiorette
f4458eea7f
fix(core): stop failing durable runs on a slow local ownership read (#30386)
### Description

A durable agent step whose process holds the run's claim no longer puts
a lease TTL deadline on its ownership checks. Such a step now stops only
when the claim itself is lost: a renewal fails or comes back superseded,
an ownership check fails, or another execution takes over the run. Steps
that only observe another process's claim (evented workers following a
claim held elsewhere) keep the deadline.

The change is in `ExecutionOwnershipGuard`
(`packages/core/src/agent/durable/execution-ownership-guard.ts`): the
deadline is armed only for remote guards, and the per-check re-arm for
local guards is gone. The "Stop work after ownership loss" passage in
`docs/src/content/en/docs/harness/durable-agents.mdx` is updated to
match.

### Why

Each fenced durable loop step checks ownership before and after it runs.
For a local guard, that check armed a timer of one lease TTL (15 s by
default, `MASTRA_AGENT_THREAD_LEASE_TTL_MS`) and then read the claim
with `getRunOwnership`. When that read took longer than the TTL, the
timer fired and the step threw `DURABLE_AGENT_EXECUTION_UNVERIFIED`, a
non-retryable error. The fence was never lost and the heartbeat kept
renewing the claim. Settlement then read ownership again, found this
execution still the owner, and settled the run as `owned`. It published
`DURABLE_AGENT_EXECUTION_UNVERIFIED` to the caller, deleted the run's
snapshots and released the claim. No other execution ever owned the run.

A local claim already has its own bound. The heartbeat renews it every 5
s by default (a third of the TTL) and marks the fence lost when renewals
fail past the lease deadline or come back superseded. The guard hears
about that loss through its `subscribe` listener. A check that throws
already marks the fence lost. So the local deadline only ever acted on a
read that was slow but would have answered "still yours".

### Known limits

- A cancelled run ends a pending check: `withExecutionFence` always
starts the ownership read but stops waiting for it when the run's abort
signal fires (or has already fired), and the step handles the abort as
it does elsewhere. A takeover that read finds still marks the fence
lost. Without a cancellation, a check read that never returns while
renewals keep succeeding still leaves the step waiting at that check,
holding a live claim. Before this change the step ended at one TTL with
`DURABLE_AGENT_EXECUTION_UNVERIFIED`, which failed the run and deleted
its snapshots.
- After a real takeover, the old execution is now stopped by its
heartbeat. The heartbeat checks on ticks, so on storage-backed claims
that is at most 2 x TTL plus one renewal interval after the last
confirmed renewal (about 35 s with defaults). On lease-backed claims it
is within one TTL plus one interval (about 20 s). Before, a step stopped
one TTL after its check started. Storage writes stay fenced by
generation; side effects outside Mastra don't.
- Claims on `NoopLeaseProvider` have no heartbeat and were never fenced.
Unchanged.
- `participateInDurableExecution` awaits `fence.verify()` before every
evented participant step
(`packages/core/src/agent/durable/evented-execution.ts:128`) with no
deadline and no abort race, so cancellation doesn't end a hanging read
there. Unchanged by this PR.
- On an evented worker's first fenced step of a run there is no registry
entry yet, so a remote cancel doesn't end a pre-check ownership read
that never returns. Losing the claim still does, and that worker's
earlier `fence.verify()` (`evented-execution.ts:128`) waits on the same
store with no deadline either.
- Remote guards are unchanged: they still stop after one TTL without a
successful check, even while verification is queued. That item stays
open in #28934.
- `settleEventedDurableExecution`
(`packages/core/src/agent/durable/evented-execution.ts:196`) still
returns early for fence errors, without emitting an error event,
deleting snapshots or releasing the claim. A local guard can now only
fail with `DURABLE_AGENT_EXECUTION_UNVERIFIED` when its fence is lost,
and for the coordinator's fence `watchExecutionLoss` then rejects the
waiting caller. Remote guards and participant fences in another process
can still reach that early return with a loss that `watchExecutionLoss`
doesn't report.

### Tests

-
`packages/core/src/agent/durable/__tests__/execution-ownership-guard.test.ts`,
"leaves a slow local read to the fence instead of a deadline": a local
guard whose read never returns is still not lost after 2 x TTL, and its
check is still pending. Firing the fence's loss listener then rejects
the check with that loss. Without the fix, the guard is already lost at
2 x TTL.
- `packages/core/src/agent/durable/execution-fence.test.ts`, "finishes a
step whose ownership read outlasts the lease TTL while the claim keeps
renewing": a storage-backed `ExecutionFence` on `InMemoryStore` with
real renewals under fake timers, and a `withExecutionFence` step whose
first `getRunOwnership` resolves after TTL + 1 s. The step returns its
output, the fence is not lost, the owner record keeps the same owner and
generation, and the run settles as `owned`. Without the fix, the step
rejects with `DURABLE_AGENT_EXECUTION_UNVERIFIED`.
- `packages/core/src/agent/durable/execution-fence.test.ts`, "ends a
cancelled step whose ownership read before/after its work never
returns": on the same storage-backed setup, the run's registry abort
signal fires while the pre-check or post-check read hangs, and the step
returns its abort output (pre-check) or its finished output (post-check)
with the fence not lost and the owner record unchanged. Without the
cancellation handling, the step call stays pending after the abort.
- `packages/core/src/agent/durable/execution-fence.test.ts`, "a step of
an already-cancelled run": the step returns its abort output without
waiting, the ownership read was still started, and a takeover that read
finds marks the fence lost before any heartbeat. "ends a cancelled
post-check on a worker whose step created the run registry entry": there
is no registry entry at the pre-check, the step creates it with
`ensureRemoteAbortListener` as the durable steps do, the post-check read
never returns, and a published abort request ends the step with its
output.

The whole `packages/core/src/agent/durable` suite passes.

### Related issue(s)

Related to #28934

### Type of change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [ ] Test update

### Checklist

- [x] I have linked the related issue(s) in the description above
- [x] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

ELI5: A durable agent run checks that it still owns its work. A slow
check used to make the run fail. Now a local run keeps going while its
claim stays valid, and it stops when the claim is lost, a check fails,
or another execution takes over.

## Overview

This change updates `ExecutionOwnershipGuard` in
`packages/core/src/agent/durable/execution-ownership-guard.ts`. Local
guards no longer apply a lease-TTL deadline to ownership checks. Remote
guards, which observe a claim held by another process, keep the
deadline. The change also adds a cancellation race in
`execution-fence.ts`, updates the durable agents documentation, and adds
a patch changeset.

## Changes

- **`execution-ownership-guard.ts`** (+2/-3): The guard arms a lease
deadline at construction only for remote observers. Local claims rely on
subscription notifications to report ownership loss. Local periodic
checks no longer arm a deadline before they run. They also no longer
clear a deadline after a successful observation.
- **`execution-fence.ts`** (+26/-2): Adds `checkUnlessAborted`. This
helper races an ownership check against an abort signal and removes its
abort listener afterward. It still throws if the guard recorded
ownership loss. `withExecutionFence` gets the signal from the run
registry, with `params.abortSignal` as fallback. It uses the helper
before and after each step.
- **`.changeset/thick-baboons-drum.md`** (+5): Adds a patch release note
for `@mastra/core`. Durable agent runs no longer fail with
`DURABLE_AGENT_EXECUTION_UNVERIFIED` and lose saved snapshots when an
ownership check exceeds the default 15-second lease TTL in the process
that holds the run.
- **`docs/src/content/en/docs/harness/durable-agents.mdx`** (+1/-1): The
"Stop work after ownership loss" section separates two cases:
- Steps in the process that holds the claim stop when an ownership check
fails or a renewal misses the lease expiry.
- Steps that only observe another process's claim stop when no check
succeeds within one lease TTL (15 seconds by default).
- **`execution-ownership-guard.test.ts`** (+25): Adds a test. A
`check()` that waits on a never-settling local ownership read stays
pending after 2× the lease TTL and does not mark ownership lost. When
the subscription reports supersession, the pending check completes with
the superseded error.
- **`execution-fence.test.ts`** (+188/-1): Adds storage-backed
`withExecutionFence` tests. They cover these cases:
- A step whose ownership read exceeds the lease TTL, while the claim
keeps renewing, returns its output. The run settles as owned.
- A step blocked before or after its work ends when the run is
cancelled. The fence stays intact.
- A step of an already-cancelled run returns without waiting for a
pending read.
- A cancelled post-check ends on a worker whose step created the run
registry entry.

## Known limits

- A local check that never returns stays pending while renewals succeed.
Cancellation ends the pending check through the abort race. Without
cancellation, the run waits for fence loss.
- Takeover detection depends on heartbeat timing.
- Remote guards keep their TTL deadline.
- Evented participant checks have no deadline or abort race.
- Storage writes stay fenced by generation. External side effects are
not fenced.

## Verification

The author reports that the `packages/core/src/agent/durable` test suite
passes. This summary does not independently confirm that result. Check
status on the current head is not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-14 00:59:18 -03:00
Arthur Fiorette
8acea4cc5b
fix(core): background task timeout 0 means none (#30381)
## Description

Setting a background task timeout to `0` killed the task right away: the
worker armed `setTimeout(abort, 0)`, so the task ended as `timed_out`
before it did any work.

Now `0` means no timeout. The task runs until it finishes or is
cancelled. This works on the tool config, the agent's per-tool config,
the enqueue payload, and the manager's `defaultTimeoutMs`. Other values
behave as before.

```ts
const longJob = createTool({
  id: 'long-job',
  // ...
  background: {
    enabled: true,
    // before: the task timed out immediately
    // after: the task has no timeout
    timeoutMs: 0,
  },
})
```

Also documents `0` on the related types and on the background tasks and
configuration docs pages.

## Related issue(s)

None. Standalone fix, independent of any other PR.

## Type of change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [ ] Test update

## Checklist

- [ ] I have linked the related issue(s) in the description above
- [x] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

**ELI5:** A background task can have a time limit. Before this change, a
limit of `0` stopped the task right away. After this change, `0` means
the task has no time limit, so it runs until it finishes or is
cancelled.

## Summary

This change makes a background task timeout of `0` mean "no timeout."
Other timeout values keep their existing behavior. The `0` rule applies
in four places:
- Tool configuration
- Agent per-tool configuration
- The enqueue payload
- The background task manager's `defaultTimeoutMs`

The change also alters how an `AbortError` from a tool is classified. A
tool that throws its own `AbortError` no longer counts as a timeout. The
task fails with the tool's own error. Retries apply if they are
configured.

The task's own signal aborts for these cases: timer expiry,
cancellation, workflow cancellation, shutdown, and ownership loss.
Executors receive that signal.

## Changes by file

- `packages/core/src/background-tasks/workflow.ts`: Does not create the
timeout timer when `task.timeoutMs` is `0`. For other values, the timer
still aborts the task after the configured duration. Timeout
classification now requires an aborted signal or the matching timeout
message. An `AbortError` name or the `"Task cancelled"` message alone no
longer counts as a timeout.
- `packages/core/src/background-tasks/types.ts`: Documentation comments
state that `0` means no timeout for task payloads, manager defaults, and
per-tool overrides. No runtime types changed.
- `packages/core/src/background-tasks/manager.test.ts`: Adds tests. One
test checks that `timeoutMs: 0` prevents a timeout after fake timers
advance 600 seconds, and that the task completes after its executor is
released. Another test checks that an executor's `AbortError` with
`timeoutMs: 0` results in status `failed` with the original error
message.
- `.changeset/wet-geckos-work.md`: Adds a patch changeset for
`@mastra/core`. It documents the `0` behavior and the `AbortError`
behavior.
- `docs/src/content/en/docs/harness/background-tasks.mdx`: The timeout
resolution order states that `0` means no timeout at the tool,
agent-tool, or manager-default level.
- `docs/src/content/en/reference/configuration.mdx`: The
`backgroundTasks.defaultTimeoutMs` entry states that `0` means no
timeout.
- `docs/src/content/en/reference/core/mastra-class.mdx`: The
`backgroundTasks.defaultTimeoutMs` description states that `0` means no
timeout.

## Public API

No exported declarations changed.

## Diff size

- 7 files changed
- 73 lines added, 11 lines removed

## Test and CI results

Not available. The new tests appear in the diff, but no completed test
or CI run results were supplied.

## Review status

Review finding counts are not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 19:35:43 -08:00
mastra-platform[bot]
ddb52d535d
Stop running unused Mastra Code session scorers (#30363)
> Opened by @abhiaiyer91

## Summary
- Stop registering the built-in outcome and efficiency scorers on Mastra
Code sessions by default.
- Allow SDK and Factory hosts to supply their own `scorers` to the
mounted coding agent.
- Preserve the public eval exports and `disableScoreStorage` behavior;
update tests, Code SDK documentation, and patch changesets.

## Test plan
- Independent review verified SDK tests (124 passed), Factory tests (162
passed), and the core live-scorer scenario (52 passed).
- SDK and Factory TypeScript checks passed; a controller probe confirmed
zero default scorers and registration of caller-provided scorers.
- In-memory score-storage tests cover default discarding and explicit
persistence.
- The full SDK suite previously exceeded the four-minute runner limit
without a reported test failure.

---------

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-13 14:40:12 -08:00
Arthur Fiorette
2788a6d959
feat(core): recover stopped workflow runs from a loop with one leader (#30281)
## Description

Restarts a default-engine workflow run whose instance crashed without
waiting for another instance to boot. `startWorkers()` now starts a
recovery loop on an instance that runs the orchestration worker, and one
instance among those that share a store runs its passes.

### The loop

Each pass applies the rule `restartAllActiveWorkflowRuns()` already uses
for default-engine runs: restart a `running` or `waiting` run once no
instance holds its run lease and `recovery.workflowGraceMs` passed since
the run or its lease last changed. The loop replaces the per-run recheck
timers the previous PR kept (`#scheduleWorkflowRecoveryRecheck`,
`#recheckWorkflowRun`); the durable-agent rechecks are unchanged.

The loop never touches evented runs. A redelivering broker recovers
them, and on other pubsubs the explicit `restartAllActiveWorkflowRuns()`
restarts them in the caller, as before. That call no longer joins a loop
pass for its evented part, so `startWorkers()` followed by the explicit
call still restarts evented runs on `EventEmitterPubSub`
(`evented/sleep-restart.test.ts` passes).

### Leader

One row, `::recovery-leader`, in the run-owners table, which gates the
loop's ticks only. Each tick of an instance tries a non-forced claim
until it gets the row, then keeps it: the lease helper renews it every
renew interval until `stopWorkers()` or `shutdown()` releases it. Only
the holder's loop lists runs; another instance's tick costs one refused
claim. When the leader dies, the row lapses after the lease TTL and the
next tick elsewhere takes it.

The leader saves listing cost only. Two restarts of one run are still
prevented by the run lease (`restart()` claims non-forced and throws
`WORKFLOW_RESTART_RUN_OWNED`).

An explicit `restartAllActiveWorkflowRuns()` never looks at the row. As
in the previous PR, each booting instance restarts its own stopped
default-engine runs (joining its own pass in flight), so another app's
workflows on the same store are recovered at its boot, and runs
recovered after a redeploy spread over the booting instances. Without
run fencing there is no leader and no loop, and the call restarts every
active run as before.

A failed probe, `getStore` or claim skips that tick and the loop keeps
going. A probe that answers "no fencing" stops the loop for good.

### Timings

- Interval: `recovery.workflowScanIntervalMs`, 60 s by default, each
wait up to 20 % shorter or longer, capped at 2^31 - 1 ms.
- A pass that saw a run in its grace period (and not live) moves the
next pass to that run's due time, no sooner than 1 s later and up to 20
% past it.
- A crashed instance's run is restarted about
`MASTRA_AGENT_THREAD_LEASE_TTL_MS` (15 s) plus `workflowGraceMs` (2 s)
after the crash, plus up to one interval for the next pass. When the
crashed instance was the leader, add up to one more interval for a
follower to take the row.
- `shutdown()` stops the loop when it starts, waits within
`drainTimeout` for the pass in flight and the restarts it handed out,
and releases the row in `stopWorkers()`.

### Config

```ts
const mastra = new Mastra({
  recovery: { workflowScanIntervalMs: 30_000 },
});
```

`recovery.workflows: 'off'` turns the loop off. `workers: false`, or a
`MASTRA_WORKERS` filter without `orchestration`, runs no loop. An
instance that registers no default-engine workflow to recover never
claims the leader row.

### LibSQL thread state writes

`@mastra/libsql` thread state writes (`setState`, `deleteState`) now
take the client write lock the store's other writes take. Without it, on
a local database file, a thread state write that ran while the same
store held a write transaction open (a lease claim or renewal) failed
with `SQLITE_BUSY`; the leader's renewals made that much more likely.

### Known limits

- A process that calls `restartAllActiveWorkflowRuns()` but never
`startWorkers()` runs no loop and doesn't recheck the runs it left
alone. A run its boot call found live or in its grace period is left to
the loop of another instance, if any.
- A leader whose renewals fail for longer than the lease TTL can run
passes alongside the new leader for up to one renew interval after the
store answers again. The run lease still prevents a run from being
restarted twice.
- Between boots, the loop recovers only the workflows the leader
registers. Instances that register different workflows on one store get
loop recovery for the leader's workflows only, and the others' runs at
their next boot.
- A run the loop restarts executes on the leader.
- A `mastra worker` process calls `stopWorkers()` on `SIGTERM` and exits
without `shutdown()`, so its default-engine runs wait as long as a
crashed instance's.
- A run that another instance won between the pass's lease read and
`restart()` is looked at again only at the next pass, up to one interval
later.
- Rolling deploys are out of scope for this stack.

### Tests

New `recovery-loop.test.ts` (in-memory store, 600 ms lease TTL):

- a crashed leader's run is restarted by the follower with no boot,
within TTL + 2 intervals + 1.2 s
- instances started together run passes on one of them, and the leader
keeps the row while passes are further apart than the lease TTL
- the leader's `stopWorkers()` releases the row and the follower takes
over
- a tick that `stopWorkers()` overtook (on the first tick's store read,
or on a later tick's probe) claims no row, so another instance can lead
- a stopped run of a workflow only another app registers is restarted by
that app's boot call while a different instance leads, and the leader's
loop never lists that workflow
- next-pass timing table (no run, live run, run in grace, grace ending
within 1 s), timers unref'd
- no listing after `stopWorkers()`; `shutdown()` waits for a pass in
flight
- a loop on `startWorkers('orchestration')`
- the loop keeps going after a failed probe, leader claim, listing or
restart
- no listing and no claim with `workflows: 'off'` or without run fencing

The recheck cases of `restart-active-runs-ownership.test.ts` and
`restart-active-runs-rule.test.ts` are gone with the rechecks: the crash
case covers a crashed owner, the failure table covers transient errors,
and the existing shutdown and limiter cases cover draining and
`workflowConcurrency`. The LibSQL cross-process cases and the MCP
survivor cases boot like a server (`startWorkers()`, then the explicit
call) and recover through the loop.

Checks: core `src/workflows src/mastra src/agent/durable src/events`
(only the two `scheduler.test.ts` retry cases that fail outside UTC),
core typecheck, LibSQL cross-process, Postgres and LibSQL
`recovery-pass.test.ts`, MCP `server-tasks-background`.

## Related issue(s)

Part of #28712. Stack 4 of 4, on top of the recovery pass PR.

## Type of change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [x] Test update

## Checklist

- [x] I have linked the related issue(s) in the description above
- [x] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

ELI5: Sometimes a workflow run stops when its server crashes. This
change adds a background check. One server regularly looks for stopped
runs and restarts them, so the work can finish without a manual restart.

## Overview

This change adds a workflow recovery loop to `@mastra/core`.
`startWorkers()` starts the loop. Among instances that share a store,
only the leader runs default-engine passes. A pass restarts
default-engine runs in `running` or `waiting` status when:

- no instance holds the run lease, and
- the configured grace period has elapsed.

The loop excludes evented runs. Explicit
`restartAllActiveWorkflowRuns()` calls handle evented runs and are not
leader-gated.

The change also updates LibSQL thread-state writes. `setState` and
`deleteState` now run through `withClientWriteLock`.

## Key changes

- `packages/core/src/mastra/index.ts`
- Adds `workflowScanIntervalMs` to `MastraRecoveryConfig`. The default
is 60000 ms. Values are capped at `2 ** 31 - 1`. A value of `-1`
resolves to the default.
- `startWorkers()` starts the loop when the selected workers handle
workflow events and recovery is enabled.
- When workflows use run fencing, the loop elects a leader through a
renewable lease. If fencing is unsupported, the loop stops.
- A pass records the earliest due time for runs still in their grace
period. The loop can then scan sooner.
- Failed lease reads leave runs for a later pass. Restart failures are
tracked per engine.
  - Per-run recheck, retry, and backoff logic is removed.
- `stopWorkers()` stops the loop and releases the leader lease. Shutdown
waits for an in-flight pass.
- Explicit `restartAllActiveWorkflowRuns()` calls share an in-flight
default-engine pass. Broker-redelivered evented runs stay broker-owned,
except process-local scheduled runs.
- `packages/core/src/workflows/run-ownership.ts`
- `repeatAfterSettled` accepts a function for the delay and an optional
`immediately` flag.
- Signature changes from `(tick, intervalMs: number)` to `(tick,
intervalMs: number | (() => number), options?)`.
- `stores/libsql/src/storage/domains/thread-state/index.ts`
- `setState` and `deleteState` use `withClientWriteLock`. SQL,
arguments, and error handling are unchanged.
- Tests
- Adds `packages/core/src/mastra/recovery-loop.test.ts`. It covers
leadership, scheduling, shutdown, and failure handling. The file is
added to `vitest.isolated-tests.ts`.
  - Removes three tests from `restart-active-runs-ownership.test.ts`.
- In `restart-active-runs-rule.test.ts`, removes one test and rewrites
the lease-read failure test to use the loop.
- Updates the recovery-defaults test, the MCP background server test,
and two LibSQL cross-process tests for the new scan interval.
- Adds a LibSQL thread-state test. It holds a write transaction open
during `setState` and `deleteState`.
- Documentation and changesets
- Updates the workers, server-adapter, workflow overview, and
`mastra-class` reference pages. Adds a "Recovery loop" section to the
workflow overview.
  - Corrects the crash-recovery text in the workers page.
  - Adds changesets for `@mastra/core` and `@mastra/libsql`.

## Public API changes

- Adds `workflowScanIntervalMs?: number` to `MastraRecoveryConfig`.
- Changes the signature of `repeatAfterSettled` in `run-ownership.ts`.

## Scope

- 19 files changed. 1028 lines added and 379 removed.

## Verification status

- Review findings: not available. No current review findings were
supplied.
- Author-reported checks: the PR objectives list core tests, core
typecheck, LibSQL cross-process tests, Postgres and LibSQL
`recovery-pass.test.ts`, and MCP `server-tasks-background`. These are
author reports. Results on the current head were not available for this
summary.
- Known failures: the author reports that two `scheduler.test.ts` retry
cases fail outside UTC.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 18:15:40 -03:00
Arthur Fiorette
6d14d0edc4
feat(core): recover an agent resume whose process died (#30330)
## Description

`approveToolCall()`, `declineToolCall()`, and `resumeStream()` kept the
decision in the memory of the process handling the resume. If that
process died after claiming the run and before the run finished, the
decision was gone: the run stayed claimed, nothing in storage could
finish it, and the thread kept showing the approval as pending.

Every resume claim now writes `resumeClaimIntent` (the resume data and
the targeted `toolCallId`) in the same compare-and-set that stamps
`resumeClaimedAt`. The new `agent.recoverResume({ runId })` reads that
intent and finishes the run through the existing takeover rules, so it
only takes over an abandoned claim (judged by claim age or lease expiry,
as for any takeover).

- The replay keeps the original target. A resume without a `toolCallId`
resumes every parked call, so storing the data alone would apply one
answer to sibling calls.
- A recovery only takes over the exact claim whose intent it read. If a
newer resume claimed the run in between, the recovery rejects with
`AGENT_RESUME_ALREADY_CLAIMED`, so a stale decision never overrides a
newer one.
- `resumeStream(undefined, ...)` keeps its meaning. Recovery is opt-in
through `recoverResume()`, which rejects with
`AGENT_RESUME_NOTHING_TO_RECOVER` when no resume claimed the run.
- No adapter changes: every adapter merges `updateWorkflowState` options
into the stored run, the same way `resumeClaimedAt` is stored.

`DurableAgent` has its own claim and is not covered here.

Supersedes #30321, which stored the data without its target and applied
a recovered answer to every parked tool call.

## Related issue(s)

Fixes #30285

## Type of change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [ ] Test update

## Checklist

- [x] I have linked the related issue(s) in the description above
- [x] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

**ELI5:** When a program stops during a user's approval, the approval
can be lost. This change saves the user's answer with the run. A new
recovery method can then finish the run without asking the user again.

## Overview

This change adds `agent.recoverResume({ runId })`. It replays a resume
that was claimed but not finished, for example after the process that
handled it stops.

## Behavior

- A resume claim stores `toolCallId` and `resumeData` in
`resumeClaimIntent`, together with `claimedAt`.
- The claim write sets `resumeClaimedAt` and `resumeClaimIntent` in one
update object. The code confirms that both fields are written together.
The storage layer's atomicity is not verified in this change.
- `recoverResume()` reads the stored intent. It takes over only the
claim whose `claimedAt` matches the snapshot's current
`resumeClaimedAt`.
- Recovery re-reads the run after it hands the run back. If the run no
longer carries the claim that recovery read, recovery fails with
`AGENT_RESUME_ALREADY_CLAIMED`. A newer pending claim can still be
displaced by the hand-back. That claim then fails in `start()`, and the
older decision does not run.
- If no resume has claimed the run, `recoverResume()` fails with
`AGENT_RESUME_NOTHING_TO_RECOVER`.
- `resumeStream(undefined, ...)` keeps its existing meaning. Recovery
runs only when a caller calls `recoverResume()`.
- The stored `toolCallId` targets one call. A recovered answer applies
only to that call, not to sibling parked calls.
- Recovery supports storage without run fencing for claims that never
started. This matches other takeovers.

## Documented takeover rules

The documentation states that recovery takes over a claim when either
condition holds:

- The claim was not started for 60 seconds.
- The run-fencing lease was not renewed for 60 seconds.

The documentation also notes that the resumed tool call runs again and
can repeat side effects.

## Changes by area

- **Core agent (`packages/core/src/agent/agent.ts`):** Adds
`recoverResume()`. Stores resume intent on each claim write. Passes
resume data into claim creation from `resumeStream()` and
`resumeGenerate()`. Adds the recovery path to the shared stream-resume
flow.
- **Until-idle helper
(`packages/core/src/agent/stream-until-idle.ts`):**
`runResumeStreamUntilIdle` accepts an optional `resume` callback for the
first turn. Without it, the function calls `agent.resumeStream` as
before. Continuation behavior is unchanged.
- **Types (`packages/core/src/workflows/types.ts`,
`packages/core/src/storage/types.ts`):** `WorkflowRunState` gains an
optional `resumeClaimIntent` with `claimedAt`, `toolCallId?`, and
`resumeData?`. `UpdateWorkflowStateOptions` accepts `resumeClaimIntent`.
- **Tests
(`packages/core/src/loop/test-utils/aimock/scenarios/resume-claim-after-crash.scenario.test.ts`):**
Adds parallel suspended tool-call support. Adds recovery tests for:
  - Replaying a dead claimant's decision.
  - Targeting only the answered call.
  - Keeping a newer decision when recovery runs late.
  - Refusing to recover a live claim.
  - Returning `AGENT_RESUME_NOTHING_TO_RECOVER` when no decision exists.
- **Documentation and changeset:** Adds a section on recovering a
decision after a crash. The changeset declares a minor release for
`@mastra/core`.

## Scope and limits

- `DurableAgent` is not covered.
- No storage adapter changes are required.
- Public API addition: `recoverResume()` on `Agent`.

## Relation to linked issue

The linked issue asks that approval decisions survive a crash after a
claim. This change persists the decision with the claim and replays it
through `recoverResume()`. It does not add a separate
`getPendingResume(runId)` helper.

## Verification

- CI and test results: not available. No completed check results were
supplied for the current head.
- Review severity counts: not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 17:42:51 -03:00
Daniel Lew
7fd6830a26
fix(core): carry request context to a wake started by a claimed owner on another server (#29447)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-13 13:55:18 -05:00
mastra-platform[bot]
70f3eba5ec
docs: fix inaccurate examples and defaults across workflows, processors, Studio, workers, sandbox, observability, channels and Next.js guide (#30331) 2026-11-13 11:38:15 -07:00
Arthur Fiorette
99ec3bd582
fix(core): leave evented workflow runs to a redelivering broker and restart only stopped runs (#30279)
## Description

This PR stops `restartAllActiveWorkflowRuns()` from restarting evented
runs that a redelivering broker (Redis Streams, Valkey Streams, Google
Cloud Pub/Sub) already recovers. It also restarts a default-engine run
only once no instance holds its run lease and a grace period (2 s by
default) has passed. Before, every booting instance restarted every
active evented run, so a step interrupted by a crash ran once per
booting pod (#28712). The broker already recovers those runs: Redis and
Valkey Streams reclaim an unacked message once it has been idle for
`reclaimIdleMs`, and Google Cloud Pub/Sub redelivers once the ack
deadline passes.

### Evented runs on a broker

`PubSub` gets a `redeliversUnacked` getter (default `false`). Redis and
Valkey Streams return `true` while the reclaim loop runs
(`reclaimIntervalMs > 0`), Google Cloud Pub/Sub returns `true`,
`CachingPubSub` forwards to its inner pubsub, and the in-process
`EventEmitterPubSub` and the Unix socket pubsub return `false`. When the
pubsub (read through `__rawBus()`) redelivers, recovery never lists,
reads or restarts an evented run. An evented workflow without
declarative schedules costs no listing at all. Evented runs on any other
pubsub (`EventEmitterPubSub`, Unix socket, or a third-party pubsub that
doesn't override the getter) are restarted at once by the explicit call,
as before.

### Scheduled runs

The `mastra.pubsub` proxy publishes every workflow event of a scheduled
run (`sched_wf_*`, plus the notification dispatcher's row) `localOnly`,
so the broker never holds it. Recovery still restarts those runs: an
evented workflow with declarative schedules gets one summary listing per
status, and only its process-local scheduled runs are kept. The proxy
and recovery share one predicate, `isProcessLocalScheduledRunId`.

### Default engine

One recovery rule: a `running` or `waiting` run is restarted when its
run lease isn't live and `recovery.workflowGraceMs` (new, 2,000 ms by
default for every pubsub) has passed since the later of the lease's last
expiry and the run's last write. The pass lists summaries
(`listWorkflowRuns({ status, perPage: false, summary: true })` per
status), reads leases in batches of 100 with `getRunOwnerships`,
re-reads a run's lease once its restart gets a slot, and runs every
restart on one instance-wide limiter (`recovery.workflowConcurrency`). A
run left live or in its grace period gets a per-run recheck timer, as on
main. Overlapping calls share one pass. `shutdown()` waits, within
`drainTimeout`, for a pass still reading the store and for the restarts
it handed out. Without run fencing every active run is restarted, as
before.

### Nested resume

A default-engine nested run resumed directly now holds its parent's run
lease while it runs and hands it to the parent's resume, so recovery
doesn't restart the parent meanwhile. On a store without
compare-and-set, where the parent stays `suspended`, the nested run
holds its own lease instead.

Also: `renewWorkflowLease` waits for each renewal to settle before
arming the next one (a self-rearming `setTimeout` instead of
`setInterval`), and `EventEmitterPubSub`'s JSDoc and the docs say it's
in-process only.

### Repro

Postgres + Valkey Streams with default reclaim settings (`reclaimIdleMs`
60 s, `reclaimIntervalMs` 30 s). Instance A starts a run whose step
records an execution and then hangs; A is killed with `SIGKILL`
mid-step. Then 2 or 4 fresh instances boot and all call
`restartAllActiveWorkflowRuns()` at once. "At boot" counts step
executions caused by those calls; "total" counts every execution after
the kill until the run succeeds; "rerun after" is seconds from the kill
to the first new execution. 3 trials per row, identical results within
each row except for timing.

| Build | Engine | Booting pods | Executions at boot | Executions total
| Rerun after |
|---|---|---|---|---|---|
| Base (this PR's base branch) | evented | 2 | 2 | 2 | 0.1 s |
| Base | evented | 4 | 4 | 4 | 0.1 s |
| Base | default | 2 | 0 | 1 | 15.1 to 16.6 s |
| Base | default | 4 | 0 | 1 | 15.0 to 16.9 s |
| This PR | evented | 2 | 0 | 1 | 60.1 to 60.2 s |
| This PR | evented | 4 | 0 | 1 | 60.1 to 60.2 s |
| This PR | default | 2 | 0 | 1 | 17.0 to 19.1 s |
| This PR | default | 4 | 0 | 1 | 17.2 to 21.0 s |

On this PR the evented rerun is the broker's reclaim: one execution on
one pod, about 60 s after the kill. The default-engine rerun comes once
A's lease (15 s TTL) lapsed plus the 2 s grace and up to 5 s of recheck
jitter.

### Benchmark

3,000 active default-engine runs that another instance holds, among
30,000 finished runs, two workflows. "Before" is a copy of main's sweep
run in the same process (full listing, then `createRun()` + `restart()`
per run, 5 at a time); "after" is `restartAllActiveWorkflowRuns()` on
this PR. Median of 5 calls.

| Store | Before | After | Listing payload before | Listing payload
after |
|---|---|---|---|---|
| Postgres | 1,773 ms | 30 ms | 1.12 MB | 0.60 MB |
| LibSQL | 627 ms | 44 ms | 1.12 MB | 0.60 MB |

The benchmark's snapshots are nearly empty, so the payload gap here is
small; with real step results it grows with snapshot size.

### Known limits

On a redelivering broker these evented runs stay `running` until someone
calls `restart()` on them. Main's boot restart recovered them; stack 2
takes them over.

- A sleeping evented run whose instance exits, after a crash or during a
normal deploy. The sleep acks its delivery and keeps a `setTimeout` in
the process that started it, so any exit of that process loses the
timer.
- A step whose message Redis or Valkey Streams trimmed. Both trim with
`MAXLEN ~ maxStreamLength` (10,000 by default), which also drops entries
that are still pending. A step that is in flight while that many newer
events are published loses its message if its instance then exits.
Deployments with long steps and a high event volume should raise
`maxStreamLength`.
- A message the broker dropped after `maxDeliveryAttempts` or at
`inFlightTimeoutMs`.
- A failure publish that gave up after its retries (the worker acks the
message).
- A run whose `running` snapshot was saved but whose start event never
reached the broker, because its process stopped between the two.
- A Redis or Valkey instance that restarted without persistence.
- A scheduled evented run that was in flight when a deploy stopped its
process and removed the workflow's schedule from code. Recovery doesn't
list it, so it stays `running` until `restart()`.

#28712 is fixed for evented runs on a redelivering broker. These are
still restarted by every booting instance, as before:

- scheduled evented runs (`sched_wf_*`), whose events never leave their
process
- evented runs on the in-process `EventEmitterPubSub` or the Unix socket
pubsub. Several instances sharing one storage with these pubsubs aren't
supported for evented workflows; the docs now say a multi-instance
evented deployment needs a broker.

With Redis or Valkey Streams, `reclaimIdleMs` must stay at 30 s or more:
a message reclaimed while the dead worker's 30 s step fence is still
held is dropped as a duplicate. The getter's JSDoc, the adapter docs and
the changeset state it.

Rolling deploys are out of scope.

Stack 3 of 4: storage projections and batch ownership reads, then the
finished-step check, then this PR, then the recovery loop with a leader.

## Related issue(s)

Fixes #28712

## Type of change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Code refactoring
- [x] Performance improvement
- [x] Test update

## Checklist

- [x] I have linked the related issue(s) in the description above
- [x] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

ELI5: When a server crashes in the middle of a workflow, Mastra can
restart that workflow. Before this change, some workflows ran their
steps again on every new server that started, so a step could run many
times. This change lets the message broker handle the restart for those
workflows, so each step runs only once. Other workflows now wait for a
short grace period before they restart.

## Summary

This pull request changes how workflow recovery restarts runs after a
crash or a boot. It targets duplicate step execution in two cases:

- **Evented runs on brokers that redeliver unacknowledged messages.**
Recovery skips these runs. The broker redelivers the unacknowledged step
events instead.
- **Default-engine runs.** Recovery restarts a run only after no live
lease remains and a grace period has passed.

### Behavior changes

- Google Cloud Pub/Sub, Redis Streams, and Valkey Streams report
`redeliversUnacked` as `true` when their reclaim or redelivery path is
active. Redis Streams and Valkey Streams report `true` only when
`reclaimIntervalMs` is above 0.
- `PubSub` adds a `redeliversUnacked` getter that defaults to `false`.
`CachingPubSub`, `EventEmitterPubSub`, and `UnixSocketPubSub` expose or
declare this capability.
- Recovery leaves evented runs to the broker when `redeliversUnacked` is
`true`. Scheduled evented runs restart on boot. Other evented runs
restart immediately when the broker does not redeliver.
- Default-engine recovery applies `recovery.workflowGraceMs`. The
default is 2,000 ms. Negative or non-finite values use the default. A
run restarts only after its lease is no longer live and the grace period
has elapsed since the later of its last update and lease expiry. Runs
that are not yet eligible are checked again later. If the storage
adapter does not support run fencing, active runs restart without the
grace rule.
- Recovery lists active run summaries, reads ownership in batches, and
rechecks ownership before each restart.
- Restarts use a shared concurrency limiter
(`createConcurrencyLimiter`). A restarted run holds its slot until it
finishes.
- Overlapping recovery passes are deduplicated. Shutdown waits for an
in-flight pass.
- Recovery skips nested runs whose active parent drives them.
- Lease renewal runs sequentially (`renewWorkflowLease`,
`repeatAfterSettled`). A new renewal starts only after the previous one
settles.
- Nested resume coordinates parent and child leases. A failed lease
claim rejects the resume and leaves the runs suspended.
- Run-claim operations return whether a status transition occurred. They
return `false` when storage is absent, running snapshots are not
persisted, or compare-and-set is not supported.

### Public API changes

- `MastraRecoveryConfig.workflowGraceMs?: number` (default 2,000 ms).
- `PubSub.redeliversUnacked` getter (default `false`).
- `redeliversUnacked` getters on `GoogleCloudPubSub`,
`RedisStreamsPubSub`, `ValkeyStreamsPubSub`, and `UnixSocketPubSub`.
- `ConcurrencyLimiter` interface and `createConcurrencyLimiter(limit)`
function in `run-with-concurrency.ts`.
- `renewWorkflowLease`, `repeatAfterSettled`, and
`WORKFLOW_RUN_LEASE_RENEW_INTERVAL_MS` in `run-ownership.ts`.
- `Run._resume(params)` accepts an optional `takeHeldOwnership`
callback.

### Documentation

- Crash-recovery and worker docs describe broker reassignment and
multi-instance limits.
- `EventEmitterPubSub` documents that delivery is process-local.
- Redis Streams and Valkey Streams docs recommend `reclaimIdleMs` of at
least 30 seconds. Reclaiming earlier can cause duplicate-dropped step
events.
- The recovery reference documents `workflowGraceMs`.

### Tests and test utilities

- New recovery tests cover exit cases (crash and shutdown), recovery
reads, rules, ownership during nested resumes, and lease renewal.
- Shared test helpers add `sharedBroker`, `killableProcess`,
`interceptStarts`, `persistActiveRun`, and `describeRecoveryExitCases`.
- Existing tests set `workflowGraceMs: 0` where they need immediate
recovery.
- Redis Streams and Valkey Streams tests check `redeliversUnacked`
against `reclaimIntervalMs`.
- LibSQL and Postgres recovery-pass suites run the shared exit cases.

### Scope and limits

- The change addresses issue `#28712` for evented runs on redelivering
brokers. Evented runs on in-process or Unix socket pubsubs still restart
at boot. The PR states that multi-instance evented deployments need a
broker.
- The PR lists unrecovered cases: sleeping evented runs, trimmed or
dropped messages, failed publishes, missing start events, non-persistent
Redis or Valkey restarts, and scheduled runs whose schedules were
removed. Rolling deploys are out of scope.
- Reported benchmark results (median recovery time on 3,000 active and
30,000 finished runs: 1,773 ms to 30 ms on Postgres, 627 ms to 44 ms on
LibSQL) and repro results are author-reported. They were not
independently verified.
- CI and test results: not available.
- Review findings and severity counts: not available.

### Size

- 41 files changed, 2,594 lines added, 209 lines removed (from the pull
request diff).

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 14:59:00 -03:00
mastra-platform[bot]
95c9fa97f1
fix(docs): fail pnpm check when Vale is missing (#30334) 2026-11-13 10:42:46 -07:00
Arthur Fiorette
a86a47c7ef
fix(core): don't rerun evented steps that already finished (#30275)
## Description

With this PR the evented processor reads the stored run before it runs a
leaf step:

- The step finished and the run went on without it: the delivery is
dropped.
- The step saved its result but nothing took the run past it (the worker
stopped between the save and the next step's message): the run continues
from the saved result without running the step.
- Otherwise the step runs as before.

A `workflow.step.run` message can reach a worker after its step already
finished: a broker redelivers a message whose ack was lost, a message
waits in a queue while `restart()` runs the step, or a restart reads the
run just before the step saves its result. Before this PR the worker ran
the step again and then every step after it.

The decision is a pure function in
`workflow-event-processor/finished-occurrence.ts`. It compares the
stored record with the one the delivery carried, so a step ID used twice
in one workflow still runs at both places. It handles loop iterations,
foreach items, parallel and conditional branches, and nested workflows.
A resume of the step and time travel skip the check and run the step
again on purpose. On stores with workflow step guards (Postgres),
records the guards stamped are left to the guards. The check reuses the
snapshot `#dispatch` already loads for the delivery, so a leaf delivery
still reads the run twice (dispatch and `runLeafStep`'s suspension
check), as on `main`.

The same check runs on an exhausted delivery. A late `workflow.step.run`
of a finished step used to fail the run once it used up its delivery
attempts. It is now dropped, or continues the run when the run still has
to continue from it.

**When the check runs**

The workflows store supports run fencing, the workflow keeps
`autoRestartActiveRuns` (durable agent workflows opt out, so they are
unchanged), it is not a processor workflow, and its
`shouldPersistSnapshot` saves running snapshots. A store whose
run-fencing probe rejects gets no check for that delivery: the processor
logs a warning and runs the step as it does today, and the next delivery
asks again.

**Known limits**

- A step whose worker stops before saving its result still runs again on
the next delivery. Make steps with side effects idempotent.
- `pruneSnapshot` must keep each step result's `status`, `startedAt` and
`endedAt`, a loop step's `metadata.iterationCount`, a foreach step's
`suspendPayload.__workflow_meta.foreachOutput`, and `activePaths`.
Without them a step delivered again after it finished can run a second
time. The changeset and the `workflow.mdx` reference say so.
- A workflow whose `shouldPersistSnapshot` returns false for a running
run gets no check: its stored active path lags behind the run, so the
check can't tell a step the run went past from one it still has to
continue from. Dropping there would leave a run stuck after a worker
stopped between a step's save and the next step's message, so such a
delivery runs the step again, as before this PR.
- Nothing serializes two deliveries of one occurrence except the
processor's event-ID step fence, which keeps same-ID deliveries apart
and only on a pubsub with a lease provider (Google Cloud Pub/Sub has
none). Deliveries it doesn't fence that read the same saved result can
both continue the run from it. The extra chain can run the next step a
second time and ends at the first step whose check finds that step
finished and the run past it.
- On a store with step guards (Postgres), loop iterations are left to
the guards. A delivery the guards don't arbitrate (a restart or a time
travel) that arrives after a loop's last iteration was saved runs that
iteration again, as before this PR. The same holds for a foreach item
that a restart queued again while another worker was still finishing it.

**Tests**

`test-utils/finished-occurrence-cases.ts` is one shared suite, run on
the in-memory store (`finished-occurrence.test.ts`) and on Postgres
(`stores/pg/.../evented-finished-occurrence.test.ts`). Replicas are
separate `Mastra` instances sharing one store, and a broker is modelled
by holding or dropping published events and by handing a recorded event
to another replica. Cases: a finished step redelivered while the run
goes on and after it ended (step, parallel, foreach, loop, nested), the
original message arriving after a restart ran the step, a restart that
read the run just before the step saved its result (plain step and a
branch), a worker that stopped after saving a step result (step,
exhausted delivery, loop's last iteration), an exhausted redelivery
while the next step is running or suspended, a step used twice, time
travel, and a step ID reused across a suspension in a workflow that
skips running snapshots.

With the two call sites reverted, 13 of the first 15 cases fail on their
assertions (the step-used-twice and time-travel cases check behavior the
fix must keep). Without the `shouldPersistSnapshot` condition, the last
case fails with the run left `running`. The known-difference blocks for
completed-transition replays in `evented-workflow-sleep-restart`,
`nested-suspend-loop-redelivery` and `parallel-suspend-redelivery` are
removed; those cells now assert the step runs once and fail without the
fix. `redelivery-contract.test.ts` opts its workflow out with
`autoRestartActiveRuns: false`, since the drain test needs a replay that
runs steps again.

Checks run locally: `pnpm build:core`, `pnpm --filter ./packages/core
check`, core vitest on `src/workflows src/mastra src/agent/durable`
(only the two `scheduler.test.ts` retry tests fail; they are time-zone
dependent and fail on a machine not set to UTC), the Postgres suite
against Docker.

Rolling deploys (old and new instances processing one run) are out of
scope.

## Related issue(s)

Part of #28712. Stack: storage projections (#30274, merged) -> this PR
-> recovery pass (#30279) -> recovery loop (#30281).

## Type of change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [x] Test update

## Checklist

- [x] I have linked the related issue(s) in the description above
- [x] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

A workflow step can finish, and then a late copy of its "run this step"
message can arrive. Before this change, the system could run the
finished step again. Now the system checks the saved run state first. It
skips the late message or continues from the saved result.

## Summary

This change stops evented workflow leaf steps from running again when a
delayed or redelivered `workflow.step.run` message arrives after that
step occurrence has finished.

### Behavior

- `WorkflowEventProcessor` checks persisted run state before it runs a
leaf step. The check uses the new `finishedOccurrence` function.
- The check applies only when all of these hold:
  - The storage adapter supports run fencing.
  - The workflow is not a processor workflow.
  - `autoRestartActiveRuns` is not disabled.
  - The workflow persists running snapshots.
- For an eligible delivery, the result is one of three outcomes:
  - **Drop:** the run has moved past the step occurrence.
- **Continue from saved result:** the run still needs to advance past
the occurrence. The processor uses the stored result and does not run
the step.
  - **Run:** the step runs as before.
- If the run-fencing probe rejects, the processor logs a warning and
runs the step as before.
- Time travel and resume executions bypass the check and run the step.
- Exhausted `workflow.step.run` deliveries get the same drop-or-continue
handling before the pending failure is published.
- The check covers repeated step IDs, loops, foreach items, branches,
and nested workflows.
- Canceled and tripwire results run only for restart targets. Success
and failure results continue from the stored result when the run has not
moved past the occurrence.
- Start-guard eligibility checks are centralized in `#guardsStepStart`.
- `routeStepRun` classifies each `workflow.step.run` delivery as a leaf,
a dispatch, or missing.

### Limits

- A worker that stops before it saves a result can still cause a rerun.
- Concurrent deliveries that the event-ID step fence does not separate
can both continue from the same saved result.
- Rolling deploys that run old and new instances on the same run are out
of scope.

### Snapshot pruning

When `pruneSnapshot` is set, the snapshot must keep these fields for the
check to work:

- Step result `status`, `startedAt`, and `endedAt`.
- Loop `metadata.iterationCount`.
- Foreach `suspendPayload.__workflow_meta.foreachOutput`.
- `activePaths`.

### Tests

- Adds `describeFinishedOccurrenceCases({ getStorage, connect? })`. This
shared suite uses separate Mastra replicas on shared storage. It
simulates redelivery, dropped messages, process crashes, and held
step-end events.
- The cases cover sequential, parallel, foreach, loop, and nested
workflows. They also cover exhausted deliveries, saved results, time
travel, and workflows that do not persist running snapshots.
- Adds an in-memory suite and a Postgres suite. Both run the shared
cases.
- Adds a parameterized `routeStepRun` test suite.
- Updates the redelivery tests in `evented-workflow-sleep-restart`,
`nested-suspend-loop-redelivery`, and `parallel-suspend-redelivery`.
Each now expects one fixed result for every redelivery mode. The
mode-dependent extra executions are removed.
- Updates `redelivery-contract`. The timer-drain test disables
`autoRestartActiveRuns`, and the duplicate-execution reason now names
the workflow without the finished-step check.
- The author reports these results. This summary did not verify them:
  - Reverting both check call sites makes 13 of the first 15 cases fail.
- Removing the `shouldPersistSnapshot` condition leaves a run in
`running` in the final case.
- `pnpm build:core`, `pnpm --filter ./packages/core check`, the core
Vitest tests, and the Postgres suite against Docker were run.
- Two `scheduler.test.ts` retry tests failed in the Vitest run. The
author attributes them to time-zone dependence on a non-UTC machine.

### Documentation

- The changeset adds a patch note for evented workflows. It limits the
claims to storage adapters with run-ownership support. It excludes
workflows with `autoRestartActiveRuns: false` and runs whose running
snapshot is rejected by `shouldPersistSnapshot`.
- `docs/.../workflows/overview.mdx`: On the evented engine, a late event
does not rerun a step whose result is saved. Workflows with
`autoRestartActiveRuns: false`, including durable agent workflows, still
rerun the step. The earlier statement that redelivery reruns the step is
removed.
- `docs/.../reference/workflows/workflow.mdx`: Returning `false` from
`shouldPersistSnapshot` for a running run can cause a redelivered
completed step and later steps to run again. The page also lists the
additional `pruneSnapshot` fields above.
- `redelivery-driver.ts` documentation: Describes when a completed-step
redelivery is dropped or resumed, and when it re-executes. The ID-keyed
`STEP_FENCE_TTL_MS` fence drops immediate duplicates but not late ones.

### Scope

- 15 files changed. Lines: +1379 and −69.
- Core logic: `finished-occurrence.ts` (+163),
`workflow-event-processor/index.ts` (+249, −26), and `routing.ts` (+68).
- Shared test suite `finished-occurrence-cases.ts` adds 747 lines.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 13:35:36 -03:00
mastra-platform[bot]
b860194516
chore(daytona): switch to the renamed @daytona/sdk package (#28995)
Co-authored-by: wardpeet <ward@coding-tech.com>
2026-11-13 17:13:01 +01:00
Ward Peeters
af1d6bdfd2
fix(mastracode): fire permission hooks for ask_user questions (#30301)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-13 14:06:30 +01:00
Ward Peeters
673c45a861
feat(factory): lower the default run cap to 10 and show the run queue on the board (#30295)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-13 13:31:58 +01:00
Arthur Fiorette
7b646a07a8
fix(core): follow session permission rules for AgentController tool approval (#30151) 2026-11-13 09:08:43 +01:00
Roamin
5266060ccd
fix(core): make webSearchTool work on Mastra gateway models (#30239)
An agent using `webSearchTool` on a Mastra gateway model failed every
turn. `webSearchTool` resolved to the provider's native tool
(`openai.web_search`, etc.), but the gateway's chat endpoint only runs
its own `openrouter:*` server tools. OpenAI models got a 400 (`unknown
tool type "openai:web_search"`), and Anthropic and Google models
silently lost the tool.

```ts
const agent = new Agent({
  id: 'research-agent',
  name: 'Research Agent',
  instructions: 'Use web search when you need current information.',
  model: 'mastra/openai/gpt-6.1-sol',
  tools: { search: webSearchTool },
})

// Before: 400 from the gateway on every turn
// After: the gateway runs the search and the agent answers
await agent.generate('What is the weather in London?')
```

Gateway chat models now get `openrouter.web_search` instead. Any other
provider-native tool on a gateway model (for example
`openai.code_interpreter`, or a native web search tool with provider
options) now fails before the request with an error naming the tool.
This is a behaviour change for Anthropic and Google gateway models,
where such tools used to be ignored without an error. The OAuth
Anthropic route is unchanged and keeps its native tool, and direct
`openrouter/...` models are not touched.

Tested with a new aimock scenario across all engines that asserts the
tool type on the wire, plus a manual run against the live gateway with
OpenAI, Anthropic and Google models. Search results come back as plain
text, so there are no tool-call or source chunks in the stream yet.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

Models that run through the Mastra gateway can now use the web search
tool. Other special provider tools on those models now stop the request
with a clear error, so they no longer fail in unclear ways or get
dropped silently.

## Summary

This change updates how `@mastra/core` prepares tools for models routed
through the Mastra gateway.

### Behavior changes

- `webSearchTool` maps to the gateway tool `openrouter.web_search` for
gateway OpenRouter models. The mapping applies only when the tool has no
arguments.
- Native web search tools with arguments throw a `MastraError`. The
error uses a web-search-specific message.
- Other provider-native tools on gateway models throw a `MastraError`
before the request is sent. One example is `openai.code_interpreter`.
The error names the tool ID and the router ID.
- The supported native web search IDs are `openai.web_search`,
`anthropic.web_search_20250305`, `google.google_search`, and
`xai.web_search`.
- The PR description reports that Anthropic and Google gateway models
previously dropped these tools without an error. It also reports that
OpenAI and xAI requests failed with a 400.
- `openrouter.*` tools and non-provider (function) tools pass through
unchanged.
- The OAuth Anthropic route keeps its native tool.
- Direct `openrouter/...` models are not affected.

### Code changes

- `packages/core/src/llm/model/mastra-gateway-provider-tools.ts` (new
logic):
- `isOpenRouterModel(model)` returns true when `model.provider` is
`openrouter` or starts with `openrouter.`.
- `prepareToolsForMastraGateway(options, routerId)` applies the mapping
and validation above. It returns the original options when no provider
tools are present.
- `packages/core/src/llm/model/router.ts`: `doGenerate` and `doStream`
call `prepareToolsForMastraGateway` when the gateway ID is `mastra` and
the model is an OpenRouter model. Other gateway and model combinations
keep the existing options.
-
`packages/core/src/loop/test-utils/aimock/scenarios/gateway-web-search.scenario.test.ts`
(new, 146 lines): tests the OpenAI, Anthropic, and Google routes. The
tests check that:
  - `webSearchTool` is sent as `openrouter:web_search`.
  - Function tools and explicit `openrouter.web_search` tools remain.
  - The OAuth Anthropic route keeps its native search tool.
- Unsupported native tools fail with an error and send no request body.
  - A direct `doGenerate` call converts the tool.
- `packages/core/src/loop/test-utils/aimock/README.md`: adds one entry
for the new scenario test.

### Documentation and release

- `docs/src/content/en/docs/agents/tools.mdx`: states that gateway
models use `openrouter.web_search`. It also states that passing another
provider-native tool fails the agent run with a `MastraError`.
- `.changeset/lazy-suns-dream.md`: adds a patch changeset for
`@mastra/core`.

### Scope

- 6 files changed, 234 lines added, 2 lines removed.

### Limitations

- The PR description reports that gateway search results arrive as plain
text. The stream does not include tool-call or source chunks.
- The PR description reports manual runs against the live gateway with
OpenAI, Anthropic, and Google models. CI and test results were not
available for this summary.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-12 19:01:39 -05:00
Tyler Barnes
175b4b5194
fix(core): Knowledge scope values are one encoded segment (#30019)
Knowledge scope patterns let the last `$` value take the rest of an
address, so `org:acme:viewer` counted as an `org:$orgId` scope. Its
holder then got that pattern's rights on what was meant to be a separate
scope. Raw ids and escaped ids also had more than one spelling.

This makes each `$` value exactly one segment and gives every scope one
canonical spelling.

## What changes

- A scope value can't contain a raw `:`. Ids with `:`, `/`, `#` or `%`
are passed encoded, via the new `encodeKnowledgeScopeId`
(`decodeKnowledgeScopeId` reverses it). Raw forms are refused with a
message naming the helper.
- Agent `scopes` are canonical addresses: `%` is always an escape.
- Memory resource and thread ids are bound encoded (`slack:U123` becomes
`resource:slack%3AU123`). Knowledge takes identity only from the run's
resolved Memory ids, never from `resource:`/`thread:` strings in the run
scopes.
- Patterns of one type with different segment counts no longer overlap,
so `org:$orgId` and `org:$orgId:viewer` can be declared together.
- A stored scope row whose address now matches a different declaration
than the one it was created under gets none of the new declaration's
access: not by exact address, prefix, `belongsTo` lookup or parent-bound
role. Holders of the old declaration's prefix grant still read it, as
before, without a role.

## Existing data

Nothing is rewritten. Ids with a raw colon in the last segment get new
addresses, so a run reaches a new, empty scope, and rows at the old
address stay unreached. Ids with `/`, `#` or `%` were already encoded
and keep their addresses when passed through `encodeKnowledgeScopeId`.
Knowledge is in alpha; this is a breaking change for colon ids.

## Testing

- New tests cover encoded and raw ids, segment counts, stale rows
(exact, prefix, `belongsTo` parent, parent-bound roles) and Memory
identity through `resolveAgentScopes`. Each fails on the commit before
its fix.
- Core Knowledge, storage and agent-scope suites: 412 passed. Server
Knowledge handlers: 48 passed. Typecheck, lint and format are clean.
- A store written by the previous release, read by this one: `/`, `#`,
`%` and literal-escape ids remain readable through
`encodeKnowledgeScopeId`; colon ids move.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

**ELI5:** Knowledge uses labels called scopes to decide who can read
which pages. This change makes each label part match exactly one piece
of the address, and it requires special characters in IDs to be encoded.
This makes access rules predictable, but old alpha data with colons in
IDs may no longer be reached.

## Overview

This change makes each scope parameter match exactly one address
segment. Scope IDs that contain reserved characters must use canonical
encoding. The change is breaking for alpha Knowledge users who have
colon IDs in the last segment.

## Core changes

- **Scope matching** (`declarations.ts`): Patterns match an exact
segment count. A final parameter no longer absorbs raw colons. Raw Agent
scopes are no longer matched. `knowledgeRunScopes` accepts only
canonical addresses that match a declaration, returns them unchanged,
and rejects reserved characters that are not encoded.
- **Overlap rules** (`declarations.ts`): Patterns overlap only when they
share a type, have no conflicting literals, and have equal segment
counts.
- **Encoding** (`scope-address.ts`):
- `encodeKnowledgeScopeValue` and `decodeKnowledgeScopeValue` always
escape `%`, `/`, `#`, and `:`. The `final` argument is removed.
- New experimental helpers `encodeKnowledgeScopeId` and
`decodeKnowledgeScopeId` encode and decode one ID segment. They are
exported from `index.ts`.
- **Memory identity** (`run.ts`): Resource and thread scopes are built
by binding Memory IDs through `knowledgeScopeAddress`. The generated
addresses must match compiled declarations. A mismatch raises
`KnowledgeDeclarationError`. The result is the deduplicated union of raw
and Memory-derived scopes.
- **Stale rows** (`scope-runtime.ts`): A stored scope is current only
when it has no declaration key or its address matches that declaration.
Stale rows are excluded from `belongsTo` parent resolution, final
readable addresses, and `roleOn`. `explainAccess` returns no grants for
a stale target. Materialization rejects a stored scope whose declaration
no longer matches its address.

## Mastra Code and Factory changes

- `knowledge-scope.ts` encodes Factory organization and project IDs
once, as single canonical segments. A host tag of the same type with a
different value is rejected.
- `knowledge-inspector.ts` encodes resource and thread IDs before
calling `readScopeFor`. It no longer uses the local `canonical` helper.
Session tags are unchanged.
- `factory/src/routes/knowledge.ts` encodes organization and project IDs
before building `org:` and `project:` principals.

## Data compatibility

- No stored data is rewritten.
- IDs with a raw colon in the last segment map to new addresses.
Existing rows at old addresses are not reached.
- IDs that contain `/`, `#`, or `%` keep their addresses when encoded.
- Stale rows do not gain access from a new matching declaration. A
prefix grant from the original declaration can still include those rows
in reads, as tested in `legacy-scope-rows.test.ts`.

## Documentation

- Scope declaration docs say each `$param` matches one segment. They
describe compatibility effects for alpha scopes and the new overlap
rule.
- Memory scope docs describe `encodeKnowledgeScopeId` and
`decodeKnowledgeScopeId`. They state that alpha scopes with a raw colon
in the last segment are not migrated.
- `explainAccess` docs describe `runScopes` as canonical addresses.
- A changeset marks a minor `@mastra/core` release. Changesets for
`@mastra/code-sdk` and `@mastra/factory` are patch releases.

## Tests

- New and updated tests cover encoded and raw IDs, segment counts, stale
rows, and Memory identity through `resolveAgentScopes`.
- A new helper, `agentRunContext`, is used by test files.
- The author reports 412 passing core Knowledge, storage, and
agent-scope tests. The author also reports 48 passing server Knowledge
handler tests, and clean typecheck, lint, and format. These results are
not verified here. CI status is not available.

## Scope of change

- Files changed: 24
- Lines added: 549
- Lines removed: 116
- Review severity counts: not available
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com>
2026-11-12 14:52:04 -07:00
Daniel Lew
9c8d3377e4
docs: built-in MCP server tools reference, client setup, and access control (#30232)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
2026-11-12 16:37:03 -05:00
dane-ai-mastra[bot]
d60ba71be0 chore: version - exit prerelease mode 2026-11-12 21:12:29 +00:00
shreyas-mastra
2a24694de4 Merge main into security/agent-browser-cli 2026-11-12 15:18:23 -05:00
Daniel Lew
18168d518c
feat(playground): show the built-in MCP server in Studio (#29932)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-12 13:40:02 -05:00
Daniel Lew
4ba2bdef89
feat(server): memory, observability and agent builder MCP toolsets (#29931)
## Description

Adds four toolsets to the built-in MCP server at `/api/mcp`:

- `memory`: threads, messages, memory search, and Observational Memory.
`get_observations` can wait for pending Observational Memory work, for
example right after `call_agent`.
- `observability`: trace and span queries, logs, and trace trajectories.
- `observability_analytics`: `analyze_metrics`, `analyze_scores`, and
`analyze_feedback`, plus metric name and label lookups.
- `stored_agents`: list allowed models, and create, read, and update
stored agents.

With these, an MCP client such as Claude Code can create an agent, talk
to it, read what it remembered, and explain a slow trace.

### Behavior

- **On by default, shown only when supported.** Each toolset appears
only when the app has the matching capability (a memory-enabled agent,
an observability store that supports the queries, an editor for stored
agents). Apps that don't set `server.mcp.toolsets` get these tools on
upgrade, including write tools. Set `toolsets` to keep the previous set,
or `readOnly: true` to drop writes. The changeset says this.
- **Same checks as HTTP.** Every tool dispatches through its HTTP route,
so authentication, RBAC, FGA, thread ownership, and agent visibility
apply unchanged. No MCP caller gets more access than the same caller
over HTTP.
- **Observability isn't scoped per user.** Anyone with
`observability:read` sees all traces, as over HTTP. The docs show how to
leave those toolsets out.
- **Builder model policy.** When the agent builder restricts models,
`create_stored_agent` and `update_stored_agent` check the model the
agent runs (`provider` and `name`, in every conditional variant; other
keys on `model` are rejected), models named in `defaultOptions` such as
`structuredOutput.model`, enabled fallback models (`modelList`), memory
models, input and output processor models, and the defaults they fall
back to. Anything the policy can't check is rejected. The HTTP
stored-agent routes don't apply this check. Agents created or updated
over MCP are published immediately (`update_stored_agent` defaults
`autoPublish` to `true`).
- **Results fit MCP client limits.** `get_trace` returns light span
records with every span id, and `get_span` returns one span in full.
Long lists are cut to fit, with a note on how to get the rest, and a
single oversized item has its long values shortened (ids are never cut).
- **Errors.** Bad `analyze_*` and `list_metric_names` arguments return
400. A failed observability store lookup is retried on the next request
instead of hiding the observability tools until restart.

Stacked on #29930.

## Test plan

- `@mastra/server`: unit tests for the toolsets, capability detection,
result sizing, and schemas.
- Shared adapter suite (`builtin-mcp-toolsets-test-suite`) on Hono,
Express, Fastify, Koa, Elysia, and NestJS: memory ownership compared
with HTTP on the same request, read-only mode, analytics dispatch, and
argument errors. Hono also runs observability tests against in-memory,
LibSQL, and DuckDB stores.
- `@mastra/editor` integration tests for the builder model policy across
`model` (static and conditional), `defaultOptions`, fallback models,
memory, and processors. They need no provider API keys.
- Run end to end with two users over MCP and HTTP, and with Claude Code.

---------

Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-12 10:25:19 -08:00
Daniel Lew
27705673f2
feat(server): OAuth discovery and audience checks for the built-in MCP endpoint (#29930)
## Description

Lets MCP clients such as Claude Code sign in to the built-in `/api/mcp`
endpoint with OAuth instead of a pasted token. The endpoint tells
clients where to sign in (RFC 9728 protected-resource metadata) and can
reject tokens issued for a different API (audience check).

```ts
new Mastra({
  server: {
    auth: new MastraJwtAuth({
      jwksUri: 'https://auth.example.com/.well-known/jwks.json',
      issuer: 'https://auth.example.com/',
    }),
    mcp: { oauth: { resource: 'https://my-app.example.com/api/mcp' } },
  },
});
```

### Behavior changes

**OAuth discovery for `/api/mcp`** (`@mastra/server`, every server
adapter including NestJS)
- When `server.auth` is set and the built-in MCP server is on, a `401`
from `/api/mcp` carries `WWW-Authenticate: Bearer
resource_metadata="…"`. `error="invalid_token"` is added only when a
token was sent (RFC 6750 §3).
- Metadata is served publicly at
`/.well-known/oauth-protected-resource/api/mcp` and
`/.well-known/oauth-protected-resource`, with `Cache-Control: no-store`.
- The document lists `resource`, `authorization_servers`, optional
`scopes_supported`, and `bearer_methods_supported: ['header']`. Without
`oauth.resource`, `resource` is derived from the request origin.
- The challenge, metadata routes and audience check apply only to
`/api/mcp`. Other routes are unaffected.

**Audience check** (`server.mcp.oauth.resource`)
- When set, the token audience must include this exact value, or the
request gets 401.
- An invalid value (non-URL, or a non-loopback `http://` URL) turns
discovery off and logs an error, but the audience check still runs
against it. Requests fail closed until the value is fixed.
- A provider that can't check audience (no `verifyAudience`, for example
`SimpleAuth` API keys) is accepted by default. The server logs a startup
warning on adapters built on `MastraServer`, not on NestJS.
- `requireAudience: true` (opt-in, only valid together with `resource`)
rejects those requests on `/api/mcp` with 401 `invalid_token`. Inside
`CompositeAuth`, only requests accepted by such providers are rejected.
Without `resource`, nothing changes.

**Config and provider hooks** (`@mastra/core`)
- New `server.mcp.oauth` config: `resource`, `requireAudience`,
`authorizationServers`, `scopesSupported`.
- Two new optional auth-provider methods:
  - `getOAuthMetadata()` advertises the provider's authorization server.
  - `verifyAudience()` checks the token audience.
- `CompositeAuth` combines the authorization servers of its providers.
It checks the audience with the provider that accepted the token, and
rejects when that provider can't be determined. The new
`getAuthenticatedProvider(user)` returns that provider.

**Providers**
| Provider | Advertises its authorization server | Checks audience
against `oauth.resource` |
|---|---|---|
| `MastraJwtAuth` | Yes, its `issuer` | Yes |
| Auth0 | Yes, the tenant | Yes |
| Clerk | Yes, the Frontend API | No |
| Okta | Yes, the issuer | No: Okta already enforces its own `audience`
(`OKTA_AUDIENCE`) |
| Supabase | Yes, `<url>/auth/v1` (requires Supabase's OAuth 2.1 server)
| No |

`MastraJwtAuth` gains `jwksUri` and `issuer` options
(`JWT_AUTH_JWKS_URI`, `JWT_AUTH_ISSUER`), so it can verify tokens signed
by an identity provider. JWKS keys are restricted to their key type, so
HMAC tokens forged with the public key are rejected.

**JWT subject as user id** (`@mastra/auth`, Auth0, Clerk)
- When a token has no `id` claim, the user's `id` is now set to `sub`.
Existing `id` claims are kept.
- Standard OIDC access tokens only carry `sub`. Before this change, RBAC
never loaded roles for these users, so every protected route returned
`403 Missing required permission`.
- Unless `mapUserToResourceId` is set, this also scopes data for these
users:
- **Owned records** (stored agents, skills, workspaces and other owned
records): each user sees their own records plus public ones. Admin
permissions keep the full list.
- **Tool-provider connections:** per-author connections move from the
shared `default` owner to the user's `sub`. Existing connections need
reconnecting, or set `mapUserToResourceId: () => 'default'` to keep the
old shared behavior.
  - **Feedback:** recorded with `sub` as the user id.
- The changeset describes all of these effects.

**Test suite** (`@mastra/server-adapters-test-suite`)
- New `createBuiltinMCPOAuthTestSuite`.
- The package now has `@mastra/auth` and `jose` as peer dependencies.

Stacked on #29929.

## Test plan

- `packages/server`: `src/server/mcp/oauth.test.ts` and
`oauth-without-mcp.test.ts`. They cover metadata, the challenge format,
resource validation, failing closed on an invalid `resource`, and
discovery working without `@mastra/mcp` installed.
- `packages/auth` `jwt.test.ts`:
  - JWKS and issuer validation
  - string and array `aud`
- an algorithm-confusion token that is HS256-signed with the real RSA
public key PEM and must be rejected
  - `sub` used as `id`
- `packages/core` `composite-auth.test.ts`: per-provider audience
delegation, rejection when the authenticating provider is unknown, and
`getAuthenticatedProvider`.
- `auth/auth0`, `auth/clerk`, `auth/okta`, `auth/supabase` unit tests.
- `createBuiltinMCPOAuthTestSuite` on Hono, Express, Fastify, Koa and
Elysia, plus the NestJS equivalent. It checks:
  - the challenge, and fetching the advertised metadata URL
  - audience accept and reject, including for an invalid `resource`
  - `requireAudience`:
    - `SimpleAuth` + `resource`: 200 by default, 401 in strict mode
    - a JWT with a matching audience: 200 in strict mode
    - a mixed `CompositeAuth`: only the API key is rejected
  - RBAC with a token that carries only `sub`
- that a `sub`-only user can't list another user's private stored agent
- `tsc --noEmit` and `oxlint` on the touched packages, plus docs `pnpm
check` and Vale.
- Manual run end to end with Claude Code against a local identity
provider: discovery, client registration, PKCE login, and two users with
different roles.

---------

Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-12 10:10:31 -08:00
Lennart
171e27355f
fix(code-sdk): fire SessionStart again when a session binds a different thread (#30223)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-12 11:03:36 -07:00
Daniel Lew
392c7b69d5
feat(server): built-in MCP endpoint for agents, workflows and tools (#29929)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-12 12:56:28 -05:00
Tyler Barnes
25ba8bae0e
docs(memory): rewrite the Observational Memory guide (#30213)
Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-12 10:38:25 -07:00
Daniel Lew
d05afc4a90
refactor(server): shared route authorization and in-process dispatchRoute across adapters (#29928)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-12 12:30:28 -05:00
shreyas-mastra
d6e2fb42d2 Merge branch 'main' into security/agent-browser-cli 2026-11-12 12:26:00 -05:00
mastra-platform[bot]
649ac25a61
docs(studio): remove outdated Studio overview and deployment details (#30220) 2026-11-12 10:19:04 -07:00
shreyas-mastra
9ad5417c63 Merge remote-tracking branch 'origin/main' into HEAD
# Conflicts:
#	pnpm-lock.yaml
2026-11-12 11:24:15 -05:00
arctic-char
c0d3bfdc43
feat: pass the authenticated user to DynamicArgument callbacks (#28612)
Dynamic agent config callbacks now get the authenticated end user as an
optional `user` argument, so you can personalize an agent without
digging auth keys out of `requestContext` yourself. It works in any
`DynamicArgument` callback: instructions, model, tools, memory,
workspace, processors, scorers, and so on.

```ts
import { Agent } from '@mastra/core/agent';

const supportAgent = new Agent({
  id: 'support-agent',
  instructions: ({ user }) =>
    user ? `You are helping ${user.name ?? user.id}.` : 'You are a helpful assistant.',
  model: ({ user }) => (user?.role === 'admin' ? 'openai/gpt-5' : 'openai/gpt-5-mini'),
  tools: ({ user }) => (user ? { accountTool, ...(user.role === 'admin' ? { adminTool } : {}) } : {}),
  memory: ({ user }) => (user ? userMemory : undefined),
});
```

`user` always has the same `MastraUser` shape, no matter which auth
provider you use, and it's `undefined` for unauthenticated requests:

```ts
import type { MastraUser } from '@mastra/core/types';

// { id: string; name?: string; email?: string; avatarUrl?: string; role?: string }
```

The server auth middleware builds it through a new overridable
`toMastraUser()` on `MastraAuthProvider`. The default handles common
shapes (`id`/`sub`/`userId`, `name`, `email`, `avatarUrl`/`picture`,
`role`), and JWT, Cloud, Supabase, Neon and Better Auth ship overrides
for their user objects. Custom providers can map their own shape:

```ts
import { MastraAuthProvider } from '@mastra/core/server';
import type { MastraUser } from '@mastra/core/types';

class MyAuth extends MastraAuthProvider<MyUser> {
  async authenticateToken(token: string) { /* ... */ }
  async authorizeUser(user: MyUser) { return true; }

  toMastraUser(user: MyUser): MastraUser {
    return { id: user.accountId, name: user.profile.displayName, role: user.permissions.primaryRole };
  }
}
```

Inline auth configs can pass `toMastraUser` too. If the provider user
has no role and an RBAC provider is configured, the primary RBAC role
fills it in, so `user.role` also works with WorkOS, Okta, Google and
Neon. The raw provider user is still available when you need
provider-specific fields, and the new context key is reserved, so
clients can't spoof it through the request body:

```ts
instructions: ({ requestContext }) => {
  const rawUser = requestContext.get('mastra__user'); // provider-specific object, unchanged
  // ...
},
```

Dynamic `vectorStore` resolvers on RAG tools get `user` as well:

```ts
createVectorQueryTool({
  indexName: 'docs',
  model: embedder,
  vectorStore: ({ user }) => (user ? tenantStores.get(user.id)! : sharedStore),
});
```

Inside core, every place that invoked a `DynamicArgument` callback
inline now goes through a single `resolveDynamicArgument` helper, which
reads the user from `requestContext` at call time. This is non-breaking:
callbacks get an extra optional field, providers get a method with a
working default. Requests served through an `MCPServer` don't get `user`
yet, since MCP builds its own request context; that's documented.

Tested with unit tests for the helper, normalization and each provider
override, middleware tests (inline config, provider role, RBAC role,
CompositeAuth ordering, unauthenticated), regression tests for the
migrated call sites, and an end-to-end test where an agent behind a fake
auth provider gets personalized instructions. Core, server, rag and the
touched auth packages typecheck clean, and their suites pass.



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5
When a signed-in user makes a request, configuration callbacks can now
use a standard set of details about that user. When a workflow resumes,
it uses the new request’s user details instead of restoring them from
the earlier run.

## Summary
- Added optional `user` data to `DynamicArgument` callbacks and RAG
vector-store resolvers. Callbacks can also use `getRequestUser()` to
read the normalized user from request context.
- Added the `MastraUser` type and `toMastraUser()` normalization hooks
for auth providers and inline auth configs. The server stores the
normalized user in a reserved request-context key and uses the primary
RBAC role when the user has no role.
- Updated core dynamic configuration paths to use the shared resolver.
Added user support to agent scopes and migrated A2A memory resolution.
- Added `withoutLiveRequestKeys()` to remove live authentication and
authorization data from saved context during workflow resume, restart,
and time travel. Saved non-authentication context, such as tenant data,
remains available.
- Added documentation and changesets. Raised relevant `@mastra/core`
peer dependency minimums to 1.81.0. MCPServer requests do not receive
the normalized `user`.

## Validation
The author reports passing typechecks and test suites across core,
server, RAG, and several auth packages. The author also reports three
failing Inngest tests and earlier failures in evented workflow and
scheduler tests, which reportedly fail the same way on comparison
sources. These results are author-reported and have not been
independently verified.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-12 10:55:12 -05:00
devin-ai-integration[bot]
731b66498c
docs(mcp): remove prebuilt Mastra apps section (#30209)
## Description

Removes the "Use prebuilt Mastra apps" section from
`docs/connections/mcp`, along with `@mastra/mcp-apps-components` in the
page's `packages` frontmatter. The package exists to support Playground
UI MCP App components and shouldn't be documented as a user-facing
package yet.

## Related issue(s)

None. Requested internally in Slack (#growth-pmm).

## Type of change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [ ] Test update

## Checklist

- [ ] I have linked the related issue(s) in the description above
- [x] I have made corresponding changes to the documentation (if
applicable)
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR

Link to Devin session:
https://app.devin.ai/sessions/e41299f3bd58441b9d227362b56641e1
Open in Devin Desktop:
https://app.devin.ai/desktop/session/e41299f3bd58441b9d227362b56641e1?variant=devin

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

**ELI5:** The docs page for MCP no longer talks about a ready-made app
package. That package is still being built for internal use, so it
should not appear in public docs yet.

## Summary

This change removes user-facing documentation for
`@mastra/mcp-apps-components` from the MCP docs page.

## Changes

- `docs/src/content/en/docs/connections/mcp.mdx`: Removes the "Use
prebuilt Mastra apps" section. This section explained how to install and
use the package's prebuilt greeting app. Removes
`@mastra/mcp-apps-components` from the page's `packages` frontmatter.

## Scope

- Files changed: 1
- Lines added: 0
- Lines removed: 46
- The rest of the MCP Apps documentation is unchanged.

## Rationale

The package supports Playground UI MCP App components. The team does not
yet want to document it as a user-facing package.

## Public API

No exported or public declarations change.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: paul <paul@mastra.ai>
2026-11-12 14:49:24 +00:00
mastra-platform[bot]
79ff445aca
fix(rag): reject JSON chunk sizes that stall string splitting (#30186)
## Summary
- Require JSON chunk `maxSize` of at least 21 in strategy validation and
direct transformer construction, preventing zero-length string splits
from stalling chunking.
- Add regressions for invalid integer and fractional limits and verify
long strings reconstruct at the supported boundary.
- Include a patch changeset for `@mastra/rag`.

## Test plan
- `cd packages/rag && ./node_modules/.bin/vitest run
src/document/document.test.ts --reporter=dot` (167 passed)
- `./node_modules/.bin/tsc --noEmit -p packages/rag/tsconfig.json`
- Type-aware oxlint for changed files; Prettier check; `git diff
--check`

Fixes #30166

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

**ELI5:** The JSON text splitter could get stuck in an endless loop when
a user set a small chunk size. This change rejects those small sizes
with a clear error, so the splitter never hangs.

## Summary

This change stops JSON chunking from hanging when `maxSize` is too
small. The splitter subtracts a fixed overhead of 20 from `maxSize` and
floors the result. At `maxSize` 20, the split point is zero. The
splitter then never consumes input. Fractional values between 20 and 21
have the same problem.

The change sets a minimum JSON `maxSize` of 21. It applies this minimum
in two places:

- **Option validation** (`packages/rag/src/document/validation.ts`): The
JSON chunk options schema adds an optional `maxSize` field. It accepts
the field only when the value is at least 21.
- **Transformer constructor**
(`packages/rag/src/document/transformers/json.ts`):
`RecursiveJsonTransformer` throws a `RangeError` when `maxSize` is not
finite or is below 21.

Other chunking strategies keep their existing size rules.

### Documentation

- `docs/src/content/en/reference/rag/chunk.mdx` states that the JSON
strategy's `maxSize` is a target maximum for each chunk. It must be at
least 21. Long values nested in objects can exceed it.
- A patch changeset for `@mastra/rag` describes the new minimum.

### Tests (`packages/rag/src/document/document.test.ts`)

- Adds cases that expect `maxSize` values `0.5`, `20`, and `20.9` to be
rejected. The cases cover both `MDocument.chunk` and direct transformer
construction. The expected error matches `maxSize.*21`.
- Adds a case that splits a 100-character JSON string with `maxSize:
21`. The test checks that the output has multiple chunks. It also checks
that the parsed string values rejoin to the original text.

### Behavior change

Callers that pass a JSON `maxSize` below 21 now receive a configuration
error. Before this change, those calls were accepted and could hang.
Default JSON chunking remains valid.

### Reported validation (not verified in this summary)

The PR description reports these checks:

- `vitest run src/document/document.test.ts`: 167 passed.
- `tsc --noEmit` for `packages/rag`.
- Type-aware oxlint on changed files, Prettier check, and `git diff
--check`.

Diff size: 5 files changed, 38 lines added, 1 line removed.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-12 06:35:07 -08:00
mastra-platform[bot]
da717f489c
Fix GraphRAG leaking documents across reused tool requests (#30185)
## Summary
- Rebuild the GraphRAG graph from each request’s retrieval results so
tenant-specific indexes cannot reuse a previous request’s documents.
- Return empty context and sources when retrieval yields no candidates.
- Add a real LibSQL and embedding-server regression covering sequential,
empty, and concurrent requests, plus a patch changeset.

## Test plan
- `pnpm build:rag`
- `pnpm test:rag` (454 tests)
- `pnpm exec tsc --noEmit -p packages/rag/tsconfig.json`
- `pnpm --filter ./packages/rag lint`
- `pnpm exec prettier --check packages/rag/src/tools/graph-rag.ts
packages/rag/src/tools/graph-rag-isolation.test.ts
.changeset/graph-rag-request-isolation.md`

Fixes #30168

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

**ELI5:** A search helper that is reused can return a document from an
earlier user's search. This change makes the helper build its answer
only from the current search. Each request now gets results for its own
search.

## Summary

This change fixes a data-isolation defect in the GraphRAG tool that
`createGraphRAGTool` creates in `@mastra/rag`. Linked issue `#30168`
reports the defect. The tool kept one `GraphRAG` instance and an
`isInitialized` flag across executions. Later requests that selected a
different index could still receive documents from the first execution.

## Changes

- `packages/rag/src/tools/graph-rag.ts` (+32/-37):
- Removes the `GraphRAG` instance and the initialization flag that
persisted across executions.
- Returns empty context and sources when vector search finds no
candidates.
- Otherwise, builds a new graph from that execution's retrieved chunks
and embeddings, then traverses it.
  - Keeps graph-build logging, tracing, and error propagation.
- `packages/rag/src/tools/graph-rag-isolation.test.ts` (+84/-0): Adds a
Vitest integration test. The test uses tenant-specific and empty vector
indexes. It serves an `LLMock` through a local HTTP proxy. It checks
that:
  - A request returns only the document from its selected index.
  - An empty index returns empty results.
  - Concurrent requests stay isolated.
- `.changeset/graph-rag-request-isolation.md` (+5/-0): Declares a patch
release for `@mastra/rag`.
- `docs/src/content/en/reference/rag/graph-rag-guide.mdx` (+1/-1):
States that each graph query tool execution builds its graph from that
request's retrieved chunks. The guide no longer says that later queries
reuse the first execution's graph.

No exported or public declarations change.

## Tradeoff

Each execution now builds a graph. The linked issue states that this
adds work per request. The request-specific candidates require this
work. A future cache would need correct candidate and scope keys.

## Verification

Not available. The PR lists planned checks: `pnpm build:rag`, `pnpm
test:rag`, a TypeScript check, lint, and a Prettier check. No completed
results for these checks were supplied.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-12 06:34:21 -08:00
Eric Pinzur
72abae68eb
feat(observability): export span-local originating error counts (#30088)
## Description

Export `errorCount` as failures originating directly on each span. Child
failures forwarded through a model step, workflow or agent add no parent
count; independent parent failures do. Failed retries remain counted
after recovery, and replay does not create new occurrences. No
parent/child totals or suspend/resume rollup are maintained.

Attribution stays in a WeakMap without modifying user errors. Explicit
internal serializers and Inngest memoized operations preserve provenance
across boundaries; public workflow results and stored workflow state
omit transport metadata. Failures before child execution count on their
actual originating span. Excluded spans preserve attribution so their
failures cannot be misassigned to visible parents, but excluded failures
are absent from exported totals.

Retry context stays in the server-only context module, while shared
origin utilities use global crypto. Guarded cross-package calls preserve
mixed-version startup compatibility. Unsupported accounting and older
snapshots remain unknown. Error-free spans do not allocate occurrence
sets.

Validation: 117 focused observability/workflow/compatibility tests, 23
real Inngest error/retry scenarios and 37 Inngest engine/nested-result
checks pass. New regressions reproduced resolver undercounting,
excluded-child misattribution and public transport-field leakage before
the fixes. Real Inngest tests cover an empty execution graph,
unavailable remote child, instrumented child errors, nested propagation,
retries, public responses and stored state. Core/observability build,
Inngest typecheck, changed-file type-aware lint and CI formatting checks
pass. The preceding merge also passed 167 core checks and redaction
integration coverage.

Playground production builds were verified on Node 24. Docs checks with
Vale report no errors (three warnings on unchanged API headings); the CI
Vale configuration passes. Standalone observability `tsc` previously
reported generic union-call errors in untouched
`workflow-control-flow-suite.ts`; package builds and focused tests pass.
No production rollout or query-table changes. CI reruns on the new head.

## Related issue(s)

Fixes https://linear.app/kepler-crm/issue/OBS-901

Follow-ups: https://linear.app/kepler-crm/issue/OBS-902 and
https://linear.app/kepler-crm/issue/OBS-903

Nonblocking review follow-up:
https://linear.app/kepler-crm/issue/OBS-905 tracks explicit propagated
classification on previously untagged errors and reuse of one Error
object across separate foreach iterations. Changesets intentionally
remain single sentences per maintainer direction.

## Type of change

- [x] New feature (non-breaking change that adds functionality)
- [x] Documentation update
- [x] Test update

## Checklist

- [x] I have linked the related issue(s) in the description above
- [x] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

This change counts each error on the span where it first happened. It
keeps the count correct when errors are wrapped, retried, or rebuilt, so
one failure is not counted twice.

## Summary

- Adds span-local `errorCount` to exported spans. The count covers
failures that start directly on that span.
- Adds `claimSpanError`, `BaseSpan.recordError`, and
`BaseSpan.restoreErrorCount`. Each claim records one occurrence ID on
the span.
- Ignores propagated failures on the receiving span. Propagated failures
count only on their origin span.
- Keeps error origin through cause chains, durable tool serialization,
and event codecs.
- Stores origin metadata in a `WeakMap` for live errors and formatted
results. Omits it from public workflow JSON.
- Stores occurrence IDs in span snapshots. Rebuilt spans restore them,
so replayed attempts are not counted again.
- Records failed attempts for default, evented, and Inngest workflow
retries. Each attempt uses an attempt-scoped occurrence ID.
- Reports `errorCount` as unknown when the runtime lacks the claim API
or when an older snapshot has no accounting history.
- Updates tracing types, the span record schema description, the span
reference documentation, and changesets for `@mastra/core`,
`@mastra/observability`, `@mastra/client-js`, and `@mastra/inngest`.
- Adds tests for repeated reports, retry scopes, concurrent async spans,
excluded children, cause-linked errors, tool and agent spans, rebuilt
and legacy snapshots, codecs, durable serialization, workflow retries,
and ESM/CJS compatibility with an older core build.

## Scope

- Parent and child totals are not computed.
- Suspend and resume rollups are not computed.
- Query-time summation, display, filtering, sorting, and aggregate-table
integration are not included. These are tracked in
[OBS-902](https://linear.app/kepler-crm/issue/OBS-902) and
[OBS-903](https://linear.app/kepler-crm/issue/OBS-903).
- Counts reflect observed exported failures only. Filtering, sampling,
or loss before export can leave counts incomplete.

## Change size

- 32 files changed
- 862 lines added
- 161 lines removed

## Validation

- Completed checks on the current head: not available from the supplied
evidence.
- Author-reported results, not verified here:
- Focused observability tests, codec and durable-serialization checks,
and engine and Inngest retry scenarios.
- Node 24 builds, 167 core checks, 125 observability and redaction
tests, and 19 real Inngest scenarios.
  - Type-aware lint and `lint:format`.
- Known limitations:
- A prior monorepo E2E attempt did not run application assertions.
Fixture dependency installation failed with a pnpm store ENOENT error.
- Earlier runtime tests ran on Node 23.11.0, which is outside the
supported engine range.
- Standalone observability `tsc` reports errors in an untouched test
utility.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-12 14:21:22 +01:00
Ward Peeters
46ce164dde
fix(factory): recover stuck environment builds and rebuild on saved changes (#30148)
Environment templates could stop rebuilding for good, and changing a
setup command never rebuilt them.

When a build start isn't recorded (for example the process restarts
mid-start), Factory marks the start uncertain and asks the provider for
its build list before building again. Mastra Platform has no build list,
so that mark could never clear: every Build now, schedule and push run
skipped with `unresolved` forever and sessions kept booting an old
template. Now an uncertain start ages out 30 minutes (the build wait
ceiling) after its lease lapsed, Factory logs once that it gave up on
it, and the next trigger builds. A new `build_start_uncertain_since`
column records when the mark started; rows already stuck get stamped on
their next read and recover 30 minutes later.

Saving a template input (setup commands, working directory, settings,
environment repositories, Build variables) also never started a build,
so each new sandbox reran the changed setup command. Saves now start a
build once the project has a template, only when the inputs actually
differ from the last build. Repeated saves don't double-build, and a
change saved during a build is built when that build ends.

Before: a project stuck on `unresolved` stayed on its Oct 28 template,
and a new setup command reran for 40s in every sandbox. After: the stuck
project builds on its next trigger after 30 minutes, and saving the
setup command rebuilds the template.

Tested with red/green tests on LibSQL and Postgres 16 using real storage
and a fake clock, plus route tests for every save path.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

**ELI5:** Sometimes the system got stuck while it prepared a project's
build environment, and it did not try again for a long time. Also,
saving changes to the project's setup did not start a new build. This
change fixes both problems. Stuck builds now recover, and saved changes
now start a new build when they affect the template.

## Overview

This change fixes two problems in Factory environment template builds.

1. **Stuck build starts.** A build start can stay `unresolved` for a
long time when the provider cannot list builds.
2. **Missing rebuild on save.** Saving changed template inputs did not
start a build.

Diff base: `main` at merge base
`e1753c7199`. The diff changes 17 files,
with 721 lines added and 118 removed.

## Changes

### Stuck build starts

- `build.ts` adds `MAX_BUILD_WAIT_MS` (30 minutes). `build-workflow.ts`
imports it and no longer declares or exports it. The workflow still uses
it as the default maximum wait.
- `build.ts` adds `UNRESOLVED_BUILD_START_TIMEOUT_MS`. It equals
`MAX_BUILD_WAIT_MS`.
- When the provider cannot list builds, triggers skip as `unresolved`
until the timeout passes after the start lease lapses.
- After the timeout, the run treats the uncertain start as failed. It
clears the mark, logs the lapse, and continues. A later trigger can then
build.
- `projects/base.ts` adds `buildStartUncertainSince: Date | null` to
`FactoryProject`. The projects schema adds a nullable timestamp column
for it.
- `claimBuildStart` keeps the later of the lapsed lease expiry and any
existing mark time. A legacy mark without a timestamp uses the current
claim time.
- `releaseBuildStart` clears the flag and the timestamp.
- Existing uncertain markers receive a timestamp on their next read.

### Rebuild on saved changes

- `build.ts` adds `settings` to `ENVIRONMENT_BUILD_TRIGGERS`. This also
adds it to `EnvironmentBuildTrigger`.
- A `settings` run skips in this order:
  1. `no_environment`: the environment has no repositories.
2. `no_template`: the project has no active template and no last build.
  3. `unchanged`: the template inputs match the last build's inputs.
- The skip check runs before the build-start lease is acquired. It runs
again after the lease is acquired.
- Build environment variables are resolved once. The same values feed
the build context and the inputs fingerprint.
- The outcome documentation adds `no_template`.
- `build-runner.ts` adds an `onSettled` handler to the workflow. The
handler runs for every run, including scheduled runs.
- After a `started` run, it clears the project's pending marker and
schedules one more `settings` run.
  - After a run skipped as `starting` or `in_progress`, it does nothing.
- After any other outcome, it schedules a `settings` run only if a save
was pending.
- `onEnvironmentSaved` starts a `settings` run.
- On `in_progress`, it marks the project pending and returns. The
current run's settlement starts the next run.
- On `starting`, it marks the project pending and retries. The first
delay is 1 second. Each retry doubles the delay, up to 30 seconds. It
stops when the pending marker is cleared.
  - Other outcomes return without a retry.
- The runner accepts an optional `sleep` function. The default uses a
timer.
- Hand-offs run in the background. Failures are logged and do not fail
the save.

### Save-path callbacks

- `factory.ts` passes `onEnvironmentSaved` to sandbox environment
routes. It also passes `onRepositoryPush` and `onEnvironmentSaved` to
project routes. Before, project routes received only `onRepositoryPush`.
- `routes/projects.ts` calls
`environmentBuilds?.onEnvironmentSaved(projectId)` in these cases:
  - After a repository link.
  - When project environment changes are saved.
  - When a repository is patched.
  - When a repository is unlinked.
- After a source-control connection is unlinked and its repositories are
deleted.
  - It removes the comment that said saving never builds.
- `routes/sandbox-envs.ts` adds optional `onEnvironmentSaved?:
(projectId: string) => void` to `SandboxEnvRoutesDeps`. It calls the
callback after a successful create, update, or delete. Validation
failures, refused writes, and unsuccessful deletes do not call it.
- `routes/source-control-settings.ts` adds optional
`onEnvironmentSaved?: (projectId: string) => void` to
`SourceControlSettingsRoutesOptions`. After a successful update, the
POST handler looks up the connection. If the connection exists, it calls
the callback with its `factoryProjectId`. A failed update still returns
404. A missing connection skips the callback.
- `routes/surface.ts` adds the same optional callback to
`FactoryApiRoutesDeps`. `assembleFactoryApiRoutes` passes it to the
source-control settings routes when present.

### Documentation and changeset

- `docs/.../factory-sandbox.mdx` says that a saved template-input change
starts a build when a template exists. It says that changes saved during
a build are built when it ends. It says that a save with no input change
does not start a build. It adds the 30-minute `unresolved` rule.
- `.changeset/calm-builds-recover.md` adds a patch changeset for
`@mastra/factory`.

### Tests

- `build.test.ts`:
- The unlistable-provider test checks recovery at the timeout boundary.
Before the timeout, manual, schedule, and push triggers stay
`unresolved`. At the timeout, a manual trigger builds and clears the
marker.
  - Adds schedule and push recovery tests at that boundary.
  - Adds a test that stamps legacy markers on first read.
- Adds `settings` tests. Builds run only for template-input changes, not
for head-only changes. Unchanged and in-progress outcomes are covered.
- `build-schedule.test.ts` adds tests for a setup-command change saved
during an active build. The changed command builds after the active run
ends. A `settings` start skipped as `starting` does not block the
changed command from building after a schedule run exits unchanged.
- `projects.test.ts`:
  - Covers a rebuild after environment changes when a template exists.
- Confirms that a save before the first build does not trigger a build.
  - Confirms that unchanged setup commands do not trigger a build.
- Covers a save during an active build, which is queued until the build
finishes.
  - The `seedBuilding` helper accepts an injectable sleep function.
- `sandbox-envs.test.ts` supplies a mock callback. It checks that the
callback receives the project ID once for each of three stored changes.
- `source-control-settings.test.ts`:
  - The helper accepts an optional callback.
  - Repository seeding returns the created project.
- The GitHub/GitLab settings test checks that the callback receives the
project ID after commands are saved.
- A new test checks that a failed connection lookup returns 500, saves
no command, and skips the callback.
- `event-worker.lifecycle.test.ts` checks that a push rebuild runs two
successful workflow runs and starts one build.

## Public API changes

- Added: `buildStartUncertainSince: Date | null` on `FactoryProject`.
- Added: `onEnvironmentSaved?: (projectId: string) => void` on
`SandboxEnvRoutesDeps`, `SourceControlSettingsRoutesOptions`, and
`FactoryApiRoutesDeps`.
- Added: `MAX_BUILD_WAIT_MS` and `UNRESOLVED_BUILD_START_TIMEOUT_MS`
exports in `build.ts`.
- Added: `onSettled?` on `EnvironmentBuildWorkflowOptions`. The runner's
options omit both `onStart` and `onSettled`.
- Changed: `ENVIRONMENT_BUILD_TRIGGERS` includes `'settings'`.
- Removed: the `MAX_BUILD_WAIT_MS` export from `build-workflow.ts`.

## Verification

- The PR description says the author tested on LibSQL and Postgres 16
with real storage and a fake clock. It says route tests cover every save
path. This review did not run these tests.
- CI and build results: not available.
- Review severity counts: not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-12 14:08:56 +01:00
Eric Pinzur
f0710446e0
feat(observability): sort spans by model and provider (#30096)
Adds ascending/descending `model` and `provider` ordering to
`querySpans()` across Core, server, Client JS, PostgreSQL, DuckDB and
ClickHouse. No schema changes or backfill.

```ts
orderBy: [{ field: 'model', direction: 'asc' }]
```

Sorting uses the dedicated columns before pagination, with
case-sensitive UTF-8 ordering, missing values last, and identity
tie-breakers. String cursors preserve their values and remain bound to
the query and trusted scope. Legacy attribute-based display values
remain available without changing cursor boundaries. Trace/thread sort
fields stay unchanged. The server advertises
`spanQueryModelProviderSort` and rejects unsupported stores.

This PR is API-only. Studio sorting controls and live-refresh changes
are tracked in
[OBS-906](https://linear.app/kepler-crm/issue/OBS-906/add-model-and-provider-sorting-to-spans-explorer).
Enabling those controls needs raw model/provider sorting values
available to the existing head-row merge, including when polls are
ordered by end time; using display values can disagree with database
ordering. The PR has no remaining Playground UI changes or release
entry.

Validation: Core/server tests, real storage conformance across
PostgreSQL, DuckDB and ClickHouse, mixed-core checks, and
Core/server/adapter typechecks passed for the API implementation. After
removing the UI changes, both existing span component/hook suites pass
(52 tests). Changed-file lint and docs checks including CI Vale pass.

A bounded local ClickHouse 26.6.8.7 comparison used 180,000 synthetic
rows, 10,000 matches per tenant, distinct tenants across cases, both
directions and continuation pages, and disabled query/condition caches.
Median CPU was 21.9 ms for start time, 19.4 ms for model, and 21.4 ms
for provider; median query memory was about 4.6 MiB for each (maximum
6.8/8.4/7.2 MiB). These are selection-query measurements on synthetic
OSS data, not a production benchmark.

Part of OBS-896. Platform API support and package pins follow this OSS
merge and snapshot publication. UI enablement is tracked separately in
OBS-906.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

**ELI5:** Trace spans can now be sorted by their model name and provider
name, in both directions. The database does the sorting, so users can
page through the results in the right order.

## Overview

This change adds `model` and `provider` as sort fields for span queries.
It applies to the Core API, the server handlers, and the PostgreSQL,
DuckDB, and ClickHouse stores. The diff has 31 changed files, with 320
lines added and 43 removed.

## Core changes

- `SPAN_QUERY_ORDER_FIELDS` now includes `model` and `provider`.
- `SpanQueryPlanOptions` adds `allowSpanModelProviderOrdering?:
boolean`. Ordering by these fields requires this option.
- Trace and thread queries do not support these fields.
- Cursor validation accepts string or null values for `model` and
`provider`. Timestamp normalization applies only to `startedAt` and
`endedAt`.
- `SpanQuerySelectedRow` adds an optional `cursorSortValue`. Response
construction removes this field from returned spans. It uses the value
for the next-page cursor, and falls back to the row's ordering field
when the value is absent.
- The base storage feature union adds `span-query-model-provider-sort`.

## Server changes

- `assertObservabilitySpanModelProviderSortSupported` returns HTTP 501
with `SPAN_QUERY_UNSUPPORTED` when a store does not declare
`span-query-model-provider-sort`. The check runs before `querySpans` is
called.
- `ObservabilityStorageCapabilities` and its empty default add
`spanQueryModelProviderSort`, which defaults to `false`.
- The capability is reported as supported only when span queries are
supported, the core order fields include both `model` and `provider`,
and the store declares the feature.
- The system capabilities schema adds `spanQueryModelProviderSort:
z.boolean().default(false)`.
- The route handler permits model and provider ordering and performs the
capability check.

## Store changes

- **PostgreSQL:** Sorts by `model` and `provider` use `COLLATE "C"`. The
cursor bounds cast sort values to `text COLLATE "C"` when `stringSort`
is enabled. The feature is advertised only when the core order fields
include both fields.
- **DuckDB:** Sorts by `model` and `provider` use `"C"` collation.
Cursor values cast as strings with the same collation. The feature is
advertised only when the core order fields include both fields.
- **ClickHouse:** `afterCursor` treats `model` and `provider` as
strings. Null rows are included after the cursor for numeric and string
sorts. The feature is advertised only when the core order fields include
both fields.
- Returned `model` and `provider` values may still come from span
attributes. The code comments state that these fallback values are not
used for filtering or sorting.

## Tests and conformance

- Core tests cover ordering gating, rejection in trace and thread
queries, string and null cursor values, and rejection of cursors when
the order, filters, authorization, or scope change.
- Server tests cover the 501 rejection for `durationMs`, `model`, and
`provider` on older stores. They also cover capability parsing and the
capability flag in the system endpoint.
- Shared conformance tests in `stores/_test-utils` cover ascending and
descending pagination at page sizes 1, 2, and 3. They cover empty
strings, missing values, case and Unicode ordering, ties, and filter
exclusion. They also check that missing values sort last and that legacy
attribute values are not treated as dedicated fields.
- Store feature-list tests for ClickHouse, DuckDB, and PostgreSQL expect
the new feature flag.

## Changesets and documentation

- Changesets add minor release notes for `@mastra/core`,
`@mastra/server`, `@mastra/client-js`, `@mastra/duckdb`,
`@mastra/clickhouse`, and `@mastra/pg`.
- The advanced query reference adds the `spanQueryModelProviderSort`
capability row. It documents `model` and `provider` ordering. It states
that the comparison is case-sensitive and that missing values sort last
in both directions.

## Scope and limits

- This diff does not include Studio UI changes. The commit history says
UI enablement is deferred to a separate issue because it needs raw model
and provider values for head-row merging.
- This diff does not include client-js source changes. Its client-js
changeset documents the new sort fields.
- The diff does not include schema changes or backfill.
- The PR description reports these results, which this review did not
run or verify:
- Core and server tests, and real shared-storage conformance and
mixed-core checks, pass.
- Core, server, and adapter typechecks pass. Staged-file lint and docs
checks, including CI Vale, pass.
  - 58 Studio and MSW tests pass.
- The full Playground UI typecheck reports four unrelated chart and form
errors, and the library declaration build fails.
- A local ClickHouse 26.6.8.7 comparison with 180,000 synthetic rows
measured median CPU of 21.9 ms for start time, 19.4 ms for model, and
21.4 ms for provider. Median query memory was about 4.6 MiB for each.
These figures are not production measurements.
- Platform backend and UI wiring, and package pins, remain under
OBS-896.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-12 13:52:37 +01:00
devin-ai-integration[bot]
ba2529c209
docs(models): explain when to use custom gateways and the base class (#30066)
## Description

The custom gateways guide showed a plain-object gateway and a
`MastraModelGateway` subclass. It didn't explain when to use a custom
gateway or why you'd extend the class.

Changes:

- `docs/src/content/en/models/gateways/custom-gateways.mdx`
- New "When to use custom gateways" section, which replaces the "Create
custom gateways to support" list.
- It describes a custom gateway as the way to connect Mastra to
open-weight models you run yourself, such as gpt-oss, Llama, Gemma,
Mistral, Qwen, or DeepSeek.
- It lists when self-hosting fits: compliance rules on sending data to
third parties, per-token pricing that costs more than running your own
GPUs, fine-tuned models, model versions that only change when you
upgrade, and offline, air-gapped, or edge deployments.
- It notes what the gateway adds: typed model IDs and a Studio model
picker, per-call credential resolution, and Studio filtering with
`AUTO_BLOCK_EXTERNAL_PROVIDERS`.
- For a single OpenAI-compatible endpoint, it points to `url` in the
model config instead. It also says to set `apiKey` there, because Mastra
doesn't read environment-variable keys for `url` models.
- New "Choose a plain object or a class" subsection. Both forms behave
the same at runtime, because the base class defaults for `getId()`,
`shouldEnable()`, `handlesModel()`, and `serializeForSpan()` match the
fallbacks in `gateway-helpers.ts`. A class makes sense when you need
constructor options, private state, or several instances.
- Replaced the class example with one that takes optional `id`,
`baseURL`, and `apiKey` constructor options. Added a two-instance
example with distinct IDs, because `GatewayManager` deduplicates
gateways by ID. Both examples now forward `headers` in
`resolveLanguageModel()`.
- "Showing only custom gateways in Studio" now says the filter also
applies to `GET /agents/providers`. It only changes which providers are
listed, and doesn't stop agents from calling other providers in code.
- `docs/src/content/en/docs/studio/overview.mdx`: the model picker
section now mentions `AUTO_BLOCK_EXTERNAL_PROVIDERS` and links to the
custom gateways section.

## Related issue(s)

Docs follow-up from a Slack question about `MastraModelGateway`.

## Type of change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [ ] Test update

## Checklist

- [ ] I have linked the related issue(s) in the description above
- [x] I have made corresponding changes to the documentation (if
applicable)
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] I have addressed all Coderabbit comments on this PR

Link to Devin session:
https://app.devin.ai/sessions/922f1f27155a4c858ce0e76079f78b2f
Open in Devin Desktop:
https://app.devin.ai/desktop/session/922f1f27155a4c858ce0e76079f78b2f?variant=devin

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: paul <paul@mastra.ai>
2026-11-12 12:48:28 +00:00
mastra-platform[bot]
686c178917
fix(voice): replace retired default OpenAI Realtime model (#30183)
## Summary
- Default new OpenAI Realtime voice connections to
`gpt-realtime-2.1-mini` instead of the retired preview model.
- Align the constructor examples, README, and voice integration
documentation with the new default.
- Cover default and explicit model selection with a real local WebSocket
handshake regression test and a patch changeset.

## Test plan
- `pnpm turbo build --filter="@mastra/voice-openai-realtime^..."`
- `pnpm --dir voice/openai-realtime-api test` (70 tests)
- `pnpm --dir voice/openai-realtime-api exec tsc --noEmit --project
tsconfig.json`
- `pnpm --dir voice/openai-realtime-api lint`
- `pnpm --dir voice/openai-realtime-api build`

Fixes #29722

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

**ELI5:** The OpenAI voice connector picked an old model by default, and
OpenAI has shut that model down. This change makes the connector pick a
current model by default. Users who set their own model still get that
model.

**Overview**

This change replaces the default OpenAI Realtime model in
`@mastra/voice-openai-realtime` with `gpt-realtime-2.1-mini`. The
previous default was `gpt-4o-mini-realtime-preview-2024-12-17`. Linked
issue `#29722` reports that OpenAI lists this snapshot as shut down.

**Changes by file (5 files, +36/-8 lines)**

- `voice/openai-realtime-api/src/index.ts` (+4/-4): Sets the default
realtime model to `gpt-realtime-2.1-mini`. Updates the class example,
the model-option documentation, and the constructor example.
- `voice/openai-realtime-api/src/connection.integration.test.ts`
(+24/-1): Adds a local WebSocket handshake test. The `setup` helper
accepts an optional model and types the connection callback with the
WebSocket and `IncomingMessage`. Parameterized tests check the model in
the connection URL for two cases: no model configured (expects
`gpt-realtime-2.1-mini`) and an explicit model (expects that model).
- `voice/openai-realtime-api/README.md` (+1/-1): Usage example uses
`gpt-realtime-2.1-mini`.
- `docs/src/content/en/integrations/voice/openai.mdx` (+2/-2): Realtime
usage example and constructor options use `gpt-realtime-2.1-mini`.
- `.changeset/realtime-mini-default.md` (+5/-0): Patch changeset for
`@mastra/voice-openai-realtime`.

**Behavior**

- Without a `model` option, the connection URL requests
`gpt-realtime-2.1-mini`.
- With a `model` option, the connection URL requests the configured
model.
- No exported or public declarations change.

**Validation**

The author reports these commands. This summary does not confirm their
results:
- `pnpm turbo build --filter="@mastra/voice-openai-realtime^..."`
- `pnpm --dir voice/openai-realtime-api test` (70 tests)
- `pnpm --dir voice/openai-realtime-api exec tsc --noEmit --project
tsconfig.json`
- `pnpm --dir voice/openai-realtime-api lint`
- `pnpm --dir voice/openai-realtime-api build`

**Scope limits**

The linked issue also mentions future shutdowns for `whisper-1` and
`tts-1`. This change does not alter those defaults.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-11 23:07:34 -08:00
mastra-platform[bot]
b75348adbd
docs: redirect commonly hallucinated docs URLs to real pages (#30126) 2026-11-11 16:07:08 -07:00