mirror of
https://github.com/mastra-ai/mastra.git
synced 2026-11-17 08:44:46 +00:00
237 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
d60ba71be0 | chore: version - exit prerelease mode | ||
|
|
055a8c3b82 | chore: version packages | ||
|
|
743a02786e |
feat(valkey): identify Mastra-created clients with a clientInfoTag (#30156) | ||
|
|
c014d187bb |
chore(deps): update docker images (major) (#19788) | ||
|
|
ce03237b34 |
fix(core): send only the response messages that changed on agent stream events (#29971)
## Description Follow-up to #29965 (merged). After that PR, each plain Agent `step-finish` on `agent.stream.<runId>` still carried the run's response messages so far, so a long run stored every tool output once per later step: the log still grew with the square of the step count, mostly from large tool results. The publisher now keeps a cursor of message fingerprints and sends only the messages from the first one that changed: ```ts // step 1: { from: 0, seq: 0, messages: [A1, T1] } // step 2: { from: 2, seq: 1, messages: [A2, T2] } // step 3: { from: 4, seq: 2, messages: [A3] } _durableResponseMessagesDiff: { producer, seq, from, base, hash, messages } ``` "From the first changed index" (not "only new messages") because earlier entries change: a step's assistant message grows across steps, and a processor retry rolls the list back. Readers rebuild the full list per step in the stream adapter, before chunks reach `MastraModelOutput`, so `onStepFinish(step).response.messages`, `output.steps` and `onFinish` still see the whole list: - `agent.observe(runId, { offset })` first folds the step-finish events before the offset from the run's history (bounded wait, doesn't delay the subscription). - A diff only applies to the list it was made against (`base` must equal the hash of the list the reader holds). One that can't be placed (a missed or reordered delivery, a late copy) is never applied, and the reader reports no messages for that step instead of a partial list. - The publisher resends the whole list once the diffs since the last whole list add up to its size, so whole lists cost at most about as much as the diffs and the log stays linear. A failed publish or diff makes the next one a whole list, so a reader that missed one recovers. - On a bus that keeps no history of the run's topic (`cache: false`, or a `CachingPubSub` whose `shouldCache` leaves the topic out), every step-finish is a whole list, so an observer that attaches mid-run can place it. An internal `PubSub.__replaysTopic(topic)` answers this per topic, since `CachingPubSub.supportsOffsets` is true even for topics it doesn't cache. Transports without offsets (Redis and Valkey Streams without a cache) get whole lists too: an `observe()` from an offset there follows the stream from its tail. - Each diff carries its publisher's id: a run resumed after approval publishes from a new publisher that starts with the whole list. Readers keep the list of the current and previous publisher apart, so a late event from the previous one doesn't break the next diff. - The diff replaces `_durableResponseMessages`, so an older reader during a rolling upgrade reports no messages rather than a partial list. DurableAgent and EventedAgent still send whole lists, which the adapter handles as before; they move to diffs in a follow-up. ## Related issue(s) Part of COR-1532 (plain Agent; DurableAgent/EventedAgent and Inngest follow in the next PRs of this stack). Follow-up to #29965 (production Redis OOM from plain Agent run logs). Related: #27475. ## Type of change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Code refactoring - [x] Performance improvement - [x] Test update ## Checklist - [x] I have linked the related issue(s) in the description above - [ ] I have made corresponding changes to the documentation (if applicable) - [x] I have added tests that prove my fix is effective or that my feature works - [ ] I have addressed all Coderabbit comments on this PR Tests: `agent-stream-log.scenario.test.ts` (all engines) adds "an early tool result is stored as often as the last one" (fails before this PR on the plain Agent; durable engines pinned as the known gap), an observer resuming mid-run at an offset (fails without the history fold), an observer joining mid-run on a bus without history (fails without the whole-list rule), and an approval suspend/resume replay compared step by step. `response-messages.test.ts` covers the diff and apply rules (edits, shrinking lists, duplicates, lost diffs, producer switches, a late whole list from the previous producer, the size-based whole list). The real-Redis test asserts the tool-result counts and that a 20-step run stores less than 2.5x the bytes of a 10-step one (2.1x with this PR, 2.9x before it). <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 Each step used to save the full list of response messages again, even when most messages had not changed. This change saves only the updates and rebuilds the full list for observers, which can reduce storage during long runs. ## Summary - Plain Agent stream events now publish response-message diffs. The publisher sends a full list first and periodically resends one as diffs accumulate. - Readers apply diffs when they match the current message list. If a reader cannot rebuild the list, it reports no response messages for that step rather than a partial list. - When `agent.observe()` starts at a numeric offset, it replays earlier events to rebuild response messages before delivering later events. Observers continue to receive the full list for each step. - DurableAgent and EventedAgent continue to publish full lists. Tests add coverage for diff replay, offset replay, approval resumption, tool-result repetition, and Redis stream storage growth. - Older observers report no response messages for steps published with diffs. The changeset recommends upgrading agent and observer processes together. Test results: not available. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
749c09cf91 |
fix(core): publish lean agent stream events instead of the loop's chunks (#29965)
## Description
A registered Agent publishes every run on `agent.stream.<runId>` so
`agent.observe()` and reconnecting clients can replay it, and on Redis
that log is stored until the run is cleaned up. The publisher forwarded
the loop's in-process chunks as they were: every `step-start` carried
the provider request, and every `step-finish` carried the prompt again
(`metadata.request`, `messages.all`/`user`) plus every earlier step with
its own prompt (`output.steps`). The log grew roughly with the cube of
the step count; in production single runs of a plain Agent took 0.3 to
3.4 GB of Redis and filled it.
The publisher now builds `step-start`, `step-finish` and `finish` field
by field (`agent/streams/wire.ts`) with what readers use, which is what
DurableAgent already publishes:
```ts
// step-finish payload on the topic, before
{ stepResult, metadata: { ...response, request: { body: '<whole prompt>' } },
output: { text, toolCalls, usage, steps: [/* every step, each with its prompt and response */] },
messages: { all, user, nonUser } }
// after
{ stepResult, metadata: { id, timestamp, modelId, headers, providerMetadata, ... },
output: { usage /*, steps: [{ tripwire }] when the step was rejected */ },
_durableStepContent: [/* this step's content */],
_durableResponseMessages: [/* response messages so far */] }
```
- The caller's own stream and callbacks are unchanged; only the
published copy is reduced.
- Observers of a plain Agent get better replays: each step's content was
`[]` before, and `onFinish` lost its text over JSON transports and threw
on in-memory ones (the finish steps carried AI SDK-shaped `toolCalls`
the reader can't read). Finish steps now carry `text`, and tool
calls/results in the record shape DurableAgent uses.
- A delegated agent's chunks inside `tool-output` /
`workflow-step-output` are slimmed by the plain publisher. Durable
engines still publish them as-is, since their callers read their own run
back from the topic; that case is pinned with `it.fails`.
- Step content is read from the run's message list when the step
finishes (one extra response-to-UI conversion per step, the same work
the caller's output does).
Still to come, stacked on this PR: each `step-finish` still sends the
response messages so far in full; follow-ups send only what changed, for
the plain Agent first and then DurableAgent/EventedAgent and Inngest
(superseding #27522).
## Related issue(s)
Fixes COR-1530. Production incident: shipyard Redis OOM from
`agent.stream.<runId>` keys of plain Agent runs (Factory). Related:
#27475 (the DurableAgent side of the per-step growth) and COR-1532
(per-step response messages, the follow-up stack), introduced with
#28424.
## Type of change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [x] Performance improvement
- [x] Test update
## Checklist
- [x] I have linked the related issue(s) in the description above
- [ ] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR
Tests: `agent-stream-log.scenario.test.ts` runs on every engine (prompt
copies on the log, delegated agent prompt, observer replay over
in-memory and JSON transports, a rejected and retried reply), and
`pubsub/redis-streams/src/agent-stream-size.test.ts` checks what real
Redis holds in both cache modes. The new plain-agent cases fail on main.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## ELI5
Agent stream logs repeat old prompt and step data, which can make them
large. This change publishes smaller event records while keeping the
caller’s stream intact and giving observers step content they need.
## Summary
- The publisher creates reduced `step-start`, `step-finish`, and
`finish` events. It removes provider requests and repeated step data.
- For `step-finish`, the publisher reads the current step’s content from
the run’s message list. It omits content for retry steps and logs a
warning if it cannot read the content.
- The publisher reduces nested agent chunks inside `tool-output` and
`workflow-step-output`. DurableAgent and EventedAgent continue to
publish their delegated-agent steps in full.
- Observer replay gains step content and richer finish data, including
run text and tool calls, and step reasoning, sources, and files. The
event still includes response messages on each `step-finish`.
- Added scenarios cover prompt retention, delegated-agent prompts,
observer replay over in-memory and JSON transports, and rejected and
retried replies. A Redis integration test checks stored stream size and
observer replay with both cache modes.
## Validation
Completed CI, build, and test results: not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
d77893aed3 |
fix(core): resume evented agent observe() at the offset on Redis Streams with the default cache (#29798)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9 ## What Fixes #28324. On an evented agent (`createEventedAgent()`) using a cross-process PubSub such as `RedisStreamsPubSub` with the default in-memory server cache, `observe(runId, { offset })` ignored the offset. A client reconnecting mid-run got earlier events again, and an offset past the end of a finished run replayed nearly the whole run. ## Why it happened Evented step frames went out unnumbered whenever an in-memory cache sat behind a cross-process transport. The concern was that a worker in another process would number the run from 0 in its own cache, and observers would then drop its frames as duplicates. That skipped numbering in the common case too, where every step runs in the process that started the run. ## Change - `CachingPubSub` gains two `@internal` options, `extendExistingLogsOnly` and `onUnlogged`. With `extendExistingLogsOnly`, a publish only appends to a topic whose log this cache already holds. Any other publish goes out live and unnumbered, and `onUnlogged` is called. - `StepExecutor` now always wraps the step bus when a server cache exists. If the cache is process-local and the bus crosses processes, it turns on `extendExistingLogsOnly`. The caller writes the run's `start` frame before any step runs, so steps in the caller's process extend that log and get numbered. Steps on another worker find no log, stay live-only as before, and log a one-time warning that recommends `RedisServerCache`. - `StepExecutor` now picks up the Mastra logger when it is constructed. Workflow event processors create executors without registering them on Mastra, so without this the warning was dropped. - Docs (`durable-agent.mdx`) list the two setups that still need a shared cache: steps running on a dedicated worker, and observers reconnecting through another replica. ## Tests - `pubsub/redis-streams/src/observe-offset.test.ts`: the evented + default cache row now passes the exact-resume checks (resume at offset, offset past end returns nothing, mid-run replay). The full matrix passes 12/12 against Redis 7. - `step-executor.test.ts`: - When the caller holds the run's log, step frames are numbered. - When a worker has no log, its frames stay unnumbered and one warning reaches the logger configured on `Mastra`, using an executor that was never registered on Mastra. - Both cases fail on main. - `caching-pubsub.test.ts`: tests `extendExistingLogsOnly` both with and without an existing log. - `stores/libsql` cross-process suite, including `xproc-input-tripwire`: 48 passed, 0 failed. - Core `tsc` is clean. oxlint and prettier pass on the changed source files. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 When an evented run starts, the process that started it can now number its events even when events travel through Redis. That lets a client resume from an offset without seeing old events again. Events from a different worker remain unnumbered when only a local cache is available, and Mastra warns about this limitation. ## Changes - Added internal `CachingPubSub` options to append only to existing cached logs and report events that cannot be numbered. - Updated `StepExecutor` to use those options with a process-local cache behind cross-process PubSub. It numbers events for runs whose log exists locally and warns once when a worker publishes events for a run it did not start. - Updated documentation to explain when a shared cache is required, including worker execution and reconnects through another server process. - Added tests for existing and absent cached logs, step numbering and warnings, and offset replay scenarios. ## Validation Test, build, and CI results: not available. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com> Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com> Co-authored-by: NikAiyer <nick.aiyer@gmail.com> |
||
|
|
f21e0742ce | chore: version packages | ||
|
|
d6a5f98df3 |
fix: index observe chunks for resume (COR-1506) (#29520)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9 Linear: [COR-1506](https://linear.app/kepler-crm/issue/COR-1506/observe-offset-cant-be-used-to-resume-it-counts-log-events-but-clients) ## Problem `observe({ runId, offset })` takes `offset` as a position in the run log. Clients can't compute that position: - Some log entries never reach the client. These include iteration-complete records after every step and ownership markers. - Updates on the wire carried no index. A client that resumes at "number of updates received" lands too early and gets repeated updates. The drift grows by one per completed step, so every tool-using run that reconnects after its first step is affected. On a 3-step tool loop, 16 of 28 cut points repeated 1–2 updates on both durable and evented engines. ## Fix - **Index on updates:** every update now carries `index`, the run-log position it came from. This covers: - `/observe` on every agent - `/stream` on durable and evented agents - core `stream()`/`observe()` chunks and client-js `observe()`/`stream()` - **Resuming:** pass `index + 1` of the last received update that has an index as `offset`. Gaps in the numbering are expected; they're log entries that send nothing. - **Unindexed updates:** the synthesized `abort` update carries no index. Its log position is on the `finish` sent beside it. Client-built structured-output `object` chunks have no index either. An `object-result` logged by a structuring-model run carries one. Unindexed updates don't move the resume point. A client that reconnects between `abort` and `finish` gets the `abort` again. - **`offset` is unchanged:** it still means a run-log position. - Core no longer strips `index` in `DurableAgent.__subscribeToRunStream`. The stream adapter and the server's `toAgentStreamChunks` copy the envelope's index onto each chunk and drop any index a forwarded sub-agent chunk carried. `index` is optional on `BaseChunkType` and `DataChunkType`. - **Also fixed:** on plain agents, `/observe` (HTTP and core `agent.observe()`) with an offset past the end of a finished run used to wait for the idle timeout or hang. It now closes immediately. It reads the log from just before the offset and only falls back to a full read when that window is empty. The check runs in the background, so it never delays returning the stream or its cancel handle. A trailing `error` chunk doesn't count as the end of the run. ## Migration `index` is a new field and `offset` keeps its meaning: - Callers that pass a log position need no change. - Callers that replay from `offset: 0` need no change. They get the same updates in the same order, now with `index` set. - Only callers that passed a count of updates received need to change. They were already getting repeats after step boundaries; switch them to `index + 1` of the last indexed update. ## Evidence | Check | Result | | --- | --- | | Red/green | With `main`'s core and server, 15 tests on numbering and `/stream`–`/observe` matching fail; with this change all pass. | | Cut points | Every cut point of a 3-step tool loop on durable and evented engines resumes from the first `/stream` connection at last index + 1. Covers the HTTP route and client-js `stream()` → `observe()`. | | `/stream` vs `/observe` match | Frames are identical JSON, including `index`. Replay from 0 equals the first `/stream`. | | End of run | Resume after finish or past the end returns the rest and closes, including far past the end (offset 100). | | Adapters | Hono 98/98, Fastify 98/98, Express 90 passed + 8 skipped (see bug 1 below). The shared suite is now wired into Express and Fastify. | | Redis restart | Real-Redis split-deploy test (in `pubsub/redis-streams/src/observe-cross-process.test.ts`) passes on both engines. The client resumes on a freshly booted API process at last index + 1 with nothing repeated or missing. Evented: the worker executes the run and the API is killed with SIGKILL. Durable runs execute in the API that started them, so that API stays up; the client drops its connection and resumes on a second API. | | Suspend | After tool approval, indexes keep increasing. Observers parked mid-suspend and every cut across the suspend resume correctly on both engines. | | Unit suites | server `agents.test.ts` 289, core `src/stream` + `src/agent/streams` 481 | | Static | tsc clean on core, server, client-js, the test suite, the Hono/Express/Fastify adapters and redis-streams. oxlint 0 warnings. | ## Scope Plain (non-durable) agents' `/stream` updates aren't numbered yet. The caller receives each update before the cache assigns its number. That's tracked in COR-1510. Plain agents' `/observe` is covered here. ## Existing bugs found (not fixed here) 1. **Express untilIdle 404:** after a client drops an untilIdle stream on a durable or evented agent, Express answers `/observe` with 404 "Run not found". It fails the same way on `main`. Those 8 cases are skipped on Express only, via `skipUntilIdleDisconnectReplay`. 2. **Finished durable run, other API:** a finished durable run returns 404 on `/observe` from any API process except the one that executed it (`authorizeDurableRun`, `handlers/agents.ts`). Resuming while the run is live works from any API. 3. **Plain replay mismatch:** on plain agents, replayed `step-finish`/`finish` payloads differ from the live ones (for example `timestamp` and `processorRetryCount`). Related to COR-1510. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 When a connection drops, clients need to know which updates they already received. This change adds positions to many updates so clients can resume from the right place. It also closes observations of finished runs when the requested position is past the end. ## Summary - Added optional `index` values to chunks from durable and evented agent streams, `observe()` streams, and corresponding HTTP routes. Clients can resume by passing the last indexed chunk’s `index + 1`. - Preserved run-log indexes through core stream adapters and server frame conversion. Sub-agent data chunks do not carry a child run’s index into the parent stream. - Some chunks remain unindexed, including synthesized `abort` chunks and client-built structured-output `object` chunks. Failed cache writes and non-atomic custom-cache indexing can affect replay. - Updated API schemas and documentation to explain index gaps, replay limits, and resumption. - Added past-end handling for finished plain-agent observations in core and HTTP `/observe`. History checks do not delay stream return or prevent cancellation. - Added tests for indexed replay, past-end closure, cancellation during history reads, sub-agent index isolation, and observation across API restarts. **Validation:** Current-head CI, build, and test results are not available. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com> Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com> Co-authored-by: NikAiyer <nick.aiyer@gmail.com> |
||
|
|
75435bc2cd |
fix(redis-streams): retry publishes on transient connection errors (#29512)
## Summary The Redis Streams pubsub now retries a publish that fails with a transient connection error, instead of failing the delivery and letting the step re-run. `#publishRemote` already retried while Redis answered `LOADING` after a restart. Any other connection failure (socket closed unexpectedly, client offline, connection or socket timeout, `ECONNRESET`, `EPIPE`, `ETIMEDOUT`, `ECONNREFUSED`) was rethrown immediately. A lost `workflow.step.end` publish was nacked and redelivered, so the step ran again. These errors now retry under the same 120 s budget (250 ms between attempts) until Redis accepts the write, the pubsub closes, or the budget runs out. This addresses the transport side of #29504. ## Limits - **Not exactly-once.** A worker that dies between finishing a step and publishing `step.end` still gets the `step.run` redelivered, and that step runs again. At-least-once delivery is accepted for that window. - **Duplicates after a lost reply.** If Redis wrote the event but the reply was lost, the retry appends it again. Subscribers can receive a second delivery with the same event id. A duplicate `workflow.step.run` is dropped by the step fence; other events are already handled as at-least-once. - **Slower failure in a long outage.** A publish that starts while connected and then loses the connection now fails after the 2-minute budget (`PublishRetryBudgetExhaustedError`) if Redis doesn't come back, where it previously failed immediately. The trade-off is that no event is lost in a short outage. ## Changes - `pubsub/redis-streams/src/index.ts`: `isTransientPublishError()` widens the publish retry from `LOADING` to transient socket errors; budget/delay constants renamed to `PUBLISH_RETRY_BUDGET_MS` / `PUBLISH_RETRY_DELAY_MS`. Each attempt races a timer for what is left of the budget, because node-redis parks commands issued while reconnecting in its offline queue and never settles them; running out throws `PublishRetryBudgetExhaustedError`, which is not retried. `publishRetryBudgetMs` overrides the 120 s default. - `pubsub/redis-streams/test-fixtures/harness.ts`: `makeXaddDropProxy()`, a TCP proxy that drops the client connection on the first XADD either before Redis sees it or after Redis applies it, optionally closing its listener after the drop so Redis looks down for good. - `pubsub/redis-streams/src/publish-transient-retry.test.ts`: both drop modes deliver the event; all copies share one event id. A third case closes the proxy listener after the drop and asserts publish rejects with `PublishRetryBudgetExhaustedError` at a 2 s budget. - Changeset: `@mastra/redis-streams` patch. ## Tests - `publish-transient-retry.test.ts`: 2/2 pass. With the check reverted to `isLoadingError`, both fail with `Socket closed unexpectedly`. - `pnpm --filter @mastra/redis-streams run check:test` passes; full `@mastra/redis-streams` test suite run against Redis 6381 / Postgres 5436. - No changes to `@mastra/core`; `pnpm build:core` succeeds. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 If Redis briefly loses a connection while receiving an event, the publisher now tries again instead of failing immediately. If Redis saved the event but its reply was lost, the retry can deliver a duplicate with the same event ID. ## Changes - Redis Streams publishing retries `LOADING` errors and recognized transient connection errors, including closed sockets, `ECONNRESET`, `EPIPE`, and timeouts. - Retries stop for non-transient errors, when the pub/sub instance is closed, or after 120 seconds. Attempts are spaced 250 ms apart. - Added a test proxy and tests for connection drops before and after Redis writes an `XADD`. - Added a patch changeset for `@mastra/redis-streams`. When Redis writes an event but the reply is lost, the retry can append the event again. Both copies have the same event ID. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com> Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com> Co-authored-by: NikAiyer <nick.aiyer@gmail.com> |
||
|
|
2f93851c37 |
fix(core): make Agent.durable configure resumable streams only (#29503)
Co-authored-by: NikAiyer <nick.aiyer@gmail.com> |
||
|
|
f94beac862 |
fix(core): unwedge restarted nested evented loops (#29431)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9
## Problem
Setup: a split deploy (API with `MASTRA_WORKERS=false` plus dedicated
workers over Redis) with `recovery.durableAgents: 'auto'`. When the API
and the worker die together mid-tool and both restart, boot recovery
claims the run under a new generation and the interrupted tool runs
again. After that, the run stops making progress. It sits in
`durable-agentic-loop:running` forever while the API keeps renewing its
lease. Because the run never ends, it also keeps its thread lease, so
any later run on the same thread is rejected with
`AgentThreadLeaseConflictError`. Mastra Doctor hits this every time,
since its thread is the deployId and every re-diagnosis lands on the
stuck thread. The `DurableExecutionFenceError` and `This workflow run
was not active` messages in the ticket are side effects of this stuck
run.
## Root cause
Recovery behaves correctly: it claims a new generation and re-drives the
run. The run gets stuck in the evented step guard in
`WorkflowEventProcessor.endWorkflow`.
When a restarted nested loop body finishes, `endWorkflow` calls
`#processLoop` for the parent loop. The `!parentWorkflow.restart`
condition there, introduced by
|
||
|
|
a0068a9ad1 |
feat(core): resumable streams for every Agent (#28424)
> Opened by @user_01KNPFT2RPETZTEP15J3AEBA08 Originally stacked on [#28416](https://github.com/mastra-ai/mastra/pull/28416), which has merged. This PR now targets `main`. ## What Any `Agent` can now have resumable streams, not only durable agents. A run is published on `agent.stream.<runId>` with a replay cache, so: - a client can reconnect after a disconnect, - another reader can follow the run with `agent.observe(runId)`, - any reader can stop it (`observe().abort()`, `agent.abortRunStream(runId)`, or the abort-run route). Durable execution (surviving process restarts) stays with `DurableAgent` / `EventedAgent` / `InngestAgent`, which keep publishing through the same stream layer. ### Config New `streams?: boolean | { pubsub, cache, shouldCache, cleanupTimeoutMs }` on `Agent`. - Omitted: on when the agent is registered on `Mastra`, off for standalone agents. - `false`: off. `{ cache: false }`: live only, no replay. - With `durable`, `streams` keys override `durable` keys. `durable: true` + `streams: false` keeps publishing, drops the replay cache, and warns. ### Bus selection - Registered agents (plain and `DurableAgent` on any engine) publish on `mastra.pubsub`, so other processes sharing the PubSub and run-record cache can observe and abort. - Precedence: `streams.pubsub` > `durable.pubsub` > `mastra.pubsub`. The evented engine still forces `mastra.pubsub` over an agent-level pubsub and warns when they differ. - Unregistered agents keep a private in-process `EventEmitterPubSub`. - The replay cache wraps the shared bus once (existing `CachingPubSub` reuse check), so nothing is double-published. ### Behaviour change With streams on, closing the HTTP stream detaches the reader instead of aborting the run. To stop a run explicitly, use `POST /agents/:agentId/runs/:runId/abort`, which now works for plain agents too (ownership-checked, including across servers sharing the PubSub and run-record cache), or the thread abort route. `streams: false` keeps the old behaviour. ## Commits 1. ` |
||
|
|
675caf0613 |
chore: version packages (alpha) (#29336)
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. ⚠️⚠️⚠️⚠️⚠️⚠️ `main` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `main`. ⚠️⚠️⚠️⚠️⚠️⚠️ # Releases > The changelog information of each package has been omitted from this message, as the content exceeds the size limit. ## @mastra/client-js@1.60.0-alpha.0 ## @mastra/react@1.15.0-alpha.0 ## @mastra/factory@0.28.0-alpha.0 ## @mastra/code-sdk@1.19.0-alpha.0 ## mastra@1.39.0-alpha.0 ## @mastra/core@1.83.0-alpha.0 ## @mastra/mcp@2.9.0-alpha.0 ## @mastra/mcp-apps-components@0.1.0-alpha.0 ## @mastra/playground-ui@69.0.0-alpha.0 ## @mastra/server@1.83.0-alpha.0 ## @mastra/nestjs@0.3.0-alpha.0 ## @mastra/clickhouse@1.30.0-alpha.0 ## @mastra/duckdb@1.19.0-alpha.0 ## @mastra/libsql@1.30.0-alpha.0 ## @mastra/pg@1.36.0-alpha.0 ## @mastra/temporal@0.6.0-alpha.0 ## @mastra/auth-better-auth@1.1.11-alpha.0 ## @mastra/deployer-cloud@1.83.0-alpha.0 ## @mastra/deployer-cloudflare@1.2.42-alpha.0 ## @mastra/deployer-netlify@1.2.42-alpha.0 ## @mastra/deployer-sandbox@0.3.27-alpha.0 ## @mastra/deployer-vercel@1.2.42-alpha.0 ## @mastra/longmemeval@1.1.42-alpha.0 ## @mastra/opencode@0.1.42-alpha.0 ## create-factory@0.4.3-alpha.0 ## mastracode@0.49.1-alpha.0 ## @mastra/arize@1.3.30-alpha.0 ## @mastra/arthur@0.4.30-alpha.0 ## @mastra/braintrust@1.3.27-alpha.0 ## @mastra/datadog@1.4.22-alpha.0 ## @mastra/deepeval@0.1.24-alpha.0 ## @mastra/laminar@1.3.32-alpha.0 ## @mastra/langfuse@1.6.6-alpha.0 ## @mastra/langsmith@1.4.3-alpha.0 ## @mastra/observability@1.23.2-alpha.0 ## @mastra/otel-bridge@1.5.22-alpha.0 ## @mastra/otel-exporter@1.5.2-alpha.0 ## @mastra/posthog@1.4.7-alpha.0 ## @mastra/sentry@1.2.32-alpha.0 ## @mastra/agent-builder@1.2.1-alpha.0 ## @mastra/connect@1.3.2-alpha.0 ## create-mastra@1.39.0-alpha.0 ## @mastra/deployer@1.83.0-alpha.0 ## @mastra/editor@0.17.1-alpha.0 ## @mastra/mcp-docs-server@1.3.13-alpha.0 ## @mastra/memory@1.41.1-alpha.0 ## @mastra/rag@2.7.3-alpha.0 ## @mastra/redis-streams@0.8.1-alpha.0 ## @mastra/elysia@0.1.23-alpha.0 ## @mastra/express@1.5.27-alpha.0 ## @mastra/fastify@1.5.27-alpha.0 ## @mastra/hono@1.7.25-alpha.0 ## @mastra/koa@1.7.27-alpha.0 ## @mastra/next@0.2.41-alpha.0 ## @mastra/tanstack-start@0.2.41-alpha.0 ## @mastra/spanner@1.11.2-alpha.0 ## @mastra/turso@0.1.21-alpha.0 ## @mastra/inngest@1.11.2-alpha.0 ## @mastra/gcs@0.3.6-alpha.0 ## @internal/playground@1.39.0-alpha.0 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
|
49eaabb19b |
fix(core,redis-streams): recover evented runs when Redis is LOADING mid-tool (COR-1502) (#29430)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9
## Summary
On the evented engine, if Redis restarts while a tool step is finishing,
it spends a while (often 30–60s) reloading its dataset. During that time
it answers every command with `LOADING Redis is loading the dataset in
memory`. When that happens, the run stays `running` forever and no
answer is saved. This PR makes the run wait out the restart and finish
normally: the answer is saved and each tool runs exactly once.
Fixes COR-1502.
## Root cause: three stacked problems
1. **Publishing never retried.** `RedisStreamsPubSub.publish()` sent the
tool result and `step.end` once. During `LOADING` that failed, so the
step was redelivered and its tools ran a second time.
2. **The thread lease expired during the window.** Redis kept counting
down the lease while it reloaded. On redelivery, Postgres still renewed
the run ownership, but `renewDurableThreadLease` found no lease key and
returned `false`. The execution fence then refused every redelivery
("could not verify that this execution still owns the run"). The event
was dropped after 5 deliveries and the run stayed `running`.
3. **The lease retake gave up when the step went idle.** `#retakeLease`
in `thread-stream-runtime` stopped trying once the tool step finished.
The run was aborted as lease-lost (`reason: 'abort'`, empty answer)
before the final model turn.
## Fix
- **`@mastra/redis-streams`:**
- `publish` retries `LOADING` errors every 250ms for up to 2 minutes,
and stops if the client closes. This covers both plain `XADD` and
`MULTI`/`EXEC` publishes.
- It implements the new `isUnavailableToAll(error)`, which returns true
for `LOADING`.
- **`@mastra/core` `events/pubsub.ts`:** adds an optional
`LeaseProvider.isUnavailableToAll?(error)`. A backend returns true when
it is refusing every client, so nobody else can be granted the lease
meanwhile. Any other error, including a connection failure, must return
`false`.
- **`@mastra/core` execution fence:** this applies only when Postgres
run ownership has just renewed for a generation-fenced claim.
- If the thread lease can't be renewed because the key is gone, the
fence takes it back with `SET NX`. It never takes a lease that another
owner holds.
- If the backend is refusing every client, the fence relies on the
Postgres run ownership instead.
- Any other lease error still fences the execution.
- **`@mastra/core` thread-stream runtime:** while the backend refuses
every client, `#retakeLease` keeps asking, within the existing
`AGENT_LEASE_LOST_TOOL_GRACE_MS`. Generic errors keep the existing
strict lease-loss policy, which stays safe for network partitions.
## Tests
- New test `pubsub/redis-streams/src/evented-redis-loading.test.ts`. It
runs separate API and worker processes against real Redis and Postgres.
They connect through a TCP proxy (`makeLoadingProxy` in
`test-fixtures/harness.ts`) that simulates a restart: it drops every
connection, then answers `-LOADING` for 20s while two tools finish.
During the window, `MULTI`/`EXEC` behave as in real Redis (`EXEC` →
`EXECABORT`).
- **Red on main:** the run never reached a terminal state, and the
worker looped on the fence error.
- **Green with the fix (~30s):** the run completes, the answer is saved,
each tool runs exactly once in `tool_executions`, and run ownership is
released.
- Related suites pass:
- core: execution-fence, thread-lease, and thread-stream suites.
- redis-streams: `pubsub`, `agent-thread-cross-process`,
`durable-worker-takeover`, `fail-publish-after-exhaustion`,
`local-redelivery-evented`.
- `tsc` (core, redis-streams, redis-streams test config) and `oxlint`
are clean on the touched files.
- Validated together with the COR-1497 branch
(`user/session-90e601e9-…`, `7c3d66f9`): the local merge had no
conflicts, and `durable-full-restart` (3 cases) plus this LOADING test
all pass.
## Out of scope / follow-up
On success, the workflow engine still publishes `workflow.step.end`
before it saves the step result. Any publish failure other than
`LOADING` still leaves a window where a redelivered `step.run` finds no
saved result and runs its tools again. Saving the result before
publishing would close that window. It is a broader engine change, so I
left it for a separate ticket.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## ELI5
When Redis restarts, it can temporarily refuse requests while it reloads
data. This change lets evented runs wait for Redis and recover their
thread lease, so they can finish without repeating completed tools.
## What changed
- `@mastra/redis-streams` retries stream publishing when Redis returns
`LOADING`, up to two minutes. Other errors still fail immediately.
- `LeaseProvider` can optionally implement `isUnavailableToAll(error)`
to identify errors that mean the backend is refusing every client.
- After Postgres run ownership is renewed, the execution fence can
reclaim a missing thread lease. If the backend reports that it is
refusing all clients, the fence relies on run ownership. Other lease
errors still prevent execution.
- Thread-lease retakes can continue during backend-wide unavailability
within the existing grace period.
- Added an integration test for a Redis loading window during tool
execution. The test checks that the run completes, saves its answer,
executes each tool once, and releases run ownership.
## Tests
The PR description reports that the integration test passes against real
Redis and Postgres. It also reports passing related core and Redis
Streams suites, TypeScript checks, and `oxlint` on touched files. These
results were not independently verified here.
## Known limitation
The PR description notes that a workflow step may still rerun tools on
redelivery if publishing its result fails for an error other than
`LOADING`, because the result is published before it is saved.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
|
||
|
|
6e8dfcb83f | chore: version - exit prerelease mode | ||
|
|
ab4d89cd38 |
test(redis-streams): run durable worker-takeover suite against real Redis and Postgres (#28952)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9
Part 3 of 3, split from #28676. Refs COR-1490. Parts 1 and 2 (#28947,
#28951) are merged into `main`. This PR is based on `main`.
## What this does
The API-loss and worker-takeover guarantees from #28951 only hold with a
shared store that supports run fencing and a transport that works across
processes. In-memory suites can't test that. This PR:
- adds `.github/workflows/test-durable-streams.yml`, which runs the
two-process takeover suite against Redis (`:6381`) and Postgres
(`:5436`) service containers. The checkout uses `persist-credentials:
false`.
- calls it from `test-suite.yml` as the `durable-streams` job. The job
reports its own check but does not gate `Merge Test Reports`, because
its flake rate hasn't been measured yet.
- runs it when `prebuild.yml` sees changes to the evented agent and
workflow code, `packages/core/src/mastra/`, `packages/memory/`,
`pubsub/redis-streams/`, `stores/pg/`, or the workflow files
- extends `pubsub/redis-streams/src/durable-worker-takeover.test.ts` and
adds the `durable-api-loss.entry.mjs` fixture. The scenarios: API
SIGTERM/SIGKILL with remote workers, a thread follow-up after API loss,
a remote abort, worker shutdown, single-process Redis shutdown, and the
original caller completing with and without a survivor that is already
running
- adds `tsconfig.test.json` and a `check:test` script, which type-check
the tests and fixtures against core's `dist`
No changeset: this PR only touches CI and tests.
## Verification
- Reviewer run at `
|
||
|
|
3bdb4ec5f4 |
chore(deps): update build tools (major) (#29026)
> [!NOTE] > This PR body was truncated due to platform limits. This PR contains the following updates: | Package | Type | Update | Change | Pending | |---|---|---|---|---| | [@vitest/coverage-v8](https://vitest.dev/guide/coverage) ([source](https://redirect.github.com/vitest-dev/vitest/tree/HEAD/packages/coverage-v8)) | pnpm.catalog.default | major | [`4.1.12` → `5.0.5`](https://renovatebot.com/diffs/npm/@vitest%2fcoverage-v8/4.1.12/5.0.5) | `5.1.0` | | [@vitest/mocker@>=2.1.0 <4.1.12](https://redirect.github.com/vitest-dev/vitest/tree/main/packages/mocker) ([source](https://redirect.github.com/vitest-dev/vitest/tree/HEAD/packages/mocker)) | pnpm-workspace.overrides | major | [`4.1.12` → `5.0.5`](https://renovatebot.com/diffs/npm/@vitest%2fmocker@>=2.1.0 <4.1.12/4.1.12/5.0.5) | `5.1.0` | | [@vitest/ui](https://vitest.dev/guide/ui) ([source](https://redirect.github.com/vitest-dev/vitest/tree/HEAD/packages/ui)) | pnpm.catalog.default | major | [`4.1.12` → `5.0.5`](https://renovatebot.com/diffs/npm/@vitest%2fui/4.1.12/5.0.5) | `5.1.0` | | [dotenv](https://redirect.github.com/motdotla/dotenv) | devDependencies | major | [`^17.4.2` → `^18.0.9`](https://renovatebot.com/diffs/npm/dotenv/17.4.2/18.0.9) | | | [dotenv](https://redirect.github.com/motdotla/dotenv) | devDependencies | major | [`17.4.2` → `18.0.9`](https://renovatebot.com/diffs/npm/dotenv/17.4.2/18.0.9) | | | [vitest](https://vitest.dev) ([source](https://redirect.github.com/vitest-dev/vitest/tree/HEAD/packages/vitest)) | devDependencies | major | [`^4.1.12` → `^5.0.5`](https://renovatebot.com/diffs/npm/vitest/4.1.12/5.0.5) | `5.1.0` | | [vitest](https://vitest.dev) ([source](https://redirect.github.com/vitest-dev/vitest/tree/HEAD/packages/vitest)) | pnpm.catalog.default | major | [`4.1.12` → `5.0.5`](https://renovatebot.com/diffs/npm/vitest/4.1.12/5.0.5) | `5.1.0` | | [vitest](https://vitest.dev) ([source](https://redirect.github.com/vitest-dev/vitest/tree/HEAD/packages/vitest)) | devDependencies | major | [`4.1.12` → `5.0.5`](https://renovatebot.com/diffs/npm/vitest/4.1.12/5.0.5) | `5.1.0` | | [vitest@>=2.1.0 <4.1.12](https://vitest.dev) ([source](https://redirect.github.com/vitest-dev/vitest/tree/HEAD/packages/vitest)) | pnpm-workspace.overrides | major | [`4.1.12` → `5.0.5`](https://renovatebot.com/diffs/npm/vitest@>=2.1.0 <4.1.12/4.1.12/5.0.5) | `5.1.0` | --- > [!WARNING] > Some dependencies could not be looked up. Check the [Dependency Dashboard](../issues/9505) for more information. --- ### Release Notes <details> <summary>vitest-dev/vitest (@​vitest/coverage-v8)</summary> ### [`v5.0.5`](https://redirect.github.com/vitest-dev/vitest/releases/tag/v5.0.5) [Compare Source](https://redirect.github.com/vitest-dev/vitest/compare/v5.0.4...v5.0.5) ##### 🐞 Bug Fixes - Keep vite default `fs.deny` with api token file - by [@​hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi Ogawa** and **Claude Opus 5.5** in [#​11578](https://redirect.github.com/vitest-dev/vitest/issues/11578) [<samp>(28dc1)</samp>]( |
||
|
|
47a6cd82e5 |
chore(deps): update ai sdk (#29242)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
|
|
30046969ad |
chore: version packages (alpha) (#29146)
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. ⚠️⚠️⚠️⚠️⚠️⚠️ `main` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `main`. ⚠️⚠️⚠️⚠️⚠️⚠️ # Releases ## @mastra/client-js@1.59.0-alpha.1 ### Minor Changes - Added the `aggregateTimeToFirstMeasures` observability capability. Trace and span aggregate requests for `timeToFirstToken.*` or `timeToFirstChunk.*` measures now return `501` against stores that can't compute them, and field discovery omits those measures for such stores. ([#29055](https://github.com/mastra-ai/mastra/pull/29055)) ```typescript const { capabilities } = await mastraClient.getObservabilityCapabilities(); if (capabilities.traceAggregate && capabilities.aggregateTimeToFirstMeasures) { const { rows } = await mastraClient.aggregateTraces({ timeRange: { from: '2026-08-01T00:00:00.000Z', to: '2026-08-02T00:00:00.000Z' }, groupBy: ['entityName'], measures: ['timeToFirstChunk.p95', 'timeToFirstChunk.count'], }); } ``` ### Patch Changes - Updated dependencies [[`8751dd7`]( |
||
|
|
7645fce513 |
chore(deps): update dependency @google-cloud/pubsub to v6 (#29113)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
|
|
eed9cc214e | chore: version packages | ||
|
|
6ede0f10f6 |
fix(valkey-streams): trim saved thread-stream events (#29095)
> Opened by @abhiaiyer91 Valkey thread streams currently retain saved run events until their size limit or idle TTL. Implement run-scoped trimming so persisted stream parts can be removed without deleting other runs, pinned entries, or retained terminal events. Closes #28974. Before: `await pubsub.trimTopic(topic, { runId })` leaves all entries behind. After: it removes that run's eligible entries; `producedBefore` bounds removal to saved parts and `retainTerminalEvents` preserves terminal markers. Cleanup now snapshots the current stream tail, so ongoing publishers cannot extend the scan indefinitely or have their later entries deleted by that scan. ## Proof - Real-Valkey concurrency regression: pause the cleanup scan, publish another target-run entry, resume, and assert only the in-flight entry remains. This test failed before the tail-snapshot fix (`[]` instead of `['in-flight']`) and passed after it. - Existing real-Valkey integration covers more than one scan page, other-run isolation, pinned entries, produced-time cutoff, and retained terminal events; the targeted integration and lease suite passed (7/7). - Full `@mastra/valkey-streams` tests passed (8 files, 42 tests); package typecheck, lint (0 errors/warnings), formatting, diff check, and ESM/CJS build passed. - Earlier local Life Coach run with Postgres and Valkey emitted a finish event and left 2 stream entries with the patched adapter, compared to 117 before. ## Unverified - The Life Coach stream iterator remained open after finish, so that isolated run was stopped by timeout; it is not a completed end-to-end test. No sustained-production-throughput or multi-process soak test was run. ## Risk: medium - Trimming is best-effort and run-scoped, but removal is irreversible. Real-Valkey tests cover the bounded concurrent-publish race and preservation predicates; production scheduling and very large/live streams have not been soak-tested. Patch changeset included for `@mastra/valkey-streams`. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 When a run is saved, Valkey can keep its stream entries around and use extra storage. This change removes eligible entries for that run while keeping entries that must remain. ## What changed - Added `ValkeyStreamsPubSub.trimTopic` to delete entries for a specified `runId`. - When `producedBefore` is set, the method deletes only entries with a numeric `data.producedAt` at or before the cutoff. It skips pinned entries. - When `retainTerminalEvents` is enabled, the method preserves `run-registered`, `run-completed`, `run-failed`, and `run-aborted` events, except those marked `suspended`. - The scan stops at the stream tail captured before scanning starts. Entries published during the scan are not included. - Trimming failures are logged as warnings and do not propagate. - Added integration tests for run-specific trimming, cutoff and pinned-entry behavior, terminal-event retention, and entries published during a scan. - Added a patch changeset for `@mastra/valkey-streams`. ## Validation The supplied report says six Valkey integration tests passed and the package TypeScript check, lint, and build succeeded. It also reports a local Postgres and Valkey test that left two stream entries after a finish event, down from 117. The stream iterator remained open, so a timeout stopped the isolated process; the Valkey regression suite completed normally. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
77076a1012 | chore: version - exit prerelease mode | ||
|
|
582ef17a2b |
chore: version packages (alpha) (#28888)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
|
b0e86c50e2 |
fix(redis-streams): stop subscribe() hanging forever when Redis stops responding before connecting (#28941) | ||
|
|
39edceea4f |
fix(core): open thread subscriptions with initial history when the lease read stalls or fails (#28924) | ||
|
|
ba8b955b1a |
fix(redis-streams,valkey-streams): expire idle streams by default and clear them on thread delete (#28968)
A production Factory deployment ran out of Redis memory. Every key was a
`mastra:topic:*` stream, and almost all of them (3,673 thread streams)
had no expiry. The deployment built `new RedisStreamsPubSub({ url })`
with defaults, and the default `streamIdleTtlMs` was `0`, meaning
streams never expire. Thread streams are open-ended and never get
`clearTopic()`, so they piled up forever. The Redis instance used
`volatile-lru`, which can't evict keys without a TTL, so writes just
started failing.
This PR makes `@mastra/redis-streams` and `@mastra/valkey-streams` give
every stream a sliding 7-day idle TTL by default. Writes refresh it, and
live subscribers keep receiving events after a stream expires because
the read loop recreates it. Streams are an event bus, not storage:
replay history lives in the caching layer and storage, so the backlog
only matters for grouped workers that are offline longer than the TTL.
```ts
// Before: streams never expire unless you opt in
new RedisStreamsPubSub({ url });
// After: 7-day idle TTL by default; opt out explicitly
new RedisStreamsPubSub({ url });
new RedisStreamsPubSub({ url, streamIdleTtlMs: 0 });
```
Both pubsubs now log one warning after they first connect if Redis has
`maxmemory` set with any policy other than `noeviction`. A TTL makes
streams, consumer groups and lease keys evictable under `volatile-*`,
which would silently drop events and leases (ElastiCache defaults to
`volatile-lru`). The check uses `INFO memory` because managed Redis
often blocks `CONFIG GET`.
`memory.deleteThread()` now also deletes the thread's event stream
through the registered Mastra pubsub. This runs fire-and-forget, so
neither deleting a thread nor `memory.settled()` waits on Redis during
an outage. It's skipped on older core versions. Agents configured with
their own pubsub still rely on the TTL.
The docs now require `noeviction`; the Helm guide used to say a
`volatile-*` policy was fine. They also add a runbook for reclaiming
streams created before the TTL: switch to `noeviction`, upgrade the
writers, then expire each stream relative to its last write. Tested
against real Redis/Valkey containers, including a real TTL expiry while
a subscriber is listening and a thread delete with Redis unreachable.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## ELI5
Redis and Valkey can keep old event streams forever, which uses storage.
This change removes streams after seven days without writes by default
and cleans up a thread’s stream when the thread is deleted.
## What changed
- Redis and Valkey streams now use a sliding seven-day idle TTL by
default. Writes refresh the TTL; reads do not. Set `streamIdleTtlMs: 0`
to disable expiry.
- The read loop recreates an expired stream so a live subscriber can
continue receiving events.
- After connecting, each pubsub warns once when `maxmemory` is enabled
with a policy other than `noeviction`. The check uses `INFO memory`.
- Thread deletion starts best-effort stream cleanup through the
registered Mastra pubsub without waiting for Redis. Older core versions
skip this cleanup.
- Documentation and the local Docker configuration now recommend
`noeviction`. The documentation also explains how to expire existing
streams that lack a TTL.
## Tests
Tests cover TTL behavior, delivery after expiry for live subscribers,
eviction-policy warnings, and thread-stream cleanup. Test execution
results were not provided.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
|
||
|
|
52265cba91 |
Fix run abort route answering false on a server that does not own the run (#28522) | ||
|
|
cf92428a76 |
chore(deps): update docker images (#28011)
This PR contains the following updates: | Package | Update | Change | |---|---|---| | amazon/aws-cli | minor | `2.36.44` → `2.39.1` | | clickhouse/clickhouse-server | minor | `26.6.1.1193` → `26.10.3.7` | | cr.weaviate.io/semitechnologies/weaviate | minor | `1.39.13` → `1.40.3` | | [docker.elastic.co/elasticsearch/elasticsearch](https://www.elastic.co/products/elasticsearch) ([source](https://redirect.github.com/elastic/elasticsearch)) | minor | `9.4.3` → `9.5.5` | | edoburu/pgbouncer | minor | `v1.25.2-p0` → `v1.26.1-p0` | | [ghcr.io/shopify/toxiproxy](https://redirect.github.com/Shopify/toxiproxy) | minor | `2.12.0` → `2.13.0` | | inngest/inngest | minor | `v1.34.0` → `v1.49.0` | | mcr.microsoft.com/azure-storage/azurite | minor | `3.35.0` → `3.37.0` | | [mongo](https://hub.docker.com/_/mongo) ([source](https://redirect.github.com/docker-library/mongo)) | patch | `7.0.37` → `7.0.45` | | pgvector/pgvector | patch | `0.8.4-pg16` → `0.8.7-pg16` | --- > [!WARNING] > Some dependencies could not be looked up. Check the [Dependency Dashboard](../issues/9505) for more information. --- ### Release Notes <details> <summary>elastic/elasticsearch (docker.elastic.co/elasticsearch/elasticsearch)</summary> ### [`v9.5.5`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.5.5): Elasticsearch 9.5.5 [Compare Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.5.4...v9.5.5) Downloads: <https://elastic.co/downloads/elasticsearch> Release notes: <https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.5.5-release-notes> ### [`v9.5.4`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.5.4): Elasticsearch 9.5.4 [Compare Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.5.3...v9.5.4) Downloads: <https://elastic.co/downloads/elasticsearch> Release notes: <https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.5.4-release-notes> ### [`v9.5.3`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.5.3): Elasticsearch 9.5.3 [Compare Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.5.2...v9.5.3) Downloads: <https://elastic.co/downloads/elasticsearch> Release notes: <https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.5.3-release-notes> ### [`v9.5.2`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.5.2): Elasticsearch 9.5.2 [Compare Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.5.1...v9.5.2) Downloads: <https://elastic.co/downloads/elasticsearch> Release notes: <https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.5.2-release-notes> ### [`v9.5.1`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.5.1): Elasticsearch 9.5.1 [Compare Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.5.0...v9.5.1) Downloads: <https://elastic.co/downloads/elasticsearch> Release notes: <https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.5.1-release-notes> ### [`v9.5.0`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.5.0): Elasticsearch 9.5.0 [Compare Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.4.8...v9.5.0) Downloads: <https://elastic.co/downloads/elasticsearch> Release notes: <https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.5.0-release-notes> ### [`v9.4.8`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.4.8): Elasticsearch 9.4.8 [Compare Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.4.7...v9.4.8) Downloads: <https://elastic.co/downloads/elasticsearch> Release notes: <https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.4.8-release-notes> ### [`v9.4.7`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.4.7): Elasticsearch 9.4.7 [Compare Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.4.6...v9.4.7) Downloads: <https://elastic.co/downloads/elasticsearch> Release notes: <https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.4.7-release-notes> ### [`v9.4.6`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.4.6): Elasticsearch 9.4.6 [Compare Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.4.5...v9.4.6) Downloads: <https://elastic.co/downloads/elasticsearch> Release notes: <https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.4.6-release-notes> ### [`v9.4.5`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.4.5): Elasticsearch 9.4.5 [Compare Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.4.4...v9.4.5) Downloads: <https://elastic.co/downloads/elasticsearch> Release notes: <https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.4.5-release-notes> ### [`v9.4.4`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.4.4): Elasticsearch 9.4.4 [Compare Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.4.3...v9.4.4) Downloads: <https://elastic.co/downloads/elasticsearch> Release notes: <https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.4.4-release-notes> </details> <details> <summary>Shopify/toxiproxy (ghcr.io/shopify/toxiproxy)</summary> ### [`v2.13.0`](https://redirect.github.com/Shopify/toxiproxy/blob/HEAD/CHANGELOG.md#2130---2026-10-14) [Compare Source](https://redirect.github.com/Shopify/toxiproxy/compare/v2.12.0...v2.13.0) - Reject slicer toxics with `average_size` <= 0 or `size_variation` outside `[0, average_size]`, and stop the slicer from recursing forever on degenerate sizes ([#​774](https://redirect.github.com/Shopify/toxiproxy/issues/774), [@​iyaz-shaikh](https://redirect.github.com/iyaz-shaikh)) - Add optional bearer token auth for the HTTP API via `TOXIPROXY_AUTH_TOKEN`, supported by the Go client and `toxiproxy-cli`, and warn when the API is unauthenticated on a non-loopback address ([#​774](https://redirect.github.com/Shopify/toxiproxy/issues/774), [@​iyaz-shaikh](https://redirect.github.com/iyaz-shaikh)) - Fix removing a toxic hanging forever, and freezing the whole HTTP API, when the downstream toxic or connection is already gone ([#​760](https://redirect.github.com/Shopify/toxiproxy/issues/760), [@​matheusgb](https://redirect.github.com/matheusgb)) - Fix the bandwidth toxic hanging or panicking when `rate` <= 0 ([#​764](https://redirect.github.com/Shopify/toxiproxy/issues/764), [@​jaideeppyne](https://redirect.github.com/jaideeppyne)) - Update go version to 1.27 ([#​777](https://redirect.github.com/Shopify/toxiproxy/issues/777), [@​The-Black-Fox](https://redirect.github.com/The-Black-Fox)) - Update various go packages </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - "before 6am every weekday" - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://redirect.github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate CLI](https://redirect.github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4xNzUuMSIsInVwZGF0ZWRJblZlciI6IjQ0LjE5Mi4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJhdXRvbWF0aW9uIl19--> --------- Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
4f12792ed1 |
chore: version packages (alpha) (#28722)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
|
1607791e91 |
fix(core): trim finished runs of memoryless agents from thread-stream topics (#28746)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
f2ad3844f3 |
chore(deps): update opentelemetry (#28581)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
6deb7c0a1e |
chore(security): Vanta 2026-10-28 - qs 6.16.1, @modelcontextprotocol/node 2.1.1 floors (#28716)
Refs mastra-ai/platform#2931 Vanta remediation pass for 2026-10-28. It follows #28325, which has merged. ## 2026-10-28 update Today's live Vanta export added 3 advisory IDs on `mastra-ai/mastra` (GitHub/Dependabot). This PR fixes the two that have a published fix and leaves deepmerge, which has none. | Package | From → To | Finding | Where | How | |---|---|---|---|---| | qs | 6.16.0 → 6.16.1 | CVE-2026-110816 / GHSA-6pq8-jhm2-2vcw (Medium ×4) | root `pnpm-lock.yaml`, 9 `e2e-tests/*` lockfiles, `scripts/gh-issue-triage/package-lock.json` | Raised the existing floors `qs@>=6.7.0 <6.16.0: 6.16.0` → `<6.16.1: 6.16.1` and `'qs@<6.7.0': 6.16.0` → `6.16.1`. Raised the npm override in gh-issue-triage. | | @modelcontextprotocol/node | 2.0.0 → 2.1.1 | GHSA-f897-62hf-6gxq (High ×2; HTTP transports had no body-size or batch-length limit) | root `pnpm-lock.yaml` (published `@mastra/mcp` 1.18.0, used as `mcp-legacy`), `scripts/gh-issue-triage/package-lock.json` | New pnpm override `'@modelcontextprotocol/node@>=2.0.0 <2.1.0': 2.1.1` and a new npm override in gh-issue-triage. I used 2.1.1 rather than 2.1.2 because 2.1.2 needs a `@modelcontextprotocol/server` peer of `^2.4.0`, while `@mastra/mcp` 1.18 resolves server 2.3.1. `packages/mcp` itself already uses node 2.1.2. | - Both versions are older than the 1-day `minimumReleaseAge`: qs 6.16.1 was published 10-14 and node 2.1.1 on 10-02. I did not use qs 6.16.2 (10-27), which is too new. - No floors or overrides were removed; existing ones were only raised. - `pnpm install --frozen-lockfile --lockfile-only` passes at the root. Each nested e2e workspace was regenerated with `pnpm install --lockfile-only`. - Added the empty changeset `.changeset/vanta-2026-10-28-security-floors.md`. No published package changes. - After this change, osv-scanner over all 33 lockfiles reports only the documented leftovers: - zod 3.25.76 (zod v3 compat line on purpose) - ai 4.3.19 / `@ai-sdk/provider-utils` 2.2.8 (vendored v4 line) - `@azure/msal-node` 5.6.0 / 6.0.1 (fix 7.1.0 needs majors in identity and teams) - `@opentelemetry/core` 1.30.1 (needs pubsub 6) - extract-zip 2.0.1 (no patched release) - deepmerge 4.3.1 (CVE-2026-93753 / GHSA-gcqx-w48m-w2h4; no patched release, 4.3.1 is the latest) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 This PR updates dependency rules and package versions to use newer releases, including security fixes for `qs` and `@modelcontextprotocol/node`. It also updates AI SDK dependencies and does not include published package changes. ## Summary - Raise the `qs` override target to `6.16.1` in the root and e2e workspaces and in `scripts/gh-issue-triage`. - Add an `@modelcontextprotocol/node` override at `2.1.1` in the root workspace and `scripts/gh-issue-triage`. - Update AI SDK dependency versions across package manifests and add a changeset describing the provider-utils update. - Add an empty changeset for the security-only dependency updates. <!-- end of auto-generated comment: release notes by coderabbit.ai --> ## Update 2026-10-28 (afternoon) - Merged origin/main (clean). - Bumped the AI SDK v5 compat line from @ai-sdk/provider-utils 3.0.41 to 3.0.42 (patch, published 2026-10-21; it blocks `.internal` hostnames in download URL validation and encodes provider-returned IDs in request paths). This addresses the Aikido /mastra Highs AIKIDO-2026-10990 and AIKIDO-2026-10981: - ai 5.0.277 → 5.0.282, @ai-sdk/gateway 2.0.168 → 2.0.173 (vendored ai_v5) - anthropic 2.0.109 → 2.0.110, google 2.0.103 → 2.0.106, groq 2.0.58 → 2.0.59, openai 2.0.135 → 2.0.137, openai-compatible 1.0.58 → 1.0.59, xai 2.0.99 → 2.0.100 - Nested lockfiles updated: memory/integration-tests and playground/e2e/kitchen-sink. - No new overrides. Changeset: `vanta-2026-10-28-ai-sdk-v5-provider-utils.md`. |
||
|
|
6a191d46dc | chore: version - exit prerelease mode | ||
|
|
e6764d26da |
chore: version packages (alpha) (#28625)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
|
a7cc444233 |
fix(core): expose recoverable run outcomes to thread subscribers (#28621)
Thread subscribers now receive an ordered `thread-run-status` frame for
success, failure, abort, or suspension without closing the subscription.
Initial history also reports `activeRunId`, so an idle snapshot no
longer has to stand in for a run outcome. Retained outcomes reach late
subscribers even after saved output has been trimmed, with execution
authority preserved to reject superseded attempts.
For example, a subscription opened after a run finishes now receives
`thread-history` followed by `thread-run-status` with `{ status:
"success" }`, rather than history alone. Recovery is bounded by PubSub
retention; upgrade both `@mastra/core` and `@mastra/redis-streams` for
recovery after trimming. Client callback types, documentation, and
package changesets are included.
Verified with core build and typecheck, 220 core tests across the
thread-stream suites and four execution engines, 91 client tests, 55
Redis integration tests, and three HTTP conformance cases covering
plain, durable, and evented agents. Real Redis regressions cover restart
after trimming and superseded completion/abort after registration
eviction. The live outcome scenario also passed 20 repeated four-engine
runs. Typed lint, formatting, MDX prose checks, and final adversarial
review passed.
Fixes COR-1500.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## ELI5
A thread subscription can now tell clients when a run succeeds, fails,
stops, or pauses without ending the subscription. This helps clients
understand what happened after reconnecting, even when the run’s saved
output is no longer available.
## Summary
- Added opt-in `withRunStatus` support to core, server, and client
thread subscriptions. Enabled subscriptions can receive ordered
`thread-run-status` chunks for `success`, `failed`, `aborted`, or
`suspended` outcomes.
- Added `activeRunId` to initial thread history so clients can identify
the run that held the thread lease during history loading.
- Added terminal-event retention during output trimming. Recovery
depends on PubSub retention. Redis Streams users must upgrade
`@mastra/core` and `@mastra/redis-streams` together.
- Updated React `useChat` to request run statuses and apply outcomes to
the matching run. Added API types, documentation, and changesets.
## Validation
The PR objectives report passing core build and typecheck, core and
client tests, Redis integration tests, HTTP conformance cases, lint and
formatting checks, and documentation checks. They also report repeated
run-outcome checks and adversarial review. These results are
author-reported.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
|
||
|
|
e718b91a64 |
fix(core): stop observe offsets past the log end replaying the whole run (#28348)
> Opened by @roaminro
Refs #28324 (partial fix; the issue stays open, see Open question)
## What changed
On transports that keep old messages, such as Redis Streams,
`observe(runId, { offset })` replayed the **whole run** when the offset
was at or past the end of the run's log. A new subscriber receives the
stream's retained frame 0. `CachingPubSub`'s log-restart check compared
the counter with the subscriber's cursor, and with a past-end offset the
counter is always below the cursor. So frame 0 looked like a log restart
and everything replayed. This affected every Redis setup, durable agents
included.
`subscribeFromOffset` in `packages/core/src/events/caching-pubsub.ts`
now decides restarts by frame 0's identity:
- On attach it records the id of the log's frame 0. A frame 0 arriving
below the cursor is a restart only if the log's frame 0 id has changed.
A retained copy of the same frame is acknowledged and dropped.
- On a restart it delivers the new segment from the cache log, not from
the frame that signalled the restart, because that frame may be a
delayed copy of the old frame 0.
- If an id can't be read, it falls back to the counter. If the read
failed at attach, the counter is compared with the log's end at that
point, so an intact log isn't mistaken for a restart.
The cursor stays at the offset, so the documented contract holds: an
offset beyond the retained index replays nothing and skips live events
below it. If the retained log expired and the run restarts numbering at
0, the observer resyncs to the new segment.
**Known limitation (not fixed):** `createEventedAgent` on a
cross-process pubsub with the default in-memory server cache, which is
the case in the issue title. Its step executor publishes unnumbered
frames there, because each process would count them separately. These
frames can't be skipped by position:
- `CachingPubSub` can't see where a transport's replay starts.
- Redis Streams trims the start of its backlog with `MAXLEN`.
- Other transports replay nothing.
That configuration now logs a one-time warning. It names both possible
causes, a failed cache append or no shared server cache, and recommends
a shared cache such as `RedisServerCache`. Both observe docs pages
(`docs/harness/durable-agents.mdx`,
`reference/agents/durable-agent.mdx`) document the requirement.
## Tests
- `packages/core/src/events/caching-pubsub.test.ts`:
- A past-end offset delivers nothing, then delivers live frames once the
log reaches the offset.
- An offset inside a short log resumes without resyncing.
- A short log past the offset expires and a new segment resyncs.
- A stale copy of the old frame 0 doesn't stand in for the new segment.
- A failed frame 0 id read falls back to the counter.
- A restart is detected by id when the counter read fails.
- A failed id read at attach doesn't replay an intact log.
- A log that is empty at attach still delivers the next run.
- Unnumbered frames produce exactly one warning.
- `pubsub/redis-streams/src/observe-offset.test.ts`: new file against
real Redis, with the model served by aimock.
- It covers four setups: durable or evented engine, with the default or
`RedisServerCache` server cache.
- Each setup runs three cases: resuming a live run, a past-end offset on
a finished run, and a mid-run offset on a finished run.
- The evented/default setup checks for the warning and that the
observation still ends on `finish`.
| Check | Result |
|---|---|
| `packages/core` `src/events` | ✅ 323/323 |
| core `tsc --noEmit` | ✅ clean |
| `pubsub/redis-streams` observe-offset, real Redis, rebuilt core | ✅
12/12 |
| oxlint, prettier, docs `format:mdx:check` + remark | ✅ clean |
## Proof
- I checked each new unit test against the previous commit or the code
before its fix, and it failed there. The empty-log test is the
exception: it passes before its fix by design. It guards against
limiting the counter check whenever no id was captured, and it fails if
that change is applied.
- With `caching-pubsub.ts` at base and core rebuilt, 6 of the 12 Redis
tests fail. These are the past-end case in the three numbered setups,
where the whole run replays from `start`, and all three evented/default
cases, where no warning is logged.
- NikAiyer independently reproduced the base failures and the fixed-head
results.
## Unverified
- Redis Cluster deployments, and transports other than Redis Streams and
the in-process pubsub.
- `durable-agent-tool-writer` and `durable-agent-client-tool-pending`
already fail on `main`, with or without this change.
## Risk: low
The change is limited to the restart check and cursor handling in
`CachingPubSub.subscribeFromOffset`, plus a warning. Offset 0 and the
default observe path (no offset) don't read the counter and behave as
before.
## Open question
#28324 stays open for evented agents on a cross-process pubsub without a
shared server cache. Should evented step frames be numbered without a
shared cache (or a shared cache be required), or is the documented
limitation enough?
**Merge note:** the first commit (`
|
||
|
|
5381b414b3 |
test(redis-streams): fix the Lambda-death signal-drain test that times out on main (#28533)
> Opened by @abhiaiyer91
## Summary
`drains signal-enqueued from losers that die immediately (Vercel
Lambda-death)` in
`pubsub/redis-streams/src/agent-thread-cross-process.test.ts` times out
on main. It never actually exercised the scenario in its name. This is a
test-only fix. Product code is unchanged, so there's no changeset.
Two fixture problems caused it:
1. **Each worker had its own agent id.** The fixture used
`${WORKER_ID}-agent` (`test-fixtures/agent-thread-worker.entry.ts:59` on
main). The losers therefore treated the winner's run as another agent's
run. The cross-agent guard
(`packages/core/src/agent/thread-stream-runtime.ts:6852-6854`,
`:6900-6905`) correctly returned `blocked / thread-blocked` and
published nothing, so the winner had nothing to drain. Lambdas are
replicas of one deployed agent, so they should share an id.
2. **Timing.** The winner's run was a fixed 1500ms, but a worker takes
about 2.3s to boot through tsx. The losers always sent after the lease
was already free, never "while the run is in flight".
Changes:
- The fixture accepts `AGENT_ID` (default is still `${WORKER_ID}-agent`,
so other tests are unaffected). The test sets one shared id.
- The fixture accepts `HOLD_SIG_IDS` plus a `finish-run` command. The
winner's sig-1 run stays open until both losers have exited, then the
test ends it explicitly.
- The test asserts the winner owns sig-1, and that both losers lost the
lease and started no runs. It then waits for sig-1/2/3 to finish on the
winner. I removed the fixed 300ms settle sleep, so the test waits on
product events only.
## Proof
- On main: fails with `waitFor timed out after 20000ms` waiting for 3
`run-finished`. Only sig-1 finishes on the winner.
- With the fix, against core built from main: **5/5 consecutive
passes**.
- Reverting only the shared `AGENT_ID` makes it fail again with the same
20s `waitFor` timeout.
- `agent-thread-cross-process.test.ts`: 10/10 pass. On main this file
failed because of this test.
- Full redis-streams suite, run per file:
- Fail both on main and on this branch: `cluster`,
`durable-worker-takeover`, `observe-cross-process`. On main this run
only checked `observe-cross-process`; the other two are already known to
fail on main.
- `agent-thread-publish-lag`: 1 test failed. Known to fail on main; I
didn't rerun it on main myself.
- `auth-e2e` and `fanout-sharing` failed once in the long serial run.
Both passed 2/2 when rerun alone on this branch, and both pass on main.
They don't load the changed fixture.
- oxlint: 0 warnings, 0 errors. prettier: no changes. `tsc --noEmit` in
redis-streams gives the same 14 errors on main and on this branch (a
`RedisStreamsPubSub` vs `PubSub` mismatch, unrelated).
## Unverified
- **A sender that exits without closing its pubsub can still drop the
forwarded signal (product gap, out of scope).** With the shared id, the
losers settle `deliver` (I logged `settled.action`). That means they
take the `ifActive: deliver` branch at `thread-stream-runtime.ts:7027`,
not the lease-loser hand-off at `:2589`, which settles `remote`. Both
branches publish `signal-enqueued` with the fire-and-forget `#publish`
(`:2690-2692`) and resolve `accepted` before the publish reaches Redis.
So my earlier claim about the pattern holds, and it applies to the path
this test actually runs. The test passes only because the fixture awaits
`pubsub.close()` before `process.exit`. `#publish` reaches
`RedisStreamsPubSub.publish` synchronously, which adds the publish to
`#pendingPublishes` (`pubsub/redis-streams/src/index.ts:431-434`) before
`accepted` resolves, and `close()` waits for those publishes through
`flush()` (`:1133`, `:894-899`). It isn't a case of the publish just
being fast locally. I checked this by temporarily skipping `close()` in
`send-and-exit`: **3/3 runs timed out** and sig-2/sig-3 never reached
the winner. So a Lambda that settles and exits (or is frozen or killed)
without closing or flushing its pubsub loses the signal. A fix would
await the publish before resolving `accepted` on the remote-owner
branches. That needs its own core PR. I didn't check whether the server
or serverless deployers close or flush the pubsub on shutdown.
- **A non-durable owner that dies mid-run loses queued signals (separate
product gap, out of scope).** In an earlier run of this test the losers
won the free lease. When the lease owner died, sig-2/sig-3 stayed
stranded after the lease expired, and a later signal woke a fresh run
without replaying them. Signals are only admitted by the process that
owns the runId (`thread-stream-runtime.ts:1444`). Dead-owner recovery
only covers durable runs (`:5574-5584`, `:5824+`). This needs its own
issue and design (orphaned-signal recovery, with duplicate-delivery
risk).
- Not checked on CI runners. The fix doesn't depend on boot timing
anymore, so I expect it to behave the same there.
## Risk
low. Only test and fixture files changed, and the new fixture options
are opt-in.
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
|
||
|
|
7c20f008c8 |
fix(core): abort evented workflow steps running on other replicas on cancel (#28489)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9 Closes #28444 ## Summary With evented workflows and workers on several replicas, `workflow.cancel` is handled by only one processor in the consumer group. That processor aborted only the AbortControllers in its own process. If the step was running on another replica, its `abortSignal` never fired and the step kept running, even though the run itself settled as `canceled`. This PR broadcasts cancellation on one shared control topic, `workflow.control`: - **One subscription per processor.** A processor opens a single ungrouped subscription to `workflow.control` when it runs its first step and keeps it for its lifetime. Transports share one reader/subscription per topic per process, so the cost does not grow with runs or steps (no per-run topics, consumer groups or Pub/Sub subscriptions to clean up). It reads from `latest`; a failed subscribe is logged and retried on the next step. - **Cancel publishes the abort before finalizing.** `processWorkflowCancel` aborts its local controllers, publishes an abort for the run, and only then calls `endWorkflow`. If the publish fails, the error propagates and the event is redelivered before anything is finalized. - **Receivers only act on runs they are executing.** Each processor counts in-flight steps per run (a nested step counts against its ancestors too). An abort for a run with no in-flight step here is ignored. Otherwise the run's own controller is marked cancel-handled (the step unwinds without a second `workflow.cancel`), and nested runs are aborted like the local `cancelRunAndChildren`, so each settles canceled. Parent/child links come from the step event's `parentWorkflow` chain, so this works even if the processor never saw the child's `workflow.start`. - **Cancels issued before a processor could hear them.** The control subscription starts at `latest`, so an abort published while it is still being opened is missed. When `#dispatch` takes a `workflow.step.run`, it records, before its state load, whether the subscription was ready and how many aborts this processor has ignored for runs with no step retained. The step re-reads the persisted status of its run and its ancestors when the subscription wasn't ready at dispatch, or when that count changed before the step was retained, and aborts if any of them is `canceled` (`EventedRun.cancel` persists that before publishing). Other steps skip the read. The count can't evict anything; when it changes, the overlapping steps re-check. - **A canceled step never starts its body.** If the pre-step check finds the step's own run canceled, the step returns without running. If it finds only an ancestor canceled, the body is skipped with a `canceled` result, and the existing post-step branch publishes this run's `workflow.cancel`, so the nested run settles canceled. - **No state left on replicas that never finalize the run.** When a run's last in-flight step is released on a processor that didn't start it, the controller and parent links recorded by the step path are dropped. - **Double delivery does nothing.** Aborting is idempotent; a redelivered or self-echoed abort finds the controllers already aborted. Tests assert exactly one `workflow.end` per run. Assumption: a shared broadcast topic over per-run topics. Cancels are rare and tiny, so every processor seeing every abort is cheaper than per-run subscriptions, which leaked topics/subscriptions on Google Cloud Pub/Sub and churned consumer groups on Redis Streams. ## Proof - `packages/core/src/workflows/evented/remote-cancel.test.ts`, suite "workers on both replicas": two Mastra instances, both running workers, over EventEmitter and UnixSocket, cancels from the owner and the other replica, flat and nested. Asserts the step's `abortSignal` fires, the run settles `canceled`, the only control subscription is `workflow.control` and there is at most one per processor across all runs, no in-flight step refs remain; with grouped delivery also exactly one `workflow.end` and at least one cancel handled on a different replica than the step. A pinned-topology test (steps on one replica, start/end on the other) asserts the step-only replica keeps no controllers or parent links. Five pinned-topology "skips a step …" tests cover the fallback: (a) the step replica's `workflow.control` subscribe is delayed and the run is canceled during the delay; (b) the abort lands after the step's dispatch load but before it is retained (also asserts a plain step on a ready subscription does no re-check); (c) same as (b) with 2000 aborts for other runs published in that window; (d) a step dispatched while the subscription is pending resumes after other steps have run on the ready subscription; (e) nested: the parent is canceled while the child's step is held after its dispatch load. Each asserts the step body never ran, the run settles `canceled` and exactly one `workflow.end` is published per run; (e) also asserts the child snapshot is `canceled`. 26/26 pass. Proofs: **with the abort publish disabled**, the 4 EventEmitter both-replica cases fail on the step-abort assertion. **With the pre-body skip disabled**, (a)–(d) fail. **On the |
||
|
|
9ffb79b17e |
fix(core): stop closeOnSuspend durable streams closing before their chunks arrive on Redis Streams (#28504) | ||
|
|
2c646b53d0 |
chore: version packages (alpha) (#28399)
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. ⚠️⚠️⚠️⚠️⚠️⚠️ `main` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `main`. ⚠️⚠️⚠️⚠️⚠️⚠️ # Releases > The changelog information of each package has been omitted from this message, as the content exceeds the size limit. ## @mastra/factory@0.25.0-alpha.5 ## @mastra/langsmith@1.4.0-alpha.2 ## @mastra/connect@1.2.0-alpha.2 ## @mastra/core@1.80.0-alpha.4 ## @mastra/playground-ui@66.0.0-alpha.4 ## @mastra/acp@0.4.4-alpha.0 ## @mastra/claude@0.3.4-alpha.0 ## @mastra/cursor@0.3.4-alpha.0 ## @mastra/openai@1.1.4-alpha.0 ## @mastra/auth-auth0@1.2.6-alpha.0 ## @mastra/auth-better-auth@1.1.9-alpha.0 ## @mastra/auth-clerk@1.3.2-alpha.0 ## @mastra/auth-cloud@1.2.8-alpha.0 ## @mastra/auth-firebase@1.1.5-alpha.0 ## @mastra/auth-google@0.1.7-alpha.0 ## @mastra/auth-neon@0.3.5-alpha.0 ## @mastra/auth-okta@0.2.6-alpha.0 ## @mastra/auth-studio@1.3.11-alpha.1 ## @mastra/auth-supabase@1.1.6-alpha.0 ## @mastra/auth-workos@1.6.8-alpha.0 ## @mastra/agent-browser@0.5.5-alpha.0 ## @mastra/browser-viewer@0.2.6-alpha.0 ## @mastra/browser-firecrawl@0.2.5-alpha.1 ## @mastra/stagehand@0.3.7-alpha.1 ## @mastra/discord@1.3.2-alpha.0 ## @mastra/slack@1.8.0-alpha.1 ## @mastra/teams@0.1.3-alpha.0 ## @mastra/telegram@0.2.4-alpha.0 ## @mastra/client-js@1.57.0-alpha.4 ## @mastra/react@1.13.0-alpha.4 ## @mastra/isolated-vm@0.1.4-alpha.0 ## @mastra/quickjs@0.1.5-alpha.0 ## @mastra/deployer-cloud@1.80.0-alpha.4 ## @mastra/deployer-cloudflare@1.2.39-alpha.1 ## @mastra/deployer-netlify@1.2.39-alpha.1 ## @mastra/deployer-sandbox@0.3.24-alpha.1 ## @mastra/deployer-vercel@1.2.39-alpha.1 ## @mastra/voyageai@0.4.3-alpha.0 ## @mastra/longmemeval@1.1.39-alpha.4 ## @mastra/livekit@0.5.2-alpha.0 ## @mastra/opencode@0.1.39-alpha.4 ## @mastra/parallel@0.1.4-alpha.0 ## create-factory@0.3.1-alpha.4 ## @mastra/code-sdk@1.15.1-alpha.5 ## mastracode@0.48.1-alpha.4 ## @mastra/arize@1.3.27-alpha.2 ## @mastra/arthur@0.4.27-alpha.2 ## @mastra/braintrust@1.3.24-alpha.2 ## @mastra/datadog@1.4.19-alpha.2 ## @mastra/deepeval@0.1.21-alpha.2 ## @mastra/laminar@1.3.29-alpha.2 ## @mastra/langfuse@1.6.3-alpha.2 ## @mastra/observability@1.21.1-alpha.2 ## @mastra/otel-bridge@1.5.19-alpha.2 ## @mastra/otel-exporter@1.4.10-alpha.2 ## @mastra/posthog@1.4.4-alpha.2 ## @mastra/sentry@1.2.29-alpha.2 ## @mastra/agent-builder@1.1.31-alpha.2 ## @mastra/auth@1.1.5-alpha.0 ## mastra@1.36.0-alpha.4 ## @mastra/codemod@1.1.8-alpha.0 ## create-mastra@1.36.0-alpha.4 ## @mastra/deployer@1.80.0-alpha.4 ## @mastra/editor@0.16.5-alpha.2 ## @mastra/evals@1.10.9-alpha.0 ## @mastra/fastembed@1.3.5-alpha.2 ## @mastra/loggers@1.3.10-alpha.1 ## @mastra/mcp@2.6.0-alpha.1 ## @mastra/mcp-docs-server@1.3.10-alpha.7 ## @mastra/mcp-registry-registry@1.1.7-alpha.0 ## @mastra/memory@1.39.1-alpha.2 ## @mastra/rag@2.7.1-alpha.0 ## @mastra/schema-compat@1.3.17-alpha.0 ## @mastra/server@1.80.0-alpha.4 ## @mastra/google-cloud-pubsub@1.1.5-alpha.1 ## @mastra/redis-streams@0.6.5-alpha.1 ## @mastra/valkey-streams@0.5.7-alpha.1 ## @mastra/elysia@0.1.20-alpha.4 ## @mastra/express@1.5.24-alpha.4 ## @mastra/fastify@1.5.24-alpha.4 ## @mastra/hono@1.7.22-alpha.4 ## @mastra/koa@1.7.24-alpha.4 ## @mastra/nestjs@0.2.39-alpha.4 ## @mastra/next@0.2.38-alpha.4 ## @mastra/server-adapters-test-suite@0.1.4-alpha.2 ## @mastra/tanstack-start@0.2.38-alpha.4 ## @mastra/astra@1.1.3-alpha.0 ## @mastra/azure-ai-search@0.1.2-alpha.1 ## @mastra/chroma@1.1.5-alpha.0 ## @mastra/clickhouse@1.27.0-alpha.1 ## @mastra/cloudflare@1.7.6-alpha.0 ## @mastra/cloudflare-d1@1.4.5-alpha.1 ## @mastra/convex@1.7.6-alpha.0 ## @mastra/couchbase@1.1.4-alpha.1 ## @mastra/dsql@1.5.5-alpha.0 ## @mastra/duckdb@1.16.0-alpha.1 ## @mastra/dynamodb@1.4.5-alpha.1 ## @mastra/elasticsearch@1.4.6-alpha.1 ## @mastra/lance@1.5.2-alpha.1 ## @mastra/libsql@1.27.1-alpha.0 ## @mastra/mongodb@1.24.1-alpha.1 ## @mastra/mssql@1.10.6-alpha.0 ## @mastra/mysql@0.14.1-alpha.0 ## @mastra/opensearch@1.1.4-alpha.1 ## @mastra/oracledb@0.5.5-alpha.0 ## @mastra/pg@1.33.0-alpha.2 ## @mastra/pinecone@1.1.3-alpha.0 ## @mastra/qdrant@1.1.6-alpha.0 ## @mastra/redis@1.4.12-alpha.1 ## @mastra/s3vectors@1.1.5-alpha.0 ## @mastra/spanner@1.10.1-alpha.1 ## @mastra/turbopuffer@1.2.3-alpha.0 ## @mastra/turso@0.1.18-alpha.0 ## @mastra/upstash@1.5.5-alpha.1 ## @mastra/valkey@0.2.8-alpha.1 ## @mastra/vectorize@1.1.4-alpha.0 ## @mastra/weaviate@0.1.4-alpha.0 ## @mastra/voice-aws-nova-sonic@0.2.6-alpha.0 ## @mastra/voice-azure@0.12.3-alpha.0 ## @mastra/voice-cloudflare@0.13.4-alpha.0 ## @mastra/voice-deepgram@0.13.3-alpha.0 ## @mastra/voice-elevenlabs@0.13.3-alpha.0 ## @mastra/voice-gladia@0.13.3-alpha.0 ## @mastra/voice-google@0.14.4-alpha.0 ## @mastra/voice-google-gemini-live@0.14.18-alpha.0 ## @mastra/voice-inworld@0.4.6-alpha.0 ## @mastra/voice-mistral@0.1.3-alpha.0 ## @mastra/voice-modelslab@0.2.3-alpha.0 ## @mastra/voice-murf@0.13.3-alpha.0 ## @mastra/voice-openai@0.13.4-alpha.0 ## @mastra/voice-openai-realtime@0.14.8-alpha.0 ## @mastra/voice-playai@0.13.3-alpha.0 ## @mastra/voice-sarvam@1.1.4-alpha.0 ## @mastra/voice-speechify@0.14.3-alpha.0 ## @mastra/voice-xai-realtime@0.2.17-alpha.0 ## @mastra/inngest@1.10.8-alpha.3 ## @mastra/temporal@0.4.18-alpha.4 ## @mastra/agentcore@0.5.3-alpha.0 ## @mastra/agentfs@0.2.3-alpha.0 ## @mastra/apple-container@0.5.2-alpha.0 ## @mastra/archil@0.3.2-alpha.0 ## @mastra/azure@0.3.3-alpha.1 ## @mastra/blaxel@0.10.2-alpha.0 ## @mastra/cloudflare-sandbox@0.5.4-alpha.0 ## @mastra/daytona@0.11.4-alpha.0 ## @mastra/docker@0.12.1-alpha.0 ## @mastra/e2b@0.15.1-alpha.1 ## @mastra/e2b-desktop@0.1.10-alpha.1 ## @mastra/files-sdk@0.3.3-alpha.0 ## @mastra/gcs@0.3.5-alpha.1 ## @mastra/google-drive@0.2.4-alpha.0 ## @mastra/mesa@0.3.3-alpha.0 ## @mastra/modal@0.7.3-alpha.0 ## @mastra/platform-workspace@1.10.0-alpha.1 ## @mastra/railway@0.10.2-alpha.1 ## @mastra/s3@0.6.5-alpha.1 ## @mastra/sprites@0.1.2-alpha.0 ## @mastra/vercel@1.6.2-alpha.0 ## @internal/playground@1.36.0-alpha.4 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
|
f51a29e82b |
fix(core): settle evented workflow runs cancelled from another instance (#28434)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9 Closes #28427 ## Summary On the evented workflow engine, cancelling a run from a different Mastra instance than the one running it (for example, the server cancel route landing on a non-owner replica) marked the run `canceled` in storage. The owner's `run.start()` promise never resolved, though. The same hang happened for runs started with `startAsync()` and then cancelled. **Cause:** `EventedRun.cancel()` wrote `canceled` to storage and then called `abortController.abort()`. It relied on an abort listener to publish `workflow.cancel`, but only `start()` and `resume()` register that listener. A run handle created any other way (another instance, or `startAsync()`) published nothing. The owner's event processor then dropped the run's later events through the canceled-status gate. As a result, `workflow.end` was never emitted and `start()` stayed pending. **Fix:** - `EventedRun.cancel()` now publishes `workflow.cancel` directly through a `publishCancel()` helper. The abort listener uses the same helper, and it publishes at most once per run handle. - `WorkflowEventProcessor` tracks the controllers it has already cancelled (`cancelHandledControllers`, a WeakSet). A step that unwinds on one of those controllers no longer publishes a second `workflow.cancel`, so a cancelled run emits exactly one `workflow.end`. This also removes a duplicate `workflow.cancel`/`workflow.end` pair that already happened on main whenever the owner cancelled its own run. ## Proof New tests: - `packages/core/src/workflows/evented/remote-cancel.test.ts` (12 tests): two Mastra instances sharing storage cancel across instances over `EventEmitterPubSub` and `UnixSocketPubSub`. The tests cover dountil, foreach, sequential, sleep and parallel shapes. The owner's `start()` settles as `canceled` and the step's abortSignal fires. Two more tests cancel after `startAsync()` and after an owner-local cancel, and each asserts exactly one `workflow.cancel` and one `workflow.end`. - `packages/server/src/server/handlers/workflows.test.ts`: `CANCEL_WORKFLOW_RUN_ROUTE` is called on a non-owner instance and the owner's run settles as `canceled`. - `pubsub/redis-streams/src/evented-remote-cancel.test.ts` (4 tests): cross-instance cancel over real Redis Streams, with each replica on its own connection. Shapes are sequential and dountil, with workers on the owner only and on both replicas. Results with the fix reverted: - Full revert: all 12 core tests fail, the server test times out (`expected 'timed out' to be 'canceled'`), and all 4 Redis tests time out. - Reverting only `workflow.ts`: 11 of the 12 core tests fail. - Reverting only the processor guard: the 2 single-event tests fail (they get 2 events instead of 1). With the fix, everything passes: - `pnpm build:core` - `tsc --noEmit` on core - eslint and oxlint (type-aware) on the changed files - oxfmt - core `src/workflows`: 102 files; `workflow.test.ts` (329 passed) and `evented-workflow.test.ts` (315 passed) were rerun alone after a heap OOM when three suites ran in parallel - server `workflows.test.ts`: 92 passed - Redis: 4 of 4 passed against a local redis-server on port 6381 ## Overlap with #24915 #24915 also edits `EventedRun.cancel()`. It calls `setupAbortHandler()` before `abort()`, guarded by `#abortHandlerRegistered`. That would also make cancel publish from a non-owner handle, so the two fixes overlap and will conflict on the same lines. Whichever PR merges second should keep a single publish path (`publishCancel()` here). The processor-side duplicate-end guard in this PR is independent of #24915. ## Unverified - **Multi-replica step abort (pre-existing, not fixed here):** when every replica runs workers, `workflow.cancel` reaches only one consumer in the group. If that consumer isn't the replica running the step, the step's abortSignal doesn't fire. `start()` still settles as `canceled`. Main behaves the same, even for local cancels. The Redis and server tests check step abort only when the owner is the sole worker. This should be tracked as a separate issue. - **Inngest and durable/evented agents:** I didn't run these. From reading the code, Inngest settles through `getRunOutput` on any terminal status, and durable agents use their own `agent.control.{runId}` abort transport. ## Risk: low The change is limited to the evented engine's cancel path. The publish is guarded to run at most once per handle. The WeakSet guard is per-process, so if two replicas process cancel for the same run at the same moment, the run could still end twice. That is no worse than main. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 When one Mastra instance cancels a workflow running on another, the workflow owner’s `run.start()` can now finish with a canceled result. The change also prevents duplicate cancellation messages from creating duplicate end events. ## What changed - `EventedRun` publishes `workflow.cancel` from both explicit cancellation and the abort handler through one helper. The helper prevents duplicate publishes and allows a retry if publishing fails. - `WorkflowEventProcessor` tracks controllers already handled by a cancellation event. When a step unwinds on one of those controllers, it does not publish another cancellation event. - Added tests for remote cancellation, `startAsync()`, server cancellation, Redis Streams, and retry after a failed cancellation publish. ## Validation The author reports passing core build, type-check, lint, formatting, and workflow tests. The author also reports passing server workflow tests and Redis Streams tests. These results are author-reported. ## Limitations When workers run on multiple replicas, a cancellation event may reach a processor other than the one running the active step. That step may not receive the abort signal. This case remains unfixed and unverified. Inngest and durable/evented agents were not tested. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
ce5d4731ee |
chore(deps): update dependencies across the monorepo (#28417)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
2ded35cb0c |
fix(core,redis,valkey): stop duplicate observe frames and save evented suspensions before publishing (#27352)
Fixes COR-1474: `Agent.observe()` / `POST /agents/:id/observe` sent durable-run frames more than once (a client saw 20–21 tool calls for a run that executed 16), and reconnecting with an `offset` did not reliably resume. The other half of the ticket, closing `observe()` for finished or unknown runs, shipped in #27358. The cause was two unshared numbering sources: the agent's `CachingPubSub` numbered its own frames, while evented durable steps published unnumbered to the transport and a follower re-cached them under new numbers. Now the cache list is the run's log and the transport is only the live tail. Each frame is appended with its index before it is published, and observers track one cursor. This PR also makes the evented engine save a tool suspension or approval request to memory before publishing its chunk. That change is needed here: the faster frame path made clients see the suspension before memory had it (see Proof 6). Scope notes: no breaking API changes. Cache keys keep `main`'s format. Atomic appends on Redis Cluster are tracked in COR-1491. ## Proof 1. **Frames arrive once, in order, and resume from an offset.** `caching-pubsub.test.ts`, `step-executor.test.ts`, `evented-split-transport.test.ts` and the aimock `observe-frames` / `observe-lifecycle` scenarios pass on durable and evented. The cursor, gap fill and log-restart guard each fail with their fix removed. 2. **Across processes with real Redis.** `pubsub/redis-streams/src/observe-cross-process.test.ts` (`RedisStreamsPubSub` + `RedisServerCache`, two processes), 8/8 on durable and evented: an unknown run closes, a finished run replays then closes, a live run is followed to its finish, and an observer stays attached across an approval suspend/resume. Each run's text and `finish` arrive exactly once. 3. **No shared cache behind a cross-process transport.** The libsql cross-process suite (API process plus dedicated worker, in-memory cache) passes, including `xproc-input-tripwire`. That test failed before steps fell back to publishing uncached in this setup, as on `main`. 4. **Rolling deploys.** Old and new `CachingPubSub` instances sharing one transport and cache delivered every frame once and in order: a reconnect after the upgrade, old and new live observers, and old and new workers alternating steps on one run. This ran as a local probe test that is not committed. 5. **Run-log TTL.** - In-memory, Redis and Valkey unit tests show `listPushIndexed` applies the requested TTL to the list and counter on each append. They also show a requested TTL only raises `ttlSeconds` and `ttlSeconds: 0` stays persistent. - `agent.stream.*` logs request 24h (`RUN_LOG_TTL_MS`). - These tests fail with the TTL math zeroed. - The Redis no-`eval` and `CROSSSLOT` fallback (`INCR` + `RPUSH`, as on `main`) keeps the requested TTL. That test fails with the fallback reverted to the default TTL. 6. **Suspensions are in memory before clients see them (evented).** - Timing probes, branch before this change: the client received `tool-call-suspended` at +0 ms, during the publish call, and read memory at +2 ms before the suspension was saved. - Same probes on `main`: the client received it at +1 ms, after the publish finished, and memory already had it. - `packages/memory` `replacement-instance-suspension.issue-27971` (evented) failed every run before the change and passes 3/3 after. - The observational-memory suites (1,881 tests), the durable/events/cache/evented core suites (2,774 tests) and the aimock scenarios pass. The 4 `background-task-lifecycle` aimock failures are pre-existing on `main` (bisected to #28162). 7. `pnpm --filter ./packages/core check`, lint and formatting clean. Redis and Valkey `tsc --noEmit` clean. ## Risk - **Evented routing.** On the evented engine, run frames, aborts and leases now use `mastra.pubsub` / `mastra.serverCache`. An evented agent with its own `pubsub` or `cache` on a different backend logs a warning and no longer uses it for run frames. The default engine is unchanged. - **Live-only without a shared cache.** `observe({ offset })` without a shared cache only sees the live tail. - **Late frames.** A frame the transport delivers late is filled in from the log at its original position instead of being dropped. This includes reasoning from a signal-preempted attempt, which clients already see when it arrives on time. It stays out of memory and the replacement request. - **Suspension order.** On the evented engine the suspension chunk now goes out after the memory save, so it arrives one memory write later. If the save throws (authorization failure or a superseded attempt), the chunk is not published. Before, the client saw a suspension it could not resume, followed by an error. - **Deprecations and custom caches.** `CachingPubSubOptions.source` / `__setSource()` are deprecated no-ops. Custom `MastraServerCache` subclasses keep the base, non-atomic `listPushIndexed` and ignore the TTL. - **Not exercised in this PR:** - **Redis Cluster.** The `CROSSSLOT` fallback is only unit-tested with a faked rejection. On a real Cluster it is not atomic: concurrent writers can interleave, so a list position may not match its index. Tracked in COR-1491. - **Long suspensions.** A suspension longer than the 300s default TTL was not run against a real Redis. The 24h TTL is covered by unit tests only. - **Inngest.** No end-to-end Inngest run. Inngest gets the 24h TTL but never deletes the run log when the run ends. - **Split deploys.** The platform team's split-deploy tests have not yet been rerun against this branch. --------- Co-authored-by: review <review@local> Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
11c83d5333 |
fix(core): recover evented agent runs after worker death (#28054)
Fixes COR-1469. Postgres-backed evented agents now send distributed worker orchestration through the shared broker, allowing Redis to reclaim unfinished work after a worker dies while the original API process remains alive. Other workflow locality and cross-process finish delivery are preserved. Postgres atomically checks step occurrence identifiers before writing results. Interrupted occurrences can be taken over, completed loop iterations reuse committed results and decisions, and stale deliveries cannot overwrite newer results. Regression coverage includes saved-result continuation, nested loop completion, retry publication, state preservation, and legitimate next iterations. Validated SIGTERM and SIGKILL with both a warm surviving worker and a replacement worker: all four cases pass, including five repeated matrix runs (20/20), with the original API caller completing without restart. Core validation passed in two groups (344 and 519 tests); Postgres workflow/run-fencing validation passed 42 tests earlier, and the final focused rerun passed 22 tests. TypeScript, touched-file type-aware lint, formatting, and whitespace checks pass. A Postgres query-plan assertion failed once during the final broad rerun and passed on the focused rerun. The Redis suite is not fully green: after building missing fixture dependencies, the seven affected files passed all 25 tests. The remaining agent-thread Lambda-death signal-drain test times out at agent-thread-cross-process.test.ts:389; removing all core changes and rebuilding reproduces the identical failure on the unchanged baseline. This limitation is not suppressed or changed by this PR. Upgrade @mastra/core and @mastra/pg together and drain older runs first. Interrupted external effects must be idempotent. Resume, individual foreach item guards, time travel, restart/redelivery races, other storage adapters, Redis persistence, and payload-size redesign remain out of scope. Tony’s fencing/recovery sign-off is required before merging and before platform#3224 lands. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 If a worker stops during an agent run, another worker can take over while the original API process stays alive. The system protects work that already finished, so recovery does not overwrite completed steps or repeat completed loop iterations. ## What changed - Routed eligible Postgres-backed evented workflow work through the shared broker so another worker can reclaim it. Workflow finish events still cross processes. - Added workflow-step guards and Postgres checks to reject stale or duplicate results. - Reused saved step results, loop decisions, and state during recovery. - Kept legacy step records without occurrence metadata on the existing unguarded path until a guarded write stamps them. - Added recovery tests, Redis Streams takeover fixtures, documentation, and coordinated upgrade guidance. ## Validation - SIGTERM and SIGKILL recovery passed with warm and replacement workers: 20/20, as reported. - Core validation passed in groups of 344 and 519 tests. Postgres workflow and run-fencing validation passed 42 tests earlier and 22 in the focused rerun, as reported. - TypeScript, touched-file type-aware lint, formatting, and whitespace checks passed, as reported. - A Postgres query-plan assertion failed in a broad rerun and passed in a focused rerun. - Redis validation remains partial. Seven affected files passed 25 tests. - The timeout at `agent-thread-cross-process.test.ts:389` also occurred on the unchanged baseline, as reported. ## Upgrade requirements and limitations - Upgrade `@mastra/core` and `@mastra/pg` together. - Drain runs started on older versions before upgrading. Older runs can finish after an upgrade, but crash recovery is not guaranteed. - Make interrupted external effects idempotent. - Resume, foreach-item guards, time travel, restart/redelivery races, other storage adapters, Redis persistence, and payload-size redesign are out of scope. - Tony’s fencing and recovery sign-off is required before merging and before `platform#3224` lands. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: NikAiyer <nick.aiyer@gmail.com> Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com> Co-authored-by: Abhi Aiyer <abhiaiyer91@gmail.com> |
||
|
|
dacb190944 |
chore: version packages (alpha) (#28009)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
|
e20bbd0ef4 |
fix(core): stop evented workflows/agents hanging when workflow.fail publish fails (#28218)
> Opened by @abhiaiyer91 ## What When an evented workflow event runs out of delivery attempts (3), or fails in a way that must not be retried (for example a throwing `stopWhen`, marked non-redeliverable since #28215), `WorkflowEventProcessor.handle()` publishes `workflow.fail` so whoever is waiting on `workflows-finish` gets an error. On `main` it marked the event terminal **before** that publish and only logged if the publish threw. Every later redelivery was then dropped as terminal, so in a live process: - `run.start()` on an evented workflow never resolves - `agent.generate()` / `agent.stream()` on an evented agent never errors or finishes - the stored run stays `running` (only crash recovery would clean it up) ## Fix `packages/core/src/workflows/evented/workflow-event-processor/index.ts` now tracks "failure not yet published" separately from "done for good": - If the `workflow.fail` publish throws, the processor returns `retry: true` and remembers the error. - On redelivery of that event it skips the handler and only retries the publish, reusing the transport's normal redelivery. This also applies to non-redeliverable failures, so a throwing `stopWhen` is still called exactly once. - The event becomes terminal once the publish lands, or after 3 publish attempts. - Publish attempts are folded into the delivery count. If the pending entry is evicted from its bounded map, the processor still knows the handler is spent and how many publishes it has tried, so it neither reruns the handler nor resets the 3-attempt cap. - The transport's `deliveryAttempt` sets a floor on that count. If a redelivery lands on a processor that never saw the event, and the budget is already spent, that processor only publishes `workflow.fail`. - 3 handler deliveries plus at most 2 republish deliveries fit the tightest transport cap (RedisStreamsPubSub, 5). UnixSocketPubSub allows 6. - `workflow.fail` events still never republish `workflow.fail`. There is no outbox, and nothing transport-specific. **If all 3 publish attempts fail,** the processor logs the error and stops retrying. That run is left `running` in storage and the live caller still gets no result, the same as on `main`. Only `recoverActiveRuns()` cleans it up on the next start. This is intentional: without an outbox, a pubsub that keeps failing has no way to reach the caller. The bounded give-up tests pin this behaviour (exactly 3 attempts, then no more publishes, no extra model calls) on EventEmitter, UnixSocket, Redis and the evented agent. ## Proof Repro first, on `main`: a step fails past the retry budget (every `workflow.step.end` publish throws) and the first N `workflow.fail` publishes throw. | Test | main (no fix) | with fix | |---|---|---| | `workflows/evented/fail-publish-after-exhaustion.test.ts`, EventEmitterPubSub + UnixSocketPubSub (control, 1 throw, 2 throws, bounded give-up) | 7 failed / 3 passed (controls pass): `start()` hangs past 5 s, storage `running`, `workflow.fail` tried once | 8 passed | | `aimock/scenarios/evented-fail-publish-after-exhaustion.scenario.test.ts` (evented agent; other engines skipped because they never use the event processor): control, 1 throw, bounded give-up, **throwing `stopWhen` + first `workflow.fail` publish throws** | 1 throw → `HANG`; give-up → `workflow.fail` tried once, not 3; throwing `stopWhen` → `HANG` | 4 passed. The `stopWhen` case rejects with `stopWhen exploded`, with `stopWhen` called once, 1 tool call and 1 model request. In the give-up case the model request count equals the control run's | | `workflow-event-processor/retry-budget.test.ts`: pending entry evicted; non-redeliverable handler with its pending entry gone; pending entry evicted before every redelivery; budget spent on another processor (`deliveryAttempt: 4`) | each fails with its guard removed (handler reruns or retries never stop) | 9 passed | | `pubsub/redis-streams/src/fail-publish-after-exhaustion.test.ts` (real Redis 6381; 0/1/2 throws, plus bounded give-up at 3) | 0/1/2 cases: 2 failed / 1 passed, `workflow.fail` tried once, hangs for 20 s (give-up case added later in review, not run on `main`) | 4 passed | What the caller sees with the fix, when the publish fails once or twice: `start()` resolves `failed`, the stored snapshot is `failed`, and `workflows-finish` carries `workflow.fail`. Related suites, rebased on `main` with #28215, with the fix: - `src/workflows/evented/**` and `src/events/**`: 59 files, 740 passed, 5 skipped. `evented-workflow.test.ts` crashes its vitest worker (OOM) in the combined run. Run alone with a 6 GB heap: 315 passed, 5 skipped. - aimock abort/error/stop-when/fail-publish scenarios on all engines: 32 files, 398 passed, 13 expected-fail, 60 skipped, 0 failed. - `pubsub/redis-streams` `fail-publish-after-exhaustion.test.ts` + `local-redelivery-evented.test.ts` against the rebuilt core: 6 passed. - `tsc --noEmit` on core: clean. Prettier: clean. `pnpm build:core`: succeeds. Inngest never reaches the changed code: - Runs are triggered with `inngest.send` (`workflows/inngest/src/run.ts:253`, `:366`; `durable-agent/create-inngest-agent.ts:837`). - They execute inside the Inngest function (`workflow.ts:446`) through `InngestExecutionEngine` (`workflow.ts:561`), which extends `DefaultExecutionEngine` (`execution-engine.ts:61`). - Nothing in `workflows/inngest/src` references `WorkflowEventProcessor` or `handleWorkflowEvent`. ## Unverified None. ## Risk: low - The change only affects events that have used up their delivery budget (or are non-redeliverable) **and** whose `workflow.fail` publish threw. The normal path and the success path of the publish behave the same as before. - Worst case: 2 extra redeliveries of that one event, still inside every transport's cap. - Known limit, no worse than `main`: a non-redeliverable failure goes terminal on attempt 1. If its `workflow.fail` publish throws and the redelivery lands on a different processor, that processor sees `deliveryAttempt: 2` and runs the handler once more. Ruling that out needs shared per-event state, which this PR deliberately leaves out. --------- Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
7d580884fb |
fix(core): call a throwing stopWhen once on evented agents (#28215) | ||
|
|
40a522c142 |
chore(deps): update build tools (#27997) |