mirror of
https://github.com/mastra-ai/mastra.git
synced 2026-11-17 08:44:46 +00:00
244 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
d929736435 |
fix(core): use one stream bus for durable agents (#30229)
Durable agents now use the wrapped Agent's stream bus instead of
constructing a second one. An Agent-level pubsub carries both run frames
and thread events, including after registration and forks; evented
execution still uses Mastra's shared transport. Standalone durable
agents remain observable, and plain-run records and abort checks stay
separate from durable execution.
Previously, observing across replicas required repeating the pubsub on
the durable wrapper. This now works with the Agent configuration alone:
```ts
const agent = createDurableAgent({
agent: new Agent({ id: 'assistant', name: 'Assistant', instructions, model, pubsub }),
});
```
Inngest explicitly exposes its detached-streaming capability without
migrating its bus. The server retains the `isDurableAgentLike` fallback
in `hasResumableStreams` for compatibility with older Inngest wrappers:
their proxy can inherit `false` from a plain Agent or lack the method
entirely. Server and Inngest versions are not coupled, so removing the
fallback would silently turn HTTP disconnects into run cancellation on
supported mixed-version installs. This is a deliberate, user-approved
exception to the ticket's shortcut-removal bullet; no peer-dependency
bump is needed while the fallback remains. Regression coverage exercises
capability `true`, `false`, and missing; the latter two fail without the
fallback.
Server branch audit (`packages/server/src/server/handlers/agents.ts`):
`validateDurableToolCallAccess` (line 298) remains because approvals
need durable workflow snapshots; `authorizeRunAccess` (2676) remains
because durable ownership uses workflow rows/local and replay owners
rather than plain run records (COR-1542). The two abort-route branches
(2724 and 2742) retain durable cancellation support and its
missing/finished-run acknowledgement semantics, not separate transports.
The observe-route branch (3141) remains for generation-fenced
`__subscribeToRunStream` and the Inngest adapter's `pubsub` fallback;
consolidating observation is COR-1541, and migrating Inngest's bus is
COR-1539. The recover-route guard (3746) remains because only durable
execution has recoverable workflow snapshots. None of these remaining
branches requires a second stream bus on core DurableAgent.
The Mastra-fallback regression also runs through a fork created after
registration, both with and without resolving the bus before
registration. Another replica must receive replay plus live tail through
finish, and the terminal frame must be published on Mastra's transport.
The latest targeted rerun passed 33 tests, including cache-policy
warnings and plain-publisher isolation across both durable storage
variants; core typecheck, targeted lint, formatting and diff checks
passed.
Verified with 1,643 core durable/stream tests, 78 AIMock parity tests,
292 server handler tests (including the mixed-version capability cases),
119 Inngest tests, and six real Redis cross-server cases covering live
observation, thread events and HTTP Stop. Typechecks, lint and
formatting pass. The Agent-pubsub regression failed before the fix; a
built-package red/green demo also confirms the transport change. An
earlier approval-resume timing failure passed in isolation and on the
final full rerun.
Fixes COR-1540
## Transport precedence and upgrade scope
Default-engine durable run transport precedence is now: explicit
`createDurableAgent` pubsub → wrapped Agent pubsub → Mastra pubsub.
Evented runs retain Mastra's shared transport. Mastra-only
configurations are unchanged.
If the wrapped Agent already specifies a different transport from Mastra
and the wrapper has no override, this fix changes the effective
run-frame and remote Stop transport. To preserve routing across an
upgrade, explicitly pass the previous transport to `createDurableAgent`
or align the Agent and Mastra transports. The wrapper configures the
shared stream bus, so its override also affects resumable streams from
direct calls on the wrapped Agent. The core changeset includes this
upgrade note.
A local investigation used separately built pre-change main
(`
|
||
|
|
d0f7dd3a80 |
chore: version packages (alpha) (#30420)
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. ⚠️⚠️⚠️⚠️⚠️⚠️ `main` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `main`. ⚠️⚠️⚠️⚠️⚠️⚠️ # Releases ## @mastra/factory@0.29.0-alpha.3 ### Minor Changes - Added opt-in health metrics to Factory, sent through `mastra.metrics`. Pick the groups you want, since every record counts towards observability usage: ([#30225](https://github.com/mastra-ai/mastra/pull/30225)) ```ts new MastraFactory({ // ... metrics: { groups: ['process', 'postgres', 'redis', 'sandbox', 'intake'] }, // or 'all' }); ``` Or set `FACTORY_METRICS=all` (or a list like `sandbox,intake`). `FACTORY_METRICS_INTERVAL_MS` (default 60s, minimum 15s) and `FACTORY_METRICS_MAX_PER_MIN` (default 500) tune how often records are sent and cap their volume. Metrics are named `mastra_factory_*` and labeled with `service` (`api`, `worker` or `all`) so split deployments can be told apart. ### Patch Changes - Fixed a pull request Factory opened for a card whose issue lives in Jira, Linear or incident.io getting a card of its own on GitHub. The pull request now attaches to that card and moves it to Review, as it already does on GitLab, so the work is no longer tracked on two cards while it is being reviewed. Fixes [#30358](https://github.com/mastra-ai/mastra/issues/30358). ([#30418](https://github.com/mastra-ai/mastra/pull/30418)) - Deliver Factory notifications when their bound session no longer exists, including concurrent notifications. Notifications for a role that has been handed off remain suppressed. ([#30422](https://github.com/mastra-ai/mastra/pull/30422)) - Updated dependencies [[`02dbde1`]( |
||
|
|
4dd77bac9f |
feat(factory): opt-in health metrics by group (#30225)
> Opened by @roaminro
## What
Opt-in health metrics for Factory, sent through `mastra.metrics`, so a
deployment can tell at a glance which component to investigate first:
the Node process, Postgres, Redis, sandboxes and session start-up, or
intake.
Every record counts towards observability usage, so nothing is collected
unless a group is turned on:
```ts
new MastraFactory({ ..., metrics: { groups: ['sandbox', 'intake'] } }); // or groups: 'all'
```
Env overrides: `FACTORY_METRICS` (`all`, `0`, or a list),
`FACTORY_METRICS_INTERVAL_MS` (default 60s, min 15s),
`FACTORY_METRICS_MAX_PER_MIN` (default 500).
| Group | Metrics (`mastra_factory_*`) |
| --- | --- |
| process | event-loop delay and utilization, heap utilization and
memory against the heap limit, `process_cpu_cores` (all threads) against
`process_cpu_limit_cores`, major/incremental GC pauses,
`process_uptime_ms` |
| postgres | pool total/idle/waiting per pool (read after the check),
`SELECT 1` check per pool, `pg_stat_activity` sessions + longest
query/idle transaction, slowest `pg_stat_statements` logged every 10 min
by query ID |
| redis | `INFO` check, memory used/max and utilization, clients,
evictions, stream backlog (only with `RedisStreamsPubSub`; only the
check on a cluster) |
| sandbox | `session_ready_ms{kind=first\|resume,outcome}` per start
(failures included), `session_sandbox_wait_ms`, `sessions_starting` +
`session_start_oldest_age_ms` every interval, provider start time and
failures (`sandbox_provider`), setup phase and command timings,
recreate/reset/retire counts, active sandboxes |
| intake | GitHub/Linear ingest outcomes, reconcile replays (missed
webhooks), sweep failures, `intake_last_sweep_age_ms` per kind (counted
from when the sweep worker starts) |
## How
- Disabled groups are no-ops: no timers, checks or records.
- Counts and timings are aggregated in-process and sent once per
interval. A counter is one record per series. A timing is
`stat=avg|p95|max` on its `_ms` series plus a `<name>_count` series with
the number of observations, so a series ending in `_ms` is always
milliseconds. p95 uses reservoir sampling. Every duration is in ms.
- Quiet intervals still send 0 for known counters (configured intake
sources, sweep failures, sandbox start failures), so an empty chart
means "nothing happened", not "not reporting".
- A per-minute cap holds over the whole minute; event counts and timings
are sent before gauges, so the cap drops gauges first. Overflow is
counted in `mastra_factory_metrics_dropped_total`.
- Check durations are sent for `db_store`, `db_vector` and `redis` only;
the server-stats checks send failures only.
- Labels are closed sets only: no IDs, paths or repository names.
- Every record has `service=api|worker|all`, derived from
`MASTRA_WORKERS`. Server-wide stats (Postgres activity, Redis
memory/backlog) run only on `worker`/`all`, so split deployments don't
report them twice. Platform's built-in `serviceName` isn't set per
record: `metrics.emit()` has no override, so that needs a core or
Platform change.
- `mastra.metrics` and the logger are read on every flush, since
Platform registers its exporter after the app module loads. A failed
tick is logged, never thrown.
- `[factory:timing]` stderr lines keep main's rule (only when boot
diagnostics are on); metrics are recorded either way, gated by their own
group.
- `RedisStreamsPubSub` gains `createMonitorClient()` + `streamKey()`, so
checks use their own connection and never queue behind pub/sub traffic.
- Session sandbox retirement counts only real, successful retirements.
## Not included
Linear/GitLab issue-reconcile sweeps, Platform poller latency, Slack
time-to-first-reply (needs a hook in `packages/core` channels),
HTTP/storage timings.
## Test plan
- [x] `pnpm check`, `pnpm lint` in `mastracode/factory`; `tsc` in
`pubsub/redis-streams`
- [x] `src/metrics/metrics.test.ts`: config parsing, service mapping,
aggregation and `_count` series, cap and send order, zero counters,
disabled groups, api skips server-wide stats, session start timing
(joined callers, failures, starts in flight), check guards, sweep age
- [x] Existing tests for touched modules (workspace, github
rules/sandbox/reconcile, linear rules, sandbox, timing)
- [x] Snapshot on shipyard `beta` with `FACTORY_METRICS=all` (single
process, `service=all`)
- [ ] Split deployment reporting both `service=api` and `service=worker`
Docs: mastra-ai/softwarefactory-docs#18
---------
Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: Roamin <97863888+roaminro@users.noreply.github.com>
|
||
|
|
ca0af9fa6e |
ci: skip memory benchmarks on PRs and run durable stream tests in parallel (#30416) | ||
|
|
1c52fdd034 |
chore: version packages (alpha) (#30261)
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. ⚠️⚠️⚠️⚠️⚠️⚠️ `main` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `main`. ⚠️⚠️⚠️⚠️⚠️⚠️ # Releases > The changelog information of each package has been omitted from this message, as the content exceeds the size limit. ## @mastra/factory@0.29.0-alpha.1 ## @mastra/core@1.84.0-alpha.1 ## @mastra/playground-ui@69.1.0-alpha.1 ## @mastra/client-js@1.60.1-alpha.1 ## @mastra/react@1.15.1-alpha.1 ## @mastra/deployer-cloud@1.84.0-alpha.1 ## @mastra/longmemeval@1.1.43-alpha.1 ## @mastra/opencode@0.1.43-alpha.1 ## create-factory@0.4.4-alpha.1 ## @mastra/code-sdk@1.19.1-alpha.1 ## mastracode@0.50.1-alpha.1 ## @mastra/arize@1.3.31-alpha.0 ## @mastra/arthur@0.4.31-alpha.0 ## @mastra/braintrust@1.3.28-alpha.0 ## @mastra/datadog@1.4.23-alpha.0 ## @mastra/deepeval@0.1.25-alpha.0 ## @mastra/laminar@1.3.33-alpha.0 ## @mastra/langfuse@1.6.7-alpha.0 ## @mastra/langsmith@1.4.4-alpha.0 ## @mastra/observability@1.24.1-alpha.0 ## @mastra/otel-bridge@1.5.23-alpha.0 ## @mastra/otel-exporter@1.5.3-alpha.0 ## @mastra/posthog@1.4.8-alpha.0 ## @mastra/sentry@1.2.33-alpha.0 ## @mastra/agent-builder@1.2.2-alpha.0 ## mastra@1.39.1-alpha.1 ## @mastra/connect@1.4.1-alpha.0 ## @mastra/deployer@1.84.0-alpha.1 ## @mastra/editor@0.17.2-alpha.0 ## @mastra/mcp-docs-server@1.3.14-alpha.1 ## @mastra/memory@1.41.2-alpha.0 ## @mastra/server@1.84.0-alpha.1 ## @mastra/google-cloud-pubsub@1.1.8-alpha.0 ## @mastra/redis-streams@0.8.2-alpha.0 ## @mastra/valkey-streams@0.6.3-alpha.0 ## @mastra/elysia@0.1.24-alpha.1 ## @mastra/express@1.5.28-alpha.1 ## @mastra/fastify@1.5.28-alpha.1 ## @mastra/hono@1.7.26-alpha.1 ## @mastra/koa@1.7.28-alpha.1 ## @mastra/nestjs@0.3.1-alpha.1 ## @mastra/next@0.2.42-alpha.1 ## @mastra/tanstack-start@0.2.42-alpha.1 ## @mastra/clickhouse@1.30.1-alpha.0 ## @mastra/cloudflare@1.8.1-alpha.0 ## @mastra/cloudflare-d1@1.5.1-alpha.0 ## @mastra/convex@1.9.1-alpha.0 ## @mastra/dsql@1.5.8-alpha.0 ## @mastra/dynamodb@1.5.2-alpha.0 ## @mastra/libsql@1.30.1-alpha.0 ## @mastra/mongodb@1.27.1-alpha.0 ## @mastra/mssql@1.11.2-alpha.0 ## @mastra/mysql@0.17.1-alpha.0 ## @mastra/oracledb@0.5.7-alpha.0 ## @mastra/pg@1.36.1-alpha.0 ## @mastra/redis@1.4.14-alpha.0 ## @mastra/spanner@1.12.1-alpha.0 ## @mastra/turso@0.1.22-alpha.0 ## @mastra/upstash@1.6.2-alpha.0 ## @mastra/valkey@0.2.10-alpha.0 ## @mastra/temporal@0.6.1-alpha.1 ## @mastra/daytona@0.11.6-alpha.0 ## create-mastra@1.39.1-alpha.1 ## @internal/playground@1.39.1-alpha.1 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
|
4f16c08068 |
fix(redis-streams,valkey-streams): release leases while close() is flushing (#30387)
## Description
On Redis Streams and Valkey Streams, `releaseLease` now keeps working
while `close()` is flushing, until the writer connection is released.
Before, it returned without doing anything as soon as `close()` started.
### What changed
In `@mastra/redis-streams`, `releaseLease` checks the existing
`#writerReleased` flag instead of `#closed`. `close()` sets that flag
right before it disconnects the writer. `@mastra/valkey-streams` had no
such flag, so this adds one, sets it right before the writer's `quit()`
in `close()`, and checks it in `releaseLease`. The other lease methods
are unchanged.
### Why
A worker is shutting down while a step it handles for another process's
run is still executing, and the step takes longer than the drain
timeout. The step finishes while `close()` is flushing, and publishing
its result throws on the closing client. The handler then releases the
step's lease, which did nothing once `close()` had started, and nacks.
The nack still republished through the open writer. Another worker
received the copy, was refused the lease, treated the delivery as a step
a live worker was still running, and acked it. Nothing ran the step
again and the run stayed `running`.
With this change the release and the nack's republish go through the
same open writer, so the worker that gets the copy can take the lease
and run the step.
### Known limits
A release that races the writer's quit now rejects instead of returning.
The nack that follows fails too, so the entry stays pending and is
reclaimed after `reclaimIdleMs` (60 s by default), by which time the 30
s lease has expired.
Other lease releases in core, on durable agent claims and thread
streams, now really release during `close()` and can reject in that same
race. Their callers catch the error or turn it into a retry.
A release that fails for another reason while the republish succeeds
still drops the step.
Google Cloud Pub/Sub has no lease provider and is not affected.
### Tests
`pubsub/redis-streams/src/close-release-lease.test.ts` runs against a
real Redis with two `RedisStreamsPubSub` instances on one key prefix.
A's grouped handler takes a lease, then releases it and nacks while A's
`close()` is still in progress, held open by a fan-out reader. B
receives the second delivery and must be able to take the same lease.
Without the fix, B gets `{ acquired: false, owner: 'a' }`. The test also
checks that `close()` had not resolved when the release ran.
`pubsub/valkey-streams/src/close-drain.test.ts` gets a case with a
mocked client: a release called while `close()` waits on its flush sends
the release script, and one called after the writer quit sends nothing
and doesn't reconnect. Without the fix, the release sent during the
flush is missing.
## Related issue(s)
None
## Type of change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [ ] Test update
## Checklist
- [ ] I have linked the related issue(s) in the description above
- [ ] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
**ELI5:** When a worker shuts down, it can now still let go of its task
lock and retry the task. This lets another worker pick up the task
sooner instead of leaving it stuck.
## Summary
This change fixes a shutdown race in the Redis Streams and Valkey
Streams pub/sub adapters. Before this change, `releaseLease` stopped
work as soon as `close()` began. The writer stayed open during the
flush, so a lease release and the republish from a nack could not use
it. A nack could then republish a delivery while the step lease was
still held. Another worker could then drop the step.
### Changes
- **`pubsub/redis-streams/src/index.ts`**: `releaseLease` checks
`#writerReleased` instead of `#closed`. During `close()`, a lease
release can still run until the writer is marked released.
- **`pubsub/valkey-streams/src/index.ts`**: Adds a `#writerReleased`
flag. `releaseLease` stays available after `close()` marks the instance
closed. `close()` sets the flag after it flushes pending publishes and
before it quits the writer.
- **`pubsub/redis-streams/src/close-release-lease.test.ts`** (new, 89
lines): Uses two workers. Worker A releases the lease and nacks the
event during shutdown. The test expects redelivery on attempt 2 and
expects worker B to acquire the lease.
- **`pubsub/valkey-streams/src/close-drain.test.ts`** (46 lines added):
Uses a mocked client. It checks that `release` runs during a close
flush, before `xAdd` and `quit`. It also checks that a release requested
after the writer quits does not run again.
- **Changesets** (`.changeset/eleven-pears-wave.md`,
`.changeset/spicy-canyons-yawn.md`): Patch releases for
`@mastra/valkey-streams` and `@mastra/redis-streams`.
### Behavior notes
- If a release races the writer quit, the release rejects. The following
nack also fails. The entry stays pending for reclaim after
`reclaimIdleMs` (60 seconds by default). The 30-second lease expires
before that.
- Other core lease releases can now run during `close()` and reject in
the same race. Their callers must catch the error or retry.
- Google Cloud Pub/Sub is not changed.
- A release that fails for another reason can still drop the step if the
republish succeeds.
### Public API
No exported declarations change.
### Diff size
6 files changed, 153 lines added, 2 lines removed.
### Test and CI status
The author reports that both test files cover release during a close
flush and behavior after the writer quits. Results for the current head
are not available in the supplied evidence.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
99ec3bd582 |
fix(core): leave evented workflow runs to a redelivering broker and restart only stopped runs (#30279)
## Description This PR stops `restartAllActiveWorkflowRuns()` from restarting evented runs that a redelivering broker (Redis Streams, Valkey Streams, Google Cloud Pub/Sub) already recovers. It also restarts a default-engine run only once no instance holds its run lease and a grace period (2 s by default) has passed. Before, every booting instance restarted every active evented run, so a step interrupted by a crash ran once per booting pod (#28712). The broker already recovers those runs: Redis and Valkey Streams reclaim an unacked message once it has been idle for `reclaimIdleMs`, and Google Cloud Pub/Sub redelivers once the ack deadline passes. ### Evented runs on a broker `PubSub` gets a `redeliversUnacked` getter (default `false`). Redis and Valkey Streams return `true` while the reclaim loop runs (`reclaimIntervalMs > 0`), Google Cloud Pub/Sub returns `true`, `CachingPubSub` forwards to its inner pubsub, and the in-process `EventEmitterPubSub` and the Unix socket pubsub return `false`. When the pubsub (read through `__rawBus()`) redelivers, recovery never lists, reads or restarts an evented run. An evented workflow without declarative schedules costs no listing at all. Evented runs on any other pubsub (`EventEmitterPubSub`, Unix socket, or a third-party pubsub that doesn't override the getter) are restarted at once by the explicit call, as before. ### Scheduled runs The `mastra.pubsub` proxy publishes every workflow event of a scheduled run (`sched_wf_*`, plus the notification dispatcher's row) `localOnly`, so the broker never holds it. Recovery still restarts those runs: an evented workflow with declarative schedules gets one summary listing per status, and only its process-local scheduled runs are kept. The proxy and recovery share one predicate, `isProcessLocalScheduledRunId`. ### Default engine One recovery rule: a `running` or `waiting` run is restarted when its run lease isn't live and `recovery.workflowGraceMs` (new, 2,000 ms by default for every pubsub) has passed since the later of the lease's last expiry and the run's last write. The pass lists summaries (`listWorkflowRuns({ status, perPage: false, summary: true })` per status), reads leases in batches of 100 with `getRunOwnerships`, re-reads a run's lease once its restart gets a slot, and runs every restart on one instance-wide limiter (`recovery.workflowConcurrency`). A run left live or in its grace period gets a per-run recheck timer, as on main. Overlapping calls share one pass. `shutdown()` waits, within `drainTimeout`, for a pass still reading the store and for the restarts it handed out. Without run fencing every active run is restarted, as before. ### Nested resume A default-engine nested run resumed directly now holds its parent's run lease while it runs and hands it to the parent's resume, so recovery doesn't restart the parent meanwhile. On a store without compare-and-set, where the parent stays `suspended`, the nested run holds its own lease instead. Also: `renewWorkflowLease` waits for each renewal to settle before arming the next one (a self-rearming `setTimeout` instead of `setInterval`), and `EventEmitterPubSub`'s JSDoc and the docs say it's in-process only. ### Repro Postgres + Valkey Streams with default reclaim settings (`reclaimIdleMs` 60 s, `reclaimIntervalMs` 30 s). Instance A starts a run whose step records an execution and then hangs; A is killed with `SIGKILL` mid-step. Then 2 or 4 fresh instances boot and all call `restartAllActiveWorkflowRuns()` at once. "At boot" counts step executions caused by those calls; "total" counts every execution after the kill until the run succeeds; "rerun after" is seconds from the kill to the first new execution. 3 trials per row, identical results within each row except for timing. | Build | Engine | Booting pods | Executions at boot | Executions total | Rerun after | |---|---|---|---|---|---| | Base (this PR's base branch) | evented | 2 | 2 | 2 | 0.1 s | | Base | evented | 4 | 4 | 4 | 0.1 s | | Base | default | 2 | 0 | 1 | 15.1 to 16.6 s | | Base | default | 4 | 0 | 1 | 15.0 to 16.9 s | | This PR | evented | 2 | 0 | 1 | 60.1 to 60.2 s | | This PR | evented | 4 | 0 | 1 | 60.1 to 60.2 s | | This PR | default | 2 | 0 | 1 | 17.0 to 19.1 s | | This PR | default | 4 | 0 | 1 | 17.2 to 21.0 s | On this PR the evented rerun is the broker's reclaim: one execution on one pod, about 60 s after the kill. The default-engine rerun comes once A's lease (15 s TTL) lapsed plus the 2 s grace and up to 5 s of recheck jitter. ### Benchmark 3,000 active default-engine runs that another instance holds, among 30,000 finished runs, two workflows. "Before" is a copy of main's sweep run in the same process (full listing, then `createRun()` + `restart()` per run, 5 at a time); "after" is `restartAllActiveWorkflowRuns()` on this PR. Median of 5 calls. | Store | Before | After | Listing payload before | Listing payload after | |---|---|---|---|---| | Postgres | 1,773 ms | 30 ms | 1.12 MB | 0.60 MB | | LibSQL | 627 ms | 44 ms | 1.12 MB | 0.60 MB | The benchmark's snapshots are nearly empty, so the payload gap here is small; with real step results it grows with snapshot size. ### Known limits On a redelivering broker these evented runs stay `running` until someone calls `restart()` on them. Main's boot restart recovered them; stack 2 takes them over. - A sleeping evented run whose instance exits, after a crash or during a normal deploy. The sleep acks its delivery and keeps a `setTimeout` in the process that started it, so any exit of that process loses the timer. - A step whose message Redis or Valkey Streams trimmed. Both trim with `MAXLEN ~ maxStreamLength` (10,000 by default), which also drops entries that are still pending. A step that is in flight while that many newer events are published loses its message if its instance then exits. Deployments with long steps and a high event volume should raise `maxStreamLength`. - A message the broker dropped after `maxDeliveryAttempts` or at `inFlightTimeoutMs`. - A failure publish that gave up after its retries (the worker acks the message). - A run whose `running` snapshot was saved but whose start event never reached the broker, because its process stopped between the two. - A Redis or Valkey instance that restarted without persistence. - A scheduled evented run that was in flight when a deploy stopped its process and removed the workflow's schedule from code. Recovery doesn't list it, so it stays `running` until `restart()`. #28712 is fixed for evented runs on a redelivering broker. These are still restarted by every booting instance, as before: - scheduled evented runs (`sched_wf_*`), whose events never leave their process - evented runs on the in-process `EventEmitterPubSub` or the Unix socket pubsub. Several instances sharing one storage with these pubsubs aren't supported for evented workflows; the docs now say a multi-instance evented deployment needs a broker. With Redis or Valkey Streams, `reclaimIdleMs` must stay at 30 s or more: a message reclaimed while the dead worker's 30 s step fence is still held is dropped as a duplicate. The getter's JSDoc, the adapter docs and the changeset state it. Rolling deploys are out of scope. Stack 3 of 4: storage projections and batch ownership reads, then the finished-step check, then this PR, then the recovery loop with a leader. ## Related issue(s) Fixes #28712 ## Type of change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Code refactoring - [x] Performance improvement - [x] Test update ## Checklist - [x] I have linked the related issue(s) in the description above - [x] I have made corresponding changes to the documentation (if applicable) - [x] I have added tests that prove my fix is effective or that my feature works - [ ] I have addressed all Coderabbit comments on this PR <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ELI5: When a server crashes in the middle of a workflow, Mastra can restart that workflow. Before this change, some workflows ran their steps again on every new server that started, so a step could run many times. This change lets the message broker handle the restart for those workflows, so each step runs only once. Other workflows now wait for a short grace period before they restart. ## Summary This pull request changes how workflow recovery restarts runs after a crash or a boot. It targets duplicate step execution in two cases: - **Evented runs on brokers that redeliver unacknowledged messages.** Recovery skips these runs. The broker redelivers the unacknowledged step events instead. - **Default-engine runs.** Recovery restarts a run only after no live lease remains and a grace period has passed. ### Behavior changes - Google Cloud Pub/Sub, Redis Streams, and Valkey Streams report `redeliversUnacked` as `true` when their reclaim or redelivery path is active. Redis Streams and Valkey Streams report `true` only when `reclaimIntervalMs` is above 0. - `PubSub` adds a `redeliversUnacked` getter that defaults to `false`. `CachingPubSub`, `EventEmitterPubSub`, and `UnixSocketPubSub` expose or declare this capability. - Recovery leaves evented runs to the broker when `redeliversUnacked` is `true`. Scheduled evented runs restart on boot. Other evented runs restart immediately when the broker does not redeliver. - Default-engine recovery applies `recovery.workflowGraceMs`. The default is 2,000 ms. Negative or non-finite values use the default. A run restarts only after its lease is no longer live and the grace period has elapsed since the later of its last update and lease expiry. Runs that are not yet eligible are checked again later. If the storage adapter does not support run fencing, active runs restart without the grace rule. - Recovery lists active run summaries, reads ownership in batches, and rechecks ownership before each restart. - Restarts use a shared concurrency limiter (`createConcurrencyLimiter`). A restarted run holds its slot until it finishes. - Overlapping recovery passes are deduplicated. Shutdown waits for an in-flight pass. - Recovery skips nested runs whose active parent drives them. - Lease renewal runs sequentially (`renewWorkflowLease`, `repeatAfterSettled`). A new renewal starts only after the previous one settles. - Nested resume coordinates parent and child leases. A failed lease claim rejects the resume and leaves the runs suspended. - Run-claim operations return whether a status transition occurred. They return `false` when storage is absent, running snapshots are not persisted, or compare-and-set is not supported. ### Public API changes - `MastraRecoveryConfig.workflowGraceMs?: number` (default 2,000 ms). - `PubSub.redeliversUnacked` getter (default `false`). - `redeliversUnacked` getters on `GoogleCloudPubSub`, `RedisStreamsPubSub`, `ValkeyStreamsPubSub`, and `UnixSocketPubSub`. - `ConcurrencyLimiter` interface and `createConcurrencyLimiter(limit)` function in `run-with-concurrency.ts`. - `renewWorkflowLease`, `repeatAfterSettled`, and `WORKFLOW_RUN_LEASE_RENEW_INTERVAL_MS` in `run-ownership.ts`. - `Run._resume(params)` accepts an optional `takeHeldOwnership` callback. ### Documentation - Crash-recovery and worker docs describe broker reassignment and multi-instance limits. - `EventEmitterPubSub` documents that delivery is process-local. - Redis Streams and Valkey Streams docs recommend `reclaimIdleMs` of at least 30 seconds. Reclaiming earlier can cause duplicate-dropped step events. - The recovery reference documents `workflowGraceMs`. ### Tests and test utilities - New recovery tests cover exit cases (crash and shutdown), recovery reads, rules, ownership during nested resumes, and lease renewal. - Shared test helpers add `sharedBroker`, `killableProcess`, `interceptStarts`, `persistActiveRun`, and `describeRecoveryExitCases`. - Existing tests set `workflowGraceMs: 0` where they need immediate recovery. - Redis Streams and Valkey Streams tests check `redeliversUnacked` against `reclaimIntervalMs`. - LibSQL and Postgres recovery-pass suites run the shared exit cases. ### Scope and limits - The change addresses issue `#28712` for evented runs on redelivering brokers. Evented runs on in-process or Unix socket pubsubs still restart at boot. The PR states that multi-instance evented deployments need a broker. - The PR lists unrecovered cases: sleeping evented runs, trimmed or dropped messages, failed publishes, missing start events, non-persistent Redis or Valkey restarts, and scheduled runs whose schedules were removed. Rolling deploys are out of scope. - Reported benchmark results (median recovery time on 3,000 active and 30,000 finished runs: 1,773 ms to 30 ms on Postgres, 627 ms to 44 ms on LibSQL) and repro results are author-reported. They were not independently verified. - CI and test results: not available. - Review findings and severity counts: not available. ### Size - 41 files changed, 2,594 lines added, 209 lines removed (from the pull request diff). <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
d60ba71be0 | chore: version - exit prerelease mode | ||
|
|
055a8c3b82 | chore: version packages | ||
|
|
743a02786e |
feat(valkey): identify Mastra-created clients with a clientInfoTag (#30156) | ||
|
|
c014d187bb |
chore(deps): update docker images (major) (#19788) | ||
|
|
ce03237b34 |
fix(core): send only the response messages that changed on agent stream events (#29971)
## Description Follow-up to #29965 (merged). After that PR, each plain Agent `step-finish` on `agent.stream.<runId>` still carried the run's response messages so far, so a long run stored every tool output once per later step: the log still grew with the square of the step count, mostly from large tool results. The publisher now keeps a cursor of message fingerprints and sends only the messages from the first one that changed: ```ts // step 1: { from: 0, seq: 0, messages: [A1, T1] } // step 2: { from: 2, seq: 1, messages: [A2, T2] } // step 3: { from: 4, seq: 2, messages: [A3] } _durableResponseMessagesDiff: { producer, seq, from, base, hash, messages } ``` "From the first changed index" (not "only new messages") because earlier entries change: a step's assistant message grows across steps, and a processor retry rolls the list back. Readers rebuild the full list per step in the stream adapter, before chunks reach `MastraModelOutput`, so `onStepFinish(step).response.messages`, `output.steps` and `onFinish` still see the whole list: - `agent.observe(runId, { offset })` first folds the step-finish events before the offset from the run's history (bounded wait, doesn't delay the subscription). - A diff only applies to the list it was made against (`base` must equal the hash of the list the reader holds). One that can't be placed (a missed or reordered delivery, a late copy) is never applied, and the reader reports no messages for that step instead of a partial list. - The publisher resends the whole list once the diffs since the last whole list add up to its size, so whole lists cost at most about as much as the diffs and the log stays linear. A failed publish or diff makes the next one a whole list, so a reader that missed one recovers. - On a bus that keeps no history of the run's topic (`cache: false`, or a `CachingPubSub` whose `shouldCache` leaves the topic out), every step-finish is a whole list, so an observer that attaches mid-run can place it. An internal `PubSub.__replaysTopic(topic)` answers this per topic, since `CachingPubSub.supportsOffsets` is true even for topics it doesn't cache. Transports without offsets (Redis and Valkey Streams without a cache) get whole lists too: an `observe()` from an offset there follows the stream from its tail. - Each diff carries its publisher's id: a run resumed after approval publishes from a new publisher that starts with the whole list. Readers keep the list of the current and previous publisher apart, so a late event from the previous one doesn't break the next diff. - The diff replaces `_durableResponseMessages`, so an older reader during a rolling upgrade reports no messages rather than a partial list. DurableAgent and EventedAgent still send whole lists, which the adapter handles as before; they move to diffs in a follow-up. ## Related issue(s) Part of COR-1532 (plain Agent; DurableAgent/EventedAgent and Inngest follow in the next PRs of this stack). Follow-up to #29965 (production Redis OOM from plain Agent run logs). Related: #27475. ## Type of change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Code refactoring - [x] Performance improvement - [x] Test update ## Checklist - [x] I have linked the related issue(s) in the description above - [ ] I have made corresponding changes to the documentation (if applicable) - [x] I have added tests that prove my fix is effective or that my feature works - [ ] I have addressed all Coderabbit comments on this PR Tests: `agent-stream-log.scenario.test.ts` (all engines) adds "an early tool result is stored as often as the last one" (fails before this PR on the plain Agent; durable engines pinned as the known gap), an observer resuming mid-run at an offset (fails without the history fold), an observer joining mid-run on a bus without history (fails without the whole-list rule), and an approval suspend/resume replay compared step by step. `response-messages.test.ts` covers the diff and apply rules (edits, shrinking lists, duplicates, lost diffs, producer switches, a late whole list from the previous producer, the size-based whole list). The real-Redis test asserts the tool-result counts and that a 20-step run stores less than 2.5x the bytes of a 10-step one (2.1x with this PR, 2.9x before it). <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 Each step used to save the full list of response messages again, even when most messages had not changed. This change saves only the updates and rebuilds the full list for observers, which can reduce storage during long runs. ## Summary - Plain Agent stream events now publish response-message diffs. The publisher sends a full list first and periodically resends one as diffs accumulate. - Readers apply diffs when they match the current message list. If a reader cannot rebuild the list, it reports no response messages for that step rather than a partial list. - When `agent.observe()` starts at a numeric offset, it replays earlier events to rebuild response messages before delivering later events. Observers continue to receive the full list for each step. - DurableAgent and EventedAgent continue to publish full lists. Tests add coverage for diff replay, offset replay, approval resumption, tool-result repetition, and Redis stream storage growth. - Older observers report no response messages for steps published with diffs. The changeset recommends upgrading agent and observer processes together. Test results: not available. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
749c09cf91 |
fix(core): publish lean agent stream events instead of the loop's chunks (#29965)
## Description
A registered Agent publishes every run on `agent.stream.<runId>` so
`agent.observe()` and reconnecting clients can replay it, and on Redis
that log is stored until the run is cleaned up. The publisher forwarded
the loop's in-process chunks as they were: every `step-start` carried
the provider request, and every `step-finish` carried the prompt again
(`metadata.request`, `messages.all`/`user`) plus every earlier step with
its own prompt (`output.steps`). The log grew roughly with the cube of
the step count; in production single runs of a plain Agent took 0.3 to
3.4 GB of Redis and filled it.
The publisher now builds `step-start`, `step-finish` and `finish` field
by field (`agent/streams/wire.ts`) with what readers use, which is what
DurableAgent already publishes:
```ts
// step-finish payload on the topic, before
{ stepResult, metadata: { ...response, request: { body: '<whole prompt>' } },
output: { text, toolCalls, usage, steps: [/* every step, each with its prompt and response */] },
messages: { all, user, nonUser } }
// after
{ stepResult, metadata: { id, timestamp, modelId, headers, providerMetadata, ... },
output: { usage /*, steps: [{ tripwire }] when the step was rejected */ },
_durableStepContent: [/* this step's content */],
_durableResponseMessages: [/* response messages so far */] }
```
- The caller's own stream and callbacks are unchanged; only the
published copy is reduced.
- Observers of a plain Agent get better replays: each step's content was
`[]` before, and `onFinish` lost its text over JSON transports and threw
on in-memory ones (the finish steps carried AI SDK-shaped `toolCalls`
the reader can't read). Finish steps now carry `text`, and tool
calls/results in the record shape DurableAgent uses.
- A delegated agent's chunks inside `tool-output` /
`workflow-step-output` are slimmed by the plain publisher. Durable
engines still publish them as-is, since their callers read their own run
back from the topic; that case is pinned with `it.fails`.
- Step content is read from the run's message list when the step
finishes (one extra response-to-UI conversion per step, the same work
the caller's output does).
Still to come, stacked on this PR: each `step-finish` still sends the
response messages so far in full; follow-ups send only what changed, for
the plain Agent first and then DurableAgent/EventedAgent and Inngest
(superseding #27522).
## Related issue(s)
Fixes COR-1530. Production incident: shipyard Redis OOM from
`agent.stream.<runId>` keys of plain Agent runs (Factory). Related:
#27475 (the DurableAgent side of the per-step growth) and COR-1532
(per-step response messages, the follow-up stack), introduced with
#28424.
## Type of change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [x] Performance improvement
- [x] Test update
## Checklist
- [x] I have linked the related issue(s) in the description above
- [ ] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR
Tests: `agent-stream-log.scenario.test.ts` runs on every engine (prompt
copies on the log, delegated agent prompt, observer replay over
in-memory and JSON transports, a rejected and retried reply), and
`pubsub/redis-streams/src/agent-stream-size.test.ts` checks what real
Redis holds in both cache modes. The new plain-agent cases fail on main.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## ELI5
Agent stream logs repeat old prompt and step data, which can make them
large. This change publishes smaller event records while keeping the
caller’s stream intact and giving observers step content they need.
## Summary
- The publisher creates reduced `step-start`, `step-finish`, and
`finish` events. It removes provider requests and repeated step data.
- For `step-finish`, the publisher reads the current step’s content from
the run’s message list. It omits content for retry steps and logs a
warning if it cannot read the content.
- The publisher reduces nested agent chunks inside `tool-output` and
`workflow-step-output`. DurableAgent and EventedAgent continue to
publish their delegated-agent steps in full.
- Observer replay gains step content and richer finish data, including
run text and tool calls, and step reasoning, sources, and files. The
event still includes response messages on each `step-finish`.
- Added scenarios cover prompt retention, delegated-agent prompts,
observer replay over in-memory and JSON transports, and rejected and
retried replies. A Redis integration test checks stored stream size and
observer replay with both cache modes.
## Validation
Completed CI, build, and test results: not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
d77893aed3 |
fix(core): resume evented agent observe() at the offset on Redis Streams with the default cache (#29798)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9 ## What Fixes #28324. On an evented agent (`createEventedAgent()`) using a cross-process PubSub such as `RedisStreamsPubSub` with the default in-memory server cache, `observe(runId, { offset })` ignored the offset. A client reconnecting mid-run got earlier events again, and an offset past the end of a finished run replayed nearly the whole run. ## Why it happened Evented step frames went out unnumbered whenever an in-memory cache sat behind a cross-process transport. The concern was that a worker in another process would number the run from 0 in its own cache, and observers would then drop its frames as duplicates. That skipped numbering in the common case too, where every step runs in the process that started the run. ## Change - `CachingPubSub` gains two `@internal` options, `extendExistingLogsOnly` and `onUnlogged`. With `extendExistingLogsOnly`, a publish only appends to a topic whose log this cache already holds. Any other publish goes out live and unnumbered, and `onUnlogged` is called. - `StepExecutor` now always wraps the step bus when a server cache exists. If the cache is process-local and the bus crosses processes, it turns on `extendExistingLogsOnly`. The caller writes the run's `start` frame before any step runs, so steps in the caller's process extend that log and get numbered. Steps on another worker find no log, stay live-only as before, and log a one-time warning that recommends `RedisServerCache`. - `StepExecutor` now picks up the Mastra logger when it is constructed. Workflow event processors create executors without registering them on Mastra, so without this the warning was dropped. - Docs (`durable-agent.mdx`) list the two setups that still need a shared cache: steps running on a dedicated worker, and observers reconnecting through another replica. ## Tests - `pubsub/redis-streams/src/observe-offset.test.ts`: the evented + default cache row now passes the exact-resume checks (resume at offset, offset past end returns nothing, mid-run replay). The full matrix passes 12/12 against Redis 7. - `step-executor.test.ts`: - When the caller holds the run's log, step frames are numbered. - When a worker has no log, its frames stay unnumbered and one warning reaches the logger configured on `Mastra`, using an executor that was never registered on Mastra. - Both cases fail on main. - `caching-pubsub.test.ts`: tests `extendExistingLogsOnly` both with and without an existing log. - `stores/libsql` cross-process suite, including `xproc-input-tripwire`: 48 passed, 0 failed. - Core `tsc` is clean. oxlint and prettier pass on the changed source files. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 When an evented run starts, the process that started it can now number its events even when events travel through Redis. That lets a client resume from an offset without seeing old events again. Events from a different worker remain unnumbered when only a local cache is available, and Mastra warns about this limitation. ## Changes - Added internal `CachingPubSub` options to append only to existing cached logs and report events that cannot be numbered. - Updated `StepExecutor` to use those options with a process-local cache behind cross-process PubSub. It numbers events for runs whose log exists locally and warns once when a worker publishes events for a run it did not start. - Updated documentation to explain when a shared cache is required, including worker execution and reconnects through another server process. - Added tests for existing and absent cached logs, step numbering and warnings, and offset replay scenarios. ## Validation Test, build, and CI results: not available. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com> Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com> Co-authored-by: NikAiyer <nick.aiyer@gmail.com> |
||
|
|
f21e0742ce | chore: version packages | ||
|
|
d6a5f98df3 |
fix: index observe chunks for resume (COR-1506) (#29520)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9 Linear: [COR-1506](https://linear.app/kepler-crm/issue/COR-1506/observe-offset-cant-be-used-to-resume-it-counts-log-events-but-clients) ## Problem `observe({ runId, offset })` takes `offset` as a position in the run log. Clients can't compute that position: - Some log entries never reach the client. These include iteration-complete records after every step and ownership markers. - Updates on the wire carried no index. A client that resumes at "number of updates received" lands too early and gets repeated updates. The drift grows by one per completed step, so every tool-using run that reconnects after its first step is affected. On a 3-step tool loop, 16 of 28 cut points repeated 1–2 updates on both durable and evented engines. ## Fix - **Index on updates:** every update now carries `index`, the run-log position it came from. This covers: - `/observe` on every agent - `/stream` on durable and evented agents - core `stream()`/`observe()` chunks and client-js `observe()`/`stream()` - **Resuming:** pass `index + 1` of the last received update that has an index as `offset`. Gaps in the numbering are expected; they're log entries that send nothing. - **Unindexed updates:** the synthesized `abort` update carries no index. Its log position is on the `finish` sent beside it. Client-built structured-output `object` chunks have no index either. An `object-result` logged by a structuring-model run carries one. Unindexed updates don't move the resume point. A client that reconnects between `abort` and `finish` gets the `abort` again. - **`offset` is unchanged:** it still means a run-log position. - Core no longer strips `index` in `DurableAgent.__subscribeToRunStream`. The stream adapter and the server's `toAgentStreamChunks` copy the envelope's index onto each chunk and drop any index a forwarded sub-agent chunk carried. `index` is optional on `BaseChunkType` and `DataChunkType`. - **Also fixed:** on plain agents, `/observe` (HTTP and core `agent.observe()`) with an offset past the end of a finished run used to wait for the idle timeout or hang. It now closes immediately. It reads the log from just before the offset and only falls back to a full read when that window is empty. The check runs in the background, so it never delays returning the stream or its cancel handle. A trailing `error` chunk doesn't count as the end of the run. ## Migration `index` is a new field and `offset` keeps its meaning: - Callers that pass a log position need no change. - Callers that replay from `offset: 0` need no change. They get the same updates in the same order, now with `index` set. - Only callers that passed a count of updates received need to change. They were already getting repeats after step boundaries; switch them to `index + 1` of the last indexed update. ## Evidence | Check | Result | | --- | --- | | Red/green | With `main`'s core and server, 15 tests on numbering and `/stream`–`/observe` matching fail; with this change all pass. | | Cut points | Every cut point of a 3-step tool loop on durable and evented engines resumes from the first `/stream` connection at last index + 1. Covers the HTTP route and client-js `stream()` → `observe()`. | | `/stream` vs `/observe` match | Frames are identical JSON, including `index`. Replay from 0 equals the first `/stream`. | | End of run | Resume after finish or past the end returns the rest and closes, including far past the end (offset 100). | | Adapters | Hono 98/98, Fastify 98/98, Express 90 passed + 8 skipped (see bug 1 below). The shared suite is now wired into Express and Fastify. | | Redis restart | Real-Redis split-deploy test (in `pubsub/redis-streams/src/observe-cross-process.test.ts`) passes on both engines. The client resumes on a freshly booted API process at last index + 1 with nothing repeated or missing. Evented: the worker executes the run and the API is killed with SIGKILL. Durable runs execute in the API that started them, so that API stays up; the client drops its connection and resumes on a second API. | | Suspend | After tool approval, indexes keep increasing. Observers parked mid-suspend and every cut across the suspend resume correctly on both engines. | | Unit suites | server `agents.test.ts` 289, core `src/stream` + `src/agent/streams` 481 | | Static | tsc clean on core, server, client-js, the test suite, the Hono/Express/Fastify adapters and redis-streams. oxlint 0 warnings. | ## Scope Plain (non-durable) agents' `/stream` updates aren't numbered yet. The caller receives each update before the cache assigns its number. That's tracked in COR-1510. Plain agents' `/observe` is covered here. ## Existing bugs found (not fixed here) 1. **Express untilIdle 404:** after a client drops an untilIdle stream on a durable or evented agent, Express answers `/observe` with 404 "Run not found". It fails the same way on `main`. Those 8 cases are skipped on Express only, via `skipUntilIdleDisconnectReplay`. 2. **Finished durable run, other API:** a finished durable run returns 404 on `/observe` from any API process except the one that executed it (`authorizeDurableRun`, `handlers/agents.ts`). Resuming while the run is live works from any API. 3. **Plain replay mismatch:** on plain agents, replayed `step-finish`/`finish` payloads differ from the live ones (for example `timestamp` and `processorRetryCount`). Related to COR-1510. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 When a connection drops, clients need to know which updates they already received. This change adds positions to many updates so clients can resume from the right place. It also closes observations of finished runs when the requested position is past the end. ## Summary - Added optional `index` values to chunks from durable and evented agent streams, `observe()` streams, and corresponding HTTP routes. Clients can resume by passing the last indexed chunk’s `index + 1`. - Preserved run-log indexes through core stream adapters and server frame conversion. Sub-agent data chunks do not carry a child run’s index into the parent stream. - Some chunks remain unindexed, including synthesized `abort` chunks and client-built structured-output `object` chunks. Failed cache writes and non-atomic custom-cache indexing can affect replay. - Updated API schemas and documentation to explain index gaps, replay limits, and resumption. - Added past-end handling for finished plain-agent observations in core and HTTP `/observe`. History checks do not delay stream return or prevent cancellation. - Added tests for indexed replay, past-end closure, cancellation during history reads, sub-agent index isolation, and observation across API restarts. **Validation:** Current-head CI, build, and test results are not available. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com> Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com> Co-authored-by: NikAiyer <nick.aiyer@gmail.com> |
||
|
|
75435bc2cd |
fix(redis-streams): retry publishes on transient connection errors (#29512)
## Summary The Redis Streams pubsub now retries a publish that fails with a transient connection error, instead of failing the delivery and letting the step re-run. `#publishRemote` already retried while Redis answered `LOADING` after a restart. Any other connection failure (socket closed unexpectedly, client offline, connection or socket timeout, `ECONNRESET`, `EPIPE`, `ETIMEDOUT`, `ECONNREFUSED`) was rethrown immediately. A lost `workflow.step.end` publish was nacked and redelivered, so the step ran again. These errors now retry under the same 120 s budget (250 ms between attempts) until Redis accepts the write, the pubsub closes, or the budget runs out. This addresses the transport side of #29504. ## Limits - **Not exactly-once.** A worker that dies between finishing a step and publishing `step.end` still gets the `step.run` redelivered, and that step runs again. At-least-once delivery is accepted for that window. - **Duplicates after a lost reply.** If Redis wrote the event but the reply was lost, the retry appends it again. Subscribers can receive a second delivery with the same event id. A duplicate `workflow.step.run` is dropped by the step fence; other events are already handled as at-least-once. - **Slower failure in a long outage.** A publish that starts while connected and then loses the connection now fails after the 2-minute budget (`PublishRetryBudgetExhaustedError`) if Redis doesn't come back, where it previously failed immediately. The trade-off is that no event is lost in a short outage. ## Changes - `pubsub/redis-streams/src/index.ts`: `isTransientPublishError()` widens the publish retry from `LOADING` to transient socket errors; budget/delay constants renamed to `PUBLISH_RETRY_BUDGET_MS` / `PUBLISH_RETRY_DELAY_MS`. Each attempt races a timer for what is left of the budget, because node-redis parks commands issued while reconnecting in its offline queue and never settles them; running out throws `PublishRetryBudgetExhaustedError`, which is not retried. `publishRetryBudgetMs` overrides the 120 s default. - `pubsub/redis-streams/test-fixtures/harness.ts`: `makeXaddDropProxy()`, a TCP proxy that drops the client connection on the first XADD either before Redis sees it or after Redis applies it, optionally closing its listener after the drop so Redis looks down for good. - `pubsub/redis-streams/src/publish-transient-retry.test.ts`: both drop modes deliver the event; all copies share one event id. A third case closes the proxy listener after the drop and asserts publish rejects with `PublishRetryBudgetExhaustedError` at a 2 s budget. - Changeset: `@mastra/redis-streams` patch. ## Tests - `publish-transient-retry.test.ts`: 2/2 pass. With the check reverted to `isLoadingError`, both fail with `Socket closed unexpectedly`. - `pnpm --filter @mastra/redis-streams run check:test` passes; full `@mastra/redis-streams` test suite run against Redis 6381 / Postgres 5436. - No changes to `@mastra/core`; `pnpm build:core` succeeds. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 If Redis briefly loses a connection while receiving an event, the publisher now tries again instead of failing immediately. If Redis saved the event but its reply was lost, the retry can deliver a duplicate with the same event ID. ## Changes - Redis Streams publishing retries `LOADING` errors and recognized transient connection errors, including closed sockets, `ECONNRESET`, `EPIPE`, and timeouts. - Retries stop for non-transient errors, when the pub/sub instance is closed, or after 120 seconds. Attempts are spaced 250 ms apart. - Added a test proxy and tests for connection drops before and after Redis writes an `XADD`. - Added a patch changeset for `@mastra/redis-streams`. When Redis writes an event but the reply is lost, the retry can append the event again. Both copies have the same event ID. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com> Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com> Co-authored-by: NikAiyer <nick.aiyer@gmail.com> |
||
|
|
2f93851c37 |
fix(core): make Agent.durable configure resumable streams only (#29503)
Co-authored-by: NikAiyer <nick.aiyer@gmail.com> |
||
|
|
f94beac862 |
fix(core): unwedge restarted nested evented loops (#29431)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9
## Problem
Setup: a split deploy (API with `MASTRA_WORKERS=false` plus dedicated
workers over Redis) with `recovery.durableAgents: 'auto'`. When the API
and the worker die together mid-tool and both restart, boot recovery
claims the run under a new generation and the interrupted tool runs
again. After that, the run stops making progress. It sits in
`durable-agentic-loop:running` forever while the API keeps renewing its
lease. Because the run never ends, it also keeps its thread lease, so
any later run on the same thread is rejected with
`AgentThreadLeaseConflictError`. Mastra Doctor hits this every time,
since its thread is the deployId and every re-diagnosis lands on the
stuck thread. The `DurableExecutionFenceError` and `This workflow run
was not active` messages in the ticket are side effects of this stuck
run.
## Root cause
Recovery behaves correctly: it claims a new generation and re-drives the
run. The run gets stuck in the evented step guard in
`WorkflowEventProcessor.endWorkflow`.
When a restarted nested loop body finishes, `endWorkflow` calls
`#processLoop` for the parent loop. The `!parentWorkflow.restart`
condition there, introduced by
|
||
|
|
a0068a9ad1 |
feat(core): resumable streams for every Agent (#28424)
> Opened by @user_01KNPFT2RPETZTEP15J3AEBA08 Originally stacked on [#28416](https://github.com/mastra-ai/mastra/pull/28416), which has merged. This PR now targets `main`. ## What Any `Agent` can now have resumable streams, not only durable agents. A run is published on `agent.stream.<runId>` with a replay cache, so: - a client can reconnect after a disconnect, - another reader can follow the run with `agent.observe(runId)`, - any reader can stop it (`observe().abort()`, `agent.abortRunStream(runId)`, or the abort-run route). Durable execution (surviving process restarts) stays with `DurableAgent` / `EventedAgent` / `InngestAgent`, which keep publishing through the same stream layer. ### Config New `streams?: boolean | { pubsub, cache, shouldCache, cleanupTimeoutMs }` on `Agent`. - Omitted: on when the agent is registered on `Mastra`, off for standalone agents. - `false`: off. `{ cache: false }`: live only, no replay. - With `durable`, `streams` keys override `durable` keys. `durable: true` + `streams: false` keeps publishing, drops the replay cache, and warns. ### Bus selection - Registered agents (plain and `DurableAgent` on any engine) publish on `mastra.pubsub`, so other processes sharing the PubSub and run-record cache can observe and abort. - Precedence: `streams.pubsub` > `durable.pubsub` > `mastra.pubsub`. The evented engine still forces `mastra.pubsub` over an agent-level pubsub and warns when they differ. - Unregistered agents keep a private in-process `EventEmitterPubSub`. - The replay cache wraps the shared bus once (existing `CachingPubSub` reuse check), so nothing is double-published. ### Behaviour change With streams on, closing the HTTP stream detaches the reader instead of aborting the run. To stop a run explicitly, use `POST /agents/:agentId/runs/:runId/abort`, which now works for plain agents too (ownership-checked, including across servers sharing the PubSub and run-record cache), or the thread abort route. `streams: false` keeps the old behaviour. ## Commits 1. ` |
||
|
|
675caf0613 |
chore: version packages (alpha) (#29336)
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. ⚠️⚠️⚠️⚠️⚠️⚠️ `main` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `main`. ⚠️⚠️⚠️⚠️⚠️⚠️ # Releases > The changelog information of each package has been omitted from this message, as the content exceeds the size limit. ## @mastra/client-js@1.60.0-alpha.0 ## @mastra/react@1.15.0-alpha.0 ## @mastra/factory@0.28.0-alpha.0 ## @mastra/code-sdk@1.19.0-alpha.0 ## mastra@1.39.0-alpha.0 ## @mastra/core@1.83.0-alpha.0 ## @mastra/mcp@2.9.0-alpha.0 ## @mastra/mcp-apps-components@0.1.0-alpha.0 ## @mastra/playground-ui@69.0.0-alpha.0 ## @mastra/server@1.83.0-alpha.0 ## @mastra/nestjs@0.3.0-alpha.0 ## @mastra/clickhouse@1.30.0-alpha.0 ## @mastra/duckdb@1.19.0-alpha.0 ## @mastra/libsql@1.30.0-alpha.0 ## @mastra/pg@1.36.0-alpha.0 ## @mastra/temporal@0.6.0-alpha.0 ## @mastra/auth-better-auth@1.1.11-alpha.0 ## @mastra/deployer-cloud@1.83.0-alpha.0 ## @mastra/deployer-cloudflare@1.2.42-alpha.0 ## @mastra/deployer-netlify@1.2.42-alpha.0 ## @mastra/deployer-sandbox@0.3.27-alpha.0 ## @mastra/deployer-vercel@1.2.42-alpha.0 ## @mastra/longmemeval@1.1.42-alpha.0 ## @mastra/opencode@0.1.42-alpha.0 ## create-factory@0.4.3-alpha.0 ## mastracode@0.49.1-alpha.0 ## @mastra/arize@1.3.30-alpha.0 ## @mastra/arthur@0.4.30-alpha.0 ## @mastra/braintrust@1.3.27-alpha.0 ## @mastra/datadog@1.4.22-alpha.0 ## @mastra/deepeval@0.1.24-alpha.0 ## @mastra/laminar@1.3.32-alpha.0 ## @mastra/langfuse@1.6.6-alpha.0 ## @mastra/langsmith@1.4.3-alpha.0 ## @mastra/observability@1.23.2-alpha.0 ## @mastra/otel-bridge@1.5.22-alpha.0 ## @mastra/otel-exporter@1.5.2-alpha.0 ## @mastra/posthog@1.4.7-alpha.0 ## @mastra/sentry@1.2.32-alpha.0 ## @mastra/agent-builder@1.2.1-alpha.0 ## @mastra/connect@1.3.2-alpha.0 ## create-mastra@1.39.0-alpha.0 ## @mastra/deployer@1.83.0-alpha.0 ## @mastra/editor@0.17.1-alpha.0 ## @mastra/mcp-docs-server@1.3.13-alpha.0 ## @mastra/memory@1.41.1-alpha.0 ## @mastra/rag@2.7.3-alpha.0 ## @mastra/redis-streams@0.8.1-alpha.0 ## @mastra/elysia@0.1.23-alpha.0 ## @mastra/express@1.5.27-alpha.0 ## @mastra/fastify@1.5.27-alpha.0 ## @mastra/hono@1.7.25-alpha.0 ## @mastra/koa@1.7.27-alpha.0 ## @mastra/next@0.2.41-alpha.0 ## @mastra/tanstack-start@0.2.41-alpha.0 ## @mastra/spanner@1.11.2-alpha.0 ## @mastra/turso@0.1.21-alpha.0 ## @mastra/inngest@1.11.2-alpha.0 ## @mastra/gcs@0.3.6-alpha.0 ## @internal/playground@1.39.0-alpha.0 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
|
49eaabb19b |
fix(core,redis-streams): recover evented runs when Redis is LOADING mid-tool (COR-1502) (#29430)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9
## Summary
On the evented engine, if Redis restarts while a tool step is finishing,
it spends a while (often 30–60s) reloading its dataset. During that time
it answers every command with `LOADING Redis is loading the dataset in
memory`. When that happens, the run stays `running` forever and no
answer is saved. This PR makes the run wait out the restart and finish
normally: the answer is saved and each tool runs exactly once.
Fixes COR-1502.
## Root cause: three stacked problems
1. **Publishing never retried.** `RedisStreamsPubSub.publish()` sent the
tool result and `step.end` once. During `LOADING` that failed, so the
step was redelivered and its tools ran a second time.
2. **The thread lease expired during the window.** Redis kept counting
down the lease while it reloaded. On redelivery, Postgres still renewed
the run ownership, but `renewDurableThreadLease` found no lease key and
returned `false`. The execution fence then refused every redelivery
("could not verify that this execution still owns the run"). The event
was dropped after 5 deliveries and the run stayed `running`.
3. **The lease retake gave up when the step went idle.** `#retakeLease`
in `thread-stream-runtime` stopped trying once the tool step finished.
The run was aborted as lease-lost (`reason: 'abort'`, empty answer)
before the final model turn.
## Fix
- **`@mastra/redis-streams`:**
- `publish` retries `LOADING` errors every 250ms for up to 2 minutes,
and stops if the client closes. This covers both plain `XADD` and
`MULTI`/`EXEC` publishes.
- It implements the new `isUnavailableToAll(error)`, which returns true
for `LOADING`.
- **`@mastra/core` `events/pubsub.ts`:** adds an optional
`LeaseProvider.isUnavailableToAll?(error)`. A backend returns true when
it is refusing every client, so nobody else can be granted the lease
meanwhile. Any other error, including a connection failure, must return
`false`.
- **`@mastra/core` execution fence:** this applies only when Postgres
run ownership has just renewed for a generation-fenced claim.
- If the thread lease can't be renewed because the key is gone, the
fence takes it back with `SET NX`. It never takes a lease that another
owner holds.
- If the backend is refusing every client, the fence relies on the
Postgres run ownership instead.
- Any other lease error still fences the execution.
- **`@mastra/core` thread-stream runtime:** while the backend refuses
every client, `#retakeLease` keeps asking, within the existing
`AGENT_LEASE_LOST_TOOL_GRACE_MS`. Generic errors keep the existing
strict lease-loss policy, which stays safe for network partitions.
## Tests
- New test `pubsub/redis-streams/src/evented-redis-loading.test.ts`. It
runs separate API and worker processes against real Redis and Postgres.
They connect through a TCP proxy (`makeLoadingProxy` in
`test-fixtures/harness.ts`) that simulates a restart: it drops every
connection, then answers `-LOADING` for 20s while two tools finish.
During the window, `MULTI`/`EXEC` behave as in real Redis (`EXEC` →
`EXECABORT`).
- **Red on main:** the run never reached a terminal state, and the
worker looped on the fence error.
- **Green with the fix (~30s):** the run completes, the answer is saved,
each tool runs exactly once in `tool_executions`, and run ownership is
released.
- Related suites pass:
- core: execution-fence, thread-lease, and thread-stream suites.
- redis-streams: `pubsub`, `agent-thread-cross-process`,
`durable-worker-takeover`, `fail-publish-after-exhaustion`,
`local-redelivery-evented`.
- `tsc` (core, redis-streams, redis-streams test config) and `oxlint`
are clean on the touched files.
- Validated together with the COR-1497 branch
(`user/session-90e601e9-…`, `7c3d66f9`): the local merge had no
conflicts, and `durable-full-restart` (3 cases) plus this LOADING test
all pass.
## Out of scope / follow-up
On success, the workflow engine still publishes `workflow.step.end`
before it saves the step result. Any publish failure other than
`LOADING` still leaves a window where a redelivered `step.run` finds no
saved result and runs its tools again. Saving the result before
publishing would close that window. It is a broader engine change, so I
left it for a separate ticket.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## ELI5
When Redis restarts, it can temporarily refuse requests while it reloads
data. This change lets evented runs wait for Redis and recover their
thread lease, so they can finish without repeating completed tools.
## What changed
- `@mastra/redis-streams` retries stream publishing when Redis returns
`LOADING`, up to two minutes. Other errors still fail immediately.
- `LeaseProvider` can optionally implement `isUnavailableToAll(error)`
to identify errors that mean the backend is refusing every client.
- After Postgres run ownership is renewed, the execution fence can
reclaim a missing thread lease. If the backend reports that it is
refusing all clients, the fence relies on run ownership. Other lease
errors still prevent execution.
- Thread-lease retakes can continue during backend-wide unavailability
within the existing grace period.
- Added an integration test for a Redis loading window during tool
execution. The test checks that the run completes, saves its answer,
executes each tool once, and releases run ownership.
## Tests
The PR description reports that the integration test passes against real
Redis and Postgres. It also reports passing related core and Redis
Streams suites, TypeScript checks, and `oxlint` on touched files. These
results were not independently verified here.
## Known limitation
The PR description notes that a workflow step may still rerun tools on
redelivery if publishing its result fails for an error other than
`LOADING`, because the result is published before it is saved.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
|
||
|
|
6e8dfcb83f | chore: version - exit prerelease mode | ||
|
|
ab4d89cd38 |
test(redis-streams): run durable worker-takeover suite against real Redis and Postgres (#28952)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9
Part 3 of 3, split from #28676. Refs COR-1490. Parts 1 and 2 (#28947,
#28951) are merged into `main`. This PR is based on `main`.
## What this does
The API-loss and worker-takeover guarantees from #28951 only hold with a
shared store that supports run fencing and a transport that works across
processes. In-memory suites can't test that. This PR:
- adds `.github/workflows/test-durable-streams.yml`, which runs the
two-process takeover suite against Redis (`:6381`) and Postgres
(`:5436`) service containers. The checkout uses `persist-credentials:
false`.
- calls it from `test-suite.yml` as the `durable-streams` job. The job
reports its own check but does not gate `Merge Test Reports`, because
its flake rate hasn't been measured yet.
- runs it when `prebuild.yml` sees changes to the evented agent and
workflow code, `packages/core/src/mastra/`, `packages/memory/`,
`pubsub/redis-streams/`, `stores/pg/`, or the workflow files
- extends `pubsub/redis-streams/src/durable-worker-takeover.test.ts` and
adds the `durable-api-loss.entry.mjs` fixture. The scenarios: API
SIGTERM/SIGKILL with remote workers, a thread follow-up after API loss,
a remote abort, worker shutdown, single-process Redis shutdown, and the
original caller completing with and without a survivor that is already
running
- adds `tsconfig.test.json` and a `check:test` script, which type-check
the tests and fixtures against core's `dist`
No changeset: this PR only touches CI and tests.
## Verification
- Reviewer run at `
|
||
|
|
3bdb4ec5f4 |
chore(deps): update build tools (major) (#29026)
> [!NOTE] > This PR body was truncated due to platform limits. This PR contains the following updates: | Package | Type | Update | Change | Pending | |---|---|---|---|---| | [@vitest/coverage-v8](https://vitest.dev/guide/coverage) ([source](https://redirect.github.com/vitest-dev/vitest/tree/HEAD/packages/coverage-v8)) | pnpm.catalog.default | major | [`4.1.12` → `5.0.5`](https://renovatebot.com/diffs/npm/@vitest%2fcoverage-v8/4.1.12/5.0.5) | `5.1.0` | | [@vitest/mocker@>=2.1.0 <4.1.12](https://redirect.github.com/vitest-dev/vitest/tree/main/packages/mocker) ([source](https://redirect.github.com/vitest-dev/vitest/tree/HEAD/packages/mocker)) | pnpm-workspace.overrides | major | [`4.1.12` → `5.0.5`](https://renovatebot.com/diffs/npm/@vitest%2fmocker@>=2.1.0 <4.1.12/4.1.12/5.0.5) | `5.1.0` | | [@vitest/ui](https://vitest.dev/guide/ui) ([source](https://redirect.github.com/vitest-dev/vitest/tree/HEAD/packages/ui)) | pnpm.catalog.default | major | [`4.1.12` → `5.0.5`](https://renovatebot.com/diffs/npm/@vitest%2fui/4.1.12/5.0.5) | `5.1.0` | | [dotenv](https://redirect.github.com/motdotla/dotenv) | devDependencies | major | [`^17.4.2` → `^18.0.9`](https://renovatebot.com/diffs/npm/dotenv/17.4.2/18.0.9) | | | [dotenv](https://redirect.github.com/motdotla/dotenv) | devDependencies | major | [`17.4.2` → `18.0.9`](https://renovatebot.com/diffs/npm/dotenv/17.4.2/18.0.9) | | | [vitest](https://vitest.dev) ([source](https://redirect.github.com/vitest-dev/vitest/tree/HEAD/packages/vitest)) | devDependencies | major | [`^4.1.12` → `^5.0.5`](https://renovatebot.com/diffs/npm/vitest/4.1.12/5.0.5) | `5.1.0` | | [vitest](https://vitest.dev) ([source](https://redirect.github.com/vitest-dev/vitest/tree/HEAD/packages/vitest)) | pnpm.catalog.default | major | [`4.1.12` → `5.0.5`](https://renovatebot.com/diffs/npm/vitest/4.1.12/5.0.5) | `5.1.0` | | [vitest](https://vitest.dev) ([source](https://redirect.github.com/vitest-dev/vitest/tree/HEAD/packages/vitest)) | devDependencies | major | [`4.1.12` → `5.0.5`](https://renovatebot.com/diffs/npm/vitest/4.1.12/5.0.5) | `5.1.0` | | [vitest@>=2.1.0 <4.1.12](https://vitest.dev) ([source](https://redirect.github.com/vitest-dev/vitest/tree/HEAD/packages/vitest)) | pnpm-workspace.overrides | major | [`4.1.12` → `5.0.5`](https://renovatebot.com/diffs/npm/vitest@>=2.1.0 <4.1.12/4.1.12/5.0.5) | `5.1.0` | --- > [!WARNING] > Some dependencies could not be looked up. Check the [Dependency Dashboard](../issues/9505) for more information. --- ### Release Notes <details> <summary>vitest-dev/vitest (@​vitest/coverage-v8)</summary> ### [`v5.0.5`](https://redirect.github.com/vitest-dev/vitest/releases/tag/v5.0.5) [Compare Source](https://redirect.github.com/vitest-dev/vitest/compare/v5.0.4...v5.0.5) ##### 🐞 Bug Fixes - Keep vite default `fs.deny` with api token file - by [@​hi-ogawa](https://redirect.github.com/hi-ogawa), **Hiroshi Ogawa** and **Claude Opus 5.5** in [#​11578](https://redirect.github.com/vitest-dev/vitest/issues/11578) [<samp>(28dc1)</samp>]( |
||
|
|
47a6cd82e5 |
chore(deps): update ai sdk (#29242)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
|
|
30046969ad |
chore: version packages (alpha) (#29146)
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. ⚠️⚠️⚠️⚠️⚠️⚠️ `main` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `main`. ⚠️⚠️⚠️⚠️⚠️⚠️ # Releases ## @mastra/client-js@1.59.0-alpha.1 ### Minor Changes - Added the `aggregateTimeToFirstMeasures` observability capability. Trace and span aggregate requests for `timeToFirstToken.*` or `timeToFirstChunk.*` measures now return `501` against stores that can't compute them, and field discovery omits those measures for such stores. ([#29055](https://github.com/mastra-ai/mastra/pull/29055)) ```typescript const { capabilities } = await mastraClient.getObservabilityCapabilities(); if (capabilities.traceAggregate && capabilities.aggregateTimeToFirstMeasures) { const { rows } = await mastraClient.aggregateTraces({ timeRange: { from: '2026-08-01T00:00:00.000Z', to: '2026-08-02T00:00:00.000Z' }, groupBy: ['entityName'], measures: ['timeToFirstChunk.p95', 'timeToFirstChunk.count'], }); } ``` ### Patch Changes - Updated dependencies [[`8751dd7`]( |
||
|
|
7645fce513 |
chore(deps): update dependency @google-cloud/pubsub to v6 (#29113)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
|
|
eed9cc214e | chore: version packages | ||
|
|
6ede0f10f6 |
fix(valkey-streams): trim saved thread-stream events (#29095)
> Opened by @abhiaiyer91 Valkey thread streams currently retain saved run events until their size limit or idle TTL. Implement run-scoped trimming so persisted stream parts can be removed without deleting other runs, pinned entries, or retained terminal events. Closes #28974. Before: `await pubsub.trimTopic(topic, { runId })` leaves all entries behind. After: it removes that run's eligible entries; `producedBefore` bounds removal to saved parts and `retainTerminalEvents` preserves terminal markers. Cleanup now snapshots the current stream tail, so ongoing publishers cannot extend the scan indefinitely or have their later entries deleted by that scan. ## Proof - Real-Valkey concurrency regression: pause the cleanup scan, publish another target-run entry, resume, and assert only the in-flight entry remains. This test failed before the tail-snapshot fix (`[]` instead of `['in-flight']`) and passed after it. - Existing real-Valkey integration covers more than one scan page, other-run isolation, pinned entries, produced-time cutoff, and retained terminal events; the targeted integration and lease suite passed (7/7). - Full `@mastra/valkey-streams` tests passed (8 files, 42 tests); package typecheck, lint (0 errors/warnings), formatting, diff check, and ESM/CJS build passed. - Earlier local Life Coach run with Postgres and Valkey emitted a finish event and left 2 stream entries with the patched adapter, compared to 117 before. ## Unverified - The Life Coach stream iterator remained open after finish, so that isolated run was stopped by timeout; it is not a completed end-to-end test. No sustained-production-throughput or multi-process soak test was run. ## Risk: medium - Trimming is best-effort and run-scoped, but removal is irreversible. Real-Valkey tests cover the bounded concurrent-publish race and preservation predicates; production scheduling and very large/live streams have not been soak-tested. Patch changeset included for `@mastra/valkey-streams`. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 When a run is saved, Valkey can keep its stream entries around and use extra storage. This change removes eligible entries for that run while keeping entries that must remain. ## What changed - Added `ValkeyStreamsPubSub.trimTopic` to delete entries for a specified `runId`. - When `producedBefore` is set, the method deletes only entries with a numeric `data.producedAt` at or before the cutoff. It skips pinned entries. - When `retainTerminalEvents` is enabled, the method preserves `run-registered`, `run-completed`, `run-failed`, and `run-aborted` events, except those marked `suspended`. - The scan stops at the stream tail captured before scanning starts. Entries published during the scan are not included. - Trimming failures are logged as warnings and do not propagate. - Added integration tests for run-specific trimming, cutoff and pinned-entry behavior, terminal-event retention, and entries published during a scan. - Added a patch changeset for `@mastra/valkey-streams`. ## Validation The supplied report says six Valkey integration tests passed and the package TypeScript check, lint, and build succeeded. It also reports a local Postgres and Valkey test that left two stream entries after a finish event, down from 117. The stream iterator remained open, so a timeout stopped the isolated process; the Valkey regression suite completed normally. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
77076a1012 | chore: version - exit prerelease mode | ||
|
|
582ef17a2b |
chore: version packages (alpha) (#28888)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
|
b0e86c50e2 |
fix(redis-streams): stop subscribe() hanging forever when Redis stops responding before connecting (#28941) | ||
|
|
39edceea4f |
fix(core): open thread subscriptions with initial history when the lease read stalls or fails (#28924) | ||
|
|
ba8b955b1a |
fix(redis-streams,valkey-streams): expire idle streams by default and clear them on thread delete (#28968)
A production Factory deployment ran out of Redis memory. Every key was a
`mastra:topic:*` stream, and almost all of them (3,673 thread streams)
had no expiry. The deployment built `new RedisStreamsPubSub({ url })`
with defaults, and the default `streamIdleTtlMs` was `0`, meaning
streams never expire. Thread streams are open-ended and never get
`clearTopic()`, so they piled up forever. The Redis instance used
`volatile-lru`, which can't evict keys without a TTL, so writes just
started failing.
This PR makes `@mastra/redis-streams` and `@mastra/valkey-streams` give
every stream a sliding 7-day idle TTL by default. Writes refresh it, and
live subscribers keep receiving events after a stream expires because
the read loop recreates it. Streams are an event bus, not storage:
replay history lives in the caching layer and storage, so the backlog
only matters for grouped workers that are offline longer than the TTL.
```ts
// Before: streams never expire unless you opt in
new RedisStreamsPubSub({ url });
// After: 7-day idle TTL by default; opt out explicitly
new RedisStreamsPubSub({ url });
new RedisStreamsPubSub({ url, streamIdleTtlMs: 0 });
```
Both pubsubs now log one warning after they first connect if Redis has
`maxmemory` set with any policy other than `noeviction`. A TTL makes
streams, consumer groups and lease keys evictable under `volatile-*`,
which would silently drop events and leases (ElastiCache defaults to
`volatile-lru`). The check uses `INFO memory` because managed Redis
often blocks `CONFIG GET`.
`memory.deleteThread()` now also deletes the thread's event stream
through the registered Mastra pubsub. This runs fire-and-forget, so
neither deleting a thread nor `memory.settled()` waits on Redis during
an outage. It's skipped on older core versions. Agents configured with
their own pubsub still rely on the TTL.
The docs now require `noeviction`; the Helm guide used to say a
`volatile-*` policy was fine. They also add a runbook for reclaiming
streams created before the TTL: switch to `noeviction`, upgrade the
writers, then expire each stream relative to its last write. Tested
against real Redis/Valkey containers, including a real TTL expiry while
a subscriber is listening and a thread delete with Redis unreachable.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## ELI5
Redis and Valkey can keep old event streams forever, which uses storage.
This change removes streams after seven days without writes by default
and cleans up a thread’s stream when the thread is deleted.
## What changed
- Redis and Valkey streams now use a sliding seven-day idle TTL by
default. Writes refresh the TTL; reads do not. Set `streamIdleTtlMs: 0`
to disable expiry.
- The read loop recreates an expired stream so a live subscriber can
continue receiving events.
- After connecting, each pubsub warns once when `maxmemory` is enabled
with a policy other than `noeviction`. The check uses `INFO memory`.
- Thread deletion starts best-effort stream cleanup through the
registered Mastra pubsub without waiting for Redis. Older core versions
skip this cleanup.
- Documentation and the local Docker configuration now recommend
`noeviction`. The documentation also explains how to expire existing
streams that lack a TTL.
## Tests
Tests cover TTL behavior, delivery after expiry for live subscribers,
eviction-policy warnings, and thread-stream cleanup. Test execution
results were not provided.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
|
||
|
|
52265cba91 |
Fix run abort route answering false on a server that does not own the run (#28522) | ||
|
|
cf92428a76 |
chore(deps): update docker images (#28011)
This PR contains the following updates: | Package | Update | Change | |---|---|---| | amazon/aws-cli | minor | `2.36.44` → `2.39.1` | | clickhouse/clickhouse-server | minor | `26.6.1.1193` → `26.10.3.7` | | cr.weaviate.io/semitechnologies/weaviate | minor | `1.39.13` → `1.40.3` | | [docker.elastic.co/elasticsearch/elasticsearch](https://www.elastic.co/products/elasticsearch) ([source](https://redirect.github.com/elastic/elasticsearch)) | minor | `9.4.3` → `9.5.5` | | edoburu/pgbouncer | minor | `v1.25.2-p0` → `v1.26.1-p0` | | [ghcr.io/shopify/toxiproxy](https://redirect.github.com/Shopify/toxiproxy) | minor | `2.12.0` → `2.13.0` | | inngest/inngest | minor | `v1.34.0` → `v1.49.0` | | mcr.microsoft.com/azure-storage/azurite | minor | `3.35.0` → `3.37.0` | | [mongo](https://hub.docker.com/_/mongo) ([source](https://redirect.github.com/docker-library/mongo)) | patch | `7.0.37` → `7.0.45` | | pgvector/pgvector | patch | `0.8.4-pg16` → `0.8.7-pg16` | --- > [!WARNING] > Some dependencies could not be looked up. Check the [Dependency Dashboard](../issues/9505) for more information. --- ### Release Notes <details> <summary>elastic/elasticsearch (docker.elastic.co/elasticsearch/elasticsearch)</summary> ### [`v9.5.5`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.5.5): Elasticsearch 9.5.5 [Compare Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.5.4...v9.5.5) Downloads: <https://elastic.co/downloads/elasticsearch> Release notes: <https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.5.5-release-notes> ### [`v9.5.4`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.5.4): Elasticsearch 9.5.4 [Compare Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.5.3...v9.5.4) Downloads: <https://elastic.co/downloads/elasticsearch> Release notes: <https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.5.4-release-notes> ### [`v9.5.3`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.5.3): Elasticsearch 9.5.3 [Compare Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.5.2...v9.5.3) Downloads: <https://elastic.co/downloads/elasticsearch> Release notes: <https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.5.3-release-notes> ### [`v9.5.2`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.5.2): Elasticsearch 9.5.2 [Compare Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.5.1...v9.5.2) Downloads: <https://elastic.co/downloads/elasticsearch> Release notes: <https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.5.2-release-notes> ### [`v9.5.1`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.5.1): Elasticsearch 9.5.1 [Compare Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.5.0...v9.5.1) Downloads: <https://elastic.co/downloads/elasticsearch> Release notes: <https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.5.1-release-notes> ### [`v9.5.0`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.5.0): Elasticsearch 9.5.0 [Compare Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.4.8...v9.5.0) Downloads: <https://elastic.co/downloads/elasticsearch> Release notes: <https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.5.0-release-notes> ### [`v9.4.8`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.4.8): Elasticsearch 9.4.8 [Compare Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.4.7...v9.4.8) Downloads: <https://elastic.co/downloads/elasticsearch> Release notes: <https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.4.8-release-notes> ### [`v9.4.7`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.4.7): Elasticsearch 9.4.7 [Compare Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.4.6...v9.4.7) Downloads: <https://elastic.co/downloads/elasticsearch> Release notes: <https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.4.7-release-notes> ### [`v9.4.6`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.4.6): Elasticsearch 9.4.6 [Compare Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.4.5...v9.4.6) Downloads: <https://elastic.co/downloads/elasticsearch> Release notes: <https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.4.6-release-notes> ### [`v9.4.5`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.4.5): Elasticsearch 9.4.5 [Compare Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.4.4...v9.4.5) Downloads: <https://elastic.co/downloads/elasticsearch> Release notes: <https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.4.5-release-notes> ### [`v9.4.4`](https://redirect.github.com/elastic/elasticsearch/releases/tag/v9.4.4): Elasticsearch 9.4.4 [Compare Source](https://redirect.github.com/elastic/elasticsearch/compare/v9.4.3...v9.4.4) Downloads: <https://elastic.co/downloads/elasticsearch> Release notes: <https://www.elastic.co/docs/release-notes/elasticsearch#elasticsearch-9.4.4-release-notes> </details> <details> <summary>Shopify/toxiproxy (ghcr.io/shopify/toxiproxy)</summary> ### [`v2.13.0`](https://redirect.github.com/Shopify/toxiproxy/blob/HEAD/CHANGELOG.md#2130---2026-10-14) [Compare Source](https://redirect.github.com/Shopify/toxiproxy/compare/v2.12.0...v2.13.0) - Reject slicer toxics with `average_size` <= 0 or `size_variation` outside `[0, average_size]`, and stop the slicer from recursing forever on degenerate sizes ([#​774](https://redirect.github.com/Shopify/toxiproxy/issues/774), [@​iyaz-shaikh](https://redirect.github.com/iyaz-shaikh)) - Add optional bearer token auth for the HTTP API via `TOXIPROXY_AUTH_TOKEN`, supported by the Go client and `toxiproxy-cli`, and warn when the API is unauthenticated on a non-loopback address ([#​774](https://redirect.github.com/Shopify/toxiproxy/issues/774), [@​iyaz-shaikh](https://redirect.github.com/iyaz-shaikh)) - Fix removing a toxic hanging forever, and freezing the whole HTTP API, when the downstream toxic or connection is already gone ([#​760](https://redirect.github.com/Shopify/toxiproxy/issues/760), [@​matheusgb](https://redirect.github.com/matheusgb)) - Fix the bandwidth toxic hanging or panicking when `rate` <= 0 ([#​764](https://redirect.github.com/Shopify/toxiproxy/issues/764), [@​jaideeppyne](https://redirect.github.com/jaideeppyne)) - Update go version to 1.27 ([#​777](https://redirect.github.com/Shopify/toxiproxy/issues/777), [@​The-Black-Fox](https://redirect.github.com/The-Black-Fox)) - Update various go packages </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - "before 6am every weekday" - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://redirect.github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate CLI](https://redirect.github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4xNzUuMSIsInVwZGF0ZWRJblZlciI6IjQ0LjE5Mi4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJhdXRvbWF0aW9uIl19--> --------- Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
4f12792ed1 |
chore: version packages (alpha) (#28722)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
|
1607791e91 |
fix(core): trim finished runs of memoryless agents from thread-stream topics (#28746)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
f2ad3844f3 |
chore(deps): update opentelemetry (#28581)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
6deb7c0a1e |
chore(security): Vanta 2026-10-28 - qs 6.16.1, @modelcontextprotocol/node 2.1.1 floors (#28716)
Refs mastra-ai/platform#2931 Vanta remediation pass for 2026-10-28. It follows #28325, which has merged. ## 2026-10-28 update Today's live Vanta export added 3 advisory IDs on `mastra-ai/mastra` (GitHub/Dependabot). This PR fixes the two that have a published fix and leaves deepmerge, which has none. | Package | From → To | Finding | Where | How | |---|---|---|---|---| | qs | 6.16.0 → 6.16.1 | CVE-2026-110816 / GHSA-6pq8-jhm2-2vcw (Medium ×4) | root `pnpm-lock.yaml`, 9 `e2e-tests/*` lockfiles, `scripts/gh-issue-triage/package-lock.json` | Raised the existing floors `qs@>=6.7.0 <6.16.0: 6.16.0` → `<6.16.1: 6.16.1` and `'qs@<6.7.0': 6.16.0` → `6.16.1`. Raised the npm override in gh-issue-triage. | | @modelcontextprotocol/node | 2.0.0 → 2.1.1 | GHSA-f897-62hf-6gxq (High ×2; HTTP transports had no body-size or batch-length limit) | root `pnpm-lock.yaml` (published `@mastra/mcp` 1.18.0, used as `mcp-legacy`), `scripts/gh-issue-triage/package-lock.json` | New pnpm override `'@modelcontextprotocol/node@>=2.0.0 <2.1.0': 2.1.1` and a new npm override in gh-issue-triage. I used 2.1.1 rather than 2.1.2 because 2.1.2 needs a `@modelcontextprotocol/server` peer of `^2.4.0`, while `@mastra/mcp` 1.18 resolves server 2.3.1. `packages/mcp` itself already uses node 2.1.2. | - Both versions are older than the 1-day `minimumReleaseAge`: qs 6.16.1 was published 10-14 and node 2.1.1 on 10-02. I did not use qs 6.16.2 (10-27), which is too new. - No floors or overrides were removed; existing ones were only raised. - `pnpm install --frozen-lockfile --lockfile-only` passes at the root. Each nested e2e workspace was regenerated with `pnpm install --lockfile-only`. - Added the empty changeset `.changeset/vanta-2026-10-28-security-floors.md`. No published package changes. - After this change, osv-scanner over all 33 lockfiles reports only the documented leftovers: - zod 3.25.76 (zod v3 compat line on purpose) - ai 4.3.19 / `@ai-sdk/provider-utils` 2.2.8 (vendored v4 line) - `@azure/msal-node` 5.6.0 / 6.0.1 (fix 7.1.0 needs majors in identity and teams) - `@opentelemetry/core` 1.30.1 (needs pubsub 6) - extract-zip 2.0.1 (no patched release) - deepmerge 4.3.1 (CVE-2026-93753 / GHSA-gcqx-w48m-w2h4; no patched release, 4.3.1 is the latest) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 This PR updates dependency rules and package versions to use newer releases, including security fixes for `qs` and `@modelcontextprotocol/node`. It also updates AI SDK dependencies and does not include published package changes. ## Summary - Raise the `qs` override target to `6.16.1` in the root and e2e workspaces and in `scripts/gh-issue-triage`. - Add an `@modelcontextprotocol/node` override at `2.1.1` in the root workspace and `scripts/gh-issue-triage`. - Update AI SDK dependency versions across package manifests and add a changeset describing the provider-utils update. - Add an empty changeset for the security-only dependency updates. <!-- end of auto-generated comment: release notes by coderabbit.ai --> ## Update 2026-10-28 (afternoon) - Merged origin/main (clean). - Bumped the AI SDK v5 compat line from @ai-sdk/provider-utils 3.0.41 to 3.0.42 (patch, published 2026-10-21; it blocks `.internal` hostnames in download URL validation and encodes provider-returned IDs in request paths). This addresses the Aikido /mastra Highs AIKIDO-2026-10990 and AIKIDO-2026-10981: - ai 5.0.277 → 5.0.282, @ai-sdk/gateway 2.0.168 → 2.0.173 (vendored ai_v5) - anthropic 2.0.109 → 2.0.110, google 2.0.103 → 2.0.106, groq 2.0.58 → 2.0.59, openai 2.0.135 → 2.0.137, openai-compatible 1.0.58 → 1.0.59, xai 2.0.99 → 2.0.100 - Nested lockfiles updated: memory/integration-tests and playground/e2e/kitchen-sink. - No new overrides. Changeset: `vanta-2026-10-28-ai-sdk-v5-provider-utils.md`. |
||
|
|
6a191d46dc | chore: version - exit prerelease mode | ||
|
|
e6764d26da |
chore: version packages (alpha) (#28625)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
|
a7cc444233 |
fix(core): expose recoverable run outcomes to thread subscribers (#28621)
Thread subscribers now receive an ordered `thread-run-status` frame for
success, failure, abort, or suspension without closing the subscription.
Initial history also reports `activeRunId`, so an idle snapshot no
longer has to stand in for a run outcome. Retained outcomes reach late
subscribers even after saved output has been trimmed, with execution
authority preserved to reject superseded attempts.
For example, a subscription opened after a run finishes now receives
`thread-history` followed by `thread-run-status` with `{ status:
"success" }`, rather than history alone. Recovery is bounded by PubSub
retention; upgrade both `@mastra/core` and `@mastra/redis-streams` for
recovery after trimming. Client callback types, documentation, and
package changesets are included.
Verified with core build and typecheck, 220 core tests across the
thread-stream suites and four execution engines, 91 client tests, 55
Redis integration tests, and three HTTP conformance cases covering
plain, durable, and evented agents. Real Redis regressions cover restart
after trimming and superseded completion/abort after registration
eviction. The live outcome scenario also passed 20 repeated four-engine
runs. Typed lint, formatting, MDX prose checks, and final adversarial
review passed.
Fixes COR-1500.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## ELI5
A thread subscription can now tell clients when a run succeeds, fails,
stops, or pauses without ending the subscription. This helps clients
understand what happened after reconnecting, even when the run’s saved
output is no longer available.
## Summary
- Added opt-in `withRunStatus` support to core, server, and client
thread subscriptions. Enabled subscriptions can receive ordered
`thread-run-status` chunks for `success`, `failed`, `aborted`, or
`suspended` outcomes.
- Added `activeRunId` to initial thread history so clients can identify
the run that held the thread lease during history loading.
- Added terminal-event retention during output trimming. Recovery
depends on PubSub retention. Redis Streams users must upgrade
`@mastra/core` and `@mastra/redis-streams` together.
- Updated React `useChat` to request run statuses and apply outcomes to
the matching run. Added API types, documentation, and changesets.
## Validation
The PR objectives report passing core build and typecheck, core and
client tests, Redis integration tests, HTTP conformance cases, lint and
formatting checks, and documentation checks. They also report repeated
run-outcome checks and adversarial review. These results are
author-reported.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
|
||
|
|
e718b91a64 |
fix(core): stop observe offsets past the log end replaying the whole run (#28348)
> Opened by @roaminro
Refs #28324 (partial fix; the issue stays open, see Open question)
## What changed
On transports that keep old messages, such as Redis Streams,
`observe(runId, { offset })` replayed the **whole run** when the offset
was at or past the end of the run's log. A new subscriber receives the
stream's retained frame 0. `CachingPubSub`'s log-restart check compared
the counter with the subscriber's cursor, and with a past-end offset the
counter is always below the cursor. So frame 0 looked like a log restart
and everything replayed. This affected every Redis setup, durable agents
included.
`subscribeFromOffset` in `packages/core/src/events/caching-pubsub.ts`
now decides restarts by frame 0's identity:
- On attach it records the id of the log's frame 0. A frame 0 arriving
below the cursor is a restart only if the log's frame 0 id has changed.
A retained copy of the same frame is acknowledged and dropped.
- On a restart it delivers the new segment from the cache log, not from
the frame that signalled the restart, because that frame may be a
delayed copy of the old frame 0.
- If an id can't be read, it falls back to the counter. If the read
failed at attach, the counter is compared with the log's end at that
point, so an intact log isn't mistaken for a restart.
The cursor stays at the offset, so the documented contract holds: an
offset beyond the retained index replays nothing and skips live events
below it. If the retained log expired and the run restarts numbering at
0, the observer resyncs to the new segment.
**Known limitation (not fixed):** `createEventedAgent` on a
cross-process pubsub with the default in-memory server cache, which is
the case in the issue title. Its step executor publishes unnumbered
frames there, because each process would count them separately. These
frames can't be skipped by position:
- `CachingPubSub` can't see where a transport's replay starts.
- Redis Streams trims the start of its backlog with `MAXLEN`.
- Other transports replay nothing.
That configuration now logs a one-time warning. It names both possible
causes, a failed cache append or no shared server cache, and recommends
a shared cache such as `RedisServerCache`. Both observe docs pages
(`docs/harness/durable-agents.mdx`,
`reference/agents/durable-agent.mdx`) document the requirement.
## Tests
- `packages/core/src/events/caching-pubsub.test.ts`:
- A past-end offset delivers nothing, then delivers live frames once the
log reaches the offset.
- An offset inside a short log resumes without resyncing.
- A short log past the offset expires and a new segment resyncs.
- A stale copy of the old frame 0 doesn't stand in for the new segment.
- A failed frame 0 id read falls back to the counter.
- A restart is detected by id when the counter read fails.
- A failed id read at attach doesn't replay an intact log.
- A log that is empty at attach still delivers the next run.
- Unnumbered frames produce exactly one warning.
- `pubsub/redis-streams/src/observe-offset.test.ts`: new file against
real Redis, with the model served by aimock.
- It covers four setups: durable or evented engine, with the default or
`RedisServerCache` server cache.
- Each setup runs three cases: resuming a live run, a past-end offset on
a finished run, and a mid-run offset on a finished run.
- The evented/default setup checks for the warning and that the
observation still ends on `finish`.
| Check | Result |
|---|---|
| `packages/core` `src/events` | ✅ 323/323 |
| core `tsc --noEmit` | ✅ clean |
| `pubsub/redis-streams` observe-offset, real Redis, rebuilt core | ✅
12/12 |
| oxlint, prettier, docs `format:mdx:check` + remark | ✅ clean |
## Proof
- I checked each new unit test against the previous commit or the code
before its fix, and it failed there. The empty-log test is the
exception: it passes before its fix by design. It guards against
limiting the counter check whenever no id was captured, and it fails if
that change is applied.
- With `caching-pubsub.ts` at base and core rebuilt, 6 of the 12 Redis
tests fail. These are the past-end case in the three numbered setups,
where the whole run replays from `start`, and all three evented/default
cases, where no warning is logged.
- NikAiyer independently reproduced the base failures and the fixed-head
results.
## Unverified
- Redis Cluster deployments, and transports other than Redis Streams and
the in-process pubsub.
- `durable-agent-tool-writer` and `durable-agent-client-tool-pending`
already fail on `main`, with or without this change.
## Risk: low
The change is limited to the restart check and cursor handling in
`CachingPubSub.subscribeFromOffset`, plus a warning. Offset 0 and the
default observe path (no offset) don't read the counter and behave as
before.
## Open question
#28324 stays open for evented agents on a cross-process pubsub without a
shared server cache. Should evented step frames be numbered without a
shared cache (or a shared cache be required), or is the documented
limitation enough?
**Merge note:** the first commit (`
|
||
|
|
5381b414b3 |
test(redis-streams): fix the Lambda-death signal-drain test that times out on main (#28533)
> Opened by @abhiaiyer91
## Summary
`drains signal-enqueued from losers that die immediately (Vercel
Lambda-death)` in
`pubsub/redis-streams/src/agent-thread-cross-process.test.ts` times out
on main. It never actually exercised the scenario in its name. This is a
test-only fix. Product code is unchanged, so there's no changeset.
Two fixture problems caused it:
1. **Each worker had its own agent id.** The fixture used
`${WORKER_ID}-agent` (`test-fixtures/agent-thread-worker.entry.ts:59` on
main). The losers therefore treated the winner's run as another agent's
run. The cross-agent guard
(`packages/core/src/agent/thread-stream-runtime.ts:6852-6854`,
`:6900-6905`) correctly returned `blocked / thread-blocked` and
published nothing, so the winner had nothing to drain. Lambdas are
replicas of one deployed agent, so they should share an id.
2. **Timing.** The winner's run was a fixed 1500ms, but a worker takes
about 2.3s to boot through tsx. The losers always sent after the lease
was already free, never "while the run is in flight".
Changes:
- The fixture accepts `AGENT_ID` (default is still `${WORKER_ID}-agent`,
so other tests are unaffected). The test sets one shared id.
- The fixture accepts `HOLD_SIG_IDS` plus a `finish-run` command. The
winner's sig-1 run stays open until both losers have exited, then the
test ends it explicitly.
- The test asserts the winner owns sig-1, and that both losers lost the
lease and started no runs. It then waits for sig-1/2/3 to finish on the
winner. I removed the fixed 300ms settle sleep, so the test waits on
product events only.
## Proof
- On main: fails with `waitFor timed out after 20000ms` waiting for 3
`run-finished`. Only sig-1 finishes on the winner.
- With the fix, against core built from main: **5/5 consecutive
passes**.
- Reverting only the shared `AGENT_ID` makes it fail again with the same
20s `waitFor` timeout.
- `agent-thread-cross-process.test.ts`: 10/10 pass. On main this file
failed because of this test.
- Full redis-streams suite, run per file:
- Fail both on main and on this branch: `cluster`,
`durable-worker-takeover`, `observe-cross-process`. On main this run
only checked `observe-cross-process`; the other two are already known to
fail on main.
- `agent-thread-publish-lag`: 1 test failed. Known to fail on main; I
didn't rerun it on main myself.
- `auth-e2e` and `fanout-sharing` failed once in the long serial run.
Both passed 2/2 when rerun alone on this branch, and both pass on main.
They don't load the changed fixture.
- oxlint: 0 warnings, 0 errors. prettier: no changes. `tsc --noEmit` in
redis-streams gives the same 14 errors on main and on this branch (a
`RedisStreamsPubSub` vs `PubSub` mismatch, unrelated).
## Unverified
- **A sender that exits without closing its pubsub can still drop the
forwarded signal (product gap, out of scope).** With the shared id, the
losers settle `deliver` (I logged `settled.action`). That means they
take the `ifActive: deliver` branch at `thread-stream-runtime.ts:7027`,
not the lease-loser hand-off at `:2589`, which settles `remote`. Both
branches publish `signal-enqueued` with the fire-and-forget `#publish`
(`:2690-2692`) and resolve `accepted` before the publish reaches Redis.
So my earlier claim about the pattern holds, and it applies to the path
this test actually runs. The test passes only because the fixture awaits
`pubsub.close()` before `process.exit`. `#publish` reaches
`RedisStreamsPubSub.publish` synchronously, which adds the publish to
`#pendingPublishes` (`pubsub/redis-streams/src/index.ts:431-434`) before
`accepted` resolves, and `close()` waits for those publishes through
`flush()` (`:1133`, `:894-899`). It isn't a case of the publish just
being fast locally. I checked this by temporarily skipping `close()` in
`send-and-exit`: **3/3 runs timed out** and sig-2/sig-3 never reached
the winner. So a Lambda that settles and exits (or is frozen or killed)
without closing or flushing its pubsub loses the signal. A fix would
await the publish before resolving `accepted` on the remote-owner
branches. That needs its own core PR. I didn't check whether the server
or serverless deployers close or flush the pubsub on shutdown.
- **A non-durable owner that dies mid-run loses queued signals (separate
product gap, out of scope).** In an earlier run of this test the losers
won the free lease. When the lease owner died, sig-2/sig-3 stayed
stranded after the lease expired, and a later signal woke a fresh run
without replaying them. Signals are only admitted by the process that
owns the runId (`thread-stream-runtime.ts:1444`). Dead-owner recovery
only covers durable runs (`:5574-5584`, `:5824+`). This needs its own
issue and design (orphaned-signal recovery, with duplicate-delivery
risk).
- Not checked on CI runners. The fix doesn't depend on boot timing
anymore, so I expect it to behave the same there.
## Risk
low. Only test and fixture files changed, and the new fixture options
are opt-in.
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
|
||
|
|
7c20f008c8 |
fix(core): abort evented workflow steps running on other replicas on cancel (#28489)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9 Closes #28444 ## Summary With evented workflows and workers on several replicas, `workflow.cancel` is handled by only one processor in the consumer group. That processor aborted only the AbortControllers in its own process. If the step was running on another replica, its `abortSignal` never fired and the step kept running, even though the run itself settled as `canceled`. This PR broadcasts cancellation on one shared control topic, `workflow.control`: - **One subscription per processor.** A processor opens a single ungrouped subscription to `workflow.control` when it runs its first step and keeps it for its lifetime. Transports share one reader/subscription per topic per process, so the cost does not grow with runs or steps (no per-run topics, consumer groups or Pub/Sub subscriptions to clean up). It reads from `latest`; a failed subscribe is logged and retried on the next step. - **Cancel publishes the abort before finalizing.** `processWorkflowCancel` aborts its local controllers, publishes an abort for the run, and only then calls `endWorkflow`. If the publish fails, the error propagates and the event is redelivered before anything is finalized. - **Receivers only act on runs they are executing.** Each processor counts in-flight steps per run (a nested step counts against its ancestors too). An abort for a run with no in-flight step here is ignored. Otherwise the run's own controller is marked cancel-handled (the step unwinds without a second `workflow.cancel`), and nested runs are aborted like the local `cancelRunAndChildren`, so each settles canceled. Parent/child links come from the step event's `parentWorkflow` chain, so this works even if the processor never saw the child's `workflow.start`. - **Cancels issued before a processor could hear them.** The control subscription starts at `latest`, so an abort published while it is still being opened is missed. When `#dispatch` takes a `workflow.step.run`, it records, before its state load, whether the subscription was ready and how many aborts this processor has ignored for runs with no step retained. The step re-reads the persisted status of its run and its ancestors when the subscription wasn't ready at dispatch, or when that count changed before the step was retained, and aborts if any of them is `canceled` (`EventedRun.cancel` persists that before publishing). Other steps skip the read. The count can't evict anything; when it changes, the overlapping steps re-check. - **A canceled step never starts its body.** If the pre-step check finds the step's own run canceled, the step returns without running. If it finds only an ancestor canceled, the body is skipped with a `canceled` result, and the existing post-step branch publishes this run's `workflow.cancel`, so the nested run settles canceled. - **No state left on replicas that never finalize the run.** When a run's last in-flight step is released on a processor that didn't start it, the controller and parent links recorded by the step path are dropped. - **Double delivery does nothing.** Aborting is idempotent; a redelivered or self-echoed abort finds the controllers already aborted. Tests assert exactly one `workflow.end` per run. Assumption: a shared broadcast topic over per-run topics. Cancels are rare and tiny, so every processor seeing every abort is cheaper than per-run subscriptions, which leaked topics/subscriptions on Google Cloud Pub/Sub and churned consumer groups on Redis Streams. ## Proof - `packages/core/src/workflows/evented/remote-cancel.test.ts`, suite "workers on both replicas": two Mastra instances, both running workers, over EventEmitter and UnixSocket, cancels from the owner and the other replica, flat and nested. Asserts the step's `abortSignal` fires, the run settles `canceled`, the only control subscription is `workflow.control` and there is at most one per processor across all runs, no in-flight step refs remain; with grouped delivery also exactly one `workflow.end` and at least one cancel handled on a different replica than the step. A pinned-topology test (steps on one replica, start/end on the other) asserts the step-only replica keeps no controllers or parent links. Five pinned-topology "skips a step …" tests cover the fallback: (a) the step replica's `workflow.control` subscribe is delayed and the run is canceled during the delay; (b) the abort lands after the step's dispatch load but before it is retained (also asserts a plain step on a ready subscription does no re-check); (c) same as (b) with 2000 aborts for other runs published in that window; (d) a step dispatched while the subscription is pending resumes after other steps have run on the ready subscription; (e) nested: the parent is canceled while the child's step is held after its dispatch load. Each asserts the step body never ran, the run settles `canceled` and exactly one `workflow.end` is published per run; (e) also asserts the child snapshot is `canceled`. 26/26 pass. Proofs: **with the abort publish disabled**, the 4 EventEmitter both-replica cases fail on the step-abort assertion. **With the pre-body skip disabled**, (a)–(d) fail. **On the |
||
|
|
9ffb79b17e |
fix(core): stop closeOnSuspend durable streams closing before their chunks arrive on Redis Streams (#28504) | ||
|
|
2c646b53d0 |
chore: version packages (alpha) (#28399)
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. ⚠️⚠️⚠️⚠️⚠️⚠️ `main` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `main`. ⚠️⚠️⚠️⚠️⚠️⚠️ # Releases > The changelog information of each package has been omitted from this message, as the content exceeds the size limit. ## @mastra/factory@0.25.0-alpha.5 ## @mastra/langsmith@1.4.0-alpha.2 ## @mastra/connect@1.2.0-alpha.2 ## @mastra/core@1.80.0-alpha.4 ## @mastra/playground-ui@66.0.0-alpha.4 ## @mastra/acp@0.4.4-alpha.0 ## @mastra/claude@0.3.4-alpha.0 ## @mastra/cursor@0.3.4-alpha.0 ## @mastra/openai@1.1.4-alpha.0 ## @mastra/auth-auth0@1.2.6-alpha.0 ## @mastra/auth-better-auth@1.1.9-alpha.0 ## @mastra/auth-clerk@1.3.2-alpha.0 ## @mastra/auth-cloud@1.2.8-alpha.0 ## @mastra/auth-firebase@1.1.5-alpha.0 ## @mastra/auth-google@0.1.7-alpha.0 ## @mastra/auth-neon@0.3.5-alpha.0 ## @mastra/auth-okta@0.2.6-alpha.0 ## @mastra/auth-studio@1.3.11-alpha.1 ## @mastra/auth-supabase@1.1.6-alpha.0 ## @mastra/auth-workos@1.6.8-alpha.0 ## @mastra/agent-browser@0.5.5-alpha.0 ## @mastra/browser-viewer@0.2.6-alpha.0 ## @mastra/browser-firecrawl@0.2.5-alpha.1 ## @mastra/stagehand@0.3.7-alpha.1 ## @mastra/discord@1.3.2-alpha.0 ## @mastra/slack@1.8.0-alpha.1 ## @mastra/teams@0.1.3-alpha.0 ## @mastra/telegram@0.2.4-alpha.0 ## @mastra/client-js@1.57.0-alpha.4 ## @mastra/react@1.13.0-alpha.4 ## @mastra/isolated-vm@0.1.4-alpha.0 ## @mastra/quickjs@0.1.5-alpha.0 ## @mastra/deployer-cloud@1.80.0-alpha.4 ## @mastra/deployer-cloudflare@1.2.39-alpha.1 ## @mastra/deployer-netlify@1.2.39-alpha.1 ## @mastra/deployer-sandbox@0.3.24-alpha.1 ## @mastra/deployer-vercel@1.2.39-alpha.1 ## @mastra/voyageai@0.4.3-alpha.0 ## @mastra/longmemeval@1.1.39-alpha.4 ## @mastra/livekit@0.5.2-alpha.0 ## @mastra/opencode@0.1.39-alpha.4 ## @mastra/parallel@0.1.4-alpha.0 ## create-factory@0.3.1-alpha.4 ## @mastra/code-sdk@1.15.1-alpha.5 ## mastracode@0.48.1-alpha.4 ## @mastra/arize@1.3.27-alpha.2 ## @mastra/arthur@0.4.27-alpha.2 ## @mastra/braintrust@1.3.24-alpha.2 ## @mastra/datadog@1.4.19-alpha.2 ## @mastra/deepeval@0.1.21-alpha.2 ## @mastra/laminar@1.3.29-alpha.2 ## @mastra/langfuse@1.6.3-alpha.2 ## @mastra/observability@1.21.1-alpha.2 ## @mastra/otel-bridge@1.5.19-alpha.2 ## @mastra/otel-exporter@1.4.10-alpha.2 ## @mastra/posthog@1.4.4-alpha.2 ## @mastra/sentry@1.2.29-alpha.2 ## @mastra/agent-builder@1.1.31-alpha.2 ## @mastra/auth@1.1.5-alpha.0 ## mastra@1.36.0-alpha.4 ## @mastra/codemod@1.1.8-alpha.0 ## create-mastra@1.36.0-alpha.4 ## @mastra/deployer@1.80.0-alpha.4 ## @mastra/editor@0.16.5-alpha.2 ## @mastra/evals@1.10.9-alpha.0 ## @mastra/fastembed@1.3.5-alpha.2 ## @mastra/loggers@1.3.10-alpha.1 ## @mastra/mcp@2.6.0-alpha.1 ## @mastra/mcp-docs-server@1.3.10-alpha.7 ## @mastra/mcp-registry-registry@1.1.7-alpha.0 ## @mastra/memory@1.39.1-alpha.2 ## @mastra/rag@2.7.1-alpha.0 ## @mastra/schema-compat@1.3.17-alpha.0 ## @mastra/server@1.80.0-alpha.4 ## @mastra/google-cloud-pubsub@1.1.5-alpha.1 ## @mastra/redis-streams@0.6.5-alpha.1 ## @mastra/valkey-streams@0.5.7-alpha.1 ## @mastra/elysia@0.1.20-alpha.4 ## @mastra/express@1.5.24-alpha.4 ## @mastra/fastify@1.5.24-alpha.4 ## @mastra/hono@1.7.22-alpha.4 ## @mastra/koa@1.7.24-alpha.4 ## @mastra/nestjs@0.2.39-alpha.4 ## @mastra/next@0.2.38-alpha.4 ## @mastra/server-adapters-test-suite@0.1.4-alpha.2 ## @mastra/tanstack-start@0.2.38-alpha.4 ## @mastra/astra@1.1.3-alpha.0 ## @mastra/azure-ai-search@0.1.2-alpha.1 ## @mastra/chroma@1.1.5-alpha.0 ## @mastra/clickhouse@1.27.0-alpha.1 ## @mastra/cloudflare@1.7.6-alpha.0 ## @mastra/cloudflare-d1@1.4.5-alpha.1 ## @mastra/convex@1.7.6-alpha.0 ## @mastra/couchbase@1.1.4-alpha.1 ## @mastra/dsql@1.5.5-alpha.0 ## @mastra/duckdb@1.16.0-alpha.1 ## @mastra/dynamodb@1.4.5-alpha.1 ## @mastra/elasticsearch@1.4.6-alpha.1 ## @mastra/lance@1.5.2-alpha.1 ## @mastra/libsql@1.27.1-alpha.0 ## @mastra/mongodb@1.24.1-alpha.1 ## @mastra/mssql@1.10.6-alpha.0 ## @mastra/mysql@0.14.1-alpha.0 ## @mastra/opensearch@1.1.4-alpha.1 ## @mastra/oracledb@0.5.5-alpha.0 ## @mastra/pg@1.33.0-alpha.2 ## @mastra/pinecone@1.1.3-alpha.0 ## @mastra/qdrant@1.1.6-alpha.0 ## @mastra/redis@1.4.12-alpha.1 ## @mastra/s3vectors@1.1.5-alpha.0 ## @mastra/spanner@1.10.1-alpha.1 ## @mastra/turbopuffer@1.2.3-alpha.0 ## @mastra/turso@0.1.18-alpha.0 ## @mastra/upstash@1.5.5-alpha.1 ## @mastra/valkey@0.2.8-alpha.1 ## @mastra/vectorize@1.1.4-alpha.0 ## @mastra/weaviate@0.1.4-alpha.0 ## @mastra/voice-aws-nova-sonic@0.2.6-alpha.0 ## @mastra/voice-azure@0.12.3-alpha.0 ## @mastra/voice-cloudflare@0.13.4-alpha.0 ## @mastra/voice-deepgram@0.13.3-alpha.0 ## @mastra/voice-elevenlabs@0.13.3-alpha.0 ## @mastra/voice-gladia@0.13.3-alpha.0 ## @mastra/voice-google@0.14.4-alpha.0 ## @mastra/voice-google-gemini-live@0.14.18-alpha.0 ## @mastra/voice-inworld@0.4.6-alpha.0 ## @mastra/voice-mistral@0.1.3-alpha.0 ## @mastra/voice-modelslab@0.2.3-alpha.0 ## @mastra/voice-murf@0.13.3-alpha.0 ## @mastra/voice-openai@0.13.4-alpha.0 ## @mastra/voice-openai-realtime@0.14.8-alpha.0 ## @mastra/voice-playai@0.13.3-alpha.0 ## @mastra/voice-sarvam@1.1.4-alpha.0 ## @mastra/voice-speechify@0.14.3-alpha.0 ## @mastra/voice-xai-realtime@0.2.17-alpha.0 ## @mastra/inngest@1.10.8-alpha.3 ## @mastra/temporal@0.4.18-alpha.4 ## @mastra/agentcore@0.5.3-alpha.0 ## @mastra/agentfs@0.2.3-alpha.0 ## @mastra/apple-container@0.5.2-alpha.0 ## @mastra/archil@0.3.2-alpha.0 ## @mastra/azure@0.3.3-alpha.1 ## @mastra/blaxel@0.10.2-alpha.0 ## @mastra/cloudflare-sandbox@0.5.4-alpha.0 ## @mastra/daytona@0.11.4-alpha.0 ## @mastra/docker@0.12.1-alpha.0 ## @mastra/e2b@0.15.1-alpha.1 ## @mastra/e2b-desktop@0.1.10-alpha.1 ## @mastra/files-sdk@0.3.3-alpha.0 ## @mastra/gcs@0.3.5-alpha.1 ## @mastra/google-drive@0.2.4-alpha.0 ## @mastra/mesa@0.3.3-alpha.0 ## @mastra/modal@0.7.3-alpha.0 ## @mastra/platform-workspace@1.10.0-alpha.1 ## @mastra/railway@0.10.2-alpha.1 ## @mastra/s3@0.6.5-alpha.1 ## @mastra/sprites@0.1.2-alpha.0 ## @mastra/vercel@1.6.2-alpha.0 ## @internal/playground@1.36.0-alpha.4 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
|
f51a29e82b |
fix(core): settle evented workflow runs cancelled from another instance (#28434)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9 Closes #28427 ## Summary On the evented workflow engine, cancelling a run from a different Mastra instance than the one running it (for example, the server cancel route landing on a non-owner replica) marked the run `canceled` in storage. The owner's `run.start()` promise never resolved, though. The same hang happened for runs started with `startAsync()` and then cancelled. **Cause:** `EventedRun.cancel()` wrote `canceled` to storage and then called `abortController.abort()`. It relied on an abort listener to publish `workflow.cancel`, but only `start()` and `resume()` register that listener. A run handle created any other way (another instance, or `startAsync()`) published nothing. The owner's event processor then dropped the run's later events through the canceled-status gate. As a result, `workflow.end` was never emitted and `start()` stayed pending. **Fix:** - `EventedRun.cancel()` now publishes `workflow.cancel` directly through a `publishCancel()` helper. The abort listener uses the same helper, and it publishes at most once per run handle. - `WorkflowEventProcessor` tracks the controllers it has already cancelled (`cancelHandledControllers`, a WeakSet). A step that unwinds on one of those controllers no longer publishes a second `workflow.cancel`, so a cancelled run emits exactly one `workflow.end`. This also removes a duplicate `workflow.cancel`/`workflow.end` pair that already happened on main whenever the owner cancelled its own run. ## Proof New tests: - `packages/core/src/workflows/evented/remote-cancel.test.ts` (12 tests): two Mastra instances sharing storage cancel across instances over `EventEmitterPubSub` and `UnixSocketPubSub`. The tests cover dountil, foreach, sequential, sleep and parallel shapes. The owner's `start()` settles as `canceled` and the step's abortSignal fires. Two more tests cancel after `startAsync()` and after an owner-local cancel, and each asserts exactly one `workflow.cancel` and one `workflow.end`. - `packages/server/src/server/handlers/workflows.test.ts`: `CANCEL_WORKFLOW_RUN_ROUTE` is called on a non-owner instance and the owner's run settles as `canceled`. - `pubsub/redis-streams/src/evented-remote-cancel.test.ts` (4 tests): cross-instance cancel over real Redis Streams, with each replica on its own connection. Shapes are sequential and dountil, with workers on the owner only and on both replicas. Results with the fix reverted: - Full revert: all 12 core tests fail, the server test times out (`expected 'timed out' to be 'canceled'`), and all 4 Redis tests time out. - Reverting only `workflow.ts`: 11 of the 12 core tests fail. - Reverting only the processor guard: the 2 single-event tests fail (they get 2 events instead of 1). With the fix, everything passes: - `pnpm build:core` - `tsc --noEmit` on core - eslint and oxlint (type-aware) on the changed files - oxfmt - core `src/workflows`: 102 files; `workflow.test.ts` (329 passed) and `evented-workflow.test.ts` (315 passed) were rerun alone after a heap OOM when three suites ran in parallel - server `workflows.test.ts`: 92 passed - Redis: 4 of 4 passed against a local redis-server on port 6381 ## Overlap with #24915 #24915 also edits `EventedRun.cancel()`. It calls `setupAbortHandler()` before `abort()`, guarded by `#abortHandlerRegistered`. That would also make cancel publish from a non-owner handle, so the two fixes overlap and will conflict on the same lines. Whichever PR merges second should keep a single publish path (`publishCancel()` here). The processor-side duplicate-end guard in this PR is independent of #24915. ## Unverified - **Multi-replica step abort (pre-existing, not fixed here):** when every replica runs workers, `workflow.cancel` reaches only one consumer in the group. If that consumer isn't the replica running the step, the step's abortSignal doesn't fire. `start()` still settles as `canceled`. Main behaves the same, even for local cancels. The Redis and server tests check step abort only when the owner is the sole worker. This should be tracked as a separate issue. - **Inngest and durable/evented agents:** I didn't run these. From reading the code, Inngest settles through `getRunOutput` on any terminal status, and durable agents use their own `agent.control.{runId}` abort transport. ## Risk: low The change is limited to the evented engine's cancel path. The publish is guarded to run at most once per handle. The WeakSet guard is per-process, so if two replicas process cancel for the same run at the same moment, the run could still end twice. That is no worse than main. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 When one Mastra instance cancels a workflow running on another, the workflow owner’s `run.start()` can now finish with a canceled result. The change also prevents duplicate cancellation messages from creating duplicate end events. ## What changed - `EventedRun` publishes `workflow.cancel` from both explicit cancellation and the abort handler through one helper. The helper prevents duplicate publishes and allows a retry if publishing fails. - `WorkflowEventProcessor` tracks controllers already handled by a cancellation event. When a step unwinds on one of those controllers, it does not publish another cancellation event. - Added tests for remote cancellation, `startAsync()`, server cancellation, Redis Streams, and retry after a failed cancellation publish. ## Validation The author reports passing core build, type-check, lint, formatting, and workflow tests. The author also reports passing server workflow tests and Redis Streams tests. These results are author-reported. ## Limitations When workers run on multiple replicas, a cancellation event may reach a processor other than the one running the active step. That step may not receive the abort signal. This case remains unfixed and unverified. Inngest and durable/evented agents were not tested. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com> |