0
Fork 0
mirror of https://github.com/mastra-ai/mastra.git synced 2026-11-17 08:44:46 +00:00
Commit graph

694 commits

Author SHA1 Message Date
dane-ai-mastra[bot]
a5859ee2b9
chore: version packages (alpha) (#30430)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-11-16 14:09:31 +01:00
Ward Peeters
cd2a326988
fix(libsql): stop background task writes stalling shutdown, and fix backend timing flakes (#30504)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-16 13:18:29 +01:00
Ward Peeters
88b4a97042
refactor(factory): skip legacy migration tables on new databases (#30488)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-16 13:11:56 +01:00
Ward Peeters
6c647e6fae
refactor(factory): stop creating columns and tables nothing reads (#30495)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-16 11:21:12 +01:00
dane-ai-mastra[bot]
1c52fdd034
chore: version packages (alpha) (#30261)
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases

> The changelog information of each package has been omitted from this
message, as the content exceeds the size limit.

## @mastra/factory@0.29.0-alpha.1


## @mastra/core@1.84.0-alpha.1


## @mastra/playground-ui@69.1.0-alpha.1


## @mastra/client-js@1.60.1-alpha.1


## @mastra/react@1.15.1-alpha.1


## @mastra/deployer-cloud@1.84.0-alpha.1


## @mastra/longmemeval@1.1.43-alpha.1


## @mastra/opencode@0.1.43-alpha.1


## create-factory@0.4.4-alpha.1


## @mastra/code-sdk@1.19.1-alpha.1


## mastracode@0.50.1-alpha.1


## @mastra/arize@1.3.31-alpha.0


## @mastra/arthur@0.4.31-alpha.0


## @mastra/braintrust@1.3.28-alpha.0


## @mastra/datadog@1.4.23-alpha.0


## @mastra/deepeval@0.1.25-alpha.0


## @mastra/laminar@1.3.33-alpha.0


## @mastra/langfuse@1.6.7-alpha.0


## @mastra/langsmith@1.4.4-alpha.0


## @mastra/observability@1.24.1-alpha.0


## @mastra/otel-bridge@1.5.23-alpha.0


## @mastra/otel-exporter@1.5.3-alpha.0


## @mastra/posthog@1.4.8-alpha.0


## @mastra/sentry@1.2.33-alpha.0


## @mastra/agent-builder@1.2.2-alpha.0


## mastra@1.39.1-alpha.1


## @mastra/connect@1.4.1-alpha.0


## @mastra/deployer@1.84.0-alpha.1


## @mastra/editor@0.17.2-alpha.0


## @mastra/mcp-docs-server@1.3.14-alpha.1


## @mastra/memory@1.41.2-alpha.0


## @mastra/server@1.84.0-alpha.1


## @mastra/google-cloud-pubsub@1.1.8-alpha.0


## @mastra/redis-streams@0.8.2-alpha.0


## @mastra/valkey-streams@0.6.3-alpha.0


## @mastra/elysia@0.1.24-alpha.1


## @mastra/express@1.5.28-alpha.1


## @mastra/fastify@1.5.28-alpha.1


## @mastra/hono@1.7.26-alpha.1


## @mastra/koa@1.7.28-alpha.1


## @mastra/nestjs@0.3.1-alpha.1


## @mastra/next@0.2.42-alpha.1


## @mastra/tanstack-start@0.2.42-alpha.1


## @mastra/clickhouse@1.30.1-alpha.0


## @mastra/cloudflare@1.8.1-alpha.0


## @mastra/cloudflare-d1@1.5.1-alpha.0


## @mastra/convex@1.9.1-alpha.0


## @mastra/dsql@1.5.8-alpha.0


## @mastra/dynamodb@1.5.2-alpha.0


## @mastra/libsql@1.30.1-alpha.0


## @mastra/mongodb@1.27.1-alpha.0


## @mastra/mssql@1.11.2-alpha.0


## @mastra/mysql@0.17.1-alpha.0


## @mastra/oracledb@0.5.7-alpha.0


## @mastra/pg@1.36.1-alpha.0


## @mastra/redis@1.4.14-alpha.0


## @mastra/spanner@1.12.1-alpha.0


## @mastra/turso@0.1.22-alpha.0


## @mastra/upstash@1.6.2-alpha.0


## @mastra/valkey@0.2.10-alpha.0


## @mastra/temporal@0.6.1-alpha.1


## @mastra/daytona@0.11.6-alpha.0


## create-mastra@1.39.1-alpha.1


## @internal/playground@1.39.1-alpha.1

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-11-13 19:58:20 -08:00
Arthur Fiorette
2788a6d959
feat(core): recover stopped workflow runs from a loop with one leader (#30281)
## Description

Restarts a default-engine workflow run whose instance crashed without
waiting for another instance to boot. `startWorkers()` now starts a
recovery loop on an instance that runs the orchestration worker, and one
instance among those that share a store runs its passes.

### The loop

Each pass applies the rule `restartAllActiveWorkflowRuns()` already uses
for default-engine runs: restart a `running` or `waiting` run once no
instance holds its run lease and `recovery.workflowGraceMs` passed since
the run or its lease last changed. The loop replaces the per-run recheck
timers the previous PR kept (`#scheduleWorkflowRecoveryRecheck`,
`#recheckWorkflowRun`); the durable-agent rechecks are unchanged.

The loop never touches evented runs. A redelivering broker recovers
them, and on other pubsubs the explicit `restartAllActiveWorkflowRuns()`
restarts them in the caller, as before. That call no longer joins a loop
pass for its evented part, so `startWorkers()` followed by the explicit
call still restarts evented runs on `EventEmitterPubSub`
(`evented/sleep-restart.test.ts` passes).

### Leader

One row, `::recovery-leader`, in the run-owners table, which gates the
loop's ticks only. Each tick of an instance tries a non-forced claim
until it gets the row, then keeps it: the lease helper renews it every
renew interval until `stopWorkers()` or `shutdown()` releases it. Only
the holder's loop lists runs; another instance's tick costs one refused
claim. When the leader dies, the row lapses after the lease TTL and the
next tick elsewhere takes it.

The leader saves listing cost only. Two restarts of one run are still
prevented by the run lease (`restart()` claims non-forced and throws
`WORKFLOW_RESTART_RUN_OWNED`).

An explicit `restartAllActiveWorkflowRuns()` never looks at the row. As
in the previous PR, each booting instance restarts its own stopped
default-engine runs (joining its own pass in flight), so another app's
workflows on the same store are recovered at its boot, and runs
recovered after a redeploy spread over the booting instances. Without
run fencing there is no leader and no loop, and the call restarts every
active run as before.

A failed probe, `getStore` or claim skips that tick and the loop keeps
going. A probe that answers "no fencing" stops the loop for good.

### Timings

- Interval: `recovery.workflowScanIntervalMs`, 60 s by default, each
wait up to 20 % shorter or longer, capped at 2^31 - 1 ms.
- A pass that saw a run in its grace period (and not live) moves the
next pass to that run's due time, no sooner than 1 s later and up to 20
% past it.
- A crashed instance's run is restarted about
`MASTRA_AGENT_THREAD_LEASE_TTL_MS` (15 s) plus `workflowGraceMs` (2 s)
after the crash, plus up to one interval for the next pass. When the
crashed instance was the leader, add up to one more interval for a
follower to take the row.
- `shutdown()` stops the loop when it starts, waits within
`drainTimeout` for the pass in flight and the restarts it handed out,
and releases the row in `stopWorkers()`.

### Config

```ts
const mastra = new Mastra({
  recovery: { workflowScanIntervalMs: 30_000 },
});
```

`recovery.workflows: 'off'` turns the loop off. `workers: false`, or a
`MASTRA_WORKERS` filter without `orchestration`, runs no loop. An
instance that registers no default-engine workflow to recover never
claims the leader row.

### LibSQL thread state writes

`@mastra/libsql` thread state writes (`setState`, `deleteState`) now
take the client write lock the store's other writes take. Without it, on
a local database file, a thread state write that ran while the same
store held a write transaction open (a lease claim or renewal) failed
with `SQLITE_BUSY`; the leader's renewals made that much more likely.

### Known limits

- A process that calls `restartAllActiveWorkflowRuns()` but never
`startWorkers()` runs no loop and doesn't recheck the runs it left
alone. A run its boot call found live or in its grace period is left to
the loop of another instance, if any.
- A leader whose renewals fail for longer than the lease TTL can run
passes alongside the new leader for up to one renew interval after the
store answers again. The run lease still prevents a run from being
restarted twice.
- Between boots, the loop recovers only the workflows the leader
registers. Instances that register different workflows on one store get
loop recovery for the leader's workflows only, and the others' runs at
their next boot.
- A run the loop restarts executes on the leader.
- A `mastra worker` process calls `stopWorkers()` on `SIGTERM` and exits
without `shutdown()`, so its default-engine runs wait as long as a
crashed instance's.
- A run that another instance won between the pass's lease read and
`restart()` is looked at again only at the next pass, up to one interval
later.
- Rolling deploys are out of scope for this stack.

### Tests

New `recovery-loop.test.ts` (in-memory store, 600 ms lease TTL):

- a crashed leader's run is restarted by the follower with no boot,
within TTL + 2 intervals + 1.2 s
- instances started together run passes on one of them, and the leader
keeps the row while passes are further apart than the lease TTL
- the leader's `stopWorkers()` releases the row and the follower takes
over
- a tick that `stopWorkers()` overtook (on the first tick's store read,
or on a later tick's probe) claims no row, so another instance can lead
- a stopped run of a workflow only another app registers is restarted by
that app's boot call while a different instance leads, and the leader's
loop never lists that workflow
- next-pass timing table (no run, live run, run in grace, grace ending
within 1 s), timers unref'd
- no listing after `stopWorkers()`; `shutdown()` waits for a pass in
flight
- a loop on `startWorkers('orchestration')`
- the loop keeps going after a failed probe, leader claim, listing or
restart
- no listing and no claim with `workflows: 'off'` or without run fencing

The recheck cases of `restart-active-runs-ownership.test.ts` and
`restart-active-runs-rule.test.ts` are gone with the rechecks: the crash
case covers a crashed owner, the failure table covers transient errors,
and the existing shutdown and limiter cases cover draining and
`workflowConcurrency`. The LibSQL cross-process cases and the MCP
survivor cases boot like a server (`startWorkers()`, then the explicit
call) and recover through the loop.

Checks: core `src/workflows src/mastra src/agent/durable src/events`
(only the two `scheduler.test.ts` retry cases that fail outside UTC),
core typecheck, LibSQL cross-process, Postgres and LibSQL
`recovery-pass.test.ts`, MCP `server-tasks-background`.

## Related issue(s)

Part of #28712. Stack 4 of 4, on top of the recovery pass PR.

## Type of change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [x] Test update

## Checklist

- [x] I have linked the related issue(s) in the description above
- [x] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

ELI5: Sometimes a workflow run stops when its server crashes. This
change adds a background check. One server regularly looks for stopped
runs and restarts them, so the work can finish without a manual restart.

## Overview

This change adds a workflow recovery loop to `@mastra/core`.
`startWorkers()` starts the loop. Among instances that share a store,
only the leader runs default-engine passes. A pass restarts
default-engine runs in `running` or `waiting` status when:

- no instance holds the run lease, and
- the configured grace period has elapsed.

The loop excludes evented runs. Explicit
`restartAllActiveWorkflowRuns()` calls handle evented runs and are not
leader-gated.

The change also updates LibSQL thread-state writes. `setState` and
`deleteState` now run through `withClientWriteLock`.

## Key changes

- `packages/core/src/mastra/index.ts`
- Adds `workflowScanIntervalMs` to `MastraRecoveryConfig`. The default
is 60000 ms. Values are capped at `2 ** 31 - 1`. A value of `-1`
resolves to the default.
- `startWorkers()` starts the loop when the selected workers handle
workflow events and recovery is enabled.
- When workflows use run fencing, the loop elects a leader through a
renewable lease. If fencing is unsupported, the loop stops.
- A pass records the earliest due time for runs still in their grace
period. The loop can then scan sooner.
- Failed lease reads leave runs for a later pass. Restart failures are
tracked per engine.
  - Per-run recheck, retry, and backoff logic is removed.
- `stopWorkers()` stops the loop and releases the leader lease. Shutdown
waits for an in-flight pass.
- Explicit `restartAllActiveWorkflowRuns()` calls share an in-flight
default-engine pass. Broker-redelivered evented runs stay broker-owned,
except process-local scheduled runs.
- `packages/core/src/workflows/run-ownership.ts`
- `repeatAfterSettled` accepts a function for the delay and an optional
`immediately` flag.
- Signature changes from `(tick, intervalMs: number)` to `(tick,
intervalMs: number | (() => number), options?)`.
- `stores/libsql/src/storage/domains/thread-state/index.ts`
- `setState` and `deleteState` use `withClientWriteLock`. SQL,
arguments, and error handling are unchanged.
- Tests
- Adds `packages/core/src/mastra/recovery-loop.test.ts`. It covers
leadership, scheduling, shutdown, and failure handling. The file is
added to `vitest.isolated-tests.ts`.
  - Removes three tests from `restart-active-runs-ownership.test.ts`.
- In `restart-active-runs-rule.test.ts`, removes one test and rewrites
the lease-read failure test to use the loop.
- Updates the recovery-defaults test, the MCP background server test,
and two LibSQL cross-process tests for the new scan interval.
- Adds a LibSQL thread-state test. It holds a write transaction open
during `setState` and `deleteState`.
- Documentation and changesets
- Updates the workers, server-adapter, workflow overview, and
`mastra-class` reference pages. Adds a "Recovery loop" section to the
workflow overview.
  - Corrects the crash-recovery text in the workers page.
  - Adds changesets for `@mastra/core` and `@mastra/libsql`.

## Public API changes

- Adds `workflowScanIntervalMs?: number` to `MastraRecoveryConfig`.
- Changes the signature of `repeatAfterSettled` in `run-ownership.ts`.

## Scope

- 19 files changed. 1028 lines added and 379 removed.

## Verification status

- Review findings: not available. No current review findings were
supplied.
- Author-reported checks: the PR objectives list core tests, core
typecheck, LibSQL cross-process tests, Postgres and LibSQL
`recovery-pass.test.ts`, and MCP `server-tasks-background`. These are
author reports. Results on the current head were not available for this
summary.
- Known failures: the author reports that two `scheduler.test.ts` retry
cases fail outside UTC.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 18:15:40 -03:00
Arthur Fiorette
99ec3bd582
fix(core): leave evented workflow runs to a redelivering broker and restart only stopped runs (#30279)
## Description

This PR stops `restartAllActiveWorkflowRuns()` from restarting evented
runs that a redelivering broker (Redis Streams, Valkey Streams, Google
Cloud Pub/Sub) already recovers. It also restarts a default-engine run
only once no instance holds its run lease and a grace period (2 s by
default) has passed. Before, every booting instance restarted every
active evented run, so a step interrupted by a crash ran once per
booting pod (#28712). The broker already recovers those runs: Redis and
Valkey Streams reclaim an unacked message once it has been idle for
`reclaimIdleMs`, and Google Cloud Pub/Sub redelivers once the ack
deadline passes.

### Evented runs on a broker

`PubSub` gets a `redeliversUnacked` getter (default `false`). Redis and
Valkey Streams return `true` while the reclaim loop runs
(`reclaimIntervalMs > 0`), Google Cloud Pub/Sub returns `true`,
`CachingPubSub` forwards to its inner pubsub, and the in-process
`EventEmitterPubSub` and the Unix socket pubsub return `false`. When the
pubsub (read through `__rawBus()`) redelivers, recovery never lists,
reads or restarts an evented run. An evented workflow without
declarative schedules costs no listing at all. Evented runs on any other
pubsub (`EventEmitterPubSub`, Unix socket, or a third-party pubsub that
doesn't override the getter) are restarted at once by the explicit call,
as before.

### Scheduled runs

The `mastra.pubsub` proxy publishes every workflow event of a scheduled
run (`sched_wf_*`, plus the notification dispatcher's row) `localOnly`,
so the broker never holds it. Recovery still restarts those runs: an
evented workflow with declarative schedules gets one summary listing per
status, and only its process-local scheduled runs are kept. The proxy
and recovery share one predicate, `isProcessLocalScheduledRunId`.

### Default engine

One recovery rule: a `running` or `waiting` run is restarted when its
run lease isn't live and `recovery.workflowGraceMs` (new, 2,000 ms by
default for every pubsub) has passed since the later of the lease's last
expiry and the run's last write. The pass lists summaries
(`listWorkflowRuns({ status, perPage: false, summary: true })` per
status), reads leases in batches of 100 with `getRunOwnerships`,
re-reads a run's lease once its restart gets a slot, and runs every
restart on one instance-wide limiter (`recovery.workflowConcurrency`). A
run left live or in its grace period gets a per-run recheck timer, as on
main. Overlapping calls share one pass. `shutdown()` waits, within
`drainTimeout`, for a pass still reading the store and for the restarts
it handed out. Without run fencing every active run is restarted, as
before.

### Nested resume

A default-engine nested run resumed directly now holds its parent's run
lease while it runs and hands it to the parent's resume, so recovery
doesn't restart the parent meanwhile. On a store without
compare-and-set, where the parent stays `suspended`, the nested run
holds its own lease instead.

Also: `renewWorkflowLease` waits for each renewal to settle before
arming the next one (a self-rearming `setTimeout` instead of
`setInterval`), and `EventEmitterPubSub`'s JSDoc and the docs say it's
in-process only.

### Repro

Postgres + Valkey Streams with default reclaim settings (`reclaimIdleMs`
60 s, `reclaimIntervalMs` 30 s). Instance A starts a run whose step
records an execution and then hangs; A is killed with `SIGKILL`
mid-step. Then 2 or 4 fresh instances boot and all call
`restartAllActiveWorkflowRuns()` at once. "At boot" counts step
executions caused by those calls; "total" counts every execution after
the kill until the run succeeds; "rerun after" is seconds from the kill
to the first new execution. 3 trials per row, identical results within
each row except for timing.

| Build | Engine | Booting pods | Executions at boot | Executions total
| Rerun after |
|---|---|---|---|---|---|
| Base (this PR's base branch) | evented | 2 | 2 | 2 | 0.1 s |
| Base | evented | 4 | 4 | 4 | 0.1 s |
| Base | default | 2 | 0 | 1 | 15.1 to 16.6 s |
| Base | default | 4 | 0 | 1 | 15.0 to 16.9 s |
| This PR | evented | 2 | 0 | 1 | 60.1 to 60.2 s |
| This PR | evented | 4 | 0 | 1 | 60.1 to 60.2 s |
| This PR | default | 2 | 0 | 1 | 17.0 to 19.1 s |
| This PR | default | 4 | 0 | 1 | 17.2 to 21.0 s |

On this PR the evented rerun is the broker's reclaim: one execution on
one pod, about 60 s after the kill. The default-engine rerun comes once
A's lease (15 s TTL) lapsed plus the 2 s grace and up to 5 s of recheck
jitter.

### Benchmark

3,000 active default-engine runs that another instance holds, among
30,000 finished runs, two workflows. "Before" is a copy of main's sweep
run in the same process (full listing, then `createRun()` + `restart()`
per run, 5 at a time); "after" is `restartAllActiveWorkflowRuns()` on
this PR. Median of 5 calls.

| Store | Before | After | Listing payload before | Listing payload
after |
|---|---|---|---|---|
| Postgres | 1,773 ms | 30 ms | 1.12 MB | 0.60 MB |
| LibSQL | 627 ms | 44 ms | 1.12 MB | 0.60 MB |

The benchmark's snapshots are nearly empty, so the payload gap here is
small; with real step results it grows with snapshot size.

### Known limits

On a redelivering broker these evented runs stay `running` until someone
calls `restart()` on them. Main's boot restart recovered them; stack 2
takes them over.

- A sleeping evented run whose instance exits, after a crash or during a
normal deploy. The sleep acks its delivery and keeps a `setTimeout` in
the process that started it, so any exit of that process loses the
timer.
- A step whose message Redis or Valkey Streams trimmed. Both trim with
`MAXLEN ~ maxStreamLength` (10,000 by default), which also drops entries
that are still pending. A step that is in flight while that many newer
events are published loses its message if its instance then exits.
Deployments with long steps and a high event volume should raise
`maxStreamLength`.
- A message the broker dropped after `maxDeliveryAttempts` or at
`inFlightTimeoutMs`.
- A failure publish that gave up after its retries (the worker acks the
message).
- A run whose `running` snapshot was saved but whose start event never
reached the broker, because its process stopped between the two.
- A Redis or Valkey instance that restarted without persistence.
- A scheduled evented run that was in flight when a deploy stopped its
process and removed the workflow's schedule from code. Recovery doesn't
list it, so it stays `running` until `restart()`.

#28712 is fixed for evented runs on a redelivering broker. These are
still restarted by every booting instance, as before:

- scheduled evented runs (`sched_wf_*`), whose events never leave their
process
- evented runs on the in-process `EventEmitterPubSub` or the Unix socket
pubsub. Several instances sharing one storage with these pubsubs aren't
supported for evented workflows; the docs now say a multi-instance
evented deployment needs a broker.

With Redis or Valkey Streams, `reclaimIdleMs` must stay at 30 s or more:
a message reclaimed while the dead worker's 30 s step fence is still
held is dropped as a duplicate. The getter's JSDoc, the adapter docs and
the changeset state it.

Rolling deploys are out of scope.

Stack 3 of 4: storage projections and batch ownership reads, then the
finished-step check, then this PR, then the recovery loop with a leader.

## Related issue(s)

Fixes #28712

## Type of change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Code refactoring
- [x] Performance improvement
- [x] Test update

## Checklist

- [x] I have linked the related issue(s) in the description above
- [x] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

ELI5: When a server crashes in the middle of a workflow, Mastra can
restart that workflow. Before this change, some workflows ran their
steps again on every new server that started, so a step could run many
times. This change lets the message broker handle the restart for those
workflows, so each step runs only once. Other workflows now wait for a
short grace period before they restart.

## Summary

This pull request changes how workflow recovery restarts runs after a
crash or a boot. It targets duplicate step execution in two cases:

- **Evented runs on brokers that redeliver unacknowledged messages.**
Recovery skips these runs. The broker redelivers the unacknowledged step
events instead.
- **Default-engine runs.** Recovery restarts a run only after no live
lease remains and a grace period has passed.

### Behavior changes

- Google Cloud Pub/Sub, Redis Streams, and Valkey Streams report
`redeliversUnacked` as `true` when their reclaim or redelivery path is
active. Redis Streams and Valkey Streams report `true` only when
`reclaimIntervalMs` is above 0.
- `PubSub` adds a `redeliversUnacked` getter that defaults to `false`.
`CachingPubSub`, `EventEmitterPubSub`, and `UnixSocketPubSub` expose or
declare this capability.
- Recovery leaves evented runs to the broker when `redeliversUnacked` is
`true`. Scheduled evented runs restart on boot. Other evented runs
restart immediately when the broker does not redeliver.
- Default-engine recovery applies `recovery.workflowGraceMs`. The
default is 2,000 ms. Negative or non-finite values use the default. A
run restarts only after its lease is no longer live and the grace period
has elapsed since the later of its last update and lease expiry. Runs
that are not yet eligible are checked again later. If the storage
adapter does not support run fencing, active runs restart without the
grace rule.
- Recovery lists active run summaries, reads ownership in batches, and
rechecks ownership before each restart.
- Restarts use a shared concurrency limiter
(`createConcurrencyLimiter`). A restarted run holds its slot until it
finishes.
- Overlapping recovery passes are deduplicated. Shutdown waits for an
in-flight pass.
- Recovery skips nested runs whose active parent drives them.
- Lease renewal runs sequentially (`renewWorkflowLease`,
`repeatAfterSettled`). A new renewal starts only after the previous one
settles.
- Nested resume coordinates parent and child leases. A failed lease
claim rejects the resume and leaves the runs suspended.
- Run-claim operations return whether a status transition occurred. They
return `false` when storage is absent, running snapshots are not
persisted, or compare-and-set is not supported.

### Public API changes

- `MastraRecoveryConfig.workflowGraceMs?: number` (default 2,000 ms).
- `PubSub.redeliversUnacked` getter (default `false`).
- `redeliversUnacked` getters on `GoogleCloudPubSub`,
`RedisStreamsPubSub`, `ValkeyStreamsPubSub`, and `UnixSocketPubSub`.
- `ConcurrencyLimiter` interface and `createConcurrencyLimiter(limit)`
function in `run-with-concurrency.ts`.
- `renewWorkflowLease`, `repeatAfterSettled`, and
`WORKFLOW_RUN_LEASE_RENEW_INTERVAL_MS` in `run-ownership.ts`.
- `Run._resume(params)` accepts an optional `takeHeldOwnership`
callback.

### Documentation

- Crash-recovery and worker docs describe broker reassignment and
multi-instance limits.
- `EventEmitterPubSub` documents that delivery is process-local.
- Redis Streams and Valkey Streams docs recommend `reclaimIdleMs` of at
least 30 seconds. Reclaiming earlier can cause duplicate-dropped step
events.
- The recovery reference documents `workflowGraceMs`.

### Tests and test utilities

- New recovery tests cover exit cases (crash and shutdown), recovery
reads, rules, ownership during nested resumes, and lease renewal.
- Shared test helpers add `sharedBroker`, `killableProcess`,
`interceptStarts`, `persistActiveRun`, and `describeRecoveryExitCases`.
- Existing tests set `workflowGraceMs: 0` where they need immediate
recovery.
- Redis Streams and Valkey Streams tests check `redeliversUnacked`
against `reclaimIntervalMs`.
- LibSQL and Postgres recovery-pass suites run the shared exit cases.

### Scope and limits

- The change addresses issue `#28712` for evented runs on redelivering
brokers. Evented runs on in-process or Unix socket pubsubs still restart
at boot. The PR states that multi-instance evented deployments need a
broker.
- The PR lists unrecovered cases: sleeping evented runs, trimmed or
dropped messages, failed publishes, missing start events, non-persistent
Redis or Valkey restarts, and scheduled runs whose schedules were
removed. Rolling deploys are out of scope.
- Reported benchmark results (median recovery time on 3,000 active and
30,000 finished runs: 1,773 ms to 30 ms on Postgres, 627 ms to 44 ms on
LibSQL) and repro results are author-reported. They were not
independently verified.
- CI and test results: not available.
- Review findings and severity counts: not available.

### Size

- 41 files changed, 2,594 lines added, 209 lines removed (from the pull
request diff).

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 14:59:00 -03:00
Arthur Fiorette
8559e74767
feat(storage): workflow run summary listings and batched run ownership reads (#30274)
## Description

Adds `summary: true` listings that return only each run's status,
timestamp and parent, and `WorkflowsStorage.getRunOwnerships()` to read
many runs' ownership at once. Nothing in core calls them yet, so no
workflow behavior changes.

### Summary listings

`listWorkflowRuns({ summary: true })` now reduces each run's snapshot
to:

```ts
{ status?, timestamp?, parentWorkflow?: { workflowId, runId } }
```

`parentWorkflow` is present only when the stored parent has both IDs,
and is never `null`. A field the run doesn't have is left out. `context`
and every other snapshot field are never returned. Run fields outside
the snapshot (`runId`, `workflowName`, `resourceId`, `createdAt`,
`updatedAt`) are unchanged, and so is `total`.

| Where the snapshot is reduced | Stores |
|---|---|
| In the query (only the summary fields leave the database) | PostgreSQL
and LibSQL (already projected `{ status, timestamp }`, now add the
parent), MySQL, MSSQL, OracleDB, Spanner, DSQL, Cloudflare D1 and
Durable Objects, MongoDB (`project()`) |
| In the app, after reading the whole snapshot | DynamoDB, Redis,
Valkey, Upstash, Convex, in-memory |

This PR doesn't change the status filter. Among the SQL stores,
PostgreSQL (expression index on `snapshot ->> 'status'`), Spanner
(generated status column) and OracleDB (function index on `JSON_VALUE`)
filter on an index, and LibSQL storage v2 runs have an indexed `status`
column. MySQL, MSSQL, DSQL and Cloudflare D1 and Durable Objects extract
the status from each row's JSON.

MSSQL now caches whether the snapshot table has a `resourceId` column.
The non-summary `resourceId` filter uses the same cache, with the same
result as before.

### Batch ownership reads

`getRunOwnerships({ runIds })` returns a `Map` from run ID to the record
`getRunOwnership()` would return for that ID; runs that were never
claimed are left out. The default reads each run with
`getRunOwnership()`, at most 8 at a time, and stops starting new reads
once one fails. Overrides:

| Store | Round trips |
|---|---|
| PostgreSQL | One query for all IDs (one array parameter) |
| LibSQL | One query per 500 IDs |
| MySQL, MSSQL | One query per 500 IDs, joining the requested IDs so the
column's collation matches them as the single-run read does |
| MongoDB | One `find` per 1,000 IDs |
| DynamoDB | One strongly consistent `BatchGetItem` per 100 IDs,
unprocessed keys retried with backoff; IDs that differ only in case name
one item, as in the single-run read |
| Upstash | One pipeline request per 100 IDs |
| Redis, Valkey | One single-key script call per ID (works across
cluster hash slots), up to 500 in flight |
| In-memory | Reads every record in one pass |
| Convex, DSQL, Spanner, OracleDB, Cloudflare D1 and Durable Objects |
Base default |

### In-memory `updatedAt`

The in-memory store now moves a run's `updatedAt` on
`updateWorkflowResults()`, `updateWorkflowState()` and
`updateWorkflowRunHeader()`, as the database adapters already did. A
write a status guard rejects leaves it alone.

### Conformance

In `@internal/storage-test-utils`, run on every store with run fencing:
the summary listing shape for v1 and storage v2 runs, including a null
parent, a parent with one ID and a missing timestamp; `getRunOwnerships`
over 1,001 IDs with live, released, expired and unclaimed runs, each
record equal to the single-run read, plus IDs that differ from a claimed
one only by a trailing space or case (MySQL, MSSQL and DynamoDB match
those in the single-run read). The `updatedAt` cases run through
`createWorkflowsTests`. Valkey and Cloudflare Durable Objects don't run
that suite: Valkey writes through the same path as Redis, and Durable
Objects don't implement partial writes.

Checks run locally: `pnpm build:core`, core `check`, lint and `tsc` of
every touched store, and the conformance suite against in-memory,
LibSQL, PostgreSQL, MySQL, MSSQL, MongoDB (replica set), Redis, Valkey,
DynamoDB Local and Upstash (serverless-redis-http). MSSQL has 15
failures outside this PR's cases that fail the same way on `main`
locally. DSQL, Convex, Spanner, OracleDB and Cloudflare were not run
locally.

Stack 1 of 4; the recovery pass in the later PRs uses these reads.

## Related issue(s)

Part of #28712

## Type of change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [x] Performance improvement
- [x] Test update

## Checklist

- [x] I have linked the related issue(s) in the description above
- [ ] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

**ELI5:** Workflow runs can now be listed in a smaller form that holds
only the status, time, and parent workflow. The PR also adds a way to
look up the owners of many runs in one call. Storage adapters return
only the fields that callers need.

## Overview

This PR adds two storage capabilities for workflow runs:

- `listWorkflowRuns({ summary: true })` returns a reduced snapshot for
each run. The snapshot contains `status` and `timestamp`. It adds
`parentWorkflow: { workflowId, runId }` only when both parent IDs are
non-empty strings.
- `WorkflowsStorage.getRunOwnerships({ runIds })` returns ownership
records for many runs. It returns a `Map` keyed by run ID and omits
unclaimed runs.

The in-memory store also changes. Successful writes refresh `updatedAt`.
A write rejected by a status guard leaves `updatedAt` unchanged.

The PR description states that core does not call these APIs yet.
Workflow behavior does not change in this PR.

## Changes by area

**Core (`@mastra/core`)**
- `storage/domains/workflows/base.ts`: Adds a default `getRunOwnerships`
implementation. It deduplicates run IDs and reads at most 8 records
concurrently through `getRunOwnership`. It omits unclaimed runs. A
failed read rejects the batch and stops further reads from starting.
- `storage/domains/workflows/inmemory.ts`: Adds `summary` support to
`listWorkflowRuns`. Summary reduction runs after filtering and
pagination. `updateWorkflowResults`, `updateWorkflowState`, and
`updateWorkflowRunHeader` refresh `updatedAt` after successful writes.
- `storage/types.ts`: Documents the fuller reduced-snapshot shape for
`summary`. It changes no runtime behavior.

**Store adapters**

Summary support (`summary: true`) is added in: Cloudflare D1, Cloudflare
Durable Objects, Convex, DSQL, DynamoDB, LibSQL, MongoDB, MSSQL, MySQL,
OracleDB, PostgreSQL, Redis, Spanner, Upstash, and Valkey.

Reduction happens in the database for these stores:
- SQL stores: PostgreSQL, LibSQL, MySQL, MSSQL, OracleDB, Spanner, DSQL
- Cloudflare D1 and Durable Objects
- MongoDB (projection)

Reduction happens in application code for these stores:
- Convex, DynamoDB, Redis, Upstash, Valkey

`getRunOwnerships` is added in: PostgreSQL, LibSQL, MySQL, MSSQL,
MongoDB, DynamoDB, Redis, Upstash, and Valkey. The batch strategy
differs by store:
- PostgreSQL: one query.
- LibSQL, MySQL, MSSQL: chunks of up to 500 IDs.
- MongoDB: batches of 1,000 IDs.
- DynamoDB: strongly consistent batch reads of up to 100 keys.
Unprocessed keys retry with exponential backoff.
- Upstash: pipeline batches of 100.
- Redis and Valkey: one single-key ownership script call per run. Calls
run in batches of up to 500.

Cloudflare D1 and Durable Objects include `resourceId` in summary
queries only when the table has that column. The column check is cached.

**Conformance test utilities (`stores/_test-utils`)**
- `createSummaryListingTests`: Covers summary shapes, parent metadata
handling, and comparison with full listings. It skips stores without
run-fencing support.
- `createUpdatedAtTests`: Covers `updatedAt` on partial writes. For
storage v2 run journals, it checks header and state updates, including
guard-rejected writes.
- Run-fencing tests: Add batch ownership read cases for live, released,
expired, and unclaimed runs. They also cover inputs larger than
MongoDB's 1,000-item chunk size, collation lookalike IDs, and empty
input. The `untilLapsed` helper polls for up to 5 seconds, checking
every 20 ms.

**Changesets**

Patch changesets cover the affected store packages. A minor changeset
covers `@mastra/core`.

## Scope

- Files changed: 30
- Lines added: 1,555
- Lines removed: 52

## Verification

The author reports these local results. They have not been checked
against the current head:

- `pnpm build:core`, core checks, lint, and TypeScript checks for
touched stores were run.
- Conformance runs were run against the listed stores. The author
reports 15 MSSQL failures outside this PR's test cases. The author
reports the same failures on `main`.
- Conformance tests were not run locally for DSQL, Convex, Spanner,
OracleDB, or Cloudflare.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 11:45:22 -03:00
NikAiyer
956143f6bd
test(libsql): guard live-run boot recovery (#30264)
Adds a two-process regression test for boot recovery with both durable
and evented agents. A peer blocks in its first tool while a second
Mastra instance opens the same LibSQL database, starts workers,
discovers the running agent, and calls `recoverAllDurableAgents()`. The
test checks that recovery skips the healthy owner, makes zero model
calls in the booting replica, and lets the original peer commit all
three steps. Reuses the T81 peer and IPC gates with no added sleeps or
skipped engine cells.

Run locally with `pnpm build:core`, then `pnpm --filter @mastra/libsql
exec vitest run
src/__tests__/cross-process/xproc-boot-recovery.test.ts`.

Three consecutive local runs passed, followed by another passing run
after restoring the red control:

```text
                   durable    evented    result
Green run 1        1901ms     1632ms     2 passed (2)
Green run 2        3186ms     1605ms     2 passed (2)
Green run 3        1399ms     2423ms     2 passed (2)
Restored green     1451ms     1798ms     2 passed (2)
```

For the red control, temporarily changed the recovery branch in
`packages/core/src/agent/durable/execution-fence.ts` from
`backend.tryRecover(executionId, args.untrackedRun)` to
`backend.tryTakeover(executionId)`, rebuilt core, and ran the same test.
Both engines failed on the intended assertions:

```text
Test Files  1 failed (1)
     Tests  2 failed (2)

Both durable and evented:
Expected: { agents: 1, recovered: 0, succeeded: 0, failed: 0, skipped: 1 }
Received: { agents: 1, recovered: 1, succeeded: 1, failed: 0, skipped: 0 }

boot recovery must not call the model: expected 4 to be +0
Original peer commits: expected [1] to deeply equal [1, 2, 3]
the peer finishes without work from the booting replica: expected 4 to be +0
```

The bypass was reverted and core rebuilt before the final green run.
There are no production-code changes and no changeset. Formatting,
type-aware lint, and `git diff --check` pass. `pnpm --filter
@mastra/libsql exec tsc --noEmit` is blocked by five existing errors in
unrelated fixtures: one in `dead-claim-pg-agent.ts:10` and four in
`foreach-billing-workflow.ts`.

Fixes COR-1492
Ref COR-1365


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

ELI5: The computer runs a task on one program. A second copy of the
program starts and must not repeat that task. This PR adds a test that
checks this rule. The second copy must skip the task, and the first copy
must finish it.

## Summary

This PR adds one new test file:
`stores/libsql/src/__tests__/cross-process/xproc-boot-recovery.test.ts`
(113 lines added, 0 removed). It contains no production-code changes and
no changeset.

The file defines a cross-process regression test for boot recovery. It
runs once for `durable` and once for `evented` agents. Each run has the
same flow:

1. **Setup.** `beforeEach` sets `MASTRA_AGENT_THREAD_LEASE_TTL_MS` to
`1500`. It then creates the test environment with workers enabled.
2. **Start the peer.** The test spawns a peer process from
`fixtures/t81-peer.ts`. It waits until the peer reports `blocked` at
step 1.
3. **Boot a replica.** The test opens the same LibSQL database from a
second `Mastra` instance. The replica registers a step agent with three
steps. It uses `MockMemory` and workers. Recovery is set to
`durableAgents: 'auto'`.
4. **Discover the run.** The test calls `listActiveRuns()`. It expects
exactly one active run with the peer's `runId`.
5. **Run recovery.** The test calls `recoverAllDurableAgents()`. It
expects `{ agents: 1, recovered: 0, succeeded: 0, failed: 0, skipped: 1
}`. It uses soft assertions for this result.
6. **Check model calls.** The test expects the replica's model call
count to be `0`.
7. **Release the peer.** The test sends `release` and reads the peer
result. It expects no errors. It expects `commit` events for steps `1,
2, 3`. It expects the peer to exit with code `0` and no signal.
8. **Teardown.** `afterEach` releases and kills the peer. It then shuts
down Mastra, closes storage, cleans up the environment, and restores
stubbed environment variables.

The test links to COR-1492 and COR-1436 in a code comment. The comment
says a running snapshot can belong to a healthy replica. Recovery must
find that run without running its tools again.

## Verification

The PR author reports:
- Four passing local runs of `pnpm --filter @mastra/libsql exec vitest
run src/__tests__/cross-process/xproc-boot-recovery.test.ts`, after
`pnpm build:core`.
- A temporary red control caused both engine tests to fail on the
intended assertions. The control was then reverted.
- Formatting, type-aware lint, and `git diff --check` pass.
- `pnpm --filter @mastra/libsql exec tsc --noEmit` reports five errors
in unrelated fixtures.

These results come from the author's report. This review did not run the
tests.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: review <review@local>
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-12 16:59:31 -08:00
dane-ai-mastra[bot]
d60ba71be0 chore: version - exit prerelease mode 2026-11-12 21:12:29 +00:00
dane-ai-mastra[bot]
9e82af0237
chore: version packages (alpha) (#30074)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-11-11 14:21:14 -08:00
mastra-platform[bot]
b0ef22e7bf
feat(schedules): Add bounded one-off schedules (#25131)
> Opened by @abhiaiyer91

Refs #24610

## Summary
Schedules can now fire once or stop after a given time, so you no longer
need to pin a cron expression to a date and clean it up yourself.

- **New `completed` status**: the scheduler and timing reconciliation
set it when no automatic fires remain. It's written in the same atomic
`nextFireAt` check-and-set used to claim a run, so a schedule still
fires only once across instances. A cron with no future occurrence left
also moves to `completed` instead of erroring. `pause`/`resume` on a
completed schedule return 409. Changing `runAt`/`endAt` reactivates it
only when the resulting timing can still fire. `list()` hides completed
schedules unless you pass `status: 'completed'`.
- **Declarative workflow schedules**: `createWorkflow({ schedule: {
runAt } })` and `{ cron, endAt }` are supported. Past times are accepted
so redeploys don't fail, and a completed schedule isn't re-armed unless
its timing changes.
- **Trigger kind**: one-off fires report `trigger.kind: 'once'`.
- `run(id)` doesn't complete a one-off schedule.

## Surfaces
- **Core**: types, validation, scheduler claim logic,
create/update/pause/resume/list, declarative sync.
- **Storage**: new nullable `run_at`/`end_at` columns for libsql, pg,
mysql, mongodb, spanner, convex and in-memory. Existing columns are
untouched: one-off schedules store `cron` as `''`.
`updateScheduleNextFire` takes an optional `newStatus`.
- **Server/client-js/CLI**: request bodies accept `runAt`/`endAt`
(one-of rule checked in the schema). Responses include the new fields
and `completed`. Route types and CLI route metadata regenerated.
- **Studio**: shows the run time for one-off schedules, shows the
`endAt` bound, adds a `completed` badge and hides pause/resume for
completed schedules.
- **Docs + changeset.**

## Not in this PR (remaining #24610 work)
- ISO 8601 string inputs for `runAt`/`endAt` (numbers and `Date` are
accepted today).
- The template `schedule_once` tool.

## Test plan
- [x] core: schedules, scheduler, cron, schedule-config and scheduler
integration tests (260 passing), clean `tsc`
- [x] Shared adapter suite gained tests for `runAt`/`endAt` round-trip
and atomic status CAS. Passing on libsql, pg, mysql and mongodb
(Docker), plus convex server tests. **Not run on Spanner (no emulator
available); typecheck only.**
- [x] server schedules handler tests (51 passing, including
one-off/bounded creation and body validation)
- [x] client-js and playground typecheck clean; playground schedules
tests passing

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

Schedules can now run once at a chosen time or repeat until a chosen end
time. They complete when their planned runs are over, and changing their
timing can make them active again.

## Summary

- Added `runAt` for one-off agent and workflow schedules. API-created
schedules require a future `runAt`; declarative workflow schedules can
register past times as `completed`.
- Added `endAt` for cron schedules. A schedule becomes `completed` when
its next run would occur after `endAt`, or when no future run remains.
- Completed schedules cannot be paused or resumed. Timing changes can
reactivate a completed schedule when a future run remains. `list()`
hides completed schedules unless callers filter with `status:
'completed'`.
- Resuming a paused one-off schedule after its `runAt` has passed allows
it to fire on the next scheduler tick. Calling `run(id)` manually does
not complete a one-off schedule.
- Added the `once` trigger kind for scheduled one-off agent fires.
Manual runs remain `manual`.
- Declarative workflow schedules preserve completed status across
redeploys when timing is unchanged. Timing changes can re-arm them.
- Updated schedule APIs, storage adapters, Studio, and documentation.
Storage persists `runAt` and `endAt` and supports atomic status updates.

## Checks

Current-head CI, build, and test results: not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Abhi Aiyer <abhiaiyer91@gmail.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: NikAiyer <nick.aiyer@gmail.com>
Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: Ehindero Israel <iehindero419@stu.ui.edu.ng>
2026-11-11 19:33:50 +01:00
dane-ai-mastra[bot]
779d2875b1
chore: version packages (alpha) (#29912)
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases

> The changelog information of each package has been omitted from this
message, as the content exceeds the size limit.

## @mastra/auth-studio@1.4.0-alpha.1


## @mastra/factory@0.28.0-alpha.5


## @mastra/mcp-apps-components@0.1.0-alpha.5


## @mastra/playground-ui@69.0.0-alpha.6


## @mastra/client-js@1.60.0-alpha.5


## @mastra/react@1.15.0-alpha.5


## @mastra/deployer-cloud@1.83.0-alpha.5


## @mastra/longmemeval@1.1.42-alpha.5


## @mastra/opencode@0.1.42-alpha.5


## create-factory@0.4.3-alpha.5


## @mastra/code-sdk@1.19.0-alpha.5


## mastracode@0.50.0-alpha.5


## @mastra/agent-builder@1.2.1-alpha.2


## mastra@1.39.0-alpha.5


## @mastra/core@1.83.0-alpha.5


## @mastra/deployer@1.83.0-alpha.5


## @mastra/editor@0.17.1-alpha.3


## @mastra/mcp@2.9.0-alpha.2


## @mastra/mcp-docs-server@1.3.13-alpha.8


## @mastra/memory@1.41.1-alpha.2


## @mastra/rag@2.7.3-alpha.1


## @mastra/server@1.83.0-alpha.5


## @mastra/elysia@0.1.23-alpha.5


## @mastra/express@1.5.27-alpha.5


## @mastra/fastify@1.5.27-alpha.5


## @mastra/hono@1.7.25-alpha.5


## @mastra/koa@1.7.27-alpha.5


## @mastra/nestjs@0.3.0-alpha.5


## @mastra/next@0.2.41-alpha.5


## @mastra/server-adapters-test-suite@0.3.0-alpha.2


## @mastra/tanstack-start@0.2.41-alpha.5


## @mastra/libsql@1.30.0-alpha.2


## @mastra/turso@0.1.21-alpha.2


## @mastra/inngest@1.11.2-alpha.3


## @mastra/temporal@0.6.0-alpha.5


## create-mastra@1.39.0-alpha.5


## @internal/playground@1.39.0-alpha.5

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-11-10 10:50:08 -08:00
Jamie Nichols
6563e7abfc
fix(libsql): keep vector indexes writable after truncateIndex (#26802)
## What broke

`LibSQLVector.truncateIndex()` ran `DELETE FROM <table>` with no `WHERE`
clause. On a table with a `libsql_vector_idx` index, that statement
empties the table but leaves the old rows in the vector index's shadow
table. The next insert then fails:

```
SQLITE_ERROR: vector index(insert): failed to insert shadow row
```

So once you truncated an index, you couldn't write to it again. I
reproduced this on both file-backed and `:memory:` databases with
`@libsql/client` 0.18: after the bare delete the shadow table still has
1 row and the reinsert fails. The same cycle with `WHERE rowid IS NOT
NULL` leaves 0 shadow rows and the reinsert succeeds. Upstream tracking:
tursodatabase/libsql#2289.

## The fix

`truncateIndex()` now runs `DELETE FROM <table> WHERE rowid IS NOT
NULL`. It still deletes every row, and in the repro it also removes the
shadow rows (0 left after the delete). The source change is one line in
`stores/libsql/src/vector/index.ts`, and it runs for every connection
type.

## Tests

New tests in `stores/libsql/src/vector/index.test.ts`. They check
behaviour only, using public methods:

- file database: insert, truncate, insert again, then query and get only
the new vector
- file database: truncate an empty index, then run repeated
truncate/insert cycles
- file database: single-row and filter deletes still work, and the index
still accepts inserts afterwards
- `:memory:` database: insert, truncate, insert again, then query

I ran these tests against the old `truncateIndex`. Both file truncate
tests and the `:memory:` test fail with the shadow-row error. The
partial-delete test passes there, because it guards existing behaviour.
With the fix, the whole file passes: 146 passed, 1 skipped (`npx vitest
run src/vector/index.test.ts`). oxlint and oxfmt are clean on the
changed files.

## Not tested

- Remote Turso/sqld URLs and embedded replicas (`syncUrl`). They run the
same statement, but I didn't test them against a server.
- Cost on large tables. I haven't measured the qualified delete's
performance.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## ELI5

Truncating a local LibSQL vector index could leave behind data that
stopped new vectors from being added. This change removes the vector
rows while keeping the index ready for new inserts.

## Changes

- Updated `LibSQLVector` truncation to delete rows with `rowid IS NOT
NULL`.
- Added tests for file-backed and in-memory databases. They cover
reinserting and querying vectors after truncation, repeated truncation,
and partial deletes.
- Added a patch changeset for `@mastra/libsql`.

## Validation

Test and lint results: not available.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Jamie Lee Nichols <10267569+CodingButter@users.noreply.github.com>
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-10 06:55:26 -08:00
dane-ai-mastra[bot]
7c4dd88400
chore: version packages (alpha) (#29889)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-11-10 15:48:17 +01:00
Eric Pinzur
832fb190f7
feat(observability): allow opting into remote storage export on Platform (#29737)
When `MASTRA_DEPLOYMENT_ID` is set, storage exporters are suppressed
even when a customer intentionally exports to both their own database
and Platform. Keep that default and add a per-config
`retainStorageExporterOnPlatform: true` opt-in for
`MastraStorageExporter`, including runtime registration.

```ts
new Observability({
  configs: {
    production: {
      serviceName: 'my-service',
      retainStorageExporterOnPlatform: true,
      exporters: [new MastraStorageExporter(), new MastraPlatformExporter()],
    },
  },
});
```

Deprecated `DefaultExporter` remains suppressed on Platform even with
the opt-in. Its implementation is unchanged.

The opt-in does not permit known local observability storage.
MastraStorageExporter warns during initialization and disables
persistence when the resolved observability domain reports local
storage. This blocks InMemoryStore (including Mastra's automatic
fallback when no storage is configured), local libSQL files, in-memory
libSQL, embedded replicas, and DuckDB while allowing remote libSQL/Turso
and PostgreSQL. Other storage domains and exporters are unaffected, and
local development retains local storage support.

Adds the optional persistenceLocation capability to observability
storage, with in-memory, libSQL, and DuckDB implementations. These
adapters must be upgraded alongside observability for the guard to
apply; older/custom adapters without location metadata retain the
existing opt-in behavior. No provider names or private connection fields
are inspected.

Constructor skip warnings are deferred until the application logger is
installed and emitted once. Docs explain the local-storage restriction
before the example, include a supported/blocked provider table, and
distinguish the implementation-specific option from the core interface.
The restriction remains explicit even for users who cannot see the
warning. Includes one minor changeset per affected package:
observability, core, libSQL, and DuckDB. Preserves the default policy
introduced in #23555.

Validation: 451 targeted observability tests pass, including regression
coverage that reproduced unwanted in-memory persistence before the fix
for both explicit InMemoryStore and the automatic fallback. The tests
also verify initialization warnings and continued local-development and
remote-storage behavior. All 756 MDX files pass formatting using the
CI-matching dependencies; focused type-aware lint, Remark, and Vale
pass. The earlier 9 adapter tests also passed. No live Platform or
PostgreSQL endpoint was exercised. Full typechecking remains limited by
workflow-helper errors previously reproduced on the base and
installed-dependency type mismatches encountered while regenerating core
declarations.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## ELI5

On Mastra Platform, storage exporters are normally turned off to prevent
observability data from going to local storage. This change lets each
observability configuration opt in to remote storage export, while still
blocking storage known to be local.

## Changes

- Added `retainStorageExporterOnPlatform` to observability
configuration. It defaults to `false` and applies to exporters
registered during setup or later.
- When enabled on Platform, the option retains `MastraStorageExporter`
but does not enable the deprecated `DefaultExporter`.
- Added `persistenceLocation` to observability storage. The base
implementation returns `undefined`; in-memory storage and DuckDB report
`local`; libSQL reports `local`, `remote`, or `undefined` based on its
client configuration.
- On Platform, `MastraStorageExporter` skips storage that reports
`local` and warns during initialization. Adapters without location
metadata retain the opt-in behavior.
- Added documentation and changesets for observability, core, libSQL,
and DuckDB.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-10 11:56:44 +01:00
dane-ai-mastra[bot]
675caf0613
chore: version packages (alpha) (#29336)
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases

> The changelog information of each package has been omitted from this
message, as the content exceeds the size limit.

## @mastra/client-js@1.60.0-alpha.0


## @mastra/react@1.15.0-alpha.0


## @mastra/factory@0.28.0-alpha.0


## @mastra/code-sdk@1.19.0-alpha.0


## mastra@1.39.0-alpha.0


## @mastra/core@1.83.0-alpha.0


## @mastra/mcp@2.9.0-alpha.0


## @mastra/mcp-apps-components@0.1.0-alpha.0


## @mastra/playground-ui@69.0.0-alpha.0


## @mastra/server@1.83.0-alpha.0


## @mastra/nestjs@0.3.0-alpha.0


## @mastra/clickhouse@1.30.0-alpha.0


## @mastra/duckdb@1.19.0-alpha.0


## @mastra/libsql@1.30.0-alpha.0


## @mastra/pg@1.36.0-alpha.0


## @mastra/temporal@0.6.0-alpha.0


## @mastra/auth-better-auth@1.1.11-alpha.0


## @mastra/deployer-cloud@1.83.0-alpha.0


## @mastra/deployer-cloudflare@1.2.42-alpha.0


## @mastra/deployer-netlify@1.2.42-alpha.0


## @mastra/deployer-sandbox@0.3.27-alpha.0


## @mastra/deployer-vercel@1.2.42-alpha.0


## @mastra/longmemeval@1.1.42-alpha.0


## @mastra/opencode@0.1.42-alpha.0


## create-factory@0.4.3-alpha.0


## mastracode@0.49.1-alpha.0


## @mastra/arize@1.3.30-alpha.0


## @mastra/arthur@0.4.30-alpha.0


## @mastra/braintrust@1.3.27-alpha.0


## @mastra/datadog@1.4.22-alpha.0


## @mastra/deepeval@0.1.24-alpha.0


## @mastra/laminar@1.3.32-alpha.0


## @mastra/langfuse@1.6.6-alpha.0


## @mastra/langsmith@1.4.3-alpha.0


## @mastra/observability@1.23.2-alpha.0


## @mastra/otel-bridge@1.5.22-alpha.0


## @mastra/otel-exporter@1.5.2-alpha.0


## @mastra/posthog@1.4.7-alpha.0


## @mastra/sentry@1.2.32-alpha.0


## @mastra/agent-builder@1.2.1-alpha.0


## @mastra/connect@1.3.2-alpha.0


## create-mastra@1.39.0-alpha.0


## @mastra/deployer@1.83.0-alpha.0


## @mastra/editor@0.17.1-alpha.0


## @mastra/mcp-docs-server@1.3.13-alpha.0


## @mastra/memory@1.41.1-alpha.0


## @mastra/rag@2.7.3-alpha.0


## @mastra/redis-streams@0.8.1-alpha.0


## @mastra/elysia@0.1.23-alpha.0


## @mastra/express@1.5.27-alpha.0


## @mastra/fastify@1.5.27-alpha.0


## @mastra/hono@1.7.25-alpha.0


## @mastra/koa@1.7.27-alpha.0


## @mastra/next@0.2.41-alpha.0


## @mastra/tanstack-start@0.2.41-alpha.0


## @mastra/spanner@1.11.2-alpha.0


## @mastra/turso@0.1.21-alpha.0


## @mastra/inngest@1.11.2-alpha.0


## @mastra/gcs@0.3.6-alpha.0


## @internal/playground@1.39.0-alpha.0

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-11-04 14:40:27 -08:00
Tony Kovanen
4afeceaf2d
feat(pg, libsql): add experimental storage v2 for workflow runs (Postgres and libSQL) (#28299)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: review <review@local>
2026-11-04 21:12:40 +01:00
Arthur Fiorette
3757223067
fix(libsql): stop JSON-parsing text columns on read (#29357) 2026-11-04 01:12:28 -03:00
dane-ai-mastra[bot]
6e8dfcb83f chore: version - exit prerelease mode 2026-11-03 23:45:58 +00:00
dane-ai-mastra[bot]
4bc3aa672f
chore: version packages (alpha) (#29255)
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases

> The changelog information of each package has been omitted from this
message, as the content exceeds the size limit.

## @mastra/factory@0.27.0-alpha.3


## @mastra/agent-builder@1.2.0-alpha.3


## @mastra/core@1.82.0-alpha.3


## @mastra/mcp@2.8.0-alpha.2


## @mastra/memory@1.41.0-alpha.3


## @mastra/playground-ui@68.0.0-alpha.3


## @mastra/ring-signals@0.1.0-alpha.0


## @mastra/libsql@1.29.0-alpha.2


## @mastra/mongodb@1.26.0-alpha.2


## @mastra/mysql@0.16.0-alpha.3


## @mastra/pg@1.35.0-alpha.3


## @mastra/discord@1.3.3-alpha.0


## @mastra/slack@1.8.1-alpha.0


## @mastra/teams@0.1.4-alpha.0


## @mastra/telegram@0.2.5-alpha.0


## @mastra/client-js@1.59.0-alpha.3


## @mastra/react@1.14.1-alpha.3


## @mastra/deployer-cloud@1.82.0-alpha.3


## @mastra/longmemeval@1.1.41-alpha.3


## @mastra/opencode@0.1.41-alpha.3


## create-factory@0.4.2-alpha.3


## @mastra/code-sdk@1.18.0-alpha.3


## mastracode@0.49.0-alpha.3


## @mastra/arize@1.3.29-alpha.2


## @mastra/arthur@0.4.29-alpha.2


## @mastra/laminar@1.3.31-alpha.2


## @mastra/langfuse@1.6.5-alpha.2


## @mastra/otel-bridge@1.5.21-alpha.2


## @mastra/otel-exporter@1.5.1-alpha.2


## @mastra/sentry@1.2.31-alpha.3


## mastra@1.38.0-alpha.3


## @mastra/connect@1.3.1-alpha.1


## @mastra/deployer@1.82.0-alpha.3


## @mastra/editor@0.17.0-alpha.3


## @mastra/evals@1.10.11-alpha.0


## @mastra/mcp-docs-server@1.3.12-alpha.4


## @mastra/server@1.82.0-alpha.3


## @mastra/elysia@0.1.22-alpha.3


## @mastra/express@1.5.26-alpha.3


## @mastra/fastify@1.5.26-alpha.3


## @mastra/hono@1.7.24-alpha.3


## @mastra/koa@1.7.26-alpha.3


## @mastra/nestjs@0.2.41-alpha.3


## @mastra/next@0.2.40-alpha.3


## @mastra/tanstack-start@0.2.40-alpha.3


## @mastra/turso@0.1.20-alpha.2


## @mastra/inngest@1.11.1-alpha.2


## @mastra/temporal@0.5.0-alpha.3


## create-mastra@1.38.0-alpha.3


## @internal/playground@1.38.0-alpha.3

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-11-03 13:59:38 -07:00
Tyler Barnes
234d7a3ee6
feat(core,stores): pin listings of the Knowledge page tree from config or knowledge_pin, and #29260 review follow-ups (#29263)
Pin part of the Knowledge page tree into an agent's context, from config
or from the agent itself with `knowledge_pin`, plus follow-ups to #29260
(Knowledge page link index and alpha review follow-ups).

> **Migration step on first start.** Pins now have a kind (`page` or
`list`). Each storage adapter upgrades its pins table once, the first
time it starts with this version. LibSQL rebuilds the table (SQLite
can't drop the old one-pin-per-page constraint in place); Postgres and
MySQL alter it in place; MongoDB backfills `kind: 'page'` and swaps the
unique index. Existing pins become page pins; no reset is needed.

## Pin a listing of the page tree

`pinnedContext` on a scope declaration now takes a listing as well as a
page:

```ts
scopes: {
  'docs:$orgId': { belongsTo: 'org:$orgId', access: [{ principal: 'org:$orgId', role: 'reader' }] },
  'org:$orgId': {
    pinnedContext: [
      'docs:$orgId#getting-started', // a page, as before
      { list: 'docs:$orgId#guides', depth: 2 }, // the pages under guides, two levels down
    ],
  },
},
```

Every run holding `org:<id>` then gets the same slim lines
`knowledge_list` returns (ref, name, child count) for the readable pages
under that point, on every turn, without having to search or list first.
It updates when pages are added, renamed or removed below it.

- A listing is checked at startup like a page pin: the scope type must
be declared and every `$param` bound. `depth` is 1 to 3, defaulting to
1.
- Only pages the run can read are listed, and a run that can't read the
listing's root gets nothing.
- A listing takes one `maxPins` slot and shares
`pinnedContext.maxTokens` with pinned pages. Its size is set by that
token share, not an entry count: it fills a level at a time (every
first-level page while room lasts, then second-level, up to `depth`) and
prints the result as an indented tree, so every listed page's parent is
listed too. When it doesn't fit, it's cut at a whole line and ends with
`…and N more; knowledge_list from: <ref>` (or `…and more` when the count
isn't known). At the default 500 tokens a wide root may show only its
first level; raise `pinnedContext.maxTokens` to see deeper levels.
- Page pins work as before.

To share the tree walk, it moves out of `knowledge_list` into one
function used by both. `knowledge_list` output doesn't change.

## Agents pin listings with `knowledge_pin`

`knowledge_pin` takes a `kind`: `'page'` (the default, as before) or
`'list'`, with an optional `depth` (1 to 3, default 1):

```json
{ "op": "pin", "kind": "list", "depth": 2, "page": "docs:acme#guides", "audience": ["org:acme"] }
```

- A page can hold a page pin and a listing pin at once; each is pinned
and unpinned on its own. Pinning a listing again sets its new depth and
adds audiences.
- Stored listings follow the config-listing rules: only readable pages,
one `maxPins` slot, shared `maxTokens`, ranked with page pins by
`pinnedContextPriority` then recency. A stored listing that repeats a
config listing is shown once.
- Pin rows store `kind` and `depth`, unique per page and kind, in
LibSQL, Postgres, MySQL, MongoDB and in-memory. `listPins` takes a
`kind` filter; Studio's pinned badge still counts page pins only.
- Adapters report the new `supportsPinKinds` capability. With an adapter
from before this release, page pins keep working and pinning a listing
is refused before anything is stored, with a message to upgrade the
storage package.

## Review follow-ups to #29260

- **`knowledge_list` checks child-scope access in core again.** The
store still filters out unreadable child scopes. Core now checks each
scope it gets back, so a custom store that ignores `read` can't expose
private scopes.
- **Fences follow CommonMark.** A fence indented four or more spaces is
indented code, so it no longer hides the headings after it from
sections, outlines and embedding chunks.
- **Chunk embedding respects the model.** A long page's chunks are
embedded one at a time when the embedding model doesn't support parallel
calls, instead of up to 4 at once.
- **Changesets** now lead with the outcome, say "16 distinct words" for
the search limit, and include usage examples for the link count and page
link client methods, server routes and core readers.

## Test plan

- Pin kinds: core tests for listing pins through the tool (8, including
an old-adapter case that stores nothing) fail without the change. Each
adapter has an upgrade test that seeds a pre-kind pins table, starts the
store, and checks the pin survives as a page pin next to a new listing
pin on the same page. Real stores: LibSQL 79, Postgres 89, MySQL 80,
MongoDB 83; shared contract suite 935 on in-memory; server Knowledge 48.
Core Knowledge and storage 303/303.

- New tests fail without each fix: fence indentation, a store that
ignores `read` on child scopes, sequential embedding when parallel calls
aren't supported, and the 11 pinned-listing tests, and a level-at-a-time
test (a root with 45 children and 3 grandchildren each shows all 45
children, then grandchildren as they fit; on the old 50-entry cap it
showed 5).
- `@mastra/core` Knowledge and storage tests: 295/295 on main
01524b22db. `tsc --noEmit` clean, `oxlint src/knowledge` 0/0, `oxfmt`
clean.

---------

Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com>
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-03 11:43:18 -07:00
Tyler Barnes
01524b22db
feat(core,stores,server): Knowledge page link index and alpha review follow-ups (#29260)
## Summary

Follow-ups to the Knowledge alpha (#29159, #29207, #29218, #29256), in
one PR:

**Page link index**
- Knowledge keeps an index of `[[links]]` in page bodies.
`countPageLinks` / `listPageLinks` on the Knowledge store return in and
out links for a set of pages, gated by the reader's scopes like every
other read; note `[[mentions]]` show up as `note` links.
- Links resolve when a page is written through the tools, `upsertPage`
or `removePage` (id, `scope#key/key` path, scope address, then page name
in the page's own scope, then store-wide). Dangling links resolve once
their target is created and follow renames, moves, merges and deletes.
`knowledge.reindexLinks()` backfills pages written before this release.
- Implemented in the in-memory, LibSQL, PostgreSQL, MySQL and MongoDB
stores (additive table/collection, no schema bump). Refs over 512
characters stay plain text; the per-request link cap comes from core
through the store loader.
- Server routes `GET /knowledge/:knowledgeKey/link-counts` and `GET
/knowledge/:knowledgeKey/pages/:pageId/links`, client-js methods, and
FGA coverage (`knowledge:read` on the knowledge key, denied caller gets
403).

**Review follow-ups**
- Fenced code and headings parse like CommonMark (closing fence must
match the opener; `## Use C#` keeps its `#`); reading the last part of a
page that ends in a newline works.
- Chunk embedding runs 4 at a time and semantic search stops at its
candidate cap.
- PostgreSQL and MongoDB refuse to remove a note that's already removed,
like the other stores.
- Search queries use at most 16 words.
- Changeset for the `reindex()` removal scoped correctly.

## Child scopes filtered by reader in the store
- `listChildScopes` takes an optional `read` scope and leaves out child
scopes the reader can't see before applying the limit and cursor. New
`countChildScopes` counts readable child scopes for many parents in one
call. All five stores implement both.
- `knowledge_list` uses them: it used to page through every child scope
under a scope and drop the unreadable ones afterwards, and list each
scope's child scopes just to count them. Under a scope with many private
child scopes that meant loading all of them.
- This changes cost only, not access: `knowledge_list` already removed
unreadable scopes before returning anything, so agents see the same
entries and counts. Without `read`, the host's full list (Studio, link
resolution, `knowledge_read`'s restricted entries) is unchanged.

## Test plan
- Core Knowledge + storage: 274 tests pass; tsc 0; oxlint 0/0; oxfmt
clean.
- Real stores (Docker): LibSQL 77, PostgreSQL 87, MySQL 78, MongoDB 81
pass.
- Server Knowledge handlers + adapter FGA: 143 pass. client-js
Knowledge: 6 pass.
- New tests fail without their fixes (link hooks, host-write links, ref
length, fences/headings, candidate cap, embed concurrency, double note
removal, FGA route mapping, child scopes filtered before the limit,
knowledge_list never receiving unreadable scopes).



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

Knowledge pages can now keep track of links to other pages, including
when those pages are added, renamed, moved, or removed. Readers can see
only links they are allowed to access, and page listings avoid loading
child scopes they cannot read.

## What changed

- Added a page-link index for `[[links]]` across in-memory, LibSQL,
PostgreSQL, MySQL, and MongoDB stores. `knowledge.reindexLinks()` can
backfill the index for existing pages.
- Added scope-filtered link counts and link-list APIs, server routes,
and client-js methods.
- Added child-scope filtering before pagination and child-scope counts
for readable scopes.
- Improved Markdown fence and heading parsing, page-range handling for
trailing newlines, embedding concurrency, and semantic-search candidate
limits.
- Made repeated removal of an already-removed note return
`KnowledgeNotFoundError` in PostgreSQL and MongoDB. Updated the
`reindex()` changeset.

## Validation

Author-reported test and tooling results are available in the PR
objectives; they were not independently verified here. Review finding
counts are unavailable.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com>
2026-11-03 01:35:34 -07:00
dane-ai-mastra[bot]
886679eb10
chore: version packages (alpha) (#29205)
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases

> The changelog information of each package has been omitted from this
message, as the content exceeds the size limit.

## @mastra/client-js@1.59.0-alpha.2


## @mastra/factory@0.27.0-alpha.2


## @mastra/code-sdk@1.18.0-alpha.2


## mastra@1.38.0-alpha.2


## @mastra/core@1.82.0-alpha.2


## @mastra/editor@0.17.0-alpha.2


## @mastra/server@1.82.0-alpha.2


## @mastra/clickhouse@1.29.0-alpha.1


## @mastra/duckdb@1.18.0-alpha.1


## @mastra/pg@1.35.0-alpha.2


## @mastra/platform-workspace@1.12.0-alpha.1


## @mastra/react@1.14.1-alpha.2


## @mastra/deployer-cloud@1.82.0-alpha.2


## @mastra/deployer-sandbox@0.3.26-alpha.1


## @mastra/longmemeval@1.1.41-alpha.2


## @mastra/opencode@0.1.41-alpha.2


## @mastra/parallel@0.1.5-alpha.0


## create-factory@0.4.2-alpha.2


## mastracode@0.49.0-alpha.2


## @mastra/arize@1.3.29-alpha.1


## @mastra/arthur@0.4.29-alpha.1


## @mastra/braintrust@1.3.26-alpha.1


## @mastra/datadog@1.4.21-alpha.1


## @mastra/deepeval@0.1.23-alpha.1


## @mastra/laminar@1.3.31-alpha.1


## @mastra/langfuse@1.6.5-alpha.1


## @mastra/langsmith@1.4.2-alpha.1


## @mastra/observability@1.23.1-alpha.1


## @mastra/otel-bridge@1.5.21-alpha.1


## @mastra/otel-exporter@1.5.1-alpha.1


## @mastra/posthog@1.4.6-alpha.1


## @mastra/sentry@1.2.31-alpha.2


## @mastra/agent-builder@1.1.33-alpha.2


## @mastra/deployer@1.82.0-alpha.2


## @mastra/mcp@2.8.0-alpha.1


## @mastra/mcp-docs-server@1.3.12-alpha.3


## @mastra/memory@1.41.0-alpha.2


## @mastra/playground-ui@68.0.0-alpha.2


## @mastra/rag@2.7.2-alpha.0


## @mastra/elysia@0.1.22-alpha.2


## @mastra/express@1.5.26-alpha.2


## @mastra/fastify@1.5.26-alpha.2


## @mastra/hono@1.7.24-alpha.2


## @mastra/koa@1.7.26-alpha.2


## @mastra/nestjs@0.2.41-alpha.2


## @mastra/next@0.2.40-alpha.2


## @mastra/tanstack-start@0.2.40-alpha.2


## @mastra/dsql@1.5.6-alpha.0


## @mastra/libsql@1.29.0-alpha.1


## @mastra/mongodb@1.26.0-alpha.1


## @mastra/mssql@1.11.1-alpha.0


## @mastra/mysql@0.16.0-alpha.2


## @mastra/oracledb@0.5.6-alpha.0


## @mastra/spanner@1.11.1-alpha.0


## @mastra/turso@0.1.20-alpha.1


## @mastra/voice-google@0.14.5-alpha.0


## @mastra/inngest@1.11.1-alpha.1


## @mastra/temporal@0.5.0-alpha.2


## @mastra/e2b@0.16.1-alpha.1


## create-mastra@1.38.0-alpha.2


## @internal/playground@1.38.0-alpha.2

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-11-02 20:29:47 -07:00
Tyler Barnes
f21823d204
feat(core): browse Knowledge with knowledge_list and read long pages in parts (#29207)
Agents using Knowledge could only find pages by searching. Results named
nested pages by their last path segment, which `knowledge_read` couldn't
resolve, and long pages came back in one piece, sometimes larger than
the useful answer. This PR lets agents browse the page tree, read every
page a result names, and read long pages a part at a time. Every result
says what it left out and how to get it.

Stacked on #29159 (Knowledge host writes for pages and notes). Review
that first; this PR's diff is against its branch.

## What changes

**Full page refs in results.** Search, read, list and page-write results
carry `ref`, the page's full readable ref:
`site:mastra#models/providers/anthropic`, a scope address, or the page
id when the page can't be named by path. The old last-segment `key`
field is gone. A read that misses names up to 3 close matches by ref.

**`knowledge_list`**, a new read tool:

```ts
knowledge_list({ from?: string, depth?: 1-3 /* default 1 */, limit?: 1-100 /* default 50 */, descriptions?: boolean, cursor?: string })
// → { from?: { ref, name },
//     entries: [{ ref, name, description?, depth, children, scope?: true }],  // description only with descriptions: true
//     nextCursor?, more?: string[] }
```

Without `from` it lists the scopes the run can read. A result stops at
`limit` entries or about 1,000 tokens, whichever comes first. When it
stops early, `more` says how many entries it left out and gives a
`cursor` that continues exactly where it stopped, at any depth.
Unreadable scopes and pages are left out, so the list doesn't reveal
they exist. `children` is a count, from a new `countChildPages` store
method implemented in all five stores (in-memory, LibSQL, Postgres,
MySQL, MongoDB) with a shared contract test.

**Path slashes.** Leading, trailing and doubled slashes in a page path
are dropped wherever a path is parsed: `upsertPage`, `removePage`, the
note methods, `[[links]]`, tool refs and `pinnedContext`. `docs/` and
`docs` are the same page.

**Reading long pages by line.** A body that fits in one read (1,000
tokens by default) comes back whole as `body`. A longer one comes back
as numbered `lines` (`12│text`) with the `range` they cover and
`lineCount`, ending at the last whole line that fits; `more` says where
to go on with `from: { line }`. The first read of a long page also lists
the headings below what it shows, with their line numbers and sizes, so
an agent can go straight to the section it needs. A line too long for
one read, such as a wide table row, is read by column (`from: { line,
col }`). Also:
- `outline`: the body's headings with their line numbers and sizes
- `section`: one section by heading, as numbered lines
- `match`: one or more terms, like `grep -C`: only the lines that
contain them, with `context` lines around each hit (default 2)

These part reads return only that part, not the notes and children
again.

**Notes and backlinks only when small.** A whole-page read keeps the
page's notes, and the notes on other pages that mention it, while each
list stays under 1,000 tokens. A longer list is left out, counted in
`more`, and read on its own with `notes: true` or `mentionedIn: true`. A
read with `cursor` or `mentionedInCursor` returns only those notes and
doesn't repeat the body.

**Editing long pages by line.** `knowledge_page` takes a `range` from a
read, with `expectedVersion`, and replaces only those lines; an empty
`body` removes them. A range may be the one a read returned or smaller,
but never wider than one read from its first line shows, so a writer
can't overwrite text it never saw. A body write without a range is
refused on a page too long for one read, and so is a body that still
carries the read's line numbers.

**Configurable read size.** `tools: { read: { maxTokens: 3000 }, write:
true }` sets how many estimated tokens one `knowledge_read` body part or
match result, and one `knowledge_list` result, returns (default 1,000,
at least 100). Range writes follow the same size.

**Search that points somewhere.** `knowledge_search` matches quoted
words (`"api" "responses"`) instead of treating the quote marks as text.
An empty result says no page has every word and suggests fewer key words
or `knowledge_list`. Passing a page ref as `scope` says it is a page,
not a scope, and how to list under it.

**Saying what was cut.** Read, search and list results list everything
they cut short in `more`, with the exact call that gets the rest.

The default instructions mention `knowledge_list` and part reads, and
tell agents to follow `more` only when what they need isn't already in
what they have read. They don't tell agents to always list first.

## Proof

Model runs on a copy of mastra.ai loaded into Knowledge (1,707 pages,
scope `site:mastra`, keyword search only): DeepSeek Flash, temperature
0, 12-step cap, 3 fresh runs per question, at `c4495284d46` (this head
minus the configurable cap, which leaves the default unchanged).

| Question | Result | Calls / input tokens per run |
|---|---|---|
| A fact in the "Select the OpenAI Responses API" section of the
589-line models page | 2 of 3 answered (the third ran out of steps
searching) | 18/160k, 13/128k, 7/35k |
| A fact 27,700 chars into the 38,820-char podcast page | 3 of 3 |
6/54k, 4/36k, 4/18k |
| Name the observability guides (6) | 6 of 6 on all 3 runs | 20/208k,
10/61k, 16/139k |
| Anthropic provider options (17 facts), at an earlier commit | 17/17
each run | 5/8/5 calls, 37–70k chars of tool results (102–221k before) |

Direct tool calls on the same data (no model):

| Read | Before | After |
|---|---|---|
| `match` "Steinberger" on the 38k-char podcast page | 20,800 chars |
3,457 |
| First read of `site:mastra#models` | 52,561 chars | 6,979 (lines 1–109
and the headings below, notes and backlinks counted) |
| Section "Select the OpenAI Responses API" | — | 1,368 |

## Test plan

- [x] `pnpm test:core` Knowledge and storage suites: 236 pass. New tests
fail on the base for refs that read back, slash paths, `knowledge_list`
paging (by count and by tokens, each entry once at depth 2) and access,
line reads and grep match, range writes (refusing whole-body,
one-line-too-wide, shifted and copied-line-number writes), leaving large
notes out, the heading list on a first long read, search hints, and
`tools.read.maxTokens` (reads, lists and the write guard follow it;
invalid values refused).
- [x] Shared store contract (including `countChildPages`) on real
stores: LibSQL 66, Postgres 76, MySQL 67, MongoDB 70
- [x] Type check, oxlint and oxfmt clean



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## ELI5

Knowledge pages are easier to find, browse, and read in useful pieces.
The update also makes page references clearer and helps prevent partial
edits from changing text that was not read.

## Summary

- Adds `knowledge_list` to browse readable scopes and pages. It supports
depth limits, descriptions, child counts, and cursor-based pagination.
- Uses full page `ref` values in search, read, list, and write results.
Failed reads can suggest close matches.
- Adds token-bounded reads with numbered line ranges, outlines,
sections, and matching lines with context. Range edits require the read
range and version; whole-body writes are blocked when the page exceeds
the read limit.
- Reports omitted notes, backlinks, or page content and explains how to
retrieve more.
- Normalizes leading, trailing, and repeated slashes in page paths.
- Updates search to match any query word and rank results by phrase
match, matched-word count, field weight, and recency.
- Adds `tools.read.maxTokens` to configure read and list result size,
and updates Knowledge instructions to guide browsing and partial reads.
- Adds `countChildPages` support to the in-memory, LibSQL, Postgres,
MySQL, and MongoDB stores.

## Validation

The PR description reports passing Knowledge and storage tests, clean
type checking, oxlint, and oxfmt, plus model runs and direct tool-call
comparisons. Review finding counts were not provided.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com>
2026-11-02 20:09:31 -07:00
Ehindero Israel
c7d27f650b
feat(editor): make model fallback lists editable and persistent (#27287)
Adds opt-in Editor ownership for static model fallback lists on
code-defined agents.

```ts
new Agent({
  id: 'support-agent',
  model: [
    { id: 'primary', model: 'openai/gpt-5.4-mini' },
    { id: 'fallback', model: 'anthropic/claude-sonnet-4-5' },
  ],
  editor: { model: true },
})
```

Studio can reorder models, change provider/model selections, and enable
or disable entries. These changes follow the configured Editor
persistence workflow:

- Database-backed Editor storage creates versioned drafts that can be
published.
- Code-backed Editor storage saves overrides to the filesystem or
through the configured pull-request workflow.

Persisted entries include their order, provider, model ID, enabled
state, and `maxRetries`. Immutable model IDs reconnect stored values to
code-defined entries while preserving non-serializable code
configuration.

Function-valued model configurations remain code-owned and read-only in
Studio.

This also adds storage support, model-list persistence through the
existing server mutation routes, generated client types, regression
coverage, documentation, and a changeset.

Closes #27089.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

This lets teams edit and save an agent’s model fallback list in Studio
without changing code. The feature is opt-in, so existing agents keep
their current behavior by default.

## Summary

- Added `editor: { model: true }` support for code-defined agents.
- Added persistent `modelList` storage with entry IDs, provider, model
ID, enabled state, and `maxRetries`.
- Applied stored overrides by matching immutable entry IDs while
preserving code-owned configuration.
- Persisted reorder and update actions through Editor drafts and
publishing workflows.
- Added filesystem, LibSQL, server route, export, versioning, and client
cache support.
- Kept function-valued model configurations code-owned and read-only in
Studio.
- Updated ownership logic, API schemas, client behavior, and
documentation.
- Added regression coverage for runtime overrides, persistence, exports,
ownership, draft activation, storage compatibility, and query
invalidation.
- Added changesets for the affected packages.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-02 18:53:39 +01:00
dane-ai-mastra[bot]
eed9cc214e chore: version packages 2026-11-01 04:20:58 +00:00
Tyler Barnes
0a553f0f6f
Knowledge alpha: standalone Knowledge for agents, Subconscious removed (#28353)
## What this is

**Knowledge** is Mastra's store for durable knowledge that agents share
across conversations. Until now its agent-facing parts lived in
`@mastra/memory`, inside an experimental feature called
**Subconscious**, a background curator that wrote to Knowledge after
observational-memory reflections.

This PR makes Knowledge stand on its own as an experimental alpha,
rebuilt around pages and declared scopes.

- **Subconscious is removed** from `@mastra/memory` and Mastra Code.
Setting `experimental_subconscious` or the Memory `knowledge` option
throws, saying to attach Knowledge to the agent instead. Old
Subconscious data is ignored.
- **Knowledge attaches to an agent directly:** `new Agent({ knowledge,
scopes })`. It brings its own tools, instructions and pinned context.
Tools work without Memory; pinned context needs Memory.
- **Knowledge is a tree of pages.** Each page lives in one scope and
holds a body plus short notes. Notes can mention other pages with
`[[wikilinks]]`. A page can have a key, unique among its siblings, and
is addressed by key path: `<scope>#<key>/<child key>`. Pages declared in
a scope's config are created when missing and can be edited but not
moved, deleted or merged away by tools.
- **Access comes from declared scopes.** Agent `scopes` are plain
`"<type>:<value>"` strings that say who the run is, set by the host and
never by the model. Knowledge's required `scopes` option declares which
scope types exist and which roles (viewer, contributor, editor, owner)
holders get on which scopes. Rules reach one hop with no inheritance. A
run holding an undeclared scope fails. `knowledgeMemoryScopes()`
declares Memory's `thread:`/`resource:` scopes and an `org:` scope.
- **Five tools:** `knowledge_search`, `knowledge_read`, `knowledge_page`
(write, move, merge, delete), `knowledge_note` (add, restamp, remove)
and `knowledge_pin` (pin, unpin). Tools are off by default: `tools:
true` enables read and write, `tools: { read: true }` read only, and
`pinnedContext` adds pins. Deleting a page needs owner on the page's
scope and is refused for pages with child pages, pages declared in a
scope's config, and scope pages. On a scope page, write accepts the body
only: its name and description come from config.
- **Limits are in tokens:** about 250 per note and 100 per page
description. Pinned context defaults to 500 tokens, with a maximum of
2,000.
- **Search:** keyword search by default: every word of the query must
appear in the page (or note), and name matches rank first, the same in
every store. With a `vector` store and an `embedder`, search also
returns semantic matches. Every semantic hit is re-checked against what
the run can read before ranking, so hidden content never takes a result
slot. `knowledge.reindex()` re-embeds everything.
- **Host APIs:** `knowledge.readScopeFor(principals)` returns what
holders of those scopes can read. `knowledge.explainAccess(runScopes,
target)` says why a run can or can't reach a scope or page.
`explainAccess` is host-only.
- **Renamed scope declarations:** if stored scopes use a declaration
name the config no longer declares, Knowledge fails loudly on first use,
naming each orphaned name and how many scopes use it.
`knowledge.renameScopeDeclaration(from, to)` renames them. It is
host-only and atomic, except on MongoDB without a replica set.
- **Per-message scopes:** `session.sendMessage({ content, scopes })`,
plus queue, steer and notifications. A message whose scopes differ from
the running run's waits for that run to end and then starts its own run.
Scopes are set in-process only.
- **Storage:** LibSQL, PostgreSQL, MySQL and MongoDB store Knowledge as
schema 4 (pages, notes, mentions, pins). On startup, the tables of the
earlier experimental Knowledge node store (shipped in 1.79.0) are
dropped with any rows in them, and the new tables are created. If an
older process recreates them during a rolling deploy, the next startup
drops them again. Schema-4 tables and non-Knowledge tables are never
touched. On MongoDB, Knowledge collections found without a schema
version are adopted as schema 4 rather than refused. Don't run
`dangerouslyReset()` from an older version against a database the new
version already uses. Development databases created from earlier commits
of this branch also need a reset, since sibling key uniqueness changed
within schema 4. `storage.stores?.knowledge?.dangerouslyReset()` clears
Knowledge rows only, never Memory. Adapter peer floors are unchanged: on
an older `@mastra/core` the adapters still load and the other storage
domains keep working.
- **Mastra Code:** Knowledge is opt-in through **Experimental
Knowledge** in `/settings`. The `/knowledge` browser and the SDK
inspector read pages. Factory's Knowledge routes are disabled until the
Studio page readers land.

## Test plan

- Shared schema-4 storage contract suite on real stores (as of
d5b72de125): LibSQL 59/59, MySQL 60/60, PostgreSQL 69/69, MongoDB
61/61. LibSQL Knowledge index suite 58/58 at 5b137a7677, including
startup over a published 1.79.0 layout (empty and with rows), old tables
recreated under a schema-4 database, and four concurrent startups. On
MySQL, concurrent scope materialization can deadlock on InnoDB gap
locks; it retries up to 5 times on duplicate-key or deadlock errors (31
consecutive passing runs of the concurrency test).
- Core Knowledge suites: declarations, access, page tools for the spec's
scenarios 1–13, pins, token limits, `explainAccess` agreement with the
tools (8 actors × 12 pages), and semantic search (25 hidden
higher-scoring pages never hide the visible result).
- Integration on real LibSQL and PostgreSQL: an agent writes a page,
note and pin. An outsider can't read them. They persist across a restart
and pinned context is delivered. On startup the 1.79.0 node-store tables
are dropped, rows included, while schema-4 and non-Knowledge tables are
left alone.
- Independent validation on real LibSQL: scenarios 1–13 including 6 and
11 with a real agent, and live semantic search through OpenRouter
`openai/text-embedding-3-small` with LibSQLVector, isolated per tenant.
- Mastra Code: inspector, TUI browser and Knowledge-browser E2E.
- `tsc`, `oxlint` and `oxfmt` clean on touched packages. Each milestone
was reviewed independently.

---------

Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com>
Co-authored-by: avis-reader[bot] <332681745+avis-reader[bot]@users.noreply.github.com>
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-30 22:24:20 -07:00
dane-ai-mastra[bot]
77076a1012 chore: version - exit prerelease mode 2026-10-30 23:02:25 +00:00
dane-ai-mastra[bot]
582ef17a2b
chore: version packages (alpha) (#28888)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-10-30 14:56:53 -07:00
mastra-platform[bot]
e02c876685
docs: point store READMEs at the new integrations vector pages (#28970)
Co-authored-by: LekoArts <lekoarts@gmail.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: LekoArts <16143594+LekoArts@users.noreply.github.com>
2026-10-30 11:31:23 +01:00
Daniel Lew
b79e61a92d
feat(core): add expectedSuspendedAt write fence for background tasks (#28590)
> Stacked on #28425 → #28507. This diff is only the change on top of
`daniel/mcp-tasks-3b2-pinned-pending`.

## Why

MCP tasks (#28521) keep a suspended task's open questions and answers in
the background-task row's `metadata`. `tasks/update` reads the row, adds
answers, and writes the whole metadata back fenced on `{ expectedStatus:
'suspended' }`.

If a tool suspends twice (question K1, then K2), a late `tasks/update`
for K1 that lands after the tool resumed and suspended again on K2 still
passes that fence: the status is `suspended` again, and owner and lease
are cleared on suspend, so they don't tell the two suspensions apart.
The stale write replaces K2's questions, and the tool resumes with K1's
answer as the answer to K2.

`suspendedAt` is the field that differs between two suspensions, and the
check has to happen inside the storage write.

## What

- `UpdateBackgroundTaskOptions` gains `expectedSuspendedAt?: Date |
null`, with the same semantics as `expectedLeaseExpiresAt` (`null` means
the row has no suspension time).
- The in-memory store and all 12 adapters implement it by mirroring
their existing lease fence, at the same atomicity level (single
conditional `UPDATE`, Lua CAS, Convex mutation, or DynamoDB condition).
Adapter-specific notes:
- **pg**: fences on the timezone-aware `suspendedAtZ` column, for the
same reason as the lease fence.
- **lance**: an unset `suspendedAt` is stored as `0`, so `null` matches
`IS NULL OR = 0`.
- **dynamodb**: `null` uses `notExists`, and the new condition counts
toward conditional-check failures returning `false`.
  - **upstash**: adds an extra ARGV pair to the Lua script.
- The manager now uses the fence when it takes a resume event
(c0f6544ee1). Before, a late `task.resume` event queued for an earlier
suspension could resume the task's next suspension with the old answer:
- `#publishResume` adds `suspendedAt: suspensionToken(task)` to the
`task.resume` payload.
- `handleResume` drops an event whose token doesn't match the task's
current suspension (the suspension it answered was already resumed),
then claims the row with `expectedSuspendedAt` as well as
`expectedStatus: 'suspended'`, so a re-suspension between its read and
its claim also loses.
- **Rolling deploys:** an event without `suspendedAt`, published by a
worker on an older version, falls back to the previous status-only
check, so mixed-version fleets keep resuming tasks.

## Tests

New `suspension fence` cases in the shared storage suite
(`stores/_test-utils`), written around the MCP scenario:
1. An answer for an earlier suspension is rejected after the task
resumes and suspends again, and the row keeps K2.
2. An answer for the current suspension is applied.
3. A `null` fence matches a task that was never suspended.
4. A `null` fence is rejected against a suspended task.
5. A `null` fence matches once the resume write clears `suspendedAt`.

New `stale resume events` cases in `manager-owners.test.ts` (real
`InMemoryStore`, two managers):
1. A late portable resume doesn't answer the next suspension.
2. A late process-affine resume doesn't answer the next suspension.
3. A resume is rejected when the task suspends again between its read
and its claim (storage wrapper re-suspends right after the read).

Reverting the `manager.ts` change fails all three. Core and storage
background-task suites: 229 passed.

To check the storage cases catch a missing fence, I turned off the
in-memory fence. Cases 1 and 4 then failed on their assertions.

| Adapter | Shared background-tasks suite |
|---|---|
| in-memory, libsql, lance, cloudflare (DO), cloudflare-d1 | ✅ ran
locally |
| pg, mysql, mssql, mongodb, dynamodb, upstash, spanner | ✅ ran against
their Docker setups |
| convex | ⚠️ not run: needs `CONVEX_TEST_URL` / admin key for a live
deployment |

`@mastra/core` typecheck, `tsc` on every touched store (no new errors),
and oxlint on the touched files are clean.



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

A delayed “resume” message could affect a task after it had already been
suspended again. This change lets the system check which suspension an
update belongs to, so old messages cannot change a newer suspension.

## Summary

- Added optional `expectedSuspendedAt` to background-task updates. The
in-memory store and 12 storage adapters use it as a conditional write
fence.
- Resume events now include `suspendedAt`. The manager rejects events
for a different suspension and claims matching tasks with both
`expectedStatus` and `expectedSuspendedAt`. Events without `suspendedAt`
retain the status-only check for rolling deployments.
- Added `pinToWorker` enqueue options. Pinned tasks stay with their
owner while its lease is active. Expired leases can be recovered with
retry handling.
- Added tests for suspension fences, stale resume events, and
worker-pinned task behavior.

## Reported validation

The author reports 229 core and storage background-task tests passed.
The five shared suspension-fence cases ran locally for in-memory,
libsql, Lance, Cloudflare DO, and Cloudflare D1, and against Docker
setups for PostgreSQL, MySQL, MSSQL, MongoDB, DynamoDB, Upstash, and
Spanner. Convex was not run because it requires a live deployment and
credentials.

The author also reports a clean `@mastra/core` typecheck, no new errors
from `tsc` on touched stores, and clean oxlint results on touched files.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-29 21:30:26 -04:00
Daniel Lew
292dd6cd97
feat(core): pin pending background tasks to the enqueuing worker (#28507)
## Summary

Background tasks can now be pinned to the worker that enqueued them, so
a queued task whose work is already running in one process isn't started
again by another process.

**Problem.** An MCP `auto` task adopts a tool call that is already
running inside the request on instance A. A enqueues a `pending` row for
it, but a `pending` row has no owner or lease until A claims it. If A is
at capacity the row waits locally, and instance B's stale-task recovery
(or a shared-topic dispatch) can claim it and run the tool a second time
with its static executor.

**Change.** `backgroundTaskManager.enqueue(payload, context, {
pinToWorker: true })`:

- writes the `pending` row with this worker as owner and an execution
lease, renewed by the existing lease heartbeat while the row waits for a
slot;
- other workers skip the row in recovery and shared-topic dispatch while
the lease is live;
- the owning worker claims its own pinned row fenced on owner only, so a
concurrent heartbeat renewal can't strand it;
- if the owner dies (lease lapses) or shuts down, the row is treated as
a **lost attempt**, like a `running` row whose worker died: it consumes
a retry and is re-dispatched if `retryCount < maxRetries`, otherwise it
fails with `Worker process terminated before task completed`. A task
with `maxRetries: 0` never runs on a peer.

**Graceful shutdown.** On shutdown a worker expires the lease on its
pinned pending rows (fenced on owner) but keeps itself as owner, rather
than clearing the pin. Peers recover the row immediately instead of
waiting for the lease to lapse, and still treat it as a lost attempt:
the adopted work died with the process, so a `maxRetries: 0` task fails
rather than reruns. Clearing the owner would make the row look like
fresh work and let a peer run it for free.

Pinning is opt-in. Every agent background task has a task context, so
pinning by default would change how agent pending rows are stored and
recovered; agent behaviour is unchanged. `pinToWorker` without a task
context throws.

No storage adapter changes: every adapter already stores
`ownerId`/`leaseExpiresAt` on create and supports status/owner/lease
fences. The shared adapter suite now proves this for pinned pending
rows.

Based on #28425. Consumed by the MCP tasks background-store PR (3c).

## Test plan

- `packages/core/src/background-tasks/manager-owners.test.ts`: two
managers on shared storage and pubsub — a pinned row on a busy worker is
never run by a peer and runs once on its owner; owner crash with
`maxRetries: 1` runs it once on the peer (`retryCount` 1); owner crash
and graceful shutdown with `maxRetries: 0` fail it without running
anywhere; default enqueue stays unpinned; `pinToWorker` without context
throws. 5 of the new tests fail without the fix.
- `stores/libsql/src/storage/background-tasks-pinned.test.ts`: the same
A/B scenarios across two Mastra instances sharing one libsql file. All 3
fail against core built without the fix.
- Shared storage suite (`execution ownership`) passes on libsql, pg,
mysql, mssql, mongodb, dynamodb, spanner, upstash, lance, cloudflare-d1,
cloudflare DO (convex not run: needs a live deployment).
- Core background-tasks and agent tool-call-step tests: 311 passed.
`@mastra/core` typecheck and oxlint clean.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

A task can wait until its worker has room to run. This change lets
callers keep that waiting task with the worker that created it, so
another worker does not start the same work while the first worker is
still active. If the first worker stops renewing its lease, the task
counts that as a lost attempt and retries or fails.

## Summary

- Add the optional `pinToWorker` setting to
`BackgroundTaskManager.enqueue()`. Pinning requires a task context.
- While a pinned task waits for capacity, its worker renews a lease.
Other workers skip the task while the lease is active.
- When a pin expires, the system consumes a retry if one is available
and a concurrency slot is free. Otherwise, the expired pin remains
pending until a slot is available. If no retries remain, the task fails
with `Worker process terminated before task completed`.
- Periodically scan for expired pins. Queue recovered tasks locally for
another dispatch attempt if publishing fails.
- Document the behavior and add manager, storage ownership, and LibSQL
integration tests.
- Pinning is opt-in. Unpinned tasks retain their existing behavior.

## Validation

The PR objectives report 311 passing core background-task and agent
tool-call-step tests, plus clean `@mastra/core` typecheck and oxlint
runs. Convex was not tested because it requires a live deployment.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-29 19:32:14 -04:00
Daniel Lew
e77a7d910b
feat(core): background tasks for non-agent owners (#28425)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-29 17:14:33 -04:00
dane-ai-mastra[bot]
6a191d46dc chore: version - exit prerelease mode 2026-10-28 13:15:52 +00:00
Damien Schneider
976fde7416
refactor(lint): enable no-else-return at root (#28601)
Enables `no-else-return: error` in the root Oxlint config and removes
186 redundant `else` branches across 111 source/test files. Existing
rule defaults are retained.

Reviewed every code diff for behavior preservation. An independent
TypeScript AST comparison confirms that changes are limited to removing
`else` after returning branches, with the same expressions, statement
order, return values, and `try`/`catch`/`finally` boundaries. All
181,158 identifier occurrences retain their bindings. The recorder
replay path keeps an explicit block: flattening it would shadow the
earlier request `body` with the response `body` and introduce a
temporal-dead-zone error.

Extends the existing monorepo optimized-dependency test to assert that a
raw TypeScript workspace package, including its enum and transitive
dependency, still executes in the production bundle.

Validation: the root rule-only scan passes on 15,273 files with zero
diagnostics; formatting and `git diff --check` pass. Schema
compatibility tests: 1,060 passed, 9 skipped. Recorder runtime tests: 21
passed. The core selection initially passed 3,513 tests; rerunning the
suites requiring local server/cache access passed 185 tests, with one
remaining cache-file assertion also reproduced on unchanged `main`.

Remaining validation limits: four asset-download cases cannot reach
`placehold.co`; the changed-file lint run reports six import-order
errors reproduced on unchanged files from `main`. Deployer unit tests
and the monorepo test could not run because this checkout lacks required
package dependencies (`fs-extra`/workspace links and `get-port`,
respectively). These limits are left visible for CI. No changeset is
included because this is a behavior-preserving lint cleanup.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## ELI5
This PR makes the code easier to read by removing `else` branches that
are not needed after a return. It also enables a rule that catches those
branches and adds a test for loading a raw TypeScript workspace package
in a production bundle.

## Summary
- Enables Oxlint’s `no-else-return` rule at error severity and removes
186 redundant `else` branches across 111 files.
- Extends the monorepo optimized-dependencies test to check a raw
TypeScript workspace package, its enum, and a transitive dependency.
- Updates the deployer’s transpile-package path pattern.
- In workflow event processing, a truthy branch index with no matching
child now falls through to publish a skipped-step end event. Previously,
this case published no event.
- Keeps the recorder replay block explicit to avoid request/response
`body` shadowing and a temporal-dead-zone error.

## Validation
Reported checks passed: the rule-only scan across 15,273 files,
formatting, `git diff --check`, schema compatibility tests (1,060
passed, 9 skipped), recorder runtime tests (21 passed), and an initial
core test selection (3,513 passed).

Some test limits remain. Four asset-download cases could not reach
`placehold.co`. Changed-file lint reported six import-order errors that
also occur on unchanged files from `main`. Deployer unit tests and the
monorepo test could not run because dependencies were unavailable. On
rerun, suites requiring local server/cache access had 185 passes; one
cache-file assertion also reproduced on unchanged `main`.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-28 12:30:27 +01:00
mastra-platform[bot]
31431019dd
fix(core): distinguish unavailable durable recovery checkpoints (#28611)
Fixes COR-1493. An interrupted durable run can leave a valid initial
pending loop row without restartable input when running-checkpoint
persistence is disabled. `recover()` now reports
`DURABLE_AGENT_RECOVER_CHECKPOINT_NOT_AVAILABLE` with actionable
persistence guidance instead of incorrectly reporting snapshot
corruption. The outer loop remains the restart authority;
malformed-input, workflow-kind, agent-identity, and ownership checks are
preserved. Claim-race refusals release ownership without publishing a
terminal error.

Documents the checkpoint source and replay semantics, adds a core patch
changeset, and restores T4 recovery coverage with an offline aimock
scenario rather than provider recordings. Enable `recovery: {
durableAgents: "auto" }` before starting recoverable runs; enabling it
after a crash cannot reconstruct a missing checkpoint.

Validation: 76 focused core tests passed, including malformed-context
and claim-race controls; the real LibSQL SIGKILL recovery test passed
(also 20/20 earlier repetitions); both offline T4 cells passed in five
repeated runs. The regression was proved red against the unfixed
implementation, producing INVALID_SNAPSHOT in the recovery-off cell.
Core and memory typechecks, focused scenario TypeScript compilation,
core lint, LibSQL test lint, and diff checks passed. Three independent
inline reviews found no remaining correctness blockers; their separate
sandboxes could not execute this checkout.

https://linear.app/kepler-crm/issue/COR-1493

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

An interrupted agent run may have a saved record but still lack the
information needed to restart. This change reports that case clearly and
explains that checkpoint saving must be enabled before the run starts.

## Summary

- `DurableAgent.recover()` reports
`DURABLE_AGENT_RECOVER_CHECKPOINT_NOT_AVAILABLE` when an initial pending
loop snapshot has no workflow input.
- Recovery retains its malformed-input, workflow-kind, agent-identity,
and ownership checks. If claim-time revalidation finds the checkpoint
missing, recovery releases the claim without publishing a terminal
error.
- The recovery route returns HTTP 500 for an unavailable checkpoint and
HTTP 409 for an inactive run.
- Documentation explains checkpoint requirements, replay behavior, and
how to enable running-checkpoint persistence before starting a
recoverable run.
- Tests cover recovery refusals, HTTP responses, offline aimock
recovery, and cross-process LibSQL recovery.

## Reported validation

The author reports that 76 focused core tests passed. The author also
reports that the real LibSQL SIGKILL recovery test passed and both
offline T4 cells passed in five repeated runs. Core and memory
typechecks, focused scenario TypeScript compilation, core lint, LibSQL
test lint, and diff checks also reportedly passed.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-27 13:53:18 -05:00
dane-ai-mastra[bot]
ef92d71d94
chore: version packages (alpha) (#28499)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-10-27 13:09:53 -05:00
Daniel Lew
d580530ff3
fix(core): stop background tasks cancelled from another process from running to completion (#28523)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-27 13:24:35 -04:00
mastra-platform[bot]
2447071b11
fix(core): report uncertain scheduler publication outcomes (#28544)
> Opened by @user_01KPS2202RYQR4JAMPVXBB9A1H

Refs COR-1363. This ships the narrowed, no-storage-change approach, not
the superseded durable-dispatch plan. The scheduler commits its claim
before publishing, so rejection or process death can leave an occurrence
unstarted. Replaying an ambiguous publication could instead start it
twice.

## Changes

- Log successful claims and accepted publications with the schedule ID,
run ID, and scheduled fire time. A claim without a later success or
final failure log identifies an unconfirmed publication, subject to
log-flush and claim-to-log crash windows.
- After all three immediate publication attempts fail, report
`publicationOutcome: 'unknown'` and `automaticReplay: false`. Existing
retries use 100 ms and 200 ms backoff and can duplicate accepted events
when acknowledgement is lost.
- Document best-effort delivery for workflow and agent schedules. After
retries are exhausted, that occurrence is not replayed; later recurring
fires continue. Exhausted one-shot cron schedules remain completed,
which describes cadence exhaustion rather than successful execution.
- No replay protocol, public API, schema, migration, or future-fire
blocking. Net production diff against integrated main: +14 / -0 lines.

## Verification

Integrated origin/main at 67b998ea via merge commit ca3abd45 to preserve
published branch history without force pushing. Review fixes are in
44da5e41.

- 53 scheduler tests pass, including all-attempt rejection, 100/200 ms
backoff, exhaustion-only failure reporting, no later replay, continued
recurring fires, and normal-path identity logs.
- Eight real two-process LibSQL kill cases pass against this checkout's
rebuilt core, covering workflow/agent targets, recurring/terminal
schedules, death after committed claim, and death after accepted
publication.
- Restoring origin/main's scheduler implementation makes eight
diagnostic/logging assertions fail while the other 45 tests pass. Crash
cases are compatibility controls, not recovery proofs.
- Core build and `tsc --noEmit`, focused type-aware lint, formatting,
Vale, remark, and MDX checks pass.
- LibSQL `tsc --noEmit` is blocked by JSON-schema
readonly-array/type-inference errors in
`src/__tests__/cross-process/fixtures/foreach-billing-workflow.ts`,
which is unchanged from integrated main. This is not reported as a
passing check.

## Accepted limitations

A crash can still skip an occurrence without a failure record, and logs
are not durable dispatch state. Its publication outcome is unknown.
Durable schedule-level diagnostics are not implemented. `runAt` is
absent from the integrated scheduler; one-shot lifecycle coverage uses
exhausted year-pinned cron and does not claim `runAt` coverage.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## ELI5

The scheduler marks a scheduled occurrence as claimed before it sends
the work. If sending fails or the scheduler stops at the wrong time, the
work may not start, so this change explains the risk and avoids
replaying an occurrence when its outcome is unclear.

## Summary

- Added logs for claimed occurrences and successful publications.
Failure logs include the scheduled fire time and report an unknown
outcome.
- Documented best-effort delivery for workflow and agent schedules.
Publication can be attempted up to three times, but the scheduler does
not replay an occurrence after those attempts fail.
- Clarified that `completed` means a schedule has no remaining
occurrences, not that its target ran successfully. Later fires can still
occur for recurring schedules.
- Added scheduler tests for publication outcomes and LibSQL
cross-process tests for publisher death after a claim or after
publication acceptance.

## Testing

The PR description reports 45 scheduler tests and eight LibSQL
two-process kill cases passing. It also reports passing Core and LibSQL
typechecks, focused type-aware lint, formatting, and MDX checks. These
results were not independently verified.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-27 12:13:50 -05:00
Leo
ab1727d5ca
fix(observability): end spans started by another exporter instance (#28147)
## Summary

Under the `batch-with-updates` and `realtime` tracing strategies, a span
started by one exporter instance couldn't be ended by another.
Durable-agent crash recovery hits this: the recovering process rebuilds
the crashed run's agent and model spans and ends them as interrupted,
but its `MastraStorageExporter` never saw `SPAN_STARTED`, so it deferred
the end until retries ran out and dropped it. The stored span stayed
`RUNNING`. `insert-only` and `event-sourced` aren't affected, since
`SPAN_ENDED` is a create there.

Storage now has an optional conditional update,
`batchUpdateUnendedSpans()`. It changes a row only if the row exists and
`endedAt` is null, never inserts, and returns the spans it changed. The
exporter defers an update only while its own create for that span is
unconfirmed, and sends everything else straight to storage:

```diff
 for each span update or end:
-  if this exporter flushed the span's create: batchUpdateSpans
-  else: defer until the create shows up, drop when retries run out
+  if the store has no batchUpdateUnendedSpans:
+    same as before
+  else if this exporter saw SPAN_STARTED but the create isn't stored yet:
+    defer
+  else:
+    batchUpdateUnendedSpans
+    updates that didn't apply -> drop event, reason span-not-updatable
```

- Ended spans stay immutable. Ending one again doesn't change the row
and is reported as a `span-not-updatable` drop.
- No reads before writes. The `getSpan` lookups from the earlier version
of this PR are gone.
- Implemented for the in-memory store, `@mastra/pg` (one `UPDATE ... AND
"endedAt" IS NULL RETURNING 1` per record in a transaction) and
`@mastra/libsql` (same condition, using `rowsAffected`).
`batchUpdateSpans` is unchanged.
- The exporter checks for the method at runtime, so older core or store
versions keep today's behavior. No peer dependency changes.
- The deprecated `DefaultExporter` isn't changed.

Known limit, out of scope: if the original process crashes before its
create is flushed, there's no row to end.

## Evidence

| Before | After |
|---|---|
| ![Drop reasons
before](https://github.com/user-attachments/assets/9c016917-93f2-44a0-88d9-b7d34ae940fa)
| ![Drop reasons
after](https://github.com/user-attachments/assets/98c96a90-14d0-4420-bf17-82869c36e239)
|

- **Before:** in `MastraStorageExporter spans from another process`
(`observability/mastra/src/exporters/mastra-storage.test.ts`), with the
exporter fix reverted, 6 of 10 cases fail for both strategies. Instance
A starts an agent span and flushes without ending it, instance B
rebuilds and ends it as interrupted, and the stored span keeps `endedAt:
null`.
**After:** all 10 pass. The stored span ends with the interrupted
output. A second end leaves the stored span unchanged and emits one
`span-not-updatable` drop. An end for a span that was never stored
creates nothing.
- Mutation check: sending updates while this exporter's own create is
still being written makes `holds back an end until its own create is
stored` fail under `batch-with-updates`.
- `vitest run src/exporters src/bus` in `@mastra/observability`: 500
passed.
- New `batchUpdateUnendedSpans` cases in the shared storage suite
(`stores/_test-utils`), which run for every adapter that implements the
method: pass on LibSQL and on PostgreSQL 16 in Docker. The full PG
storage file (`src/storage/index.test.ts` plus `src/storage/db`) passed
2316 tests after the `executeUpdate` refactor.
- Typecheck and oxlint pass for the changed packages.

Not tested: other storage adapters (MySQL, MSSQL, MongoDB, ClickHouse
and the rest). They don't implement the method, so they keep the
previous behavior.

## Merge Danger

**Door:** two-way. The new storage method is optional and additive, and
the exporter falls back to the old path when it's missing.

**Blast radius:** span updates and ends from `MastraStorageExporter`
under `batch-with-updates` and `realtime` on in-memory, PG and LibSQL
storage. If the conditional update were wrong, span ends could be
dropped and spans would stay `RUNNING` in Studio.

Fixes #27479

Created with Claude Opus 5.5 in Claude Code.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

One exporter can save a span, and another can finish it later. This
change lets supported storage providers update that saved span without
creating a duplicate or changing one that has already ended.

## Summary

- Added the optional `batchUpdateUnendedSpans` storage method.
In-memory, LibSQL, and PostgreSQL storage implement it to update only
existing spans that have not ended and report which spans changed.
- Updated `MastraStorageExporter` to use conditional updates when a
provider supports them. It reports updates that storage did not apply as
`span-not-updatable`. Providers without the method retain the previous
behavior.
- Added tests for spans restored across exporter instances, duplicate
ends, missing stored spans, creates still in progress, and providers
without conditional updates.
- Documented the new drop reason and added patch changesets for the
affected packages.

## Validation

The supplied PR objectives report 500 passing tests in
`@mastra/observability`, passing shared storage-suite cases for LibSQL
and PostgreSQL 16, and 2,316 passing tests in the PostgreSQL storage
file after the `executeUpdate` refactor. They also report passing
typechecks and oxlint for changed packages. Other storage adapters were
not tested and retain the previous behavior. A crash before the original
span create is flushed remains out of scope.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: 大补 <38815121+AuYuHui@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 16:02:33 +01:00
mastra-platform[bot]
4fc17c65e8
fix(core): preserve remote workflow cancellation (#27904)
> Opened by @abhiaiyer91

Fixes #27805. Supersedes #28174, which was closed as a duplicate. Its
cross-instance test is included here as `cancel-remote-run.test.ts`.

## Summary

You can cancel a workflow run on the default engine from another
process, meaning another `Mastra` instance that shares the same storage.
Before this PR, the run kept going after that cancel and then overwrote
the stored `canceled` status with `success`. The engine only knew about
cancellations made in its own process.

This PR makes the engine respect a cancellation it finds in storage:

- **Atomic guard on snapshot writes.** `persistWorkflowSnapshot` takes
an optional `expectedStatus` compare-and-set guard. The engine passes
every status except `canceled`, so a write is rejected only when storage
already holds `canceled`. The in-memory, LibSQL, PostgreSQL, MySQL,
MSSQL, DSQL and Cloudflare D1 stores check the guard in the same
statement as the write. Other stores read the status right before
writing.
- **Rejected writes stop the run.** A rejected write is recorded as a
cancellation. Final, paused and waiting writes report `canceled` instead
of their own status. A rejected step-start write aborts the run
controller before the step body runs. So `dountil` and `dowhile` stop at
the next iteration's start. Since #26214 on `main`, `foreach` items skip
their own start write, and the foreach checkpoints finished items
without holding the queue. This PR keeps that, so a `foreach`
(concurrency 1) still finishes the item that started while the first
checkpoint was in flight, then stops before the next one.
- **Status-only reads.** `loadWorkflowRunStatus()` reads only the status
column. The engine uses it for the check between entries and as the
read-before-write fallback, so it doesn't load the full snapshot on
every step.
- **One status list, internal to core.** The status list lives in
`storage/domains/workflows/run-statuses.ts`, typed as
`Record<WorkflowRunStatus, true>`, so adding a status to the union
without listing it is a `tsc` error. The engine derives the guard's
`expectedStatus` from it. It is not exported, and the SQL stores return
the stored status directly, as the base fallback and the in-memory store
already do. So the stores import no new core value and their
`@mastra/core` peer ranges are unchanged from `main`.
- **Time travel and runId reuse.** Replaying a run that is already
canceled still works the way it does on `main`. A fresh `start()` on a
runId whose earlier attempt was canceled also runs its steps, as on
`main`: the stored `canceled` belongs to the old attempt, so the new
attempt's first write may replace it. After that first write, a remote
cancel of the new attempt is honored again.

## Proof

- Cross-instance probe on `main` + this PR: a second `Mastra` instance
cancels during the first iteration or step. Each shape stores `canceled`
and skips the later step. `dountil` and `foreach` run the body once. The
shapes are sequential, `dountil`, `foreach`, sleep, and resumed
parallel. On `main`, all five keep running and store `success`.
- `cancel-sleep.test.ts`: 21/21. The loop tests (`dountil`/`foreach` ×
guarded/read-before-write) assert 1 `dountil` iteration and 2 `foreach`
items. With the `handlers/step.ts` cancel check reverted, they fail 4/4
(the loop runs all 3). The write-race tests (final, per-step pause,
executor-abort waiting) now also run with
`supportsGuardedSnapshotWrites()` forced to `false`. With the fallback
status reads disabled in `entry.ts` and `default.ts`, 10/21 fail.
- `cancel-remote-run.test.ts`: 6/6. This covers the sequential
cross-instance cancel from #28174, plus fresh `start()` on a canceled
runId (guarded and read-before-write, same and other instance) and a
remote cancel of that new attempt. With 7776a636's `default.ts` change
reverted, the 5 runId-reuse cases fail and
`background-tasks/manager.test.ts` fails 31 tests.
- `src/background-tasks`: 7/7 files, 171/171 tests (`manager.test.ts`
included).
- Status list: dropping `skipped` from `run-statuses.ts` makes core
`tsc` fail with `TS2741: Property 'skipped' is missing`.
- Head dbfd13b587 (after merging `main`, including #26214): `pnpm
build:core` passes. Rest of `src/workflows`: 99 files, 1143 tests. The
only failures in that run were the 2 `foreach` cases, before the
assertion was updated; `cancel-sleep.test.ts` is 21/21 now.
`workflow.test.ts`: 329 passed, 4 skipped. `evented-workflow.test.ts`:
315 passed, 5 skipped. Each big file ran on its own with a 6 GB heap.
The cancel and workflow-storage suites pass 78/78. `validate-peerdeps`
passes. `check:core-imports` passes for every changed store at its
`>=1.76` floor. It fails only for `@mastra/pg`, because it can't
download the unpublished `@mastra/core@1.80.0`; that floor comes from
`main`.
- LibSQL store: 61 files, 1391 passed, 35 skipped, 1 expected fail,
including the cross-process suites. Cloudflare D1 (miniflare): 337
passed, including the guarded-write tests. PostgreSQL guarded-write
tests passed earlier on PGlite, which is partial evidence.
- `tsc --noEmit` is clean for `packages/core` and for pg, mysql, mssql,
dsql and cloudflare-d1. For libsql, it fails only in
`src/__tests__/cross-process/fixtures/foreach-billing-workflow.ts`
(readonly JSON schema `required`). That file was added on `main` in
#27935, and this PR doesn't change it. `oxlint` and `oxfmt --check` are
clean on the changed files.
- Evented engine: a cross-instance `dountil` probe stores `canceled` and
runs the body once. The engine drops events for canceled runs in
`workflow-event-processor`. Its code is unchanged by this PR.

## Unverified

- DSQL guarded writes and status reads were not run against a real
server, because this sandbox has none. MSSQL was not run against a real
server here either: its retry after losing a concurrent first insert
(errors 1205, 2601, 2627) is covered by a unit test that fails the first
MERGE with each server error, and fails without the retry. A reviewer
ran the shared concurrent-first-write test on live SQL Server with this
retry: 10/10. PostgreSQL ran only on PGlite. MySQL ran only on MariaDB
10.11 and only through the workflows domain, because the full MySQL
store does not initialize on MariaDB (no functional indexes). On
MariaDB, 40 fresh runs with 3 concurrent guarded first writes each gave
75/120 `false` before the duplicate-key retry and 0/120 after. Three
concurrent guarded writes to a `canceled` row still all return `false`.
MySQL 8/9 itself was not run.
- A fresh `start()` that reuses the runId of a canceled attempt reads
the stored status once before its first write. A cancel that commits
between that read and the write is lost, as it is on `main`.
- MongoDB, DynamoDB, Redis, Upstash, Cloudflare KV/DO, ClickHouse,
LanceDB and the remaining adapters don't implement the guard. A cancel
that commits between their status read and their write can still be
overwritten.
- A file-backed LibSQL database shared by two OS processes was not
tested. The cross-instance tests use two `Mastra` instances in one
process.
- The Inngest engine was not run. It extends `DefaultExecutionEngine`,
so it inherits these changes.
- On the evented engine, the owner's `start()` promise never settles
after a remote cancel. This also happens on `main`, and it is not
addressed here.

## Assumptions

- The guard allows every status except `canceled`, not just the
non-terminal ones. Restart and time travel rewrite runs that already
finished, and a non-terminal allow-list would reject those writes.
- The guard applies only when a row already exists. A run's first
snapshot is always inserted.
- `isWorkflowRunCanceled` treats a failed status read as "not canceled",
the same as `main`. On guarded stores, the next write still catches the
cancellation.
- If a write is rejected while a run is suspending or failing, the run
is reported as `canceled`, because that is what storage holds.
- A cancel that lands while a step body is running in another process
does not interrupt that body. The run stops at the next write, such as
the next loop iteration's start or the check after the entry.

## Risk: medium

The change is on the hot path of every default-engine snapshot write and
step start. Runs that aren't canceled are unaffected.

## Compatibility

- `persistWorkflowSnapshot` takes an optional `expectedStatus` and now
returns `Promise<boolean | void>`. Adapters that return `void` are
treated as not guarding.
- `supportsGuardedSnapshotWrites()` defaults to `false`.
`loadWorkflowRunStatus()` falls back to loading the full snapshot.
Third-party adapters need no changes.
- Changesets: a patch for `@mastra/core`, and a patch for
`@mastra/libsql`, `@mastra/pg`, `@mastra/mysql`, `@mastra/mssql`,
`@mastra/dsql` and `@mastra/cloudflare-d1`.

---------

Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
2026-10-26 17:40:36 -05:00
Damien Schneider
7dbc6c1d18
refactor(lint): simplify unnecessary template expressions (#28447)
Enable `typescript/no-unnecessary-template-expression` in the shared
TypeScript configuration and simplify 17 redundant interpolations.
Preserve 11 intentional coercions at runtime boundaries and in mock
response text with local explanations. Existing lint scope and package
manifests are unchanged.

Reviewed every edit and compared emitted JavaScript after the approved
string simplifications, including interpolation order and tagged
templates. All 916 function return types and 1,833 variable binding
types in the 21 affected source files are unchanged. CI lint and build
pass; the local 57-task build, CLI typecheck, formatting, and whitespace
checks pass.

Validation covers 193 focused processor, CLI, Couchbase, and GraphRAG
tests, plus 902 deployer tests across the full run and an isolated retry
of one timeout. That protocol-import case passed alone in 14.55 seconds;
unchanged main took 14.61 seconds against its 15-second timeout. Both
selected published-package monorepo E2E cases pass, including the
extended HTTP check for external module resolution. The broad rule audit
found zero rule violations across 15,106 files, alongside four
TypeScript configuration diagnostics in unchanged files; the targeted
scan passes cleanly.

The DuckDB CI job also fails two existing capability-list assertions
because their expected arrays omit `span-aggregate`. Both failures were
reproduced on unchanged base `f4cbdb4d78`; DuckDB and core storage files
are unchanged by this PR.

No changeset is included for this lint maintenance, as requested; the
changeset check is expected to fail. Keep this PR as a draft.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

This PR adds a check that finds template strings with unnecessary string
insertions. It removes those insertions while preserving explicit
conversions where they may affect behavior.

## Summary

- Enable `typescript/no-unnecessary-template-expression` as an error in
the shared TypeScript configuration.
- Simplify redundant template expressions across CLI, core, deployer,
storage, test, and end-to-end test code.
- Preserve intentional string coercion at runtime boundaries and in mock
responses with local Oxlint suppression comments.
- Keep the existing lint scope and package manifests unchanged,
according to the PR objectives.

## Validation

The PR objectives report passing lint, build, typecheck, formatting,
whitespace, and focused test checks. They also report DuckDB
capability-list failures reproduced on the unchanged base. No changeset
is included as requested. The changeset check is expected to fail.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-26 22:37:47 +01:00
dane-ai-mastra[bot]
2c646b53d0
chore: version packages (alpha) (#28399)
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases

> The changelog information of each package has been omitted from this
message, as the content exceeds the size limit.

## @mastra/factory@0.25.0-alpha.5


## @mastra/langsmith@1.4.0-alpha.2


## @mastra/connect@1.2.0-alpha.2


## @mastra/core@1.80.0-alpha.4


## @mastra/playground-ui@66.0.0-alpha.4


## @mastra/acp@0.4.4-alpha.0


## @mastra/claude@0.3.4-alpha.0


## @mastra/cursor@0.3.4-alpha.0


## @mastra/openai@1.1.4-alpha.0


## @mastra/auth-auth0@1.2.6-alpha.0


## @mastra/auth-better-auth@1.1.9-alpha.0


## @mastra/auth-clerk@1.3.2-alpha.0


## @mastra/auth-cloud@1.2.8-alpha.0


## @mastra/auth-firebase@1.1.5-alpha.0


## @mastra/auth-google@0.1.7-alpha.0


## @mastra/auth-neon@0.3.5-alpha.0


## @mastra/auth-okta@0.2.6-alpha.0


## @mastra/auth-studio@1.3.11-alpha.1


## @mastra/auth-supabase@1.1.6-alpha.0


## @mastra/auth-workos@1.6.8-alpha.0


## @mastra/agent-browser@0.5.5-alpha.0


## @mastra/browser-viewer@0.2.6-alpha.0


## @mastra/browser-firecrawl@0.2.5-alpha.1


## @mastra/stagehand@0.3.7-alpha.1


## @mastra/discord@1.3.2-alpha.0


## @mastra/slack@1.8.0-alpha.1


## @mastra/teams@0.1.3-alpha.0


## @mastra/telegram@0.2.4-alpha.0


## @mastra/client-js@1.57.0-alpha.4


## @mastra/react@1.13.0-alpha.4


## @mastra/isolated-vm@0.1.4-alpha.0


## @mastra/quickjs@0.1.5-alpha.0


## @mastra/deployer-cloud@1.80.0-alpha.4


## @mastra/deployer-cloudflare@1.2.39-alpha.1


## @mastra/deployer-netlify@1.2.39-alpha.1


## @mastra/deployer-sandbox@0.3.24-alpha.1


## @mastra/deployer-vercel@1.2.39-alpha.1


## @mastra/voyageai@0.4.3-alpha.0


## @mastra/longmemeval@1.1.39-alpha.4


## @mastra/livekit@0.5.2-alpha.0


## @mastra/opencode@0.1.39-alpha.4


## @mastra/parallel@0.1.4-alpha.0


## create-factory@0.3.1-alpha.4


## @mastra/code-sdk@1.15.1-alpha.5


## mastracode@0.48.1-alpha.4


## @mastra/arize@1.3.27-alpha.2


## @mastra/arthur@0.4.27-alpha.2


## @mastra/braintrust@1.3.24-alpha.2


## @mastra/datadog@1.4.19-alpha.2


## @mastra/deepeval@0.1.21-alpha.2


## @mastra/laminar@1.3.29-alpha.2


## @mastra/langfuse@1.6.3-alpha.2


## @mastra/observability@1.21.1-alpha.2


## @mastra/otel-bridge@1.5.19-alpha.2


## @mastra/otel-exporter@1.4.10-alpha.2


## @mastra/posthog@1.4.4-alpha.2


## @mastra/sentry@1.2.29-alpha.2


## @mastra/agent-builder@1.1.31-alpha.2


## @mastra/auth@1.1.5-alpha.0


## mastra@1.36.0-alpha.4


## @mastra/codemod@1.1.8-alpha.0


## create-mastra@1.36.0-alpha.4


## @mastra/deployer@1.80.0-alpha.4


## @mastra/editor@0.16.5-alpha.2


## @mastra/evals@1.10.9-alpha.0


## @mastra/fastembed@1.3.5-alpha.2


## @mastra/loggers@1.3.10-alpha.1


## @mastra/mcp@2.6.0-alpha.1


## @mastra/mcp-docs-server@1.3.10-alpha.7


## @mastra/mcp-registry-registry@1.1.7-alpha.0


## @mastra/memory@1.39.1-alpha.2


## @mastra/rag@2.7.1-alpha.0


## @mastra/schema-compat@1.3.17-alpha.0


## @mastra/server@1.80.0-alpha.4


## @mastra/google-cloud-pubsub@1.1.5-alpha.1


## @mastra/redis-streams@0.6.5-alpha.1


## @mastra/valkey-streams@0.5.7-alpha.1


## @mastra/elysia@0.1.20-alpha.4


## @mastra/express@1.5.24-alpha.4


## @mastra/fastify@1.5.24-alpha.4


## @mastra/hono@1.7.22-alpha.4


## @mastra/koa@1.7.24-alpha.4


## @mastra/nestjs@0.2.39-alpha.4


## @mastra/next@0.2.38-alpha.4


## @mastra/server-adapters-test-suite@0.1.4-alpha.2


## @mastra/tanstack-start@0.2.38-alpha.4


## @mastra/astra@1.1.3-alpha.0


## @mastra/azure-ai-search@0.1.2-alpha.1


## @mastra/chroma@1.1.5-alpha.0


## @mastra/clickhouse@1.27.0-alpha.1


## @mastra/cloudflare@1.7.6-alpha.0


## @mastra/cloudflare-d1@1.4.5-alpha.1


## @mastra/convex@1.7.6-alpha.0


## @mastra/couchbase@1.1.4-alpha.1


## @mastra/dsql@1.5.5-alpha.0


## @mastra/duckdb@1.16.0-alpha.1


## @mastra/dynamodb@1.4.5-alpha.1


## @mastra/elasticsearch@1.4.6-alpha.1


## @mastra/lance@1.5.2-alpha.1


## @mastra/libsql@1.27.1-alpha.0


## @mastra/mongodb@1.24.1-alpha.1


## @mastra/mssql@1.10.6-alpha.0


## @mastra/mysql@0.14.1-alpha.0


## @mastra/opensearch@1.1.4-alpha.1


## @mastra/oracledb@0.5.5-alpha.0


## @mastra/pg@1.33.0-alpha.2


## @mastra/pinecone@1.1.3-alpha.0


## @mastra/qdrant@1.1.6-alpha.0


## @mastra/redis@1.4.12-alpha.1


## @mastra/s3vectors@1.1.5-alpha.0


## @mastra/spanner@1.10.1-alpha.1


## @mastra/turbopuffer@1.2.3-alpha.0


## @mastra/turso@0.1.18-alpha.0


## @mastra/upstash@1.5.5-alpha.1


## @mastra/valkey@0.2.8-alpha.1


## @mastra/vectorize@1.1.4-alpha.0


## @mastra/weaviate@0.1.4-alpha.0


## @mastra/voice-aws-nova-sonic@0.2.6-alpha.0


## @mastra/voice-azure@0.12.3-alpha.0


## @mastra/voice-cloudflare@0.13.4-alpha.0


## @mastra/voice-deepgram@0.13.3-alpha.0


## @mastra/voice-elevenlabs@0.13.3-alpha.0


## @mastra/voice-gladia@0.13.3-alpha.0


## @mastra/voice-google@0.14.4-alpha.0


## @mastra/voice-google-gemini-live@0.14.18-alpha.0


## @mastra/voice-inworld@0.4.6-alpha.0


## @mastra/voice-mistral@0.1.3-alpha.0


## @mastra/voice-modelslab@0.2.3-alpha.0


## @mastra/voice-murf@0.13.3-alpha.0


## @mastra/voice-openai@0.13.4-alpha.0


## @mastra/voice-openai-realtime@0.14.8-alpha.0


## @mastra/voice-playai@0.13.3-alpha.0


## @mastra/voice-sarvam@1.1.4-alpha.0


## @mastra/voice-speechify@0.14.3-alpha.0


## @mastra/voice-xai-realtime@0.2.17-alpha.0


## @mastra/inngest@1.10.8-alpha.3


## @mastra/temporal@0.4.18-alpha.4


## @mastra/agentcore@0.5.3-alpha.0


## @mastra/agentfs@0.2.3-alpha.0


## @mastra/apple-container@0.5.2-alpha.0


## @mastra/archil@0.3.2-alpha.0


## @mastra/azure@0.3.3-alpha.1


## @mastra/blaxel@0.10.2-alpha.0


## @mastra/cloudflare-sandbox@0.5.4-alpha.0


## @mastra/daytona@0.11.4-alpha.0


## @mastra/docker@0.12.1-alpha.0


## @mastra/e2b@0.15.1-alpha.1


## @mastra/e2b-desktop@0.1.10-alpha.1


## @mastra/files-sdk@0.3.3-alpha.0


## @mastra/gcs@0.3.5-alpha.1


## @mastra/google-drive@0.2.4-alpha.0


## @mastra/mesa@0.3.3-alpha.0


## @mastra/modal@0.7.3-alpha.0


## @mastra/platform-workspace@1.10.0-alpha.1


## @mastra/railway@0.10.2-alpha.1


## @mastra/s3@0.6.5-alpha.1


## @mastra/sprites@0.1.2-alpha.0


## @mastra/vercel@1.6.2-alpha.0


## @internal/playground@1.36.0-alpha.4

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-10-26 13:49:15 -07:00
Damien Schneider
f02d3b711e
refactor(lint): remove redundant returns while preserving exits (#28448)
Enable `no-useless-return` in the shared configuration. Remove 28
terminal bare returns, use 31 explicit switch breaks, and retain 12
value-returning contracts with explicit `undefined`. Preserve 13
intentional guards with local explanations, including index-skipping
property reads and the platform availability probe.

Reviewed every edit and checked that removed returns and new switch
breaks have no effectful continuation before function exit, including
cleanup through `finally`. All other source and emitted JavaScript
syntax is preserved apart from formatting, and all 2,020 function return
types and 3,280 variable binding types in the 59 affected source files
are unchanged. The new rule reports zero violations across 15,106 files.
CI lint and build, the local 57-task build, Factory build, core
typecheck, formatting, and whitespace checks pass.

All 756 focused tests pass across stream conversions, idle-loop cleanup,
workflow validation, Factory, Factory UI, Playground UI, Datadog,
Sentry, Cloudflare, and E2B. Both selected published-package monorepo
E2E cases pass. The extended build case covers a missing application
`.npmrc` and verifies successful builds with dependency installation
both skipped and enabled.

The DuckDB CI job also fails two existing capability-list assertions
because their expected arrays omit `span-aggregate`. Both failures were
reproduced on unchanged base `f4cbdb4d78`; DuckDB and core storage files
are unchanged by this PR.

No changeset is included for this lint maintenance, as requested; the
changeset check is expected to fail. Keep this PR as a draft.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## ELI5

This PR teaches the shared lint configuration to flag returns that do
not change what a function does. It updates the affected code so it
still exits at the right point, while keeping returns that are needed to
return a value or protect an intentional guard.

## Summary

- Enables `no-useless-return` as an error in the shared Oxlint
configuration.
- Removes redundant terminal returns, replaces switch-case returns with
`break` where execution continues after the switch, and keeps
value-returning contracts explicit.
- Adds local explanations or lint suppressions for intentional guards.
- Adds assertions that the monorepo build test leaves `.npmrc` files
absent when installation is skipped.

The author reports that source and emitted JavaScript syntax, apart from
formatting, and the return types and variable binding types across all
59 affected files remain unchanged. The author also reports reviewing
changes for effects before function exit, including cleanup through
`finally`.

## Validation and status

The author reports zero violations across 15,106 files and passing CI
lint and build, the local 57-task build, Factory build, core typecheck,
formatting, whitespace checks, 756 focused tests, and two selected
published-package monorepo E2E cases.

The author reports two DuckDB CI failures in existing capability-list
assertions because the expected arrays omit `span-aggregate`. The author
reports reproducing both failures on unchanged base `f4cbdb4d78`; DuckDB
and core storage files were not changed.

No changeset is included, as requested. The author expects the changeset
check to fail and requests that the PR remain a draft.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-26 21:03:00 +01:00
mastra-platform[bot]
358c71145d
fix(core): don't re-run finished parallel arms or foreach items when restarting after a crash (#27935)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9

Closes #26214

On the default engine, if a run crashes partway through a `.parallel()`
block or a `.foreach()` and you call `run.restart()`, the steps that had
already finished run again. That can repeat side effects such as
charges. Durable agents hit the same bug: they run parallel tool calls
through `.foreach()`, so a recovered durable agent re-ran tool calls
that had already finished.

## Root cause

- **Parallel:** a finished arm isn't checkpointed until the whole block
ends. A crash snapshot still lists the finished arm as active, so
restart runs it again.
- **Foreach:** per-item results are only persisted when the foreach
suspends, fails or completes. If it crashes in between, the snapshot has
no record of which items finished, so restart runs every item again.

## Fix

- **Parallel** (`handlers/control-flow.ts`): each arm that succeeds is
checkpointed (phase `parallel-arm-end.<i>`), so restart skips it.
- **Foreach** (`handlers/control-flow.ts`): finished items are recorded
on the running foreach as `metadata.foreachProgress = {[index]: {output,
nestedRunId?}}`. On restart, only items that are not in that record get
queued. The record is removed when the foreach ends and never reaches a
final step result or the workflow result.
- Progress is written by **one coalescing chain per foreach**: one write
in flight at a time, and items that finish during a write are folded
into the next one, which starts as soon as the previous write lands.
There is no timer. A crash can only lose items that finished while a
write was in flight, and the write rate is capped by how fast storage
completes writes. When the foreach ends, only an in-flight write is
awaited.
- **Foreach items don't write their own start snapshot** while progress
is checkpointed. Each one used to rewrite the whole snapshot with all
progress so far. Instead the foreach records itself as running once,
before its first item starts, and the chain carries progress from there.
- If a progress checkpoint fails, the chain stops writing, cleanup still
runs, and the foreach fails with that error. Before this change it would
have failed at its next write too.
- If a stored snapshot has malformed progress (an entry that isn't an
object with `output`, or a non-string `nestedRunId`), restart ignores
the progress and runs every item again, the same as a crash before any
item finished.
- **Write ordering** (`default.ts`, `handlers/entry.ts`): snapshot
writes for a run now go through a per-run queue
(`serializeSnapshotWrite`), so they land in the order they were made.
Before, concurrent full-snapshot writes could land out of order and an
older snapshot could overwrite a newer one. The suspended/paused
overwrite guard runs inside the queued write, so it is evaluated in
order too.
- **`handlers/step.ts`:** foreach items skip their start write on
engines that checkpoint progress; Inngest keeps it.
- **`handlers/foreach-progress.ts`** (new, internal, not re-exported
from `@mastra/core/workflows`): `ForeachProgress`, `getForeachProgress`,
`omitForeachProgress`. `omitForeachProgress` keeps progress out of
per-item step info.
- **Engines:** mid-block checkpoints and the write queue only apply to
engines that don't replay memoized durable operations
(`!requiresDurableContextSerialization()`), so default and the durable
agent. Inngest doesn't support `restart()` and records every write as
its own durable step, so it writes exactly what it wrote before. The
evented engine already wrote per-item foreach results and re-ran only
unfinished items; its code is unchanged.

## Proof

All tests below are new. Each fix test fails with the fix reverted
(`default.ts`, `handlers/{control-flow,entry,step,foreach-progress}.ts`
checked out from `origin/main` / before the fix) and passes with the
fix:

| Test | Fix reverted | With fix |
| --- | --- | --- |
| `restart-mid-block.test.ts` › parallel arm finished before the crash |
✗ finished arm called again | ✓ |
| › 3-arm parallel, unfinished arm between finished ones | ✗ | ✓ |
| › foreach, concurrency 1 | ✗ re-runs `[0,1,2,3,4]` instead of
`[2,3,4]` | ✓ |
| › foreach, concurrency 3 | ✗ | ✓ |
| › concurrency 5, items finish after every item has started | ✗ | ✓ |
| › billing: crash one macrotask after customer 1 is charged
(concurrency 1 and 3) → every customer charged exactly once | ✗ at
`8897c920` too: customers 0 and 1 charged twice | ✓ |
| › storage with seeded random latency (20 items, concurrency 4):
persisted progress never shrinks while the foreach runs | ✗ | ✓ |
| › a progress checkpoint write fails → foreach fails, next step doesn't
run, no hang | ✗ (succeeds) | ✓ |
| › parallel block, storage with seeded random latency: persisted
finished-arm count never shrinks | ✓ (see below) | ✓ |
| › crash before any item finished → every item runs again | ✓ (see
below) | ✓ |
| › malformed progress entry (`{"0": null}`) in the stored snapshot →
every item runs again | ✗ `TypeError: Cannot read properties of null
(reading 'output')`, 2/2 | ✓ |
| › 400 items, concurrency 8: foreach writes ≤ finished items + 1
running write, final result matches and carries no progress | ✓ (see
below) | ✓ |
| `stores/libsql` `cross-process/foreach-billing-restart.test.ts` › two
processes on one LibSQL file: process A charges customers 0 and 1, is
SIGKILLed 200 ms later, process B runs `restartAllActiveWorkflowRuns()`
→ ledger shows each customer once (concurrency 1 and 3) | ✗ at
`8897c920`: `A 0, A 1, B 0, B 1, B 2, B 3, B 4` | ✓ |
| `durable-agent-parallel-tool-restart.test.ts` › durable agent: 3
parallel tool calls, 2 finish, crash, recover → only the unfinished call
runs again (aimock `LLMock`) | ✗ re-runs `[0,1,2]` | ✓ re-runs `[1]` |
| same test, evented agent (control) | ✓ (already correct) | ✓ |

Each fix checked on its own, with everything else in place:
- **Write queue** removed from `entry.ts`: the parallel latency test
fails, 3/3 runs.
- **Per-item start writes** restored in `step.ts`: the foreach latency
test fails.
- **1 s checkpoint timer** restored (`control-flow.ts` from `8897c920`):
both billing tests fail, unit and two-process. Restart tests now crash
one macrotask after an item finishes instead of waiting for a checkpoint
to land, which is the window the timer left open.
- **Foreach running write** before the first item removed: the
crash-before-any-item test fails. Without it, the snapshot never shows
the foreach as active and restart skips it.
- The progress re-apply after `Object.assign` is gone. Nothing reads the
foreach entry in that window once items stop writing their own start
snapshots.
- **Suites, per file group** (`NODE_OPTIONS=--max-old-space-size=6144`):
  - `workflow.test.ts`: 329 passed, 4 skipped.
  - `evented/evented-workflow.test.ts`: 315 passed, 5 skipped.
- `restart-mid-block` + `durable-agent-parallel-tool-restart`: 16/16 at
the current head, after merging `main`. `foreach-billing-restart`
(libsql): 2/2.
- Every other test file under `src/workflows` and `src/agent/durable`:
272 of 274 files passed. `durable-agent-client-tool-pending.test.ts` and
`durable-agent-tool-writer.test.ts` (4 tests) also fail with this PR's
source files checked out from `origin/main`, so they are unrelated to
this change.
- After the malformed-progress fix: the restart files
(`restart-mid-block`, `restart-completed-boundary`,
`foreach-failure-progress`, `nested-restart-state`,
`parallel-nested-restart`) pass, 32/32.
- **aimock parallel-tool scenarios** (`multi-tool-parallel`,
`parity-parallel-tools`, `tool-concurrency-active-tools`,
`tool-output-siblings`):
  - core engines: 32 passed, 4 skipped.
- **Inngest against a real dev server**
(`e2e:workflows/inngest:aimock`): 7 passed, 2 skipped (that parity file
skips on Inngest).
- `tsc --noEmit` (packages/core), `oxlint` and `oxfmt --check` are
clean.
- **Snapshot writes**: foreach at concurrency 8, each item returning a
200-byte payload. Bytes are the JSON size of each
`persistWorkflowSnapshot` call. Postgres is PGlite (`pglite-server`,
in-process WASM Postgres) through `PostgresStore`; there is no real
Postgres server or docker in this sandbox.

  | store | | items | item time | writes | total bytes | wall |
  | --- | --- | --- | --- | --- | --- | --- |
  | Postgres (PGlite) | `main` | 400 | 0 ms | 402 | 1,064,465 | 631 ms |
  | Postgres (PGlite) | this PR | 400 | 0 ms | 4 | 282,860 | 67 ms |
| Postgres (PGlite) | `main` | 1,600 | 0 ms | 1,602 | 12,901,265 | 5,533
ms |
| Postgres (PGlite) | this PR | 1,600 | 0 ms | 4 | 1,134,260 | 108 ms |
| Postgres (PGlite) | `main` | 1,600 | 20 ms | 1,602 | 12,901,265 |
5,440 ms |
| Postgres (PGlite) | this PR | 1,600 | 20 ms | 347 | 69,200,000 (≈) |
4,235 ms |
  | MockStore | `main` | 400 | 0 ms | 403 | 1,064,465 | 46 ms |
  | MockStore | this PR | 400 | 0 ms | 66 | 3,500,000 (≈) | 48 ms |
  | MockStore | `main` | 1,600 | 0 ms | 1,603 | 12,901,265 | 173 ms |
  | MockStore | this PR | 1,600 | 0 ms | 253 | 53,300,000 (≈) | 287 ms |
  | MockStore | `main` | 1,600 | 20 ms | 1,603 | 12,901,265 | 4,116 ms |
| MockStore | this PR | 1,600 | 20 ms | 1,604 | 336,500,000 (≈) | 4,255
ms |

Final snapshots are identical to `main` in every row. Writes now track
storage speed: the slower the store, the more item completions fold into
one write. With items slower than writes, there is about one write per
item, and each write carries the full snapshot with all progress so far.
Bytes then grow with the square of the item count. That is the worst row
above: 336 MB vs 12.9 MB on in-memory storage. Writes stay off the item
path, so wall time is close to `main` on MockStore and better on
Postgres.

## Unverified

- Byte and timing numbers on a real Postgres server. PGlite is a
stand-in, not proof of production write cost.
- Nested workflows inside a foreach on restart: `nestedRunId` is carried
through progress, but no test restarts a foreach of nested workflows.
- `durable-agent-client-tool-pending.test.ts` and
`durable-agent-tool-writer.test.ts` fail on `main` too, so this PR's
behaviour in those paths isn't verified by them.

## Risk: medium

This changes when and how snapshots are written during parallel and
foreach execution on the default engine:
- Writes for a run are now queued, so parallel arms' writes no longer
overlap.
- A foreach writes progress whenever the previous write has landed and
items finished since. Items no longer write their own start snapshot on
the default engine. When items finish slower than storage writes, total
bytes written grow with the square of the item count (see the table).
- A failing mid-foreach checkpoint now fails the foreach.

Inngest is gated out and its writes are unchanged. Restart, resume,
time-travel and partial-concurrency suspend/resume are covered by the
existing suites above.

### Assumptions / judgment calls

- Kept `main`'s `break` in the parallel restart loop. With `break`
restored, every restart test still passed: later arms still run through
`Promise.all`, and the loop only refreshes status. Reverting it removes
an untested change instead of adding a test for it.
- Gated mid-block checkpoints and the write queue on
`requiresDurableContextSerialization()` rather than adding an engine
capability method. That avoids a cross-package version dependency with
`@mastra/inngest`.
- No minimum checkpoint interval. Any interval leaves a window where
finished items, such as charges, run again on restart. The remaining
window is one write in flight. The cost is the quadratic byte growth
above when items are slower than writes. An append-only progress API, so
snapshots stop being rewritten, can come later with the snapshot
redesign.
- Malformed progress is ignored as a whole (every item re-runs) rather
than salvaging valid entries. Only the engine writes this field, so a
bad entry means a corrupted or hand-edited snapshot, and trusting the
rest of it isn't safe.
- The agent test uses aimock directly with the restart harness instead
of `describeForAllEngines`, because recovery needs a fresh module graph
per generation. Inngest is covered by the aimock scenarios above, since
it has no `restart()`.
- The two-process billing test lives in `stores/libsql` (core can't
depend on a store). It runs against built `@mastra/core`, so its revert
proof rebuilt core with `8897c920`'s `control-flow.ts`.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## ELI5

If a workflow stops while several tasks are running, it can now save
which tasks finished. After a restart, it can skip those tasks and
continue the unfinished work, which helps prevent repeated side effects.

## Summary

- Save successful `.parallel()` arms and completed `.foreach()` items so
restarted workflows can skip completed work.
- Restore saved foreach outputs and nested run IDs. Throttle and
serialize progress checkpoints, and remove foreach progress from final
results.
- Serialize snapshot writes per run to prevent overlapping writes from
being applied out of order.
- Apply the relevant checkpoint changes only when the engine does not
require durable-context serialization. Inngest retains its prior write
behavior, and the evented engine is unchanged.
- Add restart tests for parallel arms, foreach items, checkpoint
failures, write ordering, and durable-agent parallel tool calls.

A crash between foreach checkpoints can still cause a recently completed
item to run again.

## Validation

The PR reports 329 passed and 4 skipped in `workflow.test.ts`, and 315
passed and 5 skipped in `evented-workflow.test.ts`. It also reports 32
passed and 4 skipped core aimock scenarios, and 7 passed and 2 skipped
Inngest aimock tests. The author reports that two durable-agent test
files fail, and that those files also fail with the PR’s five source
files checked out from `origin/main`. The author reports clean `tsc
--noEmit` for `packages/core`, `oxlint`, and `oxfmt --check`.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
2026-10-26 14:11:22 -05:00
Damien Schneider
0f82b38ed4
refactor(lint): flatten lone if statements in else branches (#28431)
Enable `no-lonely-if` in the shared Oxlint config and flatten 38 lone
`if` statements in `else` blocks across 35 files. For example, `else {
if (condition) { ... } }` becomes `else if (condition) { ... }`.
Conditions, evaluation order, statements, comments, and local bindings
are preserved. No package filters or lint exceptions are added.

Strengthen the existing monorepo test to execute another named export
from an optimized dependency. Comparisons of TypeScript syntax, emitted
JavaScript, raw template text, comments, and local bindings pass for all
35 migrated files after normalizing only the redundant else blocks. The
rule clears all 15,094 scanned files.

[CI lint and formatting
pass](https://github.com/mastra-ai/mastra/actions/runs/40344286057/job/121647174331)
for `36ca4471f9`. All 634 focused tests, four package typechecks (core,
playground, connect, and deployer), 57 packaged build tasks, and two
selected published-package monorepo E2E cases pass. The broader local
lint run hit a Tailwind design-system precomputation timeout; CI lint
succeeds. Live vendor and external-database suites were not run locally.

[DuckDB CI has two existing capability-list assertion
failures](https://github.com/mastra-ai/mastra/actions/runs/40344286246/job/121649643109):
its implementation advertises `span-aggregate`, while the expected lists
omit it. The failures reproduce locally, and the relevant implementation
and tests are unchanged from the base commit.

This maintenance PR intentionally contains no changeset and requests no
package releases. The missing-changeset check may fail.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## ELI5
This PR turns on a lint rule that asks for simpler branching in code. It
updates the affected code and adds a leap-year check to an existing
test.

## Changes
- Enables Oxlint’s `no-lonely-if` rule in the shared configuration.
- Flattens 38 lone `if` statements across 35 files. The PR description
reports that the changes preserve behavior.
- Extends the optimized-dependency monorepo test to check `isLeapYear`.

## Validation
- The PR description reports that CI lint and formatting pass.
- It also reports 634 focused tests, four package typechecks, 57
packaged build tasks, and two selected monorepo E2E cases pass.
- The PR description notes two existing DuckDB CI capability-list
assertion failures. It reports that the related implementation and tests
are unchanged from the base commit.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-26 18:44:48 +01:00
Ward Peeters
ce5d4731ee
chore(deps): update dependencies across the monorepo (#28417)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-26 16:59:30 +01:00
Damien Schneider
0fa30cfbea
refactor(lint): remove redundant boolean casts and rethrow catches (#27985)
**─────── TLDR ───────**
> The shared Oxlint config now errors on `!!x` inside a condition and on
a `catch` block that only rethrows.

A `catch (err) { throw err; }` reads like something is handled. Nothing
is. `!!x` inside an `if` reads like a deliberate coercion the `if`
already performs. Ten casts and twelve catches across 14 files go.
Nothing reaches a caller differently.

an error thrown inside one of the twelve removed `try` blocks
&nbsp;&nbsp;└─ **unchanged** — same object, same stack, same caller
receives it

| | | |
|---|---|---|
| **+** | `no-useless-catch` | a catch that only rethrows is dead code |
| **+** | `no-extra-boolean-cast` | `if` and `?:` already coerce |

### Decisions

- **no changeset** — no consumer can observe a difference, so no
release; `changeset add --empty` if the red status check bothers you
- **two monorepo e2e assertions for a no-op** — the gate requires
monorepo coverage for any `packages/deployer/` change; they pin the two
reads touched

### Not verified

- [ ] MSSQL, Spanner, Railway, Pinecone and Vectorize paths — their
suites need live services

---

> ██████████████████ **source** `+122 −169` — net −47
> █████ **tests** `+36 −11`
> █ **lint config** `+2`
2026-10-23 18:41:47 +02:00
Dane - AI (Mastra)
40a522c142
chore(deps): update build tools (#27997) 2026-10-22 14:33:21 +02:00