mirror of
https://github.com/mastra-ai/mastra.git
synced 2026-11-17 08:44:46 +00:00
1,845 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
a5859ee2b9 |
chore: version packages (alpha) (#30430)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
|
cd2a326988 |
fix(libsql): stop background task writes stalling shutdown, and fix backend timing flakes (#30504)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
88b4a97042 |
refactor(factory): skip legacy migration tables on new databases (#30488)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
6c647e6fae |
refactor(factory): stop creating columns and tables nothing reads (#30495)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
49de7780aa |
fix(convex): apply every operator in a vector filter field condition (#30469) | ||
|
|
20a1a404fd |
fix(pinecone): keep other filter conditions next to $all (#30468) | ||
|
|
536faa6508 |
fix(mongodb): retire older live observational memory records when startup races a rollover (#30452)
MongoDB's startup cleanup could leave an extra observational memory record live. At startup, the store finds threads that have more than one live record (older adapter versions never marked replaced records) and retires all but the current one. A rollover first marks the current record as replaced, then inserts the new one. If startup cleanup checked in between, the thread showed only one live record, the older one, and was skipped. Once the new record landed, both were live. Reads still picked the right record, but writes aimed at the older id still landed on it until the next startup. Startup cleanup now also counts the record being rolled over, so the thread is processed, the rollover is completed and the older record is retired. There's no schema or API change. ## Test plan - New test in `om-lifecycle-concurrency.test.ts` that pins the race: the rollover marks the record just before cleanup's check and inserts the new one just after. With the old code it fails 10/10 with two live records; with the fix it passes. - The existing randomized race test (25 races per run) passes 50/50 runs with the fix. The window is rare locally: 0 of 50 runs failed on main. - The observational memory lifecycle tests pass 26/26 against real MongoDB 7. Typecheck, lint and formatting are clean. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ELI5: When the database starts, it cleans up old memory records. A timing bug could leave an old record active next to its replacement. This change makes startup cleanup count the record that is being replaced, so the old record gets retired. ## Summary This change fixes a MongoDB startup-cleanup race in observational memory. Cleanup could skip a thread when it ran after a record was fenced for rollover but before its successor was inserted. The older record then stayed live until the next startup. ## Changes - `stores/mongodb/src/storage/domains/memory/index.ts`: The duplicate-live-document aggregation in `#maintainSupersededBy` now matches documents where `supersededBy` is null or `pendingSuccessor` exists. A fenced head therefore counts as live during grouping. The code comment explains the window. - `.changeset/quiet-owls-roll.md`: Adds a patch changeset for `@mastra/mongodb`. - `stores/mongodb/src/storage/domains/memory/om-lifecycle-concurrency.test.ts`: Adds a deterministic test. It fences the head while the backfill groups live documents. It then checks that the head has `generationCount` 2 and is the only live record. ## Diff scope Three files changed, with 47 lines added and 1 removed. ## Verification The PR description reports these results. I did not run them. - The deterministic test passes with the fix and fails 10/10 on the old code. - The randomized race test passed 50/50 with the fix. None of 50 runs failed on main. - Observational memory lifecycle tests passed 26/26 against MongoDB 7. - Typecheck, lint, and formatting were reported clean. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com> |
||
|
|
adc0a70bf8 |
fix(memory, couchbase): check semantic recall scope in Memory and warn on ignored Couchbase filters (#30446) | ||
|
|
1c52fdd034 |
chore: version packages (alpha) (#30261)
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. ⚠️⚠️⚠️⚠️⚠️⚠️ `main` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `main`. ⚠️⚠️⚠️⚠️⚠️⚠️ # Releases > The changelog information of each package has been omitted from this message, as the content exceeds the size limit. ## @mastra/factory@0.29.0-alpha.1 ## @mastra/core@1.84.0-alpha.1 ## @mastra/playground-ui@69.1.0-alpha.1 ## @mastra/client-js@1.60.1-alpha.1 ## @mastra/react@1.15.1-alpha.1 ## @mastra/deployer-cloud@1.84.0-alpha.1 ## @mastra/longmemeval@1.1.43-alpha.1 ## @mastra/opencode@0.1.43-alpha.1 ## create-factory@0.4.4-alpha.1 ## @mastra/code-sdk@1.19.1-alpha.1 ## mastracode@0.50.1-alpha.1 ## @mastra/arize@1.3.31-alpha.0 ## @mastra/arthur@0.4.31-alpha.0 ## @mastra/braintrust@1.3.28-alpha.0 ## @mastra/datadog@1.4.23-alpha.0 ## @mastra/deepeval@0.1.25-alpha.0 ## @mastra/laminar@1.3.33-alpha.0 ## @mastra/langfuse@1.6.7-alpha.0 ## @mastra/langsmith@1.4.4-alpha.0 ## @mastra/observability@1.24.1-alpha.0 ## @mastra/otel-bridge@1.5.23-alpha.0 ## @mastra/otel-exporter@1.5.3-alpha.0 ## @mastra/posthog@1.4.8-alpha.0 ## @mastra/sentry@1.2.33-alpha.0 ## @mastra/agent-builder@1.2.2-alpha.0 ## mastra@1.39.1-alpha.1 ## @mastra/connect@1.4.1-alpha.0 ## @mastra/deployer@1.84.0-alpha.1 ## @mastra/editor@0.17.2-alpha.0 ## @mastra/mcp-docs-server@1.3.14-alpha.1 ## @mastra/memory@1.41.2-alpha.0 ## @mastra/server@1.84.0-alpha.1 ## @mastra/google-cloud-pubsub@1.1.8-alpha.0 ## @mastra/redis-streams@0.8.2-alpha.0 ## @mastra/valkey-streams@0.6.3-alpha.0 ## @mastra/elysia@0.1.24-alpha.1 ## @mastra/express@1.5.28-alpha.1 ## @mastra/fastify@1.5.28-alpha.1 ## @mastra/hono@1.7.26-alpha.1 ## @mastra/koa@1.7.28-alpha.1 ## @mastra/nestjs@0.3.1-alpha.1 ## @mastra/next@0.2.42-alpha.1 ## @mastra/tanstack-start@0.2.42-alpha.1 ## @mastra/clickhouse@1.30.1-alpha.0 ## @mastra/cloudflare@1.8.1-alpha.0 ## @mastra/cloudflare-d1@1.5.1-alpha.0 ## @mastra/convex@1.9.1-alpha.0 ## @mastra/dsql@1.5.8-alpha.0 ## @mastra/dynamodb@1.5.2-alpha.0 ## @mastra/libsql@1.30.1-alpha.0 ## @mastra/mongodb@1.27.1-alpha.0 ## @mastra/mssql@1.11.2-alpha.0 ## @mastra/mysql@0.17.1-alpha.0 ## @mastra/oracledb@0.5.7-alpha.0 ## @mastra/pg@1.36.1-alpha.0 ## @mastra/redis@1.4.14-alpha.0 ## @mastra/spanner@1.12.1-alpha.0 ## @mastra/turso@0.1.22-alpha.0 ## @mastra/upstash@1.6.2-alpha.0 ## @mastra/valkey@0.2.10-alpha.0 ## @mastra/temporal@0.6.1-alpha.1 ## @mastra/daytona@0.11.6-alpha.0 ## create-mastra@1.39.1-alpha.1 ## @internal/playground@1.39.1-alpha.1 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
|
b33d5b13f9 |
test(pg): let approval runs finish before the stores close (#30372)
## Summary The Postgres store job fails now and then with `Cannot use a pool after calling end on the pool`, coming from `WorkflowsPG.persistWorkflowSnapshot` and attributed to `stores/pg/src/storage/approval-policy-change.test.ts`. Every test passes, but the unhandled error fails the job. Two tests in that file returned as soon as the approved tool ran. The run then goes on to a second model turn and saves its workflow snapshot. On a slow runner, that save lands after `afterAll` has closed the pool. Both tests now also wait for their run to stop before they end. The first session in the last test stays parked on purpose and never writes, so it is left alone. ## Test plan - To show the failure first, the second model turn was delayed by 500 ms to mimic a slow runner. Without the fix, 5 of 8 runs failed with the CI error; with the fix, 0 of 8. - The real file passed 10 of 10 times with the fix. - The full `@mastra/pg` suite passed 4,251 tests. One unrelated `afterAll` timeout came from `schema-setup-per-database.test.ts`. - Type check, oxlint and oxfmt are clean. Test-only change, no changeset. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> **ELI5:** The tests closed the database too early. A test could still be saving data when the database closed, which made it fail. Now each test waits until its run stops before it ends. ## Summary This change updates two tests in `stores/pg/src/storage/approval-policy-change.test.ts`. It is test-only. It does not change production code. - **First test (policy changed on a parked session):** After the tool is allowed and executes, the test waits until `session.run.isRunning()` returns false. A comment states that the run continues to a second model turn and saves its snapshot before the stores close. The test timeout increases to 60,000 ms. - **Second test (session reopened with the tool allowed):** After the tool executes, the test waits until `reopened.run.isRunning()` returns false. The test timeout increases to 60,000 ms. The diff shows +5/-2 across 1 file. The additions are two wait assertions, a comment, and two timeout arguments. The two removed lines are the closing `});` of each test, which now end with `}, 60_000);`. The first session in the last test is not changed. The commit message says each test now has up to three sequential waits of up to 10 seconds each. The longer timeout keeps Vitest's default 5-second limit from cancelling the final completion wait. ## Reported results The PR author reports these results. I did not run them. - A 500 ms delay before the second model turn reproduced the failure in 5 of 8 runs before the fix, and in 0 of 8 runs with the fix. - The affected test file passed 10 of 10 runs with the fix. - The full `@mastra/pg` suite passed 4,251 tests. One unrelated `afterAll` timeout occurred in `schema-setup-per-database.test.ts`. - Type check, oxlint, and oxfmt are reported clean. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com> |
||
|
|
8936292f8a |
chore(couchbase): fix "similiarity" typo in vector integration test comments (#30346)
> Opened by @DSchau
## Summary
Fixes a misspelling ("similiarity" → "similarity") in two comments in
`stores/couchbase/src/vector/index.integration.test.ts`.
This changes comments only, so behavior stays the same and no changeset
is needed.
## Test plan
- [x] Comment-only change; lint-staged passed on commit.
Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
|
||
|
|
0c32e32f8b |
test(clickhouse): run store test files in parallel on per-worker databases (#29096)
## What this does
The ClickHouse store tests ran every file one after another
(`fileParallelism: false`) because they shared one database: migration
tests drop tables other files use. This runs them on 4 workers, each
with its own database:
- `vitest.global-setup.ts` creates one ClickHouse **user** per worker
(`mastra_test_wN`), each with `DEFAULT DATABASE mastra_test_wN` and its
own database. The number of users comes from the resolved `maxWorkers`,
so a CLI override can't leave a worker without one. Teardown drops each
database before its user, because the user owns materialized views in
it.
- `vitest.setup.ts` points each worker at its user via
`CLICKHOUSE_USERNAME` and `VITEST_POOL_ID`.
- Why a user and not a database in `CLICKHOUSE_URL`: a database in the
URL overrides the explicit `database` option. That broke the
feedback-deletion-race tests (lagging replicas) and the
database-fallback test, 12 failures. There's a comment next to the setup
so nobody simplifies it back.
- `migration.test.ts` sorting-key lookups now filter on `database =
currentDatabase()`, so they don't read another worker's tables.
- The shared vNext conformance suite moves out of `v-next/index.test.ts`
into `v-next/shared-suite.test.ts`. It's a pure move with no
module-level state shared between the halves, and the same 465 test
names. That file was the tail that capped the parallel run.
Test config and test files only: no source changes, no changeset.
## Numbers (local)
| | wall |
|---|---:|
| sequential (main) | 467 s |
| 4 workers, per-worker users | 239 / 240 / 243 s |
| + shared-suite split | 155 / 148 / 149 s |
Every run: 1140 passed / 415 skipped. CI clickhouse job today: 390 s;
estimate after this ~125–135 s (unverified until CI runs).
Finding for the ClickHouse store owner, not changed here: ClickHouse 26
defaults to `async_insert=1, wait_for_async_insert=1`, so each insert
waits 50–200 ms for a flush. Turning it off in the test container gave
191 s, but that would hide a real cost, so it's left on.
## Test plan
- [x] 3 green full runs per step above
- [x] 2, 4 and 6 workers green after the maxWorkers change (148.8 s at
4); no leftover users or databases
- [x] Test names before and after the split identical (465)
- [x] tsc and oxlint clean
- [ ] CI: clickhouse store job green
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## ELI5
This change lets ClickHouse store tests run at the same time. Each test
worker gets its own ClickHouse user and database, so workers do not
change each other's data.
## What changed
- **Per-worker database and user.**
`stores/clickhouse/vitest.global-setup.ts` creates one database and one
user for each configured worker. Setup sets each user's password and
default database. It grants current grants on all databases with grant
option. The ClickHouse admin client closes in a `finally` block.
Returned cleanup drops each worker's database before its user.
- **Worker identity.** `stores/clickhouse/vitest.setup.ts` adds
`workerName(worker)`, which returns `mastra_test_w${worker}`, and
`workerPassword()`, which returns `CLICKHOUSE_PASSWORD` or falls back to
`password`. When `VITEST_POOL_ID` is set, the setup assigns the matching
worker name and password to the ClickHouse environment variables. If
`CLICKHOUSE_URL` is set, the setup removes its pathname, username, and
password before it saves the URL. A commit in the PR removes URL
credentials so that the explicit connection options, including the
per-worker user and shared password, apply.
- **Vitest configuration.** `stores/clickhouse/vitest.config.ts` no
longer disables file parallelism. It allows up to four workers and
registers the global setup and setup files.
`stores/clickhouse/tsconfig.json` includes `vitest.global-setup.ts` and
`vitest.setup.ts`.
- **Shared vNext suite.**
`stores/clickhouse/src/storage/domains/observability/v-next/shared-suite.test.ts`
adds the ClickHouse vNext conformance suite, with 60-second test and
hook timeouts. The suite reuses one storage instance and clears storage
between tests. Before discovery checks, it refreshes and waits for both
discovery materialized views. Retry-idempotency checks run `OPTIMIZE
TABLE … FINAL` on five signal tables. `v-next/index.test.ts` drops the
import and registration of this suite. Its other tests remain.
- **Migration queries.**
`stores/clickhouse/src/storage/migration.test.ts` filters
`system.tables` sorting-key lookups by `database = currentDatabase()` in
addition to the table name. Test assertions and migration scenarios are
otherwise unchanged.
- **Scope.** The PR changes test configuration and test files only. It
has no source changes and no changeset.
## Reported results
The author reports the following. These results are not verified in this
review.
- Three green full runs at each performance step. Each run reported
1,140 passed and 415 skipped tests.
- Green runs with 2, 4, and 6 workers.
- No leftover users or databases after teardown.
- The same 465 test names before and after the split.
- Clean `tsc` and `oxlint` checks.
- Local wall time fell from 467 seconds (sequential) to 148–155 seconds
after the suite split.
The ClickHouse store CI job is not checked. Its runtime reduction is not
verified until CI runs.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com>
|
||
|
|
5056a2b1ff |
fix(clickhouse): return tied scores in a stable order from listScores (#29102)
## What `listScores` in the ClickHouse store sorted only by the requested field (`timestamp` by default, or `score`). When several scores share that value, ClickHouse returns them in whatever order it happens to read the data parts, so a page can differ between two identical calls, and paging through ties can repeat or skip scores. Ties are now ordered by `scoreId` in the same direction as the requested sort. `scoreId` is unique per current score, so the order is total and the same on every call. ## Why now #29096 (parallel ClickHouse store tests) failed one test, `keeps ordinary current-score reads proportional to logical rows after merges`. The test writes 5,000 scores with the same timestamp and expects the first page of 10 to match before and after `OPTIMIZE ... FINAL`. Running in parallel changed when parts merged, so the tie order changed. The bug is in the store, not in that PR. ## Test plan - New test `pages through scores with tied timestamps and values in a stable order`: five scores, same timestamp and value, paged 2 at a time in both directions. Fails without the fix (ties come back in scoreId-ascending part order for a DESC sort), passes with it. - Full `stores/clickhouse` suite locally: 22 files, 1,141 passed, 415 skipped. - `tsc --noEmit`, oxlint and prettier clean. Changeset: `@mastra/clickhouse` patch. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ### ELI5 When several scores have the same timestamp or value, the database can return them in a different order on each page. This change adds the score ID as a tie-breaker, so tied scores keep a fixed order and paging is less likely to repeat or skip scores. ### Changes - `stores/clickhouse/src/storage/domains/observability/v-next/scores.ts`: `listScores` keeps the existing timestamp/score `ORDER BY` clause. It appends `s.scoreId`. The direction is `ASC` only when the requested direction is `ASC`. Otherwise, including when no `orderBy` is supplied, the direction is `DESC`. - `stores/clickhouse/src/storage/domains/observability/v-next/index.test.ts`: Adds a test, `pages through scores with tied timestamps and values in a stable order`. It writes five scores with the same timestamp, traceId, scorerId, and value. It pages through them two at a time over three pages. It checks that the default order is the reverse of insertion order. It checks that `score` ascending returns insertion order. - `.changeset/tangy-lands-love.md`: Adds a patch changeset for `@mastra/clickhouse`. ### Validation - The PR reports that the full `stores/clickhouse` suite passed locally (22 files, 1,141 passed, 415 skipped). It also reports that `tsc --noEmit`, oxlint, and Prettier were clean. These results were not independently verified. - The diff does not include a check for the PR `#29096` test, `keeps ordinary current-score reads proportional to logical rows after merges`. The PR reports that this change addresses an order-dependent failure in that test. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com> |
||
|
|
2788a6d959 |
feat(core): recover stopped workflow runs from a loop with one leader (#30281)
## Description
Restarts a default-engine workflow run whose instance crashed without
waiting for another instance to boot. `startWorkers()` now starts a
recovery loop on an instance that runs the orchestration worker, and one
instance among those that share a store runs its passes.
### The loop
Each pass applies the rule `restartAllActiveWorkflowRuns()` already uses
for default-engine runs: restart a `running` or `waiting` run once no
instance holds its run lease and `recovery.workflowGraceMs` passed since
the run or its lease last changed. The loop replaces the per-run recheck
timers the previous PR kept (`#scheduleWorkflowRecoveryRecheck`,
`#recheckWorkflowRun`); the durable-agent rechecks are unchanged.
The loop never touches evented runs. A redelivering broker recovers
them, and on other pubsubs the explicit `restartAllActiveWorkflowRuns()`
restarts them in the caller, as before. That call no longer joins a loop
pass for its evented part, so `startWorkers()` followed by the explicit
call still restarts evented runs on `EventEmitterPubSub`
(`evented/sleep-restart.test.ts` passes).
### Leader
One row, `::recovery-leader`, in the run-owners table, which gates the
loop's ticks only. Each tick of an instance tries a non-forced claim
until it gets the row, then keeps it: the lease helper renews it every
renew interval until `stopWorkers()` or `shutdown()` releases it. Only
the holder's loop lists runs; another instance's tick costs one refused
claim. When the leader dies, the row lapses after the lease TTL and the
next tick elsewhere takes it.
The leader saves listing cost only. Two restarts of one run are still
prevented by the run lease (`restart()` claims non-forced and throws
`WORKFLOW_RESTART_RUN_OWNED`).
An explicit `restartAllActiveWorkflowRuns()` never looks at the row. As
in the previous PR, each booting instance restarts its own stopped
default-engine runs (joining its own pass in flight), so another app's
workflows on the same store are recovered at its boot, and runs
recovered after a redeploy spread over the booting instances. Without
run fencing there is no leader and no loop, and the call restarts every
active run as before.
A failed probe, `getStore` or claim skips that tick and the loop keeps
going. A probe that answers "no fencing" stops the loop for good.
### Timings
- Interval: `recovery.workflowScanIntervalMs`, 60 s by default, each
wait up to 20 % shorter or longer, capped at 2^31 - 1 ms.
- A pass that saw a run in its grace period (and not live) moves the
next pass to that run's due time, no sooner than 1 s later and up to 20
% past it.
- A crashed instance's run is restarted about
`MASTRA_AGENT_THREAD_LEASE_TTL_MS` (15 s) plus `workflowGraceMs` (2 s)
after the crash, plus up to one interval for the next pass. When the
crashed instance was the leader, add up to one more interval for a
follower to take the row.
- `shutdown()` stops the loop when it starts, waits within
`drainTimeout` for the pass in flight and the restarts it handed out,
and releases the row in `stopWorkers()`.
### Config
```ts
const mastra = new Mastra({
recovery: { workflowScanIntervalMs: 30_000 },
});
```
`recovery.workflows: 'off'` turns the loop off. `workers: false`, or a
`MASTRA_WORKERS` filter without `orchestration`, runs no loop. An
instance that registers no default-engine workflow to recover never
claims the leader row.
### LibSQL thread state writes
`@mastra/libsql` thread state writes (`setState`, `deleteState`) now
take the client write lock the store's other writes take. Without it, on
a local database file, a thread state write that ran while the same
store held a write transaction open (a lease claim or renewal) failed
with `SQLITE_BUSY`; the leader's renewals made that much more likely.
### Known limits
- A process that calls `restartAllActiveWorkflowRuns()` but never
`startWorkers()` runs no loop and doesn't recheck the runs it left
alone. A run its boot call found live or in its grace period is left to
the loop of another instance, if any.
- A leader whose renewals fail for longer than the lease TTL can run
passes alongside the new leader for up to one renew interval after the
store answers again. The run lease still prevents a run from being
restarted twice.
- Between boots, the loop recovers only the workflows the leader
registers. Instances that register different workflows on one store get
loop recovery for the leader's workflows only, and the others' runs at
their next boot.
- A run the loop restarts executes on the leader.
- A `mastra worker` process calls `stopWorkers()` on `SIGTERM` and exits
without `shutdown()`, so its default-engine runs wait as long as a
crashed instance's.
- A run that another instance won between the pass's lease read and
`restart()` is looked at again only at the next pass, up to one interval
later.
- Rolling deploys are out of scope for this stack.
### Tests
New `recovery-loop.test.ts` (in-memory store, 600 ms lease TTL):
- a crashed leader's run is restarted by the follower with no boot,
within TTL + 2 intervals + 1.2 s
- instances started together run passes on one of them, and the leader
keeps the row while passes are further apart than the lease TTL
- the leader's `stopWorkers()` releases the row and the follower takes
over
- a tick that `stopWorkers()` overtook (on the first tick's store read,
or on a later tick's probe) claims no row, so another instance can lead
- a stopped run of a workflow only another app registers is restarted by
that app's boot call while a different instance leads, and the leader's
loop never lists that workflow
- next-pass timing table (no run, live run, run in grace, grace ending
within 1 s), timers unref'd
- no listing after `stopWorkers()`; `shutdown()` waits for a pass in
flight
- a loop on `startWorkers('orchestration')`
- the loop keeps going after a failed probe, leader claim, listing or
restart
- no listing and no claim with `workflows: 'off'` or without run fencing
The recheck cases of `restart-active-runs-ownership.test.ts` and
`restart-active-runs-rule.test.ts` are gone with the rechecks: the crash
case covers a crashed owner, the failure table covers transient errors,
and the existing shutdown and limiter cases cover draining and
`workflowConcurrency`. The LibSQL cross-process cases and the MCP
survivor cases boot like a server (`startWorkers()`, then the explicit
call) and recover through the loop.
Checks: core `src/workflows src/mastra src/agent/durable src/events`
(only the two `scheduler.test.ts` retry cases that fail outside UTC),
core typecheck, LibSQL cross-process, Postgres and LibSQL
`recovery-pass.test.ts`, MCP `server-tasks-background`.
## Related issue(s)
Part of #28712. Stack 4 of 4, on top of the recovery pass PR.
## Type of change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [x] Test update
## Checklist
- [x] I have linked the related issue(s) in the description above
- [x] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
ELI5: Sometimes a workflow run stops when its server crashes. This
change adds a background check. One server regularly looks for stopped
runs and restarts them, so the work can finish without a manual restart.
## Overview
This change adds a workflow recovery loop to `@mastra/core`.
`startWorkers()` starts the loop. Among instances that share a store,
only the leader runs default-engine passes. A pass restarts
default-engine runs in `running` or `waiting` status when:
- no instance holds the run lease, and
- the configured grace period has elapsed.
The loop excludes evented runs. Explicit
`restartAllActiveWorkflowRuns()` calls handle evented runs and are not
leader-gated.
The change also updates LibSQL thread-state writes. `setState` and
`deleteState` now run through `withClientWriteLock`.
## Key changes
- `packages/core/src/mastra/index.ts`
- Adds `workflowScanIntervalMs` to `MastraRecoveryConfig`. The default
is 60000 ms. Values are capped at `2 ** 31 - 1`. A value of `-1`
resolves to the default.
- `startWorkers()` starts the loop when the selected workers handle
workflow events and recovery is enabled.
- When workflows use run fencing, the loop elects a leader through a
renewable lease. If fencing is unsupported, the loop stops.
- A pass records the earliest due time for runs still in their grace
period. The loop can then scan sooner.
- Failed lease reads leave runs for a later pass. Restart failures are
tracked per engine.
- Per-run recheck, retry, and backoff logic is removed.
- `stopWorkers()` stops the loop and releases the leader lease. Shutdown
waits for an in-flight pass.
- Explicit `restartAllActiveWorkflowRuns()` calls share an in-flight
default-engine pass. Broker-redelivered evented runs stay broker-owned,
except process-local scheduled runs.
- `packages/core/src/workflows/run-ownership.ts`
- `repeatAfterSettled` accepts a function for the delay and an optional
`immediately` flag.
- Signature changes from `(tick, intervalMs: number)` to `(tick,
intervalMs: number | (() => number), options?)`.
- `stores/libsql/src/storage/domains/thread-state/index.ts`
- `setState` and `deleteState` use `withClientWriteLock`. SQL,
arguments, and error handling are unchanged.
- Tests
- Adds `packages/core/src/mastra/recovery-loop.test.ts`. It covers
leadership, scheduling, shutdown, and failure handling. The file is
added to `vitest.isolated-tests.ts`.
- Removes three tests from `restart-active-runs-ownership.test.ts`.
- In `restart-active-runs-rule.test.ts`, removes one test and rewrites
the lease-read failure test to use the loop.
- Updates the recovery-defaults test, the MCP background server test,
and two LibSQL cross-process tests for the new scan interval.
- Adds a LibSQL thread-state test. It holds a write transaction open
during `setState` and `deleteState`.
- Documentation and changesets
- Updates the workers, server-adapter, workflow overview, and
`mastra-class` reference pages. Adds a "Recovery loop" section to the
workflow overview.
- Corrects the crash-recovery text in the workers page.
- Adds changesets for `@mastra/core` and `@mastra/libsql`.
## Public API changes
- Adds `workflowScanIntervalMs?: number` to `MastraRecoveryConfig`.
- Changes the signature of `repeatAfterSettled` in `run-ownership.ts`.
## Scope
- 19 files changed. 1028 lines added and 379 removed.
## Verification status
- Review findings: not available. No current review findings were
supplied.
- Author-reported checks: the PR objectives list core tests, core
typecheck, LibSQL cross-process tests, Postgres and LibSQL
`recovery-pass.test.ts`, and MCP `server-tasks-background`. These are
author reports. Results on the current head were not available for this
summary.
- Known failures: the author reports that two `scheduler.test.ts` retry
cases fail outside UTC.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
99ec3bd582 |
fix(core): leave evented workflow runs to a redelivering broker and restart only stopped runs (#30279)
## Description This PR stops `restartAllActiveWorkflowRuns()` from restarting evented runs that a redelivering broker (Redis Streams, Valkey Streams, Google Cloud Pub/Sub) already recovers. It also restarts a default-engine run only once no instance holds its run lease and a grace period (2 s by default) has passed. Before, every booting instance restarted every active evented run, so a step interrupted by a crash ran once per booting pod (#28712). The broker already recovers those runs: Redis and Valkey Streams reclaim an unacked message once it has been idle for `reclaimIdleMs`, and Google Cloud Pub/Sub redelivers once the ack deadline passes. ### Evented runs on a broker `PubSub` gets a `redeliversUnacked` getter (default `false`). Redis and Valkey Streams return `true` while the reclaim loop runs (`reclaimIntervalMs > 0`), Google Cloud Pub/Sub returns `true`, `CachingPubSub` forwards to its inner pubsub, and the in-process `EventEmitterPubSub` and the Unix socket pubsub return `false`. When the pubsub (read through `__rawBus()`) redelivers, recovery never lists, reads or restarts an evented run. An evented workflow without declarative schedules costs no listing at all. Evented runs on any other pubsub (`EventEmitterPubSub`, Unix socket, or a third-party pubsub that doesn't override the getter) are restarted at once by the explicit call, as before. ### Scheduled runs The `mastra.pubsub` proxy publishes every workflow event of a scheduled run (`sched_wf_*`, plus the notification dispatcher's row) `localOnly`, so the broker never holds it. Recovery still restarts those runs: an evented workflow with declarative schedules gets one summary listing per status, and only its process-local scheduled runs are kept. The proxy and recovery share one predicate, `isProcessLocalScheduledRunId`. ### Default engine One recovery rule: a `running` or `waiting` run is restarted when its run lease isn't live and `recovery.workflowGraceMs` (new, 2,000 ms by default for every pubsub) has passed since the later of the lease's last expiry and the run's last write. The pass lists summaries (`listWorkflowRuns({ status, perPage: false, summary: true })` per status), reads leases in batches of 100 with `getRunOwnerships`, re-reads a run's lease once its restart gets a slot, and runs every restart on one instance-wide limiter (`recovery.workflowConcurrency`). A run left live or in its grace period gets a per-run recheck timer, as on main. Overlapping calls share one pass. `shutdown()` waits, within `drainTimeout`, for a pass still reading the store and for the restarts it handed out. Without run fencing every active run is restarted, as before. ### Nested resume A default-engine nested run resumed directly now holds its parent's run lease while it runs and hands it to the parent's resume, so recovery doesn't restart the parent meanwhile. On a store without compare-and-set, where the parent stays `suspended`, the nested run holds its own lease instead. Also: `renewWorkflowLease` waits for each renewal to settle before arming the next one (a self-rearming `setTimeout` instead of `setInterval`), and `EventEmitterPubSub`'s JSDoc and the docs say it's in-process only. ### Repro Postgres + Valkey Streams with default reclaim settings (`reclaimIdleMs` 60 s, `reclaimIntervalMs` 30 s). Instance A starts a run whose step records an execution and then hangs; A is killed with `SIGKILL` mid-step. Then 2 or 4 fresh instances boot and all call `restartAllActiveWorkflowRuns()` at once. "At boot" counts step executions caused by those calls; "total" counts every execution after the kill until the run succeeds; "rerun after" is seconds from the kill to the first new execution. 3 trials per row, identical results within each row except for timing. | Build | Engine | Booting pods | Executions at boot | Executions total | Rerun after | |---|---|---|---|---|---| | Base (this PR's base branch) | evented | 2 | 2 | 2 | 0.1 s | | Base | evented | 4 | 4 | 4 | 0.1 s | | Base | default | 2 | 0 | 1 | 15.1 to 16.6 s | | Base | default | 4 | 0 | 1 | 15.0 to 16.9 s | | This PR | evented | 2 | 0 | 1 | 60.1 to 60.2 s | | This PR | evented | 4 | 0 | 1 | 60.1 to 60.2 s | | This PR | default | 2 | 0 | 1 | 17.0 to 19.1 s | | This PR | default | 4 | 0 | 1 | 17.2 to 21.0 s | On this PR the evented rerun is the broker's reclaim: one execution on one pod, about 60 s after the kill. The default-engine rerun comes once A's lease (15 s TTL) lapsed plus the 2 s grace and up to 5 s of recheck jitter. ### Benchmark 3,000 active default-engine runs that another instance holds, among 30,000 finished runs, two workflows. "Before" is a copy of main's sweep run in the same process (full listing, then `createRun()` + `restart()` per run, 5 at a time); "after" is `restartAllActiveWorkflowRuns()` on this PR. Median of 5 calls. | Store | Before | After | Listing payload before | Listing payload after | |---|---|---|---|---| | Postgres | 1,773 ms | 30 ms | 1.12 MB | 0.60 MB | | LibSQL | 627 ms | 44 ms | 1.12 MB | 0.60 MB | The benchmark's snapshots are nearly empty, so the payload gap here is small; with real step results it grows with snapshot size. ### Known limits On a redelivering broker these evented runs stay `running` until someone calls `restart()` on them. Main's boot restart recovered them; stack 2 takes them over. - A sleeping evented run whose instance exits, after a crash or during a normal deploy. The sleep acks its delivery and keeps a `setTimeout` in the process that started it, so any exit of that process loses the timer. - A step whose message Redis or Valkey Streams trimmed. Both trim with `MAXLEN ~ maxStreamLength` (10,000 by default), which also drops entries that are still pending. A step that is in flight while that many newer events are published loses its message if its instance then exits. Deployments with long steps and a high event volume should raise `maxStreamLength`. - A message the broker dropped after `maxDeliveryAttempts` or at `inFlightTimeoutMs`. - A failure publish that gave up after its retries (the worker acks the message). - A run whose `running` snapshot was saved but whose start event never reached the broker, because its process stopped between the two. - A Redis or Valkey instance that restarted without persistence. - A scheduled evented run that was in flight when a deploy stopped its process and removed the workflow's schedule from code. Recovery doesn't list it, so it stays `running` until `restart()`. #28712 is fixed for evented runs on a redelivering broker. These are still restarted by every booting instance, as before: - scheduled evented runs (`sched_wf_*`), whose events never leave their process - evented runs on the in-process `EventEmitterPubSub` or the Unix socket pubsub. Several instances sharing one storage with these pubsubs aren't supported for evented workflows; the docs now say a multi-instance evented deployment needs a broker. With Redis or Valkey Streams, `reclaimIdleMs` must stay at 30 s or more: a message reclaimed while the dead worker's 30 s step fence is still held is dropped as a duplicate. The getter's JSDoc, the adapter docs and the changeset state it. Rolling deploys are out of scope. Stack 3 of 4: storage projections and batch ownership reads, then the finished-step check, then this PR, then the recovery loop with a leader. ## Related issue(s) Fixes #28712 ## Type of change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Code refactoring - [x] Performance improvement - [x] Test update ## Checklist - [x] I have linked the related issue(s) in the description above - [x] I have made corresponding changes to the documentation (if applicable) - [x] I have added tests that prove my fix is effective or that my feature works - [ ] I have addressed all Coderabbit comments on this PR <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ELI5: When a server crashes in the middle of a workflow, Mastra can restart that workflow. Before this change, some workflows ran their steps again on every new server that started, so a step could run many times. This change lets the message broker handle the restart for those workflows, so each step runs only once. Other workflows now wait for a short grace period before they restart. ## Summary This pull request changes how workflow recovery restarts runs after a crash or a boot. It targets duplicate step execution in two cases: - **Evented runs on brokers that redeliver unacknowledged messages.** Recovery skips these runs. The broker redelivers the unacknowledged step events instead. - **Default-engine runs.** Recovery restarts a run only after no live lease remains and a grace period has passed. ### Behavior changes - Google Cloud Pub/Sub, Redis Streams, and Valkey Streams report `redeliversUnacked` as `true` when their reclaim or redelivery path is active. Redis Streams and Valkey Streams report `true` only when `reclaimIntervalMs` is above 0. - `PubSub` adds a `redeliversUnacked` getter that defaults to `false`. `CachingPubSub`, `EventEmitterPubSub`, and `UnixSocketPubSub` expose or declare this capability. - Recovery leaves evented runs to the broker when `redeliversUnacked` is `true`. Scheduled evented runs restart on boot. Other evented runs restart immediately when the broker does not redeliver. - Default-engine recovery applies `recovery.workflowGraceMs`. The default is 2,000 ms. Negative or non-finite values use the default. A run restarts only after its lease is no longer live and the grace period has elapsed since the later of its last update and lease expiry. Runs that are not yet eligible are checked again later. If the storage adapter does not support run fencing, active runs restart without the grace rule. - Recovery lists active run summaries, reads ownership in batches, and rechecks ownership before each restart. - Restarts use a shared concurrency limiter (`createConcurrencyLimiter`). A restarted run holds its slot until it finishes. - Overlapping recovery passes are deduplicated. Shutdown waits for an in-flight pass. - Recovery skips nested runs whose active parent drives them. - Lease renewal runs sequentially (`renewWorkflowLease`, `repeatAfterSettled`). A new renewal starts only after the previous one settles. - Nested resume coordinates parent and child leases. A failed lease claim rejects the resume and leaves the runs suspended. - Run-claim operations return whether a status transition occurred. They return `false` when storage is absent, running snapshots are not persisted, or compare-and-set is not supported. ### Public API changes - `MastraRecoveryConfig.workflowGraceMs?: number` (default 2,000 ms). - `PubSub.redeliversUnacked` getter (default `false`). - `redeliversUnacked` getters on `GoogleCloudPubSub`, `RedisStreamsPubSub`, `ValkeyStreamsPubSub`, and `UnixSocketPubSub`. - `ConcurrencyLimiter` interface and `createConcurrencyLimiter(limit)` function in `run-with-concurrency.ts`. - `renewWorkflowLease`, `repeatAfterSettled`, and `WORKFLOW_RUN_LEASE_RENEW_INTERVAL_MS` in `run-ownership.ts`. - `Run._resume(params)` accepts an optional `takeHeldOwnership` callback. ### Documentation - Crash-recovery and worker docs describe broker reassignment and multi-instance limits. - `EventEmitterPubSub` documents that delivery is process-local. - Redis Streams and Valkey Streams docs recommend `reclaimIdleMs` of at least 30 seconds. Reclaiming earlier can cause duplicate-dropped step events. - The recovery reference documents `workflowGraceMs`. ### Tests and test utilities - New recovery tests cover exit cases (crash and shutdown), recovery reads, rules, ownership during nested resumes, and lease renewal. - Shared test helpers add `sharedBroker`, `killableProcess`, `interceptStarts`, `persistActiveRun`, and `describeRecoveryExitCases`. - Existing tests set `workflowGraceMs: 0` where they need immediate recovery. - Redis Streams and Valkey Streams tests check `redeliversUnacked` against `reclaimIntervalMs`. - LibSQL and Postgres recovery-pass suites run the shared exit cases. ### Scope and limits - The change addresses issue `#28712` for evented runs on redelivering brokers. Evented runs on in-process or Unix socket pubsubs still restart at boot. The PR states that multi-instance evented deployments need a broker. - The PR lists unrecovered cases: sleeping evented runs, trimmed or dropped messages, failed publishes, missing start events, non-persistent Redis or Valkey restarts, and scheduled runs whose schedules were removed. Rolling deploys are out of scope. - Reported benchmark results (median recovery time on 3,000 active and 30,000 finished runs: 1,773 ms to 30 ms on Postgres, 627 ms to 44 ms on LibSQL) and repro results are author-reported. They were not independently verified. - CI and test results: not available. - Review findings and severity counts: not available. ### Size - 41 files changed, 2,594 lines added, 209 lines removed (from the pull request diff). <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
a86a47c7ef |
fix(core): don't rerun evented steps that already finished (#30275)
## Description With this PR the evented processor reads the stored run before it runs a leaf step: - The step finished and the run went on without it: the delivery is dropped. - The step saved its result but nothing took the run past it (the worker stopped between the save and the next step's message): the run continues from the saved result without running the step. - Otherwise the step runs as before. A `workflow.step.run` message can reach a worker after its step already finished: a broker redelivers a message whose ack was lost, a message waits in a queue while `restart()` runs the step, or a restart reads the run just before the step saves its result. Before this PR the worker ran the step again and then every step after it. The decision is a pure function in `workflow-event-processor/finished-occurrence.ts`. It compares the stored record with the one the delivery carried, so a step ID used twice in one workflow still runs at both places. It handles loop iterations, foreach items, parallel and conditional branches, and nested workflows. A resume of the step and time travel skip the check and run the step again on purpose. On stores with workflow step guards (Postgres), records the guards stamped are left to the guards. The check reuses the snapshot `#dispatch` already loads for the delivery, so a leaf delivery still reads the run twice (dispatch and `runLeafStep`'s suspension check), as on `main`. The same check runs on an exhausted delivery. A late `workflow.step.run` of a finished step used to fail the run once it used up its delivery attempts. It is now dropped, or continues the run when the run still has to continue from it. **When the check runs** The workflows store supports run fencing, the workflow keeps `autoRestartActiveRuns` (durable agent workflows opt out, so they are unchanged), it is not a processor workflow, and its `shouldPersistSnapshot` saves running snapshots. A store whose run-fencing probe rejects gets no check for that delivery: the processor logs a warning and runs the step as it does today, and the next delivery asks again. **Known limits** - A step whose worker stops before saving its result still runs again on the next delivery. Make steps with side effects idempotent. - `pruneSnapshot` must keep each step result's `status`, `startedAt` and `endedAt`, a loop step's `metadata.iterationCount`, a foreach step's `suspendPayload.__workflow_meta.foreachOutput`, and `activePaths`. Without them a step delivered again after it finished can run a second time. The changeset and the `workflow.mdx` reference say so. - A workflow whose `shouldPersistSnapshot` returns false for a running run gets no check: its stored active path lags behind the run, so the check can't tell a step the run went past from one it still has to continue from. Dropping there would leave a run stuck after a worker stopped between a step's save and the next step's message, so such a delivery runs the step again, as before this PR. - Nothing serializes two deliveries of one occurrence except the processor's event-ID step fence, which keeps same-ID deliveries apart and only on a pubsub with a lease provider (Google Cloud Pub/Sub has none). Deliveries it doesn't fence that read the same saved result can both continue the run from it. The extra chain can run the next step a second time and ends at the first step whose check finds that step finished and the run past it. - On a store with step guards (Postgres), loop iterations are left to the guards. A delivery the guards don't arbitrate (a restart or a time travel) that arrives after a loop's last iteration was saved runs that iteration again, as before this PR. The same holds for a foreach item that a restart queued again while another worker was still finishing it. **Tests** `test-utils/finished-occurrence-cases.ts` is one shared suite, run on the in-memory store (`finished-occurrence.test.ts`) and on Postgres (`stores/pg/.../evented-finished-occurrence.test.ts`). Replicas are separate `Mastra` instances sharing one store, and a broker is modelled by holding or dropping published events and by handing a recorded event to another replica. Cases: a finished step redelivered while the run goes on and after it ended (step, parallel, foreach, loop, nested), the original message arriving after a restart ran the step, a restart that read the run just before the step saved its result (plain step and a branch), a worker that stopped after saving a step result (step, exhausted delivery, loop's last iteration), an exhausted redelivery while the next step is running or suspended, a step used twice, time travel, and a step ID reused across a suspension in a workflow that skips running snapshots. With the two call sites reverted, 13 of the first 15 cases fail on their assertions (the step-used-twice and time-travel cases check behavior the fix must keep). Without the `shouldPersistSnapshot` condition, the last case fails with the run left `running`. The known-difference blocks for completed-transition replays in `evented-workflow-sleep-restart`, `nested-suspend-loop-redelivery` and `parallel-suspend-redelivery` are removed; those cells now assert the step runs once and fail without the fix. `redelivery-contract.test.ts` opts its workflow out with `autoRestartActiveRuns: false`, since the drain test needs a replay that runs steps again. Checks run locally: `pnpm build:core`, `pnpm --filter ./packages/core check`, core vitest on `src/workflows src/mastra src/agent/durable` (only the two `scheduler.test.ts` retry tests fail; they are time-zone dependent and fail on a machine not set to UTC), the Postgres suite against Docker. Rolling deploys (old and new instances processing one run) are out of scope. ## Related issue(s) Part of #28712. Stack: storage projections (#30274, merged) -> this PR -> recovery pass (#30279) -> recovery loop (#30281). ## Type of change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Code refactoring - [ ] Performance improvement - [x] Test update ## Checklist - [x] I have linked the related issue(s) in the description above - [x] I have made corresponding changes to the documentation (if applicable) - [x] I have added tests that prove my fix is effective or that my feature works - [ ] I have addressed all Coderabbit comments on this PR <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 A workflow step can finish, and then a late copy of its "run this step" message can arrive. Before this change, the system could run the finished step again. Now the system checks the saved run state first. It skips the late message or continues from the saved result. ## Summary This change stops evented workflow leaf steps from running again when a delayed or redelivered `workflow.step.run` message arrives after that step occurrence has finished. ### Behavior - `WorkflowEventProcessor` checks persisted run state before it runs a leaf step. The check uses the new `finishedOccurrence` function. - The check applies only when all of these hold: - The storage adapter supports run fencing. - The workflow is not a processor workflow. - `autoRestartActiveRuns` is not disabled. - The workflow persists running snapshots. - For an eligible delivery, the result is one of three outcomes: - **Drop:** the run has moved past the step occurrence. - **Continue from saved result:** the run still needs to advance past the occurrence. The processor uses the stored result and does not run the step. - **Run:** the step runs as before. - If the run-fencing probe rejects, the processor logs a warning and runs the step as before. - Time travel and resume executions bypass the check and run the step. - Exhausted `workflow.step.run` deliveries get the same drop-or-continue handling before the pending failure is published. - The check covers repeated step IDs, loops, foreach items, branches, and nested workflows. - Canceled and tripwire results run only for restart targets. Success and failure results continue from the stored result when the run has not moved past the occurrence. - Start-guard eligibility checks are centralized in `#guardsStepStart`. - `routeStepRun` classifies each `workflow.step.run` delivery as a leaf, a dispatch, or missing. ### Limits - A worker that stops before it saves a result can still cause a rerun. - Concurrent deliveries that the event-ID step fence does not separate can both continue from the same saved result. - Rolling deploys that run old and new instances on the same run are out of scope. ### Snapshot pruning When `pruneSnapshot` is set, the snapshot must keep these fields for the check to work: - Step result `status`, `startedAt`, and `endedAt`. - Loop `metadata.iterationCount`. - Foreach `suspendPayload.__workflow_meta.foreachOutput`. - `activePaths`. ### Tests - Adds `describeFinishedOccurrenceCases({ getStorage, connect? })`. This shared suite uses separate Mastra replicas on shared storage. It simulates redelivery, dropped messages, process crashes, and held step-end events. - The cases cover sequential, parallel, foreach, loop, and nested workflows. They also cover exhausted deliveries, saved results, time travel, and workflows that do not persist running snapshots. - Adds an in-memory suite and a Postgres suite. Both run the shared cases. - Adds a parameterized `routeStepRun` test suite. - Updates the redelivery tests in `evented-workflow-sleep-restart`, `nested-suspend-loop-redelivery`, and `parallel-suspend-redelivery`. Each now expects one fixed result for every redelivery mode. The mode-dependent extra executions are removed. - Updates `redelivery-contract`. The timer-drain test disables `autoRestartActiveRuns`, and the duplicate-execution reason now names the workflow without the finished-step check. - The author reports these results. This summary did not verify them: - Reverting both check call sites makes 13 of the first 15 cases fail. - Removing the `shouldPersistSnapshot` condition leaves a run in `running` in the final case. - `pnpm build:core`, `pnpm --filter ./packages/core check`, the core Vitest tests, and the Postgres suite against Docker were run. - Two `scheduler.test.ts` retry tests failed in the Vitest run. The author attributes them to time-zone dependence on a non-UTC machine. ### Documentation - The changeset adds a patch note for evented workflows. It limits the claims to storage adapters with run-ownership support. It excludes workflows with `autoRestartActiveRuns: false` and runs whose running snapshot is rejected by `shouldPersistSnapshot`. - `docs/.../workflows/overview.mdx`: On the evented engine, a late event does not rerun a step whose result is saved. Workflows with `autoRestartActiveRuns: false`, including durable agent workflows, still rerun the step. The earlier statement that redelivery reruns the step is removed. - `docs/.../reference/workflows/workflow.mdx`: Returning `false` from `shouldPersistSnapshot` for a running run can cause a redelivered completed step and later steps to run again. The page also lists the additional `pruneSnapshot` fields above. - `redelivery-driver.ts` documentation: Describes when a completed-step redelivery is dropped or resumed, and when it re-executes. The ID-keyed `STEP_FENCE_TTL_MS` fence drops immediate duplicates but not late ones. ### Scope - 15 files changed. Lines: +1379 and −69. - Core logic: `finished-occurrence.ts` (+163), `workflow-event-processor/index.ts` (+249, −26), and `routing.ts` (+68). - Shared test suite `finished-occurrence-cases.ts` adds 747 lines. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
8559e74767 |
feat(storage): workflow run summary listings and batched run ownership reads (#30274)
## Description
Adds `summary: true` listings that return only each run's status,
timestamp and parent, and `WorkflowsStorage.getRunOwnerships()` to read
many runs' ownership at once. Nothing in core calls them yet, so no
workflow behavior changes.
### Summary listings
`listWorkflowRuns({ summary: true })` now reduces each run's snapshot
to:
```ts
{ status?, timestamp?, parentWorkflow?: { workflowId, runId } }
```
`parentWorkflow` is present only when the stored parent has both IDs,
and is never `null`. A field the run doesn't have is left out. `context`
and every other snapshot field are never returned. Run fields outside
the snapshot (`runId`, `workflowName`, `resourceId`, `createdAt`,
`updatedAt`) are unchanged, and so is `total`.
| Where the snapshot is reduced | Stores |
|---|---|
| In the query (only the summary fields leave the database) | PostgreSQL
and LibSQL (already projected `{ status, timestamp }`, now add the
parent), MySQL, MSSQL, OracleDB, Spanner, DSQL, Cloudflare D1 and
Durable Objects, MongoDB (`project()`) |
| In the app, after reading the whole snapshot | DynamoDB, Redis,
Valkey, Upstash, Convex, in-memory |
This PR doesn't change the status filter. Among the SQL stores,
PostgreSQL (expression index on `snapshot ->> 'status'`), Spanner
(generated status column) and OracleDB (function index on `JSON_VALUE`)
filter on an index, and LibSQL storage v2 runs have an indexed `status`
column. MySQL, MSSQL, DSQL and Cloudflare D1 and Durable Objects extract
the status from each row's JSON.
MSSQL now caches whether the snapshot table has a `resourceId` column.
The non-summary `resourceId` filter uses the same cache, with the same
result as before.
### Batch ownership reads
`getRunOwnerships({ runIds })` returns a `Map` from run ID to the record
`getRunOwnership()` would return for that ID; runs that were never
claimed are left out. The default reads each run with
`getRunOwnership()`, at most 8 at a time, and stops starting new reads
once one fails. Overrides:
| Store | Round trips |
|---|---|
| PostgreSQL | One query for all IDs (one array parameter) |
| LibSQL | One query per 500 IDs |
| MySQL, MSSQL | One query per 500 IDs, joining the requested IDs so the
column's collation matches them as the single-run read does |
| MongoDB | One `find` per 1,000 IDs |
| DynamoDB | One strongly consistent `BatchGetItem` per 100 IDs,
unprocessed keys retried with backoff; IDs that differ only in case name
one item, as in the single-run read |
| Upstash | One pipeline request per 100 IDs |
| Redis, Valkey | One single-key script call per ID (works across
cluster hash slots), up to 500 in flight |
| In-memory | Reads every record in one pass |
| Convex, DSQL, Spanner, OracleDB, Cloudflare D1 and Durable Objects |
Base default |
### In-memory `updatedAt`
The in-memory store now moves a run's `updatedAt` on
`updateWorkflowResults()`, `updateWorkflowState()` and
`updateWorkflowRunHeader()`, as the database adapters already did. A
write a status guard rejects leaves it alone.
### Conformance
In `@internal/storage-test-utils`, run on every store with run fencing:
the summary listing shape for v1 and storage v2 runs, including a null
parent, a parent with one ID and a missing timestamp; `getRunOwnerships`
over 1,001 IDs with live, released, expired and unclaimed runs, each
record equal to the single-run read, plus IDs that differ from a claimed
one only by a trailing space or case (MySQL, MSSQL and DynamoDB match
those in the single-run read). The `updatedAt` cases run through
`createWorkflowsTests`. Valkey and Cloudflare Durable Objects don't run
that suite: Valkey writes through the same path as Redis, and Durable
Objects don't implement partial writes.
Checks run locally: `pnpm build:core`, core `check`, lint and `tsc` of
every touched store, and the conformance suite against in-memory,
LibSQL, PostgreSQL, MySQL, MSSQL, MongoDB (replica set), Redis, Valkey,
DynamoDB Local and Upstash (serverless-redis-http). MSSQL has 15
failures outside this PR's cases that fail the same way on `main`
locally. DSQL, Convex, Spanner, OracleDB and Cloudflare were not run
locally.
Stack 1 of 4; the recovery pass in the later PRs uses these reads.
## Related issue(s)
Part of #28712
## Type of change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [x] Performance improvement
- [x] Test update
## Checklist
- [x] I have linked the related issue(s) in the description above
- [ ] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
**ELI5:** Workflow runs can now be listed in a smaller form that holds
only the status, time, and parent workflow. The PR also adds a way to
look up the owners of many runs in one call. Storage adapters return
only the fields that callers need.
## Overview
This PR adds two storage capabilities for workflow runs:
- `listWorkflowRuns({ summary: true })` returns a reduced snapshot for
each run. The snapshot contains `status` and `timestamp`. It adds
`parentWorkflow: { workflowId, runId }` only when both parent IDs are
non-empty strings.
- `WorkflowsStorage.getRunOwnerships({ runIds })` returns ownership
records for many runs. It returns a `Map` keyed by run ID and omits
unclaimed runs.
The in-memory store also changes. Successful writes refresh `updatedAt`.
A write rejected by a status guard leaves `updatedAt` unchanged.
The PR description states that core does not call these APIs yet.
Workflow behavior does not change in this PR.
## Changes by area
**Core (`@mastra/core`)**
- `storage/domains/workflows/base.ts`: Adds a default `getRunOwnerships`
implementation. It deduplicates run IDs and reads at most 8 records
concurrently through `getRunOwnership`. It omits unclaimed runs. A
failed read rejects the batch and stops further reads from starting.
- `storage/domains/workflows/inmemory.ts`: Adds `summary` support to
`listWorkflowRuns`. Summary reduction runs after filtering and
pagination. `updateWorkflowResults`, `updateWorkflowState`, and
`updateWorkflowRunHeader` refresh `updatedAt` after successful writes.
- `storage/types.ts`: Documents the fuller reduced-snapshot shape for
`summary`. It changes no runtime behavior.
**Store adapters**
Summary support (`summary: true`) is added in: Cloudflare D1, Cloudflare
Durable Objects, Convex, DSQL, DynamoDB, LibSQL, MongoDB, MSSQL, MySQL,
OracleDB, PostgreSQL, Redis, Spanner, Upstash, and Valkey.
Reduction happens in the database for these stores:
- SQL stores: PostgreSQL, LibSQL, MySQL, MSSQL, OracleDB, Spanner, DSQL
- Cloudflare D1 and Durable Objects
- MongoDB (projection)
Reduction happens in application code for these stores:
- Convex, DynamoDB, Redis, Upstash, Valkey
`getRunOwnerships` is added in: PostgreSQL, LibSQL, MySQL, MSSQL,
MongoDB, DynamoDB, Redis, Upstash, and Valkey. The batch strategy
differs by store:
- PostgreSQL: one query.
- LibSQL, MySQL, MSSQL: chunks of up to 500 IDs.
- MongoDB: batches of 1,000 IDs.
- DynamoDB: strongly consistent batch reads of up to 100 keys.
Unprocessed keys retry with exponential backoff.
- Upstash: pipeline batches of 100.
- Redis and Valkey: one single-key ownership script call per run. Calls
run in batches of up to 500.
Cloudflare D1 and Durable Objects include `resourceId` in summary
queries only when the table has that column. The column check is cached.
**Conformance test utilities (`stores/_test-utils`)**
- `createSummaryListingTests`: Covers summary shapes, parent metadata
handling, and comparison with full listings. It skips stores without
run-fencing support.
- `createUpdatedAtTests`: Covers `updatedAt` on partial writes. For
storage v2 run journals, it checks header and state updates, including
guard-rejected writes.
- Run-fencing tests: Add batch ownership read cases for live, released,
expired, and unclaimed runs. They also cover inputs larger than
MongoDB's 1,000-item chunk size, collation lookalike IDs, and empty
input. The `untilLapsed` helper polls for up to 5 seconds, checking
every 20 ms.
**Changesets**
Patch changesets cover the affected store packages. A minor changeset
covers `@mastra/core`.
## Scope
- Files changed: 30
- Lines added: 1,555
- Lines removed: 52
## Verification
The author reports these local results. They have not been checked
against the current head:
- `pnpm build:core`, core checks, lint, and TypeScript checks for
touched stores were run.
- Conformance runs were run against the listed stores. The author
reports 15 MSSQL failures outside this PR's test cases. The author
reports the same failures on `main`.
- Conformance tests were not run locally for DSQL, Convex, Spanner,
OracleDB, or Cloudflare.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
956143f6bd |
test(libsql): guard live-run boot recovery (#30264)
Adds a two-process regression test for boot recovery with both durable
and evented agents. A peer blocks in its first tool while a second
Mastra instance opens the same LibSQL database, starts workers,
discovers the running agent, and calls `recoverAllDurableAgents()`. The
test checks that recovery skips the healthy owner, makes zero model
calls in the booting replica, and lets the original peer commit all
three steps. Reuses the T81 peer and IPC gates with no added sleeps or
skipped engine cells.
Run locally with `pnpm build:core`, then `pnpm --filter @mastra/libsql
exec vitest run
src/__tests__/cross-process/xproc-boot-recovery.test.ts`.
Three consecutive local runs passed, followed by another passing run
after restoring the red control:
```text
durable evented result
Green run 1 1901ms 1632ms 2 passed (2)
Green run 2 3186ms 1605ms 2 passed (2)
Green run 3 1399ms 2423ms 2 passed (2)
Restored green 1451ms 1798ms 2 passed (2)
```
For the red control, temporarily changed the recovery branch in
`packages/core/src/agent/durable/execution-fence.ts` from
`backend.tryRecover(executionId, args.untrackedRun)` to
`backend.tryTakeover(executionId)`, rebuilt core, and ran the same test.
Both engines failed on the intended assertions:
```text
Test Files 1 failed (1)
Tests 2 failed (2)
Both durable and evented:
Expected: { agents: 1, recovered: 0, succeeded: 0, failed: 0, skipped: 1 }
Received: { agents: 1, recovered: 1, succeeded: 1, failed: 0, skipped: 0 }
boot recovery must not call the model: expected 4 to be +0
Original peer commits: expected [1] to deeply equal [1, 2, 3]
the peer finishes without work from the booting replica: expected 4 to be +0
```
The bypass was reverted and core rebuilt before the final green run.
There are no production-code changes and no changeset. Formatting,
type-aware lint, and `git diff --check` pass. `pnpm --filter
@mastra/libsql exec tsc --noEmit` is blocked by five existing errors in
unrelated fixtures: one in `dead-claim-pg-agent.ts:10` and four in
`foreach-billing-workflow.ts`.
Fixes COR-1492
Ref COR-1365
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
ELI5: The computer runs a task on one program. A second copy of the
program starts and must not repeat that task. This PR adds a test that
checks this rule. The second copy must skip the task, and the first copy
must finish it.
## Summary
This PR adds one new test file:
`stores/libsql/src/__tests__/cross-process/xproc-boot-recovery.test.ts`
(113 lines added, 0 removed). It contains no production-code changes and
no changeset.
The file defines a cross-process regression test for boot recovery. It
runs once for `durable` and once for `evented` agents. Each run has the
same flow:
1. **Setup.** `beforeEach` sets `MASTRA_AGENT_THREAD_LEASE_TTL_MS` to
`1500`. It then creates the test environment with workers enabled.
2. **Start the peer.** The test spawns a peer process from
`fixtures/t81-peer.ts`. It waits until the peer reports `blocked` at
step 1.
3. **Boot a replica.** The test opens the same LibSQL database from a
second `Mastra` instance. The replica registers a step agent with three
steps. It uses `MockMemory` and workers. Recovery is set to
`durableAgents: 'auto'`.
4. **Discover the run.** The test calls `listActiveRuns()`. It expects
exactly one active run with the peer's `runId`.
5. **Run recovery.** The test calls `recoverAllDurableAgents()`. It
expects `{ agents: 1, recovered: 0, succeeded: 0, failed: 0, skipped: 1
}`. It uses soft assertions for this result.
6. **Check model calls.** The test expects the replica's model call
count to be `0`.
7. **Release the peer.** The test sends `release` and reads the peer
result. It expects no errors. It expects `commit` events for steps `1,
2, 3`. It expects the peer to exit with code `0` and no signal.
8. **Teardown.** `afterEach` releases and kills the peer. It then shuts
down Mastra, closes storage, cleans up the environment, and restores
stubbed environment variables.
The test links to COR-1492 and COR-1436 in a code comment. The comment
says a running snapshot can belong to a healthy replica. Recovery must
find that run without running its tools again.
## Verification
The PR author reports:
- Four passing local runs of `pnpm --filter @mastra/libsql exec vitest
run src/__tests__/cross-process/xproc-boot-recovery.test.ts`, after
`pnpm build:core`.
- A temporary red control caused both engine tests to fail on the
intended assertions. The control was then reverted.
- Formatting, type-aware lint, and `git diff --check` pass.
- `pnpm --filter @mastra/libsql exec tsc --noEmit` reports five errors
in unrelated fixtures.
These results come from the author's report. This review did not run the
tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Co-authored-by: review <review@local>
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
|
||
|
|
d60ba71be0 | chore: version - exit prerelease mode | ||
|
|
5251e7199e |
Fix atomic pubsub replay writes on Redis Cluster (#30174)
> Opened by @abhiaiyer91
## Summary
- Preserve existing `CachingPubSub` replay keys by default, so upgrading
does not make in-flight logs unreadable for other cache backends or
Redis standalone.
- Add `keyFormat: 'redis-cluster'` as an explicit option for new Redis
Cluster deployments. It encodes the topic into a hash tag shared by the
replay list and counter, allowing the atomic Redis Lua append instead of
the `CROSSSLOT` fallback. PubSub topic names remain unchanged.
- Retain the Redis cache fallback for callers that still supply untagged
keys; add coverage for both key formats and edge-case topics.
## Usage
Pass `{ keyFormat: 'redis-cluster' }` as the third argument to `new
CachingPubSub(innerPubSub, redisServerCache, options)` on **all workers
sharing the same cache**. Roll it out for new logs, or after existing
replay logs expire. The default remains the original
`${keyPrefix}${topic}` format; opt-in keys use
`${keyPrefix}{${encodeURIComponent(topic)}}` (with `%EMPTY` for an empty
topic).
## Test plan
- Seven focused core suites: 169 tests passed, with no type errors; core
build/type generation and targeted oxlint passed.
- Convex cache suite: 21 tests passed with original key expectations.
- Redis Docker Compose integration: 14 tests passed.
- Real slot-assigned Redis Cluster: confirmed untagged two-key `EVAL`
fails with `CROSSSLOT`; opted-in replay successfully publishes and reads
indexes 0–2.
Fixes #30149
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
ELI5: Redis Cluster sometimes refuses to run one command that touches
two related cache keys. This PR puts those keys in the same place, so
the command works and event indexes stay in order.
## Summary
This PR changes `CachingPubSub` so that a topic's replay list and its
index counter share a Redis Cluster hash tag.
`RedisServerCache.listPushIndexed` can then run its Lua script on both
keys without a `CROSSSLOT` error.
## Changes
- **`packages/core/src/events/caching-pubsub.ts`**
- Adds an optional `keyFormat?: 'redis-cluster'` option. The default is
`'legacy'`.
- Legacy mode keeps the list key as `keyPrefix + topic`.
- `redis-cluster` mode builds the list key as `keyPrefix + {tag}`. The
tag is `encodeURIComponent(topic)`, or `%EMPTY` when that value is
empty.
- The counter key is now `<list key>:counter`. In `redis-cluster` mode,
it uses the same hash tag as the list key.
- **`stores/redis/src/cache.ts`**
- Updates the comment on the `CROSSSLOT` fallback. The fallback logic is
unchanged. Untagged keys can still trigger `CROSSSLOT`, so the fallback
remains for callers that do not use tagged keys.
- **`.changeset/redis-cluster-atomic-pubsub-replay.md`**
- Adds a patch changeset for `@mastra/core`.
## Behavior and compatibility
- Topic names used for publishing are unchanged.
- The default `legacy` format keeps existing replay keys. Existing logs
stay readable with the default setting.
- Opting into `keyFormat: 'redis-cluster'` changes replay key names.
Workers that use the new format cannot read logs written under the old
names until those logs expire by TTL.
## Scope limits
- The change co-locates only the replay list and counter for each topic.
It does not co-locate those keys with transport streams or other keys
for the same run.
- The change does not add a new default for `keyFormat`.
## Tests and checks
- The diff adds no test file. The PR objectives report test runs, but
this summary does not verify them.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
|
||
|
|
05a6931471 |
chore: version packages (alpha) (#30172)
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. ⚠️⚠️⚠️⚠️⚠️⚠️ `main` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `main`. ⚠️⚠️⚠️⚠️⚠️⚠️ # Releases ## @mastra/client-js@1.60.0-alpha.12 ### Minor Changes - Allow `querySpans()` to sort by `model` or `provider`. ([#30096](https://github.com/mastra-ai/mastra/pull/30096)) ```ts orderBy: [{ field: 'model', direction: 'asc' }]; ``` ### Patch Changes - The generated route types for scores now include the `SCORER` and `PROCESSOR` entity types, matching the entity types experiments can emit. ([#30145](https://github.com/mastra-ai/mastra/pull/30145)) - Add span-local error counts. ([#30088](https://github.com/mastra-ai/mastra/pull/30088)) - Updated dependencies [[`56f654f`]( |
||
|
|
f0710446e0 |
feat(observability): sort spans by model and provider (#30096)
Adds ascending/descending `model` and `provider` ordering to
`querySpans()` across Core, server, Client JS, PostgreSQL, DuckDB and
ClickHouse. No schema changes or backfill.
```ts
orderBy: [{ field: 'model', direction: 'asc' }]
```
Sorting uses the dedicated columns before pagination, with
case-sensitive UTF-8 ordering, missing values last, and identity
tie-breakers. String cursors preserve their values and remain bound to
the query and trusted scope. Legacy attribute-based display values
remain available without changing cursor boundaries. Trace/thread sort
fields stay unchanged. The server advertises
`spanQueryModelProviderSort` and rejects unsupported stores.
This PR is API-only. Studio sorting controls and live-refresh changes
are tracked in
[OBS-906](https://linear.app/kepler-crm/issue/OBS-906/add-model-and-provider-sorting-to-spans-explorer).
Enabling those controls needs raw model/provider sorting values
available to the existing head-row merge, including when polls are
ordered by end time; using display values can disagree with database
ordering. The PR has no remaining Playground UI changes or release
entry.
Validation: Core/server tests, real storage conformance across
PostgreSQL, DuckDB and ClickHouse, mixed-core checks, and
Core/server/adapter typechecks passed for the API implementation. After
removing the UI changes, both existing span component/hook suites pass
(52 tests). Changed-file lint and docs checks including CI Vale pass.
A bounded local ClickHouse 26.6.8.7 comparison used 180,000 synthetic
rows, 10,000 matches per tenant, distinct tenants across cases, both
directions and continuation pages, and disabled query/condition caches.
Median CPU was 21.9 ms for start time, 19.4 ms for model, and 21.4 ms
for provider; median query memory was about 4.6 MiB for each (maximum
6.8/8.4/7.2 MiB). These are selection-query measurements on synthetic
OSS data, not a production benchmark.
Part of OBS-896. Platform API support and package pins follow this OSS
merge and snapshot publication. UI enablement is tracked separately in
OBS-906.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
**ELI5:** Trace spans can now be sorted by their model name and provider
name, in both directions. The database does the sorting, so users can
page through the results in the right order.
## Overview
This change adds `model` and `provider` as sort fields for span queries.
It applies to the Core API, the server handlers, and the PostgreSQL,
DuckDB, and ClickHouse stores. The diff has 31 changed files, with 320
lines added and 43 removed.
## Core changes
- `SPAN_QUERY_ORDER_FIELDS` now includes `model` and `provider`.
- `SpanQueryPlanOptions` adds `allowSpanModelProviderOrdering?:
boolean`. Ordering by these fields requires this option.
- Trace and thread queries do not support these fields.
- Cursor validation accepts string or null values for `model` and
`provider`. Timestamp normalization applies only to `startedAt` and
`endedAt`.
- `SpanQuerySelectedRow` adds an optional `cursorSortValue`. Response
construction removes this field from returned spans. It uses the value
for the next-page cursor, and falls back to the row's ordering field
when the value is absent.
- The base storage feature union adds `span-query-model-provider-sort`.
## Server changes
- `assertObservabilitySpanModelProviderSortSupported` returns HTTP 501
with `SPAN_QUERY_UNSUPPORTED` when a store does not declare
`span-query-model-provider-sort`. The check runs before `querySpans` is
called.
- `ObservabilityStorageCapabilities` and its empty default add
`spanQueryModelProviderSort`, which defaults to `false`.
- The capability is reported as supported only when span queries are
supported, the core order fields include both `model` and `provider`,
and the store declares the feature.
- The system capabilities schema adds `spanQueryModelProviderSort:
z.boolean().default(false)`.
- The route handler permits model and provider ordering and performs the
capability check.
## Store changes
- **PostgreSQL:** Sorts by `model` and `provider` use `COLLATE "C"`. The
cursor bounds cast sort values to `text COLLATE "C"` when `stringSort`
is enabled. The feature is advertised only when the core order fields
include both fields.
- **DuckDB:** Sorts by `model` and `provider` use `"C"` collation.
Cursor values cast as strings with the same collation. The feature is
advertised only when the core order fields include both fields.
- **ClickHouse:** `afterCursor` treats `model` and `provider` as
strings. Null rows are included after the cursor for numeric and string
sorts. The feature is advertised only when the core order fields include
both fields.
- Returned `model` and `provider` values may still come from span
attributes. The code comments state that these fallback values are not
used for filtering or sorting.
## Tests and conformance
- Core tests cover ordering gating, rejection in trace and thread
queries, string and null cursor values, and rejection of cursors when
the order, filters, authorization, or scope change.
- Server tests cover the 501 rejection for `durationMs`, `model`, and
`provider` on older stores. They also cover capability parsing and the
capability flag in the system endpoint.
- Shared conformance tests in `stores/_test-utils` cover ascending and
descending pagination at page sizes 1, 2, and 3. They cover empty
strings, missing values, case and Unicode ordering, ties, and filter
exclusion. They also check that missing values sort last and that legacy
attribute values are not treated as dedicated fields.
- Store feature-list tests for ClickHouse, DuckDB, and PostgreSQL expect
the new feature flag.
## Changesets and documentation
- Changesets add minor release notes for `@mastra/core`,
`@mastra/server`, `@mastra/client-js`, `@mastra/duckdb`,
`@mastra/clickhouse`, and `@mastra/pg`.
- The advanced query reference adds the `spanQueryModelProviderSort`
capability row. It documents `model` and `provider` ordering. It states
that the comparison is case-sensitive and that missing values sort last
in both directions.
## Scope and limits
- This diff does not include Studio UI changes. The commit history says
UI enablement is deferred to a separate issue because it needs raw model
and provider values for head-row merging.
- This diff does not include client-js source changes. Its client-js
changeset documents the new sort fields.
- The diff does not include schema changes or backfill.
- The PR description reports these results, which this review did not
run or verify:
- Core and server tests, and real shared-storage conformance and
mixed-core checks, pass.
- Core, server, and adapter typechecks pass. Staged-file lint and docs
checks, including CI Vale, pass.
- 58 Studio and MSW tests pass.
- The full Playground UI typecheck reports four unrelated chart and form
errors, and the library declaration build fails.
- A local ClickHouse 26.6.8.7 comparison with 180,000 synthetic rows
measured median CPU of 21.9 ms for start time, 19.4 ms for model, and
21.4 ms for provider. Median query memory was about 4.6 MiB for each.
These figures are not production measurements.
- Platform backend and UI wiring, and package pins, remain under
OBS-896.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
a2998f4084 |
test(storage): verify rollback reporters settle before client release (#30160)
## Summary - Make PostgreSQL pooled/pinned and DSQL pooled async rollback reporter tests assert that reporting finishes before transaction settlement and pooled release. - Use plain async reporters to catch unhandled promise rejections rather than `vi.fn` wrappers. ## Test plan - PostgreSQL transaction tests: 18 passed; DSQL transaction tests: 9 passed. - Both package TypeScript checks, targeted Oxlint, Prettier and `git diff --check` passed. - Confirmed the changed tests fail against the parent of merged #30157: PG 2 failures, DSQL 1 failure (plus unhandled rejections). Follow-up to #30157; test-only correction to the regression coverage for its already-merged implementation. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ELI5: The change makes the database tests check the right order of events. A rollback error must be reported before the database connection is released and before the transaction ends. The tests now fail if that order is wrong. ## Summary This change updates two test files for the rollback reporter behavior in PostgreSQL and DSQL transaction clients. It changes test code only. No production code changes. Diff stats: 2 files changed, 33 insertions, 9 deletions. ### `stores/dsql/src/storage/client.tx-serialize.test.ts` - The reporter is now a plain async function, not a `vi.fn` wrapper. - The reporter stores the error it receives. - The reporter waits 10 ms, records `reported`, and then throws `async logger failed`. - The test mocks `client.release` to record `released`. - The test records `tx-settled` after the transaction call rejects with the original error. - Assertions: - The reporter received `rollbackError`. - `client.release` was called with `true`. - The event order is `['reported', 'released', 'tx-settled']`. ### `stores/pg/src/storage/client.tx-serialize.test.ts` - The same reporter change applies to the shared async rollback reporter test. - For the pooled case (`kind === 'pool'`): - `client.release` records `released`. - Assertions check that `client.release` was called with `true`. - The event order is `['reported', 'released', 'tx-settled']`. - For the pinned case: - No release tracking is added. - Assertions check that `adapter.shouldDestroyClient` is `true`. - The event order is `['reported', 'tx-settled']`. ## Not verified - The PR reports test runs, type checks, lint, and format checks. I did not run them. Their results are not available from this review. - The PR reports that the changed tests fail against the parent of the referenced prior change. I did not verify this. It is not available from this review. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com> Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
055a8c3b82 | chore: version packages | ||
|
|
b1948fb073 |
fix(storage): await rollback diagnostics before releasing clients (#30157) | ||
|
|
743a02786e |
feat(valkey): identify Mastra-created clients with a clientInfoTag (#30156) | ||
|
|
ce631e94c6 |
fix(storage): prevent poisoned PostgreSQL transactions from returning to the pool (#30144) | ||
|
|
c014d187bb |
chore(deps): update docker images (major) (#19788) | ||
|
|
9e82af0237 |
chore: version packages (alpha) (#30074)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
|
4884ee08a0 |
fix(agent-controller): answer parked tool approvals when the permission rules allow or deny them (#30114)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
1689ce38e1 |
perf(agent-controller): read only the newest page when listing threads with a limit (#30090)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com> Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com> |
||
|
|
259f142f46 |
fix(pg): support scalar and nested operators in vector elemMatch filters (#30095) | ||
|
|
cd041a5bb0 |
perf(pg): index thread metadata for listThreads metadata filters (#30086)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
c8fecd5d6f |
fix(pg): preserve all nested $elemMatch conditions in vector filters (#30091) | ||
|
|
aa49b049d0 |
feat(observability): order completed spans by duration (#30085)
Adds `durationMs` ascending/descending ordering to `querySpans()`, combined with existing duration filters and stable cursor pagination. Duration is the completed span’s elapsed time; zero is valid. Trace ordering and trace runtime semantics are unchanged. The server advertises `spanQueryDurationSort` and rejects unsupported stores before execution. A span-specific field list and default-off planner option protect old servers paired with newer Core. Updated adapters advertise support only when Core understands it. PostgreSQL and DuckDB reuse their numeric ordering paths; ClickHouse handles numeric cursor parameters and normalizes duration values returned as Int64 strings. Includes generated API types, documentation and package changesets. No schema migration. Implements the OSS portion of [OBS-899](https://linear.app/kepler-crm/issue/OBS-899). [OBS-904](https://linear.app/kepler-crm/issue/OBS-904) tracks Platform integration after merge and a snapshot from main. UI is outside this backend change. Validation: 134 Core, 112 server, 125 Client JS, 100 PostgreSQL, 94 DuckDB and 103 ClickHouse tests passed (one existing ClickHouse skip). Shared conformance exercises both directions, filters, zero durations and ties across page sizes. The new Core tests fail before the implementation; reverting the ClickHouse cursor fix makes both new pagination cases fail by skipping rows. Old-Core adapters omit the new feature; old stores receive a structured 501. Core/server/store typechecks, Core/server/SDK builds, changed-file lint/format and required docs checks passed. Vale reports six warnings in unchanged text. Local Node 23.11.0 is outside the supported 22/24 versions; CI must verify supported runtimes. A bounded local ClickHouse 26.6.8.7 experiment used 20,000 synthetic completed spans, the same duration filter and 50-row first/second pages in both directions. Duration ordering peaked at 3.43–3.67 MiB versus 3.42–3.66 MiB for timestamp ordering; both read 20,000 rows/1,488,890 bytes. Duration query CPU was 7–25 ms. These are single-run local measurements, not production capacity evidence; no ingestion or storage layout changed. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 This PR lets users sort completed spans by how long they took. It keeps pagination stable and enables the feature only when Core and the storage adapter support it. ## Changes - Added ascending and descending `durationMs` ordering to `querySpans()`. - Added duration-aware cursor validation and stable pagination. - Added `spanQueryDurationSort` capability reporting. - Added HTTP 501 handling for unsupported stores. - Added compatibility gates for older Core versions. - Updated PostgreSQL, DuckDB, and ClickHouse adapters. - Converted ClickHouse duration values to numbers. - Added Core, server, adapter, and conformance tests. - Updated generated API types, documentation, and package changesets. - No schema migration was added. ## Validation - Reported passing tests: Core 134, server 112, Client JS 125, PostgreSQL 100, DuckDB 94, and ClickHouse 103. - One existing ClickHouse test remains skipped. - Reported typechecks, builds, lint, formatting, and documentation checks passed. - Vale reported six warnings in unchanged text. - CI verification on supported Node.js versions is not available. - Diff size: 28 files changed, with 220 additions and 8 deletions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
b0ef22e7bf |
feat(schedules): Add bounded one-off schedules (#25131)
> Opened by @abhiaiyer91 Refs #24610 ## Summary Schedules can now fire once or stop after a given time, so you no longer need to pin a cron expression to a date and clean it up yourself. - **New `completed` status**: the scheduler and timing reconciliation set it when no automatic fires remain. It's written in the same atomic `nextFireAt` check-and-set used to claim a run, so a schedule still fires only once across instances. A cron with no future occurrence left also moves to `completed` instead of erroring. `pause`/`resume` on a completed schedule return 409. Changing `runAt`/`endAt` reactivates it only when the resulting timing can still fire. `list()` hides completed schedules unless you pass `status: 'completed'`. - **Declarative workflow schedules**: `createWorkflow({ schedule: { runAt } })` and `{ cron, endAt }` are supported. Past times are accepted so redeploys don't fail, and a completed schedule isn't re-armed unless its timing changes. - **Trigger kind**: one-off fires report `trigger.kind: 'once'`. - `run(id)` doesn't complete a one-off schedule. ## Surfaces - **Core**: types, validation, scheduler claim logic, create/update/pause/resume/list, declarative sync. - **Storage**: new nullable `run_at`/`end_at` columns for libsql, pg, mysql, mongodb, spanner, convex and in-memory. Existing columns are untouched: one-off schedules store `cron` as `''`. `updateScheduleNextFire` takes an optional `newStatus`. - **Server/client-js/CLI**: request bodies accept `runAt`/`endAt` (one-of rule checked in the schema). Responses include the new fields and `completed`. Route types and CLI route metadata regenerated. - **Studio**: shows the run time for one-off schedules, shows the `endAt` bound, adds a `completed` badge and hides pause/resume for completed schedules. - **Docs + changeset.** ## Not in this PR (remaining #24610 work) - ISO 8601 string inputs for `runAt`/`endAt` (numbers and `Date` are accepted today). - The template `schedule_once` tool. ## Test plan - [x] core: schedules, scheduler, cron, schedule-config and scheduler integration tests (260 passing), clean `tsc` - [x] Shared adapter suite gained tests for `runAt`/`endAt` round-trip and atomic status CAS. Passing on libsql, pg, mysql and mongodb (Docker), plus convex server tests. **Not run on Spanner (no emulator available); typecheck only.** - [x] server schedules handler tests (51 passing, including one-off/bounded creation and body validation) - [x] client-js and playground typecheck clean; playground schedules tests passing <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 Schedules can now run once at a chosen time or repeat until a chosen end time. They complete when their planned runs are over, and changing their timing can make them active again. ## Summary - Added `runAt` for one-off agent and workflow schedules. API-created schedules require a future `runAt`; declarative workflow schedules can register past times as `completed`. - Added `endAt` for cron schedules. A schedule becomes `completed` when its next run would occur after `endAt`, or when no future run remains. - Completed schedules cannot be paused or resumed. Timing changes can reactivate a completed schedule when a future run remains. `list()` hides completed schedules unless callers filter with `status: 'completed'`. - Resuming a paused one-off schedule after its `runAt` has passed allows it to fire on the next scheduler tick. Calling `run(id)` manually does not complete a one-off schedule. - Added the `once` trigger kind for scheduled one-off agent fires. Manual runs remain `manual`. - Declarative workflow schedules preserve completed status across redeploys when timing is unchanged. Timing changes can re-arm them. - Updated schedule APIs, storage adapters, Studio, and documentation. Storage persists `runAt` and `endAt` and supports atomic status updates. ## Checks Current-head CI, build, and test results: not available. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Abhi Aiyer <abhiaiyer91@gmail.com> Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com> Co-authored-by: NikAiyer <nick.aiyer@gmail.com> Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com> Co-authored-by: Ehindero Israel <iehindero419@stu.ui.edu.ng> |
||
|
|
3a6ca42696 |
chore: version packages (alpha) (#30026)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
|
c860b9bb43 |
feat(observability): support OSS feedback replacement (OBS-887) (#30059)
Implements [OBS-887](https://linear.app/kepler-crm/issue/OBS-887/add-oss-feedback-replacement-and-export). Add OSS `PUT /api/observability/feedback/:feedbackId` with the existing wrapped client request. Replacements retain the original feedback ID, trace ID, timestamp, and tenant; omitted optional fields are cleared. A full replacement can create a missing record. OSS persists locally before responding, then independently notifies supporting exporters. Resolve mode still requires the local anchor. The SDK also exposes optional `replaceFeedback()` for independent exporter delivery without a producer-side storage lookup. Replacement events use a separate optional exporter hook. `MastraPlatformExporter` sends PUT with stable identity across bounded anchor retries, and flush/shutdown drain deliveries. Older exporters skip replacements rather than replaying them as creates. Unsupported storage/package combinations return 501 or skip unavailable export support. No UI changes; review-status PATCH propagation remains tracked in OBS-785. In-memory, PostgreSQL vNext, DuckDB, and ClickHouse vNext support replacement. In-memory, PostgreSQL, and DuckDB enforce replacement identity at the write. PostgreSQL coordinates writes by feedback ID, including duplicates submitted with a different timestamp. Duplicate HTTP creates return the stored identity and timestamp and re-notify exporters with accepted values. Tenant mismatches remain hidden, and an omitted SDK replacement author stays null. HTTP/exporter creates use `onConflict: 'ignore'` so a delayed original create doesn't undo an edit; direct storage calls keep their legacy upsert default. **ClickHouse requires no migration:** it uses the existing schema and read-before-insert protection. Sequential delayed creates are covered, but overlapping writes from separate processes can race; this is not a globally ordered or exactly-once protocol. Latest review fixes: DuckDB's lazy facade delegates replacement to its guarded writer; concurrent coverage runs 20 overlapping request pairs per identity variant. Direct storage replacements normalize an omitted author to null. PostgreSQL single-feedback reads use the primary for read-after-write consistency, while list/aggregate reads retain the configured read client. Batch advisory locks use one ordered query with unchanged lock keys. Validation: the stronger DuckDB regression failed on all four identity variants before the facade fix. Afterward, the entire DuckDB package passed 718 tests across 21 files in three consecutive runs. PostgreSQL passed 57 feedback tests, including a real stale read snapshot and overlapping reversed 1,000-ID batches. Shared replacement tests passed on in-memory (12), and the new author regression passed on ClickHouse without a migration. Core/storage builds and declarations, typed lint, and formatting passed. Earlier HTTP (85 cases on each of four adapters), generic Hono routes (2,848), server/system, exporter, and docs checks remain recorded in the PR history. Local tests use Node 23.11.0; CI validates the supported runtime. The unrelated Traces UI fade assertion failure was fixed in https://github.com/mastra-ai/mastra/pull/30068; this branch includes that fix from main. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 This PR lets users replace feedback without changing its identity. It stores the replacement locally and sends it to supported exporters without allowing delayed creates to overwrite it. ## Changes - Added `PUT /api/observability/feedback/:feedbackId`. - Preserved the feedback ID, trace ID, timestamp, and tenant. - Cleared omitted optional fields. - Added capability detection and `501` responses for unsupported storage. - Added replacement event types, SDK support, client types, route metadata, and documentation. - Added replacement delivery, stable retries, flush handling, and shutdown draining for `MastraPlatformExporter`. - Added replacement support for in-memory, PostgreSQL vNext, DuckDB, and ClickHouse vNext. - Added `onConflict: 'ignore'` for HTTP and exporter creates. Direct storage calls retain their upsert default. - Older exporters skip replacements instead of replaying them as creates. - Added ClickHouse support without a migration. Concurrent writes from separate processes can still race. - No UI changes were included. ## Validation The author reports passing the HTTP route and contract suites, feedback storage suites, partition and SQL tests, server and system tests, record-builder tests, build, typecheck, lint, and documentation checks. For the current head, the available CI result is `2027dev Evals: skipped`. Other listed CI checks target different commits or remain in progress. Current-head CI results are otherwise not available. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
3b27a10322 |
fix(pg): keep thread owner and createdAt when saveThread hits an existing id (#29149) | ||
|
|
562a451e22 |
fix(pg): stop concurrent thread metadata updates from dropping keys across processes (#29150) | ||
|
|
23cc34a9c5 |
chore: version packages (alpha) (#29938)
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. ⚠️⚠️⚠️⚠️⚠️⚠️ `main` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `main`. ⚠️⚠️⚠️⚠️⚠️⚠️ # Releases > The changelog information of each package has been omitted from this message, as the content exceeds the size limit. ## @mastra/factory@0.28.0-alpha.6 ## @mastra/code-sdk@1.19.0-alpha.6 ## @mastra/playground-ui@69.0.0-alpha.7 ## @mastra/client-js@1.60.0-alpha.6 ## @mastra/react@1.15.0-alpha.6 ## @mastra/deployer-cloud@1.83.0-alpha.6 ## @mastra/longmemeval@1.1.42-alpha.6 ## @mastra/opencode@0.1.42-alpha.6 ## create-factory@0.4.3-alpha.6 ## mastracode@0.50.0-alpha.6 ## @mastra/agent-builder@1.2.1-alpha.3 ## mastra@1.39.0-alpha.6 ## @mastra/core@1.83.0-alpha.6 ## @mastra/deployer@1.83.0-alpha.6 ## @mastra/editor@0.17.1-alpha.4 ## @mastra/mcp@2.9.0-alpha.3 ## @mastra/mcp-docs-server@1.3.13-alpha.9 ## @mastra/memory@1.41.1-alpha.3 ## @mastra/rag@2.7.3-alpha.2 ## @mastra/schema-compat@1.3.18-alpha.0 ## @mastra/server@1.83.0-alpha.6 ## @mastra/elysia@0.1.23-alpha.6 ## @mastra/express@1.5.27-alpha.6 ## @mastra/fastify@1.5.27-alpha.6 ## @mastra/hono@1.7.25-alpha.6 ## @mastra/koa@1.7.27-alpha.6 ## @mastra/nestjs@0.3.0-alpha.6 ## @mastra/next@0.2.41-alpha.6 ## @mastra/server-adapters-test-suite@0.3.0-alpha.3 ## @mastra/tanstack-start@0.2.41-alpha.6 ## @mastra/upstash@1.6.1-alpha.0 ## @mastra/vectorize@1.1.5-alpha.0 ## @mastra/voice-google-gemini-live@0.14.19-alpha.0 ## @mastra/voice-openai-realtime@0.14.9-alpha.0 ## @mastra/voice-xai-realtime@0.2.18-alpha.0 ## @mastra/inngest@1.11.2-alpha.4 ## @mastra/temporal@0.6.0-alpha.6 ## create-mastra@1.39.0-alpha.6 ## @internal/playground@1.39.0-alpha.6 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
|
0af43a79d4 |
test(pg): run store test files in parallel on per-worker databases (#29093)
> **Stacked on #29092** (pg: build indexes on newly created tables
without CONCURRENTLY). The base branch is that PR's branch, so the diff
here is only this change. Merge that one first: without it, parallel
files cold-booting the same schema hit ~1 s waits per index.
## What this does
The pg store tests ran every file one after another (`fileParallelism:
false`) against a single database. This runs them on 4 workers, each
with its own database:
- `stores/pg/vitest.config.ts`: two projects. `e2e:stores/pg` runs files
in parallel; `e2e:stores/pg-sequential` runs `integration.test.ts`
afterwards on its own (`groupOrder` 1). Its delta polling reads a
cluster-wide snapshot horizon (`pg_snapshot_xmin`), so it can't share
the server with other writers.
- `vitest.global-setup.ts` creates one database per worker
(`mastra_test_w1..wN`) on both the pg and timescale containers, sized
from the resolved `maxWorkers`, so a CLI override of the worker count
can't leave a worker without a database. Teardown drops them.
- `vitest.setup.ts` points each worker at its database (`POSTGRES_DB`,
`DB_URL`, `POSTGRES_CONNECTION_STRING`, `PG_VNEXT_TIMESCALE_URL`) using
`VITEST_POOL_ID`.
- The bulk-status notification test gets a 20 s timeout. It went over 5
s under parallel load.
Test config only: no source changes, no changeset.
## Numbers
| | pg suite |
|---|---:|
| sequential, before the cold-boot fix | 701 s |
| sequential, with the cold-boot fix | ~323 s |
| this PR (4 workers) | 110–133 s |
Every run: 3905 passed / 20 skipped. CI pg job estimate: ~5.4 → ~1.9 min
(unverified until CI runs).
## Test plan
- [x] 3 full green runs at 4 workers (3905 / 20 each), plus one after
the maxWorkers change (133 s)
- [x] 6-worker subset green; no leftover databases after teardown
- [x] Root `vitest list`: 85 parallel + 1 sequential file, same as
before
- [ ] CI: pg store job green
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## ELI5
The PostgreSQL store tests now run in parallel. Each worker uses its own
database. One test file runs afterward by itself because it reads
cluster-wide state that other writers can affect.
## Changes
- Adds global setup in `stores/pg/vitest.global-setup.ts`. Before each
run, it creates one database per configured worker on the PostgreSQL and
Timescale servers. It force-drops any existing database with the same
name first. The returned teardown force-drops these databases. The
PostgreSQL server installs the `vector` extension. The Timescale server
installs no extensions.
- Adds `stores/pg/vitest.setup.ts`, which exports `workerDatabaseName`.
The name format is `mastra_test_w{worker}`. When `VITEST_POOL_ID` is
set, the setup sets `POSTGRES_DB` to the worker database. It also
updates `DB_URL`, `POSTGRES_CONNECTION_STRING`, and
`PG_VNEXT_TIMESCALE_URL` to use it. When the variable is unset, the
setup changes nothing.
- Splits `stores/pg/vitest.config.ts` into two projects:
- `e2e:stores/pg` runs in parallel with up to four workers. It excludes
performance, pooler, and cluster-wide integration tests.
- `e2e:stores/pg-sequential` runs the cluster-wide integration file with
file parallelism disabled. It uses group order 1, so it runs after the
parallel project at group order 0.
- Removes the previous `fileParallelism: false` setting and the coverage
reporter settings.
- Increases the bulk-status-update notification test timeout to 20
seconds in `stores/pg/src/storage/domains/notifications/index.test.ts`.
It also documents the load-dependent runtime of its 1,200 sequential
inserts.
- Adds `vitest.global-setup.ts` and `vitest.setup.ts` to the `include`
array in `stores/pg/tsconfig.json`.
## Test and CI results
- Author-reported, not verified in this review:
- Three full runs at four workers passed (3,905 passed, 20 skipped
each).
- A subset run at six workers passed, and no databases remained after
teardown.
- Suite time was 110–133 seconds with four workers.
- CI results for the pg store job: not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com>
|
||
|
|
22b24cc40a |
fix(core): apply semanticRecall.filter during agent runs (#29418)
## Description
`semanticRecall.filter` worked in `Memory.recall()` but not during
`agent.generate()` or `agent.stream()`. During an agent run, recall
happens in the `SemanticRecall` processor, which never read `filter` and
queried the vector store with the scope filter only. So messages from
threads that didn't match the filter ended up in the prompt.
This PR changes
`packages/core/src/processors/memory/semantic-recall.ts`:
```diff
query filter
- scopeFilter
+ filter ? { $and: [scopeFilter, filter] } : scopeFilter // same as Memory.recall()
embedding metadata written during agent runs
- { message_id, thread_id, resource_id, ... }
+ { ...threadMetadata (only when filter is set), message_id, thread_id, resource_id, ... }
```
- The filter applies to both query paths: client-side embedder and
self-embedding stores.
- Embeddings written during agent runs had no thread metadata, so even
with the query fixed, a filter like `{ mode: { $eq: 'work' } }` could
never match messages that agents saved. When a filter is set, the
processor now reads the thread from storage and stores its metadata on
the embedding, as `Memory.saveMessages()` already does. It reads from
storage because durable and recovered runs put only the thread id in the
memory context.
- Without a filter, nothing about writes changes. Some vector stores
reject nested metadata values and this processor only logs a failed
upsert, so the metadata is written only when a filter needs it.
- Docs: added `filter` to the processor reference and a note on the
semantic recall page that messages agents saved before a filter was set
don't match it until they're saved or indexed again.
## Related issue(s)
Closes #29352
## Type of change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [x] Test update
## Checklist
- [x] I have linked the related issue(s) in the description above
- [x] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR
## Evidence
Before = `semantic-recall.ts` from `origin/main`, with the same tests.
After = this branch.
- **Core, real Agent + MockMemory + in-memory vector store that applies
filters**
(`packages/core/src/processors/memory/semantic-recall-filter.test.ts`,
new): generate and stream, filter vs no filter, self-embedding store,
thread scope, durable-shaped context (`thread: { id }` only), thread
metadata written only with a filter.
**Before:** `Tests 7 failed | 2 passed (9)` (the two no-filter cases
pass on both).
**After:** `Tests 9 passed (9)`.
- **Memory integration, real `Memory` + LibSQL store and vector +
Agent**
(`packages/memory/integration-tests/src/semantic-recall-metadata-filter.test.ts`):
generate and stream; messages saved with `saveMessages()`, by earlier
agent runs, and in a thread the run created with metadata. Also checks
`Memory.recall()` agrees with the agent run.
**Before:** `Tests 6 failed | 2 passed (8)` (the two existing `recall()`
cases pass on both).
**After:** `Tests 8 passed (8)`.
- **The issue's own repro script** (LibSQL, mock embedder and model),
run against local builds:
**Before:** `agent prompt contains personal message: true`
**After:** `agent prompt contains personal message: false`
- Neighbouring core suites (`semantic-recall.test.ts`,
`processor-caching.test.ts`, `memory-config.test.ts`,
`embedding-dimension.test.ts`): 69/69 pass. `@mastra/core` typecheck
passes; oxlint, oxfmt and lint-staged are clean on the changed files.
Not tested: a real DurableAgent run end to end. The durable context
shape is covered by calling the Memory-built processor with `thread: {
id }` only, as `durable-agent.ts` and `finalize-run.ts` set it.
## Merge Danger
**Door:** two-way. A revert restores the old behavior. Embeddings
already written with thread metadata stay in the vector store but are
harmless extra fields.
**Blast radius:** agents with `semanticRecall.filter` configured. They
now recall fewer messages (only matching threads), which is the
documented behavior. Messages saved by older versions have no thread
metadata, so they won't match a filter until re-saved or re-indexed; the
changeset and docs say so. Users without a filter see no change in
queries.
**After merging main (#29826):** main's shared embedding helper now
always stores thread metadata on agent-run embeddings, so this PR no
longer adds its own filter-only metadata read; docs, changeset and the
unit test were updated to match.
**Cloudflare Vectorize (`@mastra/vectorize`):** the scope and user
filter are combined as `{ $and: [...] }` (the same shape
`Memory.recall()` already sends on main), which the Vectorize translator
rejected, so recall with a filter returned nothing there. The translator
now merges a filter that is exactly `{ $and: [...] }` into one object.
Several conditions on one field merge only as one lower plus one upper
bound; other same-field combinations and non-object clauses throw. Not
run against a live Vectorize index; checked against Cloudflare's
metadata-filtering docs and 38 translator tests (Sol reviewed 3 rounds).
## Council review
Rounds: 4
| Seat | Round 1 | Round 2 | Round 3 | Round 4 |
| --- | --- | --- | --- | --- |
| Claude Opus 5.5 | not run | not run | 8.5/10 | 9/10 |
| GPT-6.1 Sol | 9/10 (Tests 8, Code quality 8) | 9.6/10 | 9.5/10 |
9.5/10 |
| Grok 4.7 | empty seat | empty seat | empty seat | empty seat |
Created with Claude Opus 5.5 in T3 Code.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## ELI5
Agents now use the configured metadata filter when they search for
relevant messages. This keeps messages from unrelated threads out of the
agent’s prompt. Cloudflare Vectorize can also process the combined
filters used for these searches.
## Changes
- `SemanticRecall` combines the configured `VectorFilter` with the
thread or resource scope filter using `$and`.
- When a filter is configured, the processor reads thread metadata from
storage and writes it to message embeddings. Older embeddings may need
to be saved or indexed again.
- The Vectorize filter translator merges supported top-level `$and`
clauses, including repeated conditions with the same value. It rejects
unsupported combinations on the same field.
- Updated documentation and added tests for `generate()`, `stream()`,
self-embedding queries, and Vectorize filter translation.
## Verification
- Reported: 9/9 core filter tests, 8/8 integration tests, 69/69
neighboring core tests, 38 Vectorize translator tests, and the
`@mastra/core` typecheck passed.
- A real DurableAgent run was not tested end to end. A core test covers
a processor request context that contains only a thread ID.
- Live Cloudflare Vectorize testing: not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
01b2b958af |
fix(core): fail the run when a loop callback throws, instead of retrying it or leaving it running (#29850) | ||
|
|
b4152bd5da |
fix(upstash): negate every key of multi-key $not, $nor and $and/$or vector filters (#29760) | ||
|
|
cf2cd73b85 |
Revert "fix(core): snapshot agent stream events when they are publish… (#29947)
…ed (#29803)"
This reverts commit
|
||
|
|
779d2875b1 |
chore: version packages (alpha) (#29912)
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. ⚠️⚠️⚠️⚠️⚠️⚠️ `main` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `main`. ⚠️⚠️⚠️⚠️⚠️⚠️ # Releases > The changelog information of each package has been omitted from this message, as the content exceeds the size limit. ## @mastra/auth-studio@1.4.0-alpha.1 ## @mastra/factory@0.28.0-alpha.5 ## @mastra/mcp-apps-components@0.1.0-alpha.5 ## @mastra/playground-ui@69.0.0-alpha.6 ## @mastra/client-js@1.60.0-alpha.5 ## @mastra/react@1.15.0-alpha.5 ## @mastra/deployer-cloud@1.83.0-alpha.5 ## @mastra/longmemeval@1.1.42-alpha.5 ## @mastra/opencode@0.1.42-alpha.5 ## create-factory@0.4.3-alpha.5 ## @mastra/code-sdk@1.19.0-alpha.5 ## mastracode@0.50.0-alpha.5 ## @mastra/agent-builder@1.2.1-alpha.2 ## mastra@1.39.0-alpha.5 ## @mastra/core@1.83.0-alpha.5 ## @mastra/deployer@1.83.0-alpha.5 ## @mastra/editor@0.17.1-alpha.3 ## @mastra/mcp@2.9.0-alpha.2 ## @mastra/mcp-docs-server@1.3.13-alpha.8 ## @mastra/memory@1.41.1-alpha.2 ## @mastra/rag@2.7.3-alpha.1 ## @mastra/server@1.83.0-alpha.5 ## @mastra/elysia@0.1.23-alpha.5 ## @mastra/express@1.5.27-alpha.5 ## @mastra/fastify@1.5.27-alpha.5 ## @mastra/hono@1.7.25-alpha.5 ## @mastra/koa@1.7.27-alpha.5 ## @mastra/nestjs@0.3.0-alpha.5 ## @mastra/next@0.2.41-alpha.5 ## @mastra/server-adapters-test-suite@0.3.0-alpha.2 ## @mastra/tanstack-start@0.2.41-alpha.5 ## @mastra/libsql@1.30.0-alpha.2 ## @mastra/turso@0.1.21-alpha.2 ## @mastra/inngest@1.11.2-alpha.3 ## @mastra/temporal@0.6.0-alpha.5 ## create-mastra@1.39.0-alpha.5 ## @internal/playground@1.39.0-alpha.5 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
|
6563e7abfc |
fix(libsql): keep vector indexes writable after truncateIndex (#26802)
## What broke `LibSQLVector.truncateIndex()` ran `DELETE FROM <table>` with no `WHERE` clause. On a table with a `libsql_vector_idx` index, that statement empties the table but leaves the old rows in the vector index's shadow table. The next insert then fails: ``` SQLITE_ERROR: vector index(insert): failed to insert shadow row ``` So once you truncated an index, you couldn't write to it again. I reproduced this on both file-backed and `:memory:` databases with `@libsql/client` 0.18: after the bare delete the shadow table still has 1 row and the reinsert fails. The same cycle with `WHERE rowid IS NOT NULL` leaves 0 shadow rows and the reinsert succeeds. Upstream tracking: tursodatabase/libsql#2289. ## The fix `truncateIndex()` now runs `DELETE FROM <table> WHERE rowid IS NOT NULL`. It still deletes every row, and in the repro it also removes the shadow rows (0 left after the delete). The source change is one line in `stores/libsql/src/vector/index.ts`, and it runs for every connection type. ## Tests New tests in `stores/libsql/src/vector/index.test.ts`. They check behaviour only, using public methods: - file database: insert, truncate, insert again, then query and get only the new vector - file database: truncate an empty index, then run repeated truncate/insert cycles - file database: single-row and filter deletes still work, and the index still accepts inserts afterwards - `:memory:` database: insert, truncate, insert again, then query I ran these tests against the old `truncateIndex`. Both file truncate tests and the `:memory:` test fail with the shadow-row error. The partial-delete test passes there, because it guards existing behaviour. With the fix, the whole file passes: 146 passed, 1 skipped (`npx vitest run src/vector/index.test.ts`). oxlint and oxfmt are clean on the changed files. ## Not tested - Remote Turso/sqld URLs and embedded replicas (`syncUrl`). They run the same statement, but I didn't test them against a server. - Cost on large tables. I haven't measured the qualified delete's performance. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 Truncating a local LibSQL vector index could leave behind data that stopped new vectors from being added. This change removes the vector rows while keeping the index ready for new inserts. ## Changes - Updated `LibSQLVector` truncation to delete rows with `rowid IS NOT NULL`. - Added tests for file-backed and in-memory databases. They cover reinserting and querying vectors after truncation, repeated truncation, and partial deletes. - Added a patch changeset for `@mastra/libsql`. ## Validation Test and lint results: not available. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Jamie Lee Nichols <10267569+CodingButter@users.noreply.github.com> Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
7c4dd88400 |
chore: version packages (alpha) (#29889)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
|
80db9c3ba2 |
chore: run oxfmt instead of prettier in the last six pre-commit configs (#29892)
We moved from Prettier to oxfmt a while ago and CI checks formatting
with `oxfmt --check`, but six packages still ran `prettier --write` in
their pre-commit hook: the Discord, Teams and Telegram channels, the
Turso and OracleDB stores, and the DeepEval observability package. A
commit touching those could get reformatted by Prettier and then fail
the format check in CI.
They now use the same command as every other package.
```diff
- '*.{json,md,yml,yaml}': ['prettier --write'],
+ '*.{json,md,yml,yaml}': ['oxfmt --no-error-on-unmatched-pattern'],
```
`explorations/longmemeval` still uses Prettier in its hook. I left it
alone because `explorations/` is excluded from the oxfmt check.
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
|