0
Fork 0
mirror of https://github.com/mastra-ai/mastra.git synced 2026-11-17 08:44:46 +00:00
Commit graph

1,827 commits

Author SHA1 Message Date
dane-ai-mastra[bot]
d60ba71be0 chore: version - exit prerelease mode 2026-11-12 21:12:29 +00:00
mastra-platform[bot]
5251e7199e
Fix atomic pubsub replay writes on Redis Cluster (#30174)
> Opened by @abhiaiyer91

## Summary
- Preserve existing `CachingPubSub` replay keys by default, so upgrading
does not make in-flight logs unreadable for other cache backends or
Redis standalone.
- Add `keyFormat: 'redis-cluster'` as an explicit option for new Redis
Cluster deployments. It encodes the topic into a hash tag shared by the
replay list and counter, allowing the atomic Redis Lua append instead of
the `CROSSSLOT` fallback. PubSub topic names remain unchanged.
- Retain the Redis cache fallback for callers that still supply untagged
keys; add coverage for both key formats and edge-case topics.

## Usage
Pass `{ keyFormat: 'redis-cluster' }` as the third argument to `new
CachingPubSub(innerPubSub, redisServerCache, options)` on **all workers
sharing the same cache**. Roll it out for new logs, or after existing
replay logs expire. The default remains the original
`${keyPrefix}${topic}` format; opt-in keys use
`${keyPrefix}{${encodeURIComponent(topic)}}` (with `%EMPTY` for an empty
topic).

## Test plan
- Seven focused core suites: 169 tests passed, with no type errors; core
build/type generation and targeted oxlint passed.
- Convex cache suite: 21 tests passed with original key expectations.
- Redis Docker Compose integration: 14 tests passed.
- Real slot-assigned Redis Cluster: confirmed untagged two-key `EVAL`
fails with `CROSSSLOT`; opted-in replay successfully publishes and reads
indexes 0–2.

Fixes #30149

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

ELI5: Redis Cluster sometimes refuses to run one command that touches
two related cache keys. This PR puts those keys in the same place, so
the command works and event indexes stay in order.

## Summary

This PR changes `CachingPubSub` so that a topic's replay list and its
index counter share a Redis Cluster hash tag.
`RedisServerCache.listPushIndexed` can then run its Lua script on both
keys without a `CROSSSLOT` error.

## Changes

- **`packages/core/src/events/caching-pubsub.ts`**
- Adds an optional `keyFormat?: 'redis-cluster'` option. The default is
`'legacy'`.
  - Legacy mode keeps the list key as `keyPrefix + topic`.
- `redis-cluster` mode builds the list key as `keyPrefix + {tag}`. The
tag is `encodeURIComponent(topic)`, or `%EMPTY` when that value is
empty.
- The counter key is now `<list key>:counter`. In `redis-cluster` mode,
it uses the same hash tag as the list key.
- **`stores/redis/src/cache.ts`**
- Updates the comment on the `CROSSSLOT` fallback. The fallback logic is
unchanged. Untagged keys can still trigger `CROSSSLOT`, so the fallback
remains for callers that do not use tagged keys.
- **`.changeset/redis-cluster-atomic-pubsub-replay.md`**
  - Adds a patch changeset for `@mastra/core`.

## Behavior and compatibility

- Topic names used for publishing are unchanged.
- The default `legacy` format keeps existing replay keys. Existing logs
stay readable with the default setting.
- Opting into `keyFormat: 'redis-cluster'` changes replay key names.
Workers that use the new format cannot read logs written under the old
names until those logs expire by TTL.

## Scope limits

- The change co-locates only the replay list and counter for each topic.
It does not co-locate those keys with transport streams or other keys
for the same run.
- The change does not add a new default for `keyFormat`.

## Tests and checks

- The diff adds no test file. The PR objectives report test runs, but
this summary does not verify them.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-12 06:27:56 -08:00
dane-ai-mastra[bot]
05a6931471
chore: version packages (alpha) (#30172)
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases
## @mastra/client-js@1.60.0-alpha.12

### Minor Changes

- Allow `querySpans()` to sort by `model` or `provider`.
([#30096](https://github.com/mastra-ai/mastra/pull/30096))

  ```ts
  orderBy: [{ field: 'model', direction: 'asc' }];
  ```

### Patch Changes

- The generated route types for scores now include the `SCORER` and
`PROCESSOR` entity types, matching the entity types experiments can
emit. ([#30145](https://github.com/mastra-ai/mastra/pull/30145))

- Add span-local error counts.
([#30088](https://github.com/mastra-ai/mastra/pull/30088))

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
## @mastra/factory@0.28.0-alpha.14

### Minor Changes

- A card's sandbox now stops as soon as the card is done with that
context, instead of idling until the provider times it out.
([#29981](https://github.com/mastra-ai/mastra/pull/29981))

- When a card moves out of the last phase of a context into a phase that
runs in another one, Factory stops that context's sandbox. Triage →
Planning stops the triage sandbox; Planning → Building keeps the shared
`work` sandbox running. A run still in flight on the session keeps its
sandbox until it ends.
- When a card finishes, Factory stops every sandbox it holds, from any
Factory process, by identity through the configured sandbox provider.
Before, only the process that had opened a sandbox stopped it.
- A card that comes back to a context resumes the same sandbox with its
files.

Stopping never fails the move. Sandboxes are stopped, not destroyed;
deleting a session still destroys its sandbox.

- Added two read routes that answer what the new session page shows
without sending a project's cards.
([#30097](https://github.com/mastra-ai/mastra/pull/30097))

`GET /web/factory/projects/:id/work-items/starters` counts the cards
behind each starter on the project's default board, such as the cards
whose run failed, the pull requests awaiting review and the cards not
yet triaged.

`GET /web/factory/projects/:id/work-items/activity?since=<instant>`
returns how many cards the project has, and when cards were created and
completed since that instant. `since` is required and must be an ISO
date-time such as `2026-06-01T00:00:00.000Z`. The answer carries an
`ETag`, so asking again for the same `since` with `If-None-Match`
returns a 304 with no body when nothing changed.

### Patch Changes

- Fixed environment templates that could stop rebuilding for good. When
a build start was never recorded and the sandbox provider can't list its
builds (Mastra Platform), every later Build now, schedule and push run
used to skip with `unresolved` forever. Factory now treats that start as
failed 30 minutes after its lease lapsed and the next trigger builds.
([#30148](https://github.com/mastra-ai/mastra/pull/30148))

Saving a change to a template input (setup commands, the working
directory, settings, environment repositories, source-control
connections or Build variables) now rebuilds the template once the
project has one, so new sandboxes stop rerunning setup on every boot.
Saves that change nothing don't build, and a change saved during any
build, scheduled ones included, is built when that build ends.

- Factory sessions and MCP clients can now read the Factory's audit
trail with `factory_list_audit_events`: who did what to which work item,
run, branch or setting, and when. Filter by actor (`me`, a user id, or
an agent, optionally with the agent runs a person started), namespace or
exact actions, work item and time range, and page back with a cursor.
The tool only returns this Factory's events and never returns request
context (IP address, user agent), actor emails or event metadata beyond
who started an agent run. It leaves out events from or about other
people's private sessions.
([#30165](https://github.com/mastra-ai/mastra/pull/30165))

- Added a "Join the issue's card instead" switch to intake routes in the
Factory UI. A route for pull requests that close an issue can now put
the pull request on that issue's card instead of giving it a card of its
own; until now only the route Factory seeds could do this. The switch
shows on new and saved routes whose conditions include "Source kind is
Pull request" and "Pull request closes an issue is Yes".
([#30197](https://github.com/mastra-ai/mastra/pull/30197))

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/code-sdk@1.19.0-alpha.14
  - @mastra/mcp@2.9.0-alpha.4
## @mastra/observability@1.24.0-alpha.7

### Minor Changes

- Add span-local error counts.
([#30088](https://github.com/mastra-ai/mastra/pull/30088))

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
## @mastra/core@1.83.0-alpha.12

### Minor Changes

- Add model and provider as span query sort fields.
([#30096](https://github.com/mastra-ai/mastra/pull/30096))

  ```ts
  orderBy: [{ field: 'model', direction: 'asc' }];
  ```

- Add span-local error counts.
([#30088](https://github.com/mastra-ai/mastra/pull/30088))

### Patch Changes

- Name the agent in the missing-storage error thrown by `Memory`. When
an agent's memory never received storage because the agent was never
attached to a Mastra instance, accessing `memory.storage` now fails with
`[Agent:<name>] Memory requires a storage provider to function. ...`, so
it is clear which agent is misconfigured. The registration and
standalone-storage guidance added in #29972 is unchanged, and agents
that inherit storage from their Mastra or configure it directly on
`Memory` are unaffected.
([#30164](https://github.com/mastra-ai/mastra/pull/30164))

- Experiments with a scorer or processor target now save their scores,
and workflow step scorers in experiments now save their step scores.
Previously these scores were silently discarded.
([#30145](https://github.com/mastra-ai/mastra/pull/30145))
## @mastra/mcp-apps-components@0.1.0-alpha.7

### Minor Changes

- Dashboards authored in chat now accept the same widgets as the
dashboard editor: KPIs that combine values (ratio, difference, max,
fallback), read trace, span or score aggregates (for example p95 span
duration) and use duration, percent or score formats; line charts of
scores; six more chart colors (blue, orange, pink, yellow, red, purple);
and optional KPI icon and detail, series color and table empty message.
([#30192](https://github.com/mastra-ai/mastra/pull/30192))

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
## @mastra/playground-ui@69.0.0-alpha.14

### Minor Changes

- Widened what dashboard widgets can show:
([#30192](https://github.com/mastra-ai/mastra/pull/30192))

- KPIs can combine values with `ratio`, `minus`, `max` and `coalesce`,
read trace and span aggregates (for example `duration.p95`) and score
aggregates, and are formatted as `duration`, `percent` or `score`.
Combined values are compared with the previous window.
- Charts accept six more colors: `blue`, `orange`, `pink`, `yellow`,
`red` and `purple`.
- KPI `icon` and `detail`, series `color` and table `emptyMessage` are
optional. Series without a color pick one from a palette, and tables
without an empty message show "No data".

### Patch Changes

- Fixed hover highlights on cards and other surfaces stopping short of
the edge. The highlight now covers the whole surface and follows its
rounded corners exactly, including the sidebar credits card and
clickable cards. Hover highlights on these surfaces now switch instantly
instead of fading.
([#30047](https://github.com/mastra-ai/mastra/pull/30047))

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`f071044`](f0710446e0),
[`987566e`](987566e882),
[`72abae6`](72abae68eb),
[`987566e`](987566e882),
[`f071044`](f0710446e0)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/client-js@1.60.0-alpha.12
  - @mastra/react@1.15.0-alpha.12
## @mastra/server@1.83.0-alpha.12

### Minor Changes

- Support model and provider sorting in span queries.
([#30096](https://github.com/mastra-ai/mastra/pull/30096))

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
## @mastra/clickhouse@1.30.0-alpha.4

### Minor Changes

- Sort span queries by model or provider in ascending or descending
order. ([#30096](https://github.com/mastra-ai/mastra/pull/30096))

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
## @mastra/duckdb@1.19.0-alpha.5

### Minor Changes

- Sort span queries by model or provider in ascending or descending
order. ([#30096](https://github.com/mastra-ai/mastra/pull/30096))

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
## @mastra/pg@1.36.0-alpha.6

### Minor Changes

- Sort span queries by model or provider in ascending or descending
order. ([#30096](https://github.com/mastra-ai/mastra/pull/30096))

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
## @mastra/react@1.15.0-alpha.12

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`f071044`](f0710446e0),
[`987566e`](987566e882),
[`72abae6`](72abae68eb),
[`987566e`](987566e882),
[`f071044`](f0710446e0)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/client-js@1.60.0-alpha.12
## @mastra/deployer-cloud@1.83.0-alpha.12

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/deployer@1.83.0-alpha.12
## @mastra/longmemeval@1.1.42-alpha.12

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/libsql@1.30.0-alpha.3
## @mastra/opencode@0.1.42-alpha.12

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/libsql@1.30.0-alpha.3
## create-factory@0.4.3-alpha.14

### Patch Changes

- Updated dependencies
[[`1fbc69a`](1fbc69aa9a),
[`4e378c3`](4e378c3e26),
[`1fbc69a`](1fbc69aa9a)]:
  - mastra@1.39.0-alpha.14
## @mastra/code-sdk@1.19.0-alpha.14

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`72abae6`](72abae68eb),
[`f071044`](f0710446e0),
[`f071044`](f0710446e0),
[`987566e`](987566e882),
[`72abae6`](72abae68eb),
[`987566e`](987566e882),
[`f071044`](f0710446e0),
[`f071044`](f0710446e0)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/observability@1.24.0-alpha.7
  - @mastra/duckdb@1.19.0-alpha.5
  - @mastra/client-js@1.60.0-alpha.12
  - @mastra/pg@1.36.0-alpha.6
  - @mastra/mcp@2.9.0-alpha.4
  - @mastra/libsql@1.30.0-alpha.3
## mastracode@0.50.0-alpha.13

### Patch Changes

- Fix `/settings` crashing when mouse preferences are missing from saved
settings by defaulting to Off.
([#30184](https://github.com/mastra-ai/mastra/pull/30184))

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`72abae6`](72abae68eb),
[`f071044`](f0710446e0),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882),
[`f071044`](f0710446e0)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/observability@1.24.0-alpha.7
  - @mastra/duckdb@1.19.0-alpha.5
  - @mastra/pg@1.36.0-alpha.6
  - @mastra/code-sdk@1.19.0-alpha.14
  - @mastra/mcp@2.9.0-alpha.4
  - @mastra/libsql@1.30.0-alpha.3
## @mastra/arize@1.3.30-alpha.7

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/otel-exporter@1.5.2-alpha.7
## @mastra/arthur@0.4.30-alpha.7

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/otel-exporter@1.5.2-alpha.7
## @mastra/braintrust@1.3.27-alpha.7

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`72abae6`](72abae68eb),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/observability@1.24.0-alpha.7
## @mastra/datadog@1.4.22-alpha.7

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`72abae6`](72abae68eb),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/observability@1.24.0-alpha.7
## @mastra/deepeval@0.1.24-alpha.7

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`72abae6`](72abae68eb),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/observability@1.24.0-alpha.7
## @mastra/laminar@1.3.32-alpha.7

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`72abae6`](72abae68eb),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/observability@1.24.0-alpha.7
## @mastra/langfuse@1.6.6-alpha.7

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`72abae6`](72abae68eb),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/observability@1.24.0-alpha.7
  - @mastra/otel-exporter@1.5.2-alpha.7
## @mastra/langsmith@1.4.3-alpha.7

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`72abae6`](72abae68eb),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/observability@1.24.0-alpha.7
## @mastra/otel-bridge@1.5.22-alpha.7

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`72abae6`](72abae68eb),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/observability@1.24.0-alpha.7
  - @mastra/otel-exporter@1.5.2-alpha.7
## @mastra/otel-exporter@1.5.2-alpha.7

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`72abae6`](72abae68eb),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/observability@1.24.0-alpha.7
## @mastra/posthog@1.4.7-alpha.7

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`72abae6`](72abae68eb),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/observability@1.24.0-alpha.7
## @mastra/sentry@1.2.32-alpha.7

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`72abae6`](72abae68eb),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/observability@1.24.0-alpha.7
  - @mastra/otel-exporter@1.5.2-alpha.7
## mastra@1.39.0-alpha.14

### Patch Changes

- Added `mastra api factory work-item starters` and `mastra api factory
work-item activity`. The first counts the cards behind each new-session
starter on a project's default board. The second lists when a project's
cards were created and completed since an instant:
([#30097](https://github.com/mastra-ai/mastra/pull/30097))

  ```bash
  mastra api factory work-item starters <project-id>
mastra api factory work-item activity <project-id>
'{"since":"2026-06-01T00:00:00.000Z"}'
  ```

- Added a "Join the issue's card instead" switch to intake routes in the
Factory UI. A route for pull requests that close an issue can now put
the pull request on that issue's card instead of giving it a card of its
own; until now only the route Factory seeds could do this. The switch
shows on new and saved routes whose conditions include "Source kind is
Pull request" and "Pull request closes an issue is Yes".
([#30197](https://github.com/mastra-ai/mastra/pull/30197))

- Improved the Factory new session page on large boards. Its starters
and its activity band now load the counts and dates they show instead of
downloading every card of the project. They show the same content as
before. ([#30097](https://github.com/mastra-ai/mastra/pull/30097))

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/deployer@1.83.0-alpha.12
## @mastra/deployer@1.83.0-alpha.12

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`f071044`](f0710446e0),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/server@1.83.0-alpha.12
## @mastra/mcp-docs-server@1.3.13-alpha.16

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
## @mastra/elysia@0.1.23-alpha.12

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`f071044`](f0710446e0),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/server@1.83.0-alpha.12
## @mastra/express@1.5.27-alpha.12

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`f071044`](f0710446e0),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/server@1.83.0-alpha.12
## @mastra/fastify@1.5.27-alpha.12

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`f071044`](f0710446e0),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/server@1.83.0-alpha.12
## @mastra/hono@1.7.25-alpha.12

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`f071044`](f0710446e0),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/server@1.83.0-alpha.12
## @mastra/koa@1.7.27-alpha.12

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`f071044`](f0710446e0),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/server@1.83.0-alpha.12
## @mastra/nestjs@0.3.0-alpha.12

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`f071044`](f0710446e0),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/server@1.83.0-alpha.12
## @mastra/next@0.2.41-alpha.12

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`f071044`](f0710446e0),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/server@1.83.0-alpha.12
  - @mastra/hono@1.7.25-alpha.12
## @mastra/tanstack-start@0.2.41-alpha.12

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`f071044`](f0710446e0),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/server@1.83.0-alpha.12
  - @mastra/hono@1.7.25-alpha.12
## @mastra/voice-openai-realtime@0.14.9-alpha.2

### Patch Changes

- Use gpt-realtime-2.1-mini by default for OpenAI Realtime voice
connections instead of the retired
gpt-4o-mini-realtime-preview-2024-12-17 model.
([#30183](https://github.com/mastra-ai/mastra/pull/30183))
## @mastra/inngest@1.11.2-alpha.8

### Patch Changes

- Support span-local error counts in Inngest workflows.
([#30088](https://github.com/mastra-ai/mastra/pull/30088))

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
## @mastra/temporal@0.6.0-alpha.12

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`f071044`](f0710446e0),
[`72abae6`](72abae68eb),
[`987566e`](987566e882)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/deployer@1.83.0-alpha.12
## create-mastra@1.39.0-alpha.14


## @internal/playground@1.39.0-alpha.14

### Patch Changes

- Updated dependencies
[[`56f654f`](56f654fc0d),
[`f071044`](f0710446e0),
[`987566e`](987566e882),
[`72abae6`](72abae68eb),
[`f9088d8`](f9088d8510),
[`987566e`](987566e882),
[`f071044`](f0710446e0),
[`43d1e9c`](43d1e9c16d)]:
  - @mastra/core@1.83.0-alpha.12
  - @mastra/client-js@1.60.0-alpha.12
  - @mastra/playground-ui@69.0.0-alpha.14
  - @mastra/react@1.15.0-alpha.12

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-11-12 14:42:02 +01:00
Eric Pinzur
f0710446e0
feat(observability): sort spans by model and provider (#30096)
Adds ascending/descending `model` and `provider` ordering to
`querySpans()` across Core, server, Client JS, PostgreSQL, DuckDB and
ClickHouse. No schema changes or backfill.

```ts
orderBy: [{ field: 'model', direction: 'asc' }]
```

Sorting uses the dedicated columns before pagination, with
case-sensitive UTF-8 ordering, missing values last, and identity
tie-breakers. String cursors preserve their values and remain bound to
the query and trusted scope. Legacy attribute-based display values
remain available without changing cursor boundaries. Trace/thread sort
fields stay unchanged. The server advertises
`spanQueryModelProviderSort` and rejects unsupported stores.

This PR is API-only. Studio sorting controls and live-refresh changes
are tracked in
[OBS-906](https://linear.app/kepler-crm/issue/OBS-906/add-model-and-provider-sorting-to-spans-explorer).
Enabling those controls needs raw model/provider sorting values
available to the existing head-row merge, including when polls are
ordered by end time; using display values can disagree with database
ordering. The PR has no remaining Playground UI changes or release
entry.

Validation: Core/server tests, real storage conformance across
PostgreSQL, DuckDB and ClickHouse, mixed-core checks, and
Core/server/adapter typechecks passed for the API implementation. After
removing the UI changes, both existing span component/hook suites pass
(52 tests). Changed-file lint and docs checks including CI Vale pass.

A bounded local ClickHouse 26.6.8.7 comparison used 180,000 synthetic
rows, 10,000 matches per tenant, distinct tenants across cases, both
directions and continuation pages, and disabled query/condition caches.
Median CPU was 21.9 ms for start time, 19.4 ms for model, and 21.4 ms
for provider; median query memory was about 4.6 MiB for each (maximum
6.8/8.4/7.2 MiB). These are selection-query measurements on synthetic
OSS data, not a production benchmark.

Part of OBS-896. Platform API support and package pins follow this OSS
merge and snapshot publication. UI enablement is tracked separately in
OBS-906.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

**ELI5:** Trace spans can now be sorted by their model name and provider
name, in both directions. The database does the sorting, so users can
page through the results in the right order.

## Overview

This change adds `model` and `provider` as sort fields for span queries.
It applies to the Core API, the server handlers, and the PostgreSQL,
DuckDB, and ClickHouse stores. The diff has 31 changed files, with 320
lines added and 43 removed.

## Core changes

- `SPAN_QUERY_ORDER_FIELDS` now includes `model` and `provider`.
- `SpanQueryPlanOptions` adds `allowSpanModelProviderOrdering?:
boolean`. Ordering by these fields requires this option.
- Trace and thread queries do not support these fields.
- Cursor validation accepts string or null values for `model` and
`provider`. Timestamp normalization applies only to `startedAt` and
`endedAt`.
- `SpanQuerySelectedRow` adds an optional `cursorSortValue`. Response
construction removes this field from returned spans. It uses the value
for the next-page cursor, and falls back to the row's ordering field
when the value is absent.
- The base storage feature union adds `span-query-model-provider-sort`.

## Server changes

- `assertObservabilitySpanModelProviderSortSupported` returns HTTP 501
with `SPAN_QUERY_UNSUPPORTED` when a store does not declare
`span-query-model-provider-sort`. The check runs before `querySpans` is
called.
- `ObservabilityStorageCapabilities` and its empty default add
`spanQueryModelProviderSort`, which defaults to `false`.
- The capability is reported as supported only when span queries are
supported, the core order fields include both `model` and `provider`,
and the store declares the feature.
- The system capabilities schema adds `spanQueryModelProviderSort:
z.boolean().default(false)`.
- The route handler permits model and provider ordering and performs the
capability check.

## Store changes

- **PostgreSQL:** Sorts by `model` and `provider` use `COLLATE "C"`. The
cursor bounds cast sort values to `text COLLATE "C"` when `stringSort`
is enabled. The feature is advertised only when the core order fields
include both fields.
- **DuckDB:** Sorts by `model` and `provider` use `"C"` collation.
Cursor values cast as strings with the same collation. The feature is
advertised only when the core order fields include both fields.
- **ClickHouse:** `afterCursor` treats `model` and `provider` as
strings. Null rows are included after the cursor for numeric and string
sorts. The feature is advertised only when the core order fields include
both fields.
- Returned `model` and `provider` values may still come from span
attributes. The code comments state that these fallback values are not
used for filtering or sorting.

## Tests and conformance

- Core tests cover ordering gating, rejection in trace and thread
queries, string and null cursor values, and rejection of cursors when
the order, filters, authorization, or scope change.
- Server tests cover the 501 rejection for `durationMs`, `model`, and
`provider` on older stores. They also cover capability parsing and the
capability flag in the system endpoint.
- Shared conformance tests in `stores/_test-utils` cover ascending and
descending pagination at page sizes 1, 2, and 3. They cover empty
strings, missing values, case and Unicode ordering, ties, and filter
exclusion. They also check that missing values sort last and that legacy
attribute values are not treated as dedicated fields.
- Store feature-list tests for ClickHouse, DuckDB, and PostgreSQL expect
the new feature flag.

## Changesets and documentation

- Changesets add minor release notes for `@mastra/core`,
`@mastra/server`, `@mastra/client-js`, `@mastra/duckdb`,
`@mastra/clickhouse`, and `@mastra/pg`.
- The advanced query reference adds the `spanQueryModelProviderSort`
capability row. It documents `model` and `provider` ordering. It states
that the comparison is case-sensitive and that missing values sort last
in both directions.

## Scope and limits

- This diff does not include Studio UI changes. The commit history says
UI enablement is deferred to a separate issue because it needs raw model
and provider values for head-row merging.
- This diff does not include client-js source changes. Its client-js
changeset documents the new sort fields.
- The diff does not include schema changes or backfill.
- The PR description reports these results, which this review did not
run or verify:
- Core and server tests, and real shared-storage conformance and
mixed-core checks, pass.
- Core, server, and adapter typechecks pass. Staged-file lint and docs
checks, including CI Vale, pass.
  - 58 Studio and MSW tests pass.
- The full Playground UI typecheck reports four unrelated chart and form
errors, and the library declaration build fails.
- A local ClickHouse 26.6.8.7 comparison with 180,000 synthetic rows
measured median CPU of 21.9 ms for start time, 19.4 ms for model, and
21.4 ms for provider. Median query memory was about 4.6 MiB for each.
These figures are not production measurements.
- Platform backend and UI wiring, and package pins, remain under
OBS-896.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-12 13:52:37 +01:00
mastra-platform[bot]
a2998f4084
test(storage): verify rollback reporters settle before client release (#30160)
## Summary
- Make PostgreSQL pooled/pinned and DSQL pooled async rollback reporter
tests assert that reporting finishes before transaction settlement and
pooled release.
- Use plain async reporters to catch unhandled promise rejections rather
than `vi.fn` wrappers.

## Test plan
- PostgreSQL transaction tests: 18 passed; DSQL transaction tests: 9
passed.
- Both package TypeScript checks, targeted Oxlint, Prettier and `git
diff --check` passed.
- Confirmed the changed tests fail against the parent of merged #30157:
PG 2 failures, DSQL 1 failure (plus unhandled rejections).

Follow-up to #30157; test-only correction to the regression coverage for
its already-merged implementation.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

ELI5: The change makes the database tests check the right order of
events. A rollback error must be reported before the database connection
is released and before the transaction ends. The tests now fail if that
order is wrong.

## Summary

This change updates two test files for the rollback reporter behavior in
PostgreSQL and DSQL transaction clients. It changes test code only. No
production code changes.

Diff stats: 2 files changed, 33 insertions, 9 deletions.

### `stores/dsql/src/storage/client.tx-serialize.test.ts`

- The reporter is now a plain async function, not a `vi.fn` wrapper.
- The reporter stores the error it receives.
- The reporter waits 10 ms, records `reported`, and then throws `async
logger failed`.
- The test mocks `client.release` to record `released`.
- The test records `tx-settled` after the transaction call rejects with
the original error.
- Assertions:
  - The reporter received `rollbackError`.
  - `client.release` was called with `true`.
  - The event order is `['reported', 'released', 'tx-settled']`.

### `stores/pg/src/storage/client.tx-serialize.test.ts`

- The same reporter change applies to the shared async rollback reporter
test.
- For the pooled case (`kind === 'pool'`):
  - `client.release` records `released`.
  - Assertions check that `client.release` was called with `true`.
  - The event order is `['reported', 'released', 'tx-settled']`.
- For the pinned case:
  - No release tracking is added.
  - Assertions check that `adapter.shouldDestroyClient` is `true`.
  - The event order is `['reported', 'tx-settled']`.

## Not verified

- The PR reports test runs, type checks, lint, and format checks. I did
not run them. Their results are not available from this review.
- The PR reports that the changed tests fail against the parent of the
referenced prior change. I did not verify this. It is not available from
this review.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-11 20:50:47 -08:00
dane-ai-mastra[bot]
055a8c3b82 chore: version packages 2026-11-12 04:17:40 +00:00
mastra-platform[bot]
b1948fb073
fix(storage): await rollback diagnostics before releasing clients (#30157) 2026-11-11 18:18:44 -08:00
mastra-platform[bot]
743a02786e
feat(valkey): identify Mastra-created clients with a clientInfoTag (#30156) 2026-11-11 17:57:11 -08:00
mastra-platform[bot]
ce631e94c6
fix(storage): prevent poisoned PostgreSQL transactions from returning to the pool (#30144) 2026-11-11 17:55:44 -08:00
Dane - AI (Mastra)
c014d187bb
chore(deps): update docker images (major) (#19788) 2026-11-11 17:08:49 -08:00
dane-ai-mastra[bot]
9e82af0237
chore: version packages (alpha) (#30074)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-11-11 14:21:14 -08:00
Abhi Aiyer
4884ee08a0
fix(agent-controller): answer parked tool approvals when the permission rules allow or deny them (#30114)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-11 14:18:01 -08:00
Abhi Aiyer
1689ce38e1
perf(agent-controller): read only the newest page when listing threads with a limit (#30090)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
2026-11-11 13:53:07 -08:00
mastra-platform[bot]
259f142f46
fix(pg): support scalar and nested operators in vector elemMatch filters (#30095) 2026-11-11 11:31:00 -08:00
Abhi Aiyer
cd041a5bb0
perf(pg): index thread metadata for listThreads metadata filters (#30086)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-11 10:56:41 -08:00
mastra-platform[bot]
c8fecd5d6f
fix(pg): preserve all nested $elemMatch conditions in vector filters (#30091) 2026-11-11 10:56:24 -08:00
Eric Pinzur
aa49b049d0
feat(observability): order completed spans by duration (#30085)
Adds `durationMs` ascending/descending ordering to `querySpans()`,
combined with existing duration filters and stable cursor pagination.
Duration is the completed span’s elapsed time; zero is valid. Trace
ordering and trace runtime semantics are unchanged.

The server advertises `spanQueryDurationSort` and rejects unsupported
stores before execution. A span-specific field list and default-off
planner option protect old servers paired with newer Core. Updated
adapters advertise support only when Core understands it. PostgreSQL and
DuckDB reuse their numeric ordering paths; ClickHouse handles numeric
cursor parameters and normalizes duration values returned as Int64
strings. Includes generated API types, documentation and package
changesets. No schema migration.

Implements the OSS portion of
[OBS-899](https://linear.app/kepler-crm/issue/OBS-899).
[OBS-904](https://linear.app/kepler-crm/issue/OBS-904) tracks Platform
integration after merge and a snapshot from main. UI is outside this
backend change.

Validation: 134 Core, 112 server, 125 Client JS, 100 PostgreSQL, 94
DuckDB and 103 ClickHouse tests passed (one existing ClickHouse skip).
Shared conformance exercises both directions, filters, zero durations
and ties across page sizes. The new Core tests fail before the
implementation; reverting the ClickHouse cursor fix makes both new
pagination cases fail by skipping rows. Old-Core adapters omit the new
feature; old stores receive a structured 501. Core/server/store
typechecks, Core/server/SDK builds, changed-file lint/format and
required docs checks passed. Vale reports six warnings in unchanged
text. Local Node 23.11.0 is outside the supported 22/24 versions; CI
must verify supported runtimes.

A bounded local ClickHouse 26.6.8.7 experiment used 20,000 synthetic
completed spans, the same duration filter and 50-row first/second pages
in both directions. Duration ordering peaked at 3.43–3.67 MiB versus
3.42–3.66 MiB for timestamp ordering; both read 20,000 rows/1,488,890
bytes. Duration query CPU was 7–25 ms. These are single-run local
measurements, not production capacity evidence; no ingestion or storage
layout changed.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

This PR lets users sort completed spans by how long they took. It keeps
pagination stable and enables the feature only when Core and the storage
adapter support it.

## Changes

- Added ascending and descending `durationMs` ordering to
`querySpans()`.
- Added duration-aware cursor validation and stable pagination.
- Added `spanQueryDurationSort` capability reporting.
- Added HTTP 501 handling for unsupported stores.
- Added compatibility gates for older Core versions.
- Updated PostgreSQL, DuckDB, and ClickHouse adapters.
- Converted ClickHouse duration values to numbers.
- Added Core, server, adapter, and conformance tests.
- Updated generated API types, documentation, and package changesets.
- No schema migration was added.

## Validation

- Reported passing tests: Core 134, server 112, Client JS 125,
PostgreSQL 100, DuckDB 94, and ClickHouse 103.
- One existing ClickHouse test remains skipped.
- Reported typechecks, builds, lint, formatting, and documentation
checks passed.
- Vale reported six warnings in unchanged text.
- CI verification on supported Node.js versions is not available.
- Diff size: 28 files changed, with 220 additions and 8 deletions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-11 19:54:11 +01:00
mastra-platform[bot]
b0ef22e7bf
feat(schedules): Add bounded one-off schedules (#25131)
> Opened by @abhiaiyer91

Refs #24610

## Summary
Schedules can now fire once or stop after a given time, so you no longer
need to pin a cron expression to a date and clean it up yourself.

- **New `completed` status**: the scheduler and timing reconciliation
set it when no automatic fires remain. It's written in the same atomic
`nextFireAt` check-and-set used to claim a run, so a schedule still
fires only once across instances. A cron with no future occurrence left
also moves to `completed` instead of erroring. `pause`/`resume` on a
completed schedule return 409. Changing `runAt`/`endAt` reactivates it
only when the resulting timing can still fire. `list()` hides completed
schedules unless you pass `status: 'completed'`.
- **Declarative workflow schedules**: `createWorkflow({ schedule: {
runAt } })` and `{ cron, endAt }` are supported. Past times are accepted
so redeploys don't fail, and a completed schedule isn't re-armed unless
its timing changes.
- **Trigger kind**: one-off fires report `trigger.kind: 'once'`.
- `run(id)` doesn't complete a one-off schedule.

## Surfaces
- **Core**: types, validation, scheduler claim logic,
create/update/pause/resume/list, declarative sync.
- **Storage**: new nullable `run_at`/`end_at` columns for libsql, pg,
mysql, mongodb, spanner, convex and in-memory. Existing columns are
untouched: one-off schedules store `cron` as `''`.
`updateScheduleNextFire` takes an optional `newStatus`.
- **Server/client-js/CLI**: request bodies accept `runAt`/`endAt`
(one-of rule checked in the schema). Responses include the new fields
and `completed`. Route types and CLI route metadata regenerated.
- **Studio**: shows the run time for one-off schedules, shows the
`endAt` bound, adds a `completed` badge and hides pause/resume for
completed schedules.
- **Docs + changeset.**

## Not in this PR (remaining #24610 work)
- ISO 8601 string inputs for `runAt`/`endAt` (numbers and `Date` are
accepted today).
- The template `schedule_once` tool.

## Test plan
- [x] core: schedules, scheduler, cron, schedule-config and scheduler
integration tests (260 passing), clean `tsc`
- [x] Shared adapter suite gained tests for `runAt`/`endAt` round-trip
and atomic status CAS. Passing on libsql, pg, mysql and mongodb
(Docker), plus convex server tests. **Not run on Spanner (no emulator
available); typecheck only.**
- [x] server schedules handler tests (51 passing, including
one-off/bounded creation and body validation)
- [x] client-js and playground typecheck clean; playground schedules
tests passing

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

Schedules can now run once at a chosen time or repeat until a chosen end
time. They complete when their planned runs are over, and changing their
timing can make them active again.

## Summary

- Added `runAt` for one-off agent and workflow schedules. API-created
schedules require a future `runAt`; declarative workflow schedules can
register past times as `completed`.
- Added `endAt` for cron schedules. A schedule becomes `completed` when
its next run would occur after `endAt`, or when no future run remains.
- Completed schedules cannot be paused or resumed. Timing changes can
reactivate a completed schedule when a future run remains. `list()`
hides completed schedules unless callers filter with `status:
'completed'`.
- Resuming a paused one-off schedule after its `runAt` has passed allows
it to fire on the next scheduler tick. Calling `run(id)` manually does
not complete a one-off schedule.
- Added the `once` trigger kind for scheduled one-off agent fires.
Manual runs remain `manual`.
- Declarative workflow schedules preserve completed status across
redeploys when timing is unchanged. Timing changes can re-arm them.
- Updated schedule APIs, storage adapters, Studio, and documentation.
Storage persists `runAt` and `endAt` and supports atomic status updates.

## Checks

Current-head CI, build, and test results: not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Abhi Aiyer <abhiaiyer91@gmail.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: NikAiyer <nick.aiyer@gmail.com>
Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: Ehindero Israel <iehindero419@stu.ui.edu.ng>
2026-11-11 19:33:50 +01:00
dane-ai-mastra[bot]
3a6ca42696
chore: version packages (alpha) (#30026)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-11-11 08:16:57 -08:00
Eric Pinzur
c860b9bb43
feat(observability): support OSS feedback replacement (OBS-887) (#30059)
Implements
[OBS-887](https://linear.app/kepler-crm/issue/OBS-887/add-oss-feedback-replacement-and-export).

Add OSS `PUT /api/observability/feedback/:feedbackId` with the existing
wrapped client request. Replacements retain the original feedback ID,
trace ID, timestamp, and tenant; omitted optional fields are cleared. A
full replacement can create a missing record. OSS persists locally
before responding, then independently notifies supporting exporters.
Resolve mode still requires the local anchor. The SDK also exposes
optional `replaceFeedback()` for independent exporter delivery without a
producer-side storage lookup.

Replacement events use a separate optional exporter hook.
`MastraPlatformExporter` sends PUT with stable identity across bounded
anchor retries, and flush/shutdown drain deliveries. Older exporters
skip replacements rather than replaying them as creates. Unsupported
storage/package combinations return 501 or skip unavailable export
support. No UI changes; review-status PATCH propagation remains tracked
in OBS-785.

In-memory, PostgreSQL vNext, DuckDB, and ClickHouse vNext support
replacement. In-memory, PostgreSQL, and DuckDB enforce replacement
identity at the write. PostgreSQL coordinates writes by feedback ID,
including duplicates submitted with a different timestamp. Duplicate
HTTP creates return the stored identity and timestamp and re-notify
exporters with accepted values. Tenant mismatches remain hidden, and an
omitted SDK replacement author stays null. HTTP/exporter creates use
`onConflict: 'ignore'` so a delayed original create doesn't undo an
edit; direct storage calls keep their legacy upsert default.
**ClickHouse requires no migration:** it uses the existing schema and
read-before-insert protection. Sequential delayed creates are covered,
but overlapping writes from separate processes can race; this is not a
globally ordered or exactly-once protocol.

Latest review fixes: DuckDB's lazy facade delegates replacement to its
guarded writer; concurrent coverage runs 20 overlapping request pairs
per identity variant. Direct storage replacements normalize an omitted
author to null. PostgreSQL single-feedback reads use the primary for
read-after-write consistency, while list/aggregate reads retain the
configured read client. Batch advisory locks use one ordered query with
unchanged lock keys.

Validation: the stronger DuckDB regression failed on all four identity
variants before the facade fix. Afterward, the entire DuckDB package
passed 718 tests across 21 files in three consecutive runs. PostgreSQL
passed 57 feedback tests, including a real stale read snapshot and
overlapping reversed 1,000-ID batches. Shared replacement tests passed
on in-memory (12), and the new author regression passed on ClickHouse
without a migration. Core/storage builds and declarations, typed lint,
and formatting passed. Earlier HTTP (85 cases on each of four adapters),
generic Hono routes (2,848), server/system, exporter, and docs checks
remain recorded in the PR history. Local tests use Node 23.11.0; CI
validates the supported runtime.

The unrelated Traces UI fade assertion failure was fixed in
https://github.com/mastra-ai/mastra/pull/30068; this branch includes
that fix from main.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

This PR lets users replace feedback without changing its identity. It
stores the replacement locally and sends it to supported exporters
without allowing delayed creates to overwrite it.

## Changes

- Added `PUT /api/observability/feedback/:feedbackId`.
- Preserved the feedback ID, trace ID, timestamp, and tenant.
- Cleared omitted optional fields.
- Added capability detection and `501` responses for unsupported
storage.
- Added replacement event types, SDK support, client types, route
metadata, and documentation.
- Added replacement delivery, stable retries, flush handling, and
shutdown draining for `MastraPlatformExporter`.
- Added replacement support for in-memory, PostgreSQL vNext, DuckDB, and
ClickHouse vNext.
- Added `onConflict: 'ignore'` for HTTP and exporter creates. Direct
storage calls retain their upsert default.
- Older exporters skip replacements instead of replaying them as
creates.
- Added ClickHouse support without a migration. Concurrent writes from
separate processes can still race.
- No UI changes were included.

## Validation

The author reports passing the HTTP route and contract suites, feedback
storage suites, partition and SQL tests, server and system tests,
record-builder tests, build, typecheck, lint, and documentation checks.

For the current head, the available CI result is `2027dev Evals:
skipped`. Other listed CI checks target different commits or remain in
progress. Current-head CI results are otherwise not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-11 17:04:57 +01:00
mastra-platform[bot]
3b27a10322
fix(pg): keep thread owner and createdAt when saveThread hits an existing id (#29149) 2026-11-11 01:44:31 -08:00
mastra-platform[bot]
562a451e22
fix(pg): stop concurrent thread metadata updates from dropping keys across processes (#29150) 2026-11-11 01:25:57 -08:00
dane-ai-mastra[bot]
23cc34a9c5
chore: version packages (alpha) (#29938)
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases

> The changelog information of each package has been omitted from this
message, as the content exceeds the size limit.

## @mastra/factory@0.28.0-alpha.6


## @mastra/code-sdk@1.19.0-alpha.6


## @mastra/playground-ui@69.0.0-alpha.7


## @mastra/client-js@1.60.0-alpha.6


## @mastra/react@1.15.0-alpha.6


## @mastra/deployer-cloud@1.83.0-alpha.6


## @mastra/longmemeval@1.1.42-alpha.6


## @mastra/opencode@0.1.42-alpha.6


## create-factory@0.4.3-alpha.6


## mastracode@0.50.0-alpha.6


## @mastra/agent-builder@1.2.1-alpha.3


## mastra@1.39.0-alpha.6


## @mastra/core@1.83.0-alpha.6


## @mastra/deployer@1.83.0-alpha.6


## @mastra/editor@0.17.1-alpha.4


## @mastra/mcp@2.9.0-alpha.3


## @mastra/mcp-docs-server@1.3.13-alpha.9


## @mastra/memory@1.41.1-alpha.3


## @mastra/rag@2.7.3-alpha.2


## @mastra/schema-compat@1.3.18-alpha.0


## @mastra/server@1.83.0-alpha.6


## @mastra/elysia@0.1.23-alpha.6


## @mastra/express@1.5.27-alpha.6


## @mastra/fastify@1.5.27-alpha.6


## @mastra/hono@1.7.25-alpha.6


## @mastra/koa@1.7.27-alpha.6


## @mastra/nestjs@0.3.0-alpha.6


## @mastra/next@0.2.41-alpha.6


## @mastra/server-adapters-test-suite@0.3.0-alpha.3


## @mastra/tanstack-start@0.2.41-alpha.6


## @mastra/upstash@1.6.1-alpha.0


## @mastra/vectorize@1.1.5-alpha.0


## @mastra/voice-google-gemini-live@0.14.19-alpha.0


## @mastra/voice-openai-realtime@0.14.9-alpha.0


## @mastra/voice-xai-realtime@0.2.18-alpha.0


## @mastra/inngest@1.11.2-alpha.4


## @mastra/temporal@0.6.0-alpha.6


## create-mastra@1.39.0-alpha.6


## @internal/playground@1.39.0-alpha.6

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-11-10 15:51:29 -08:00
Tyler Barnes
0af43a79d4
test(pg): run store test files in parallel on per-worker databases (#29093)
> **Stacked on #29092** (pg: build indexes on newly created tables
without CONCURRENTLY). The base branch is that PR's branch, so the diff
here is only this change. Merge that one first: without it, parallel
files cold-booting the same schema hit ~1 s waits per index.

## What this does

The pg store tests ran every file one after another (`fileParallelism:
false`) against a single database. This runs them on 4 workers, each
with its own database:

- `stores/pg/vitest.config.ts`: two projects. `e2e:stores/pg` runs files
in parallel; `e2e:stores/pg-sequential` runs `integration.test.ts`
afterwards on its own (`groupOrder` 1). Its delta polling reads a
cluster-wide snapshot horizon (`pg_snapshot_xmin`), so it can't share
the server with other writers.
- `vitest.global-setup.ts` creates one database per worker
(`mastra_test_w1..wN`) on both the pg and timescale containers, sized
from the resolved `maxWorkers`, so a CLI override of the worker count
can't leave a worker without a database. Teardown drops them.
- `vitest.setup.ts` points each worker at its database (`POSTGRES_DB`,
`DB_URL`, `POSTGRES_CONNECTION_STRING`, `PG_VNEXT_TIMESCALE_URL`) using
`VITEST_POOL_ID`.
- The bulk-status notification test gets a 20 s timeout. It went over 5
s under parallel load.

Test config only: no source changes, no changeset.

## Numbers

| | pg suite |
|---|---:|
| sequential, before the cold-boot fix | 701 s |
| sequential, with the cold-boot fix | ~323 s |
| this PR (4 workers) | 110–133 s |

Every run: 3905 passed / 20 skipped. CI pg job estimate: ~5.4 → ~1.9 min
(unverified until CI runs).

## Test plan

- [x] 3 full green runs at 4 workers (3905 / 20 each), plus one after
the maxWorkers change (133 s)
- [x] 6-worker subset green; no leftover databases after teardown
- [x] Root `vitest list`: 85 parallel + 1 sequential file, same as
before
- [ ] CI: pg store job green


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

The PostgreSQL store tests now run in parallel. Each worker uses its own
database. One test file runs afterward by itself because it reads
cluster-wide state that other writers can affect.

## Changes

- Adds global setup in `stores/pg/vitest.global-setup.ts`. Before each
run, it creates one database per configured worker on the PostgreSQL and
Timescale servers. It force-drops any existing database with the same
name first. The returned teardown force-drops these databases. The
PostgreSQL server installs the `vector` extension. The Timescale server
installs no extensions.
- Adds `stores/pg/vitest.setup.ts`, which exports `workerDatabaseName`.
The name format is `mastra_test_w{worker}`. When `VITEST_POOL_ID` is
set, the setup sets `POSTGRES_DB` to the worker database. It also
updates `DB_URL`, `POSTGRES_CONNECTION_STRING`, and
`PG_VNEXT_TIMESCALE_URL` to use it. When the variable is unset, the
setup changes nothing.
- Splits `stores/pg/vitest.config.ts` into two projects:
- `e2e:stores/pg` runs in parallel with up to four workers. It excludes
performance, pooler, and cluster-wide integration tests.
- `e2e:stores/pg-sequential` runs the cluster-wide integration file with
file parallelism disabled. It uses group order 1, so it runs after the
parallel project at group order 0.
- Removes the previous `fileParallelism: false` setting and the coverage
reporter settings.
- Increases the bulk-status-update notification test timeout to 20
seconds in `stores/pg/src/storage/domains/notifications/index.test.ts`.
It also documents the load-dependent runtime of its 1,200 sequential
inserts.
- Adds `vitest.global-setup.ts` and `vitest.setup.ts` to the `include`
array in `stores/pg/tsconfig.json`.

## Test and CI results

- Author-reported, not verified in this review:
- Three full runs at four workers passed (3,905 passed, 20 skipped
each).
- A subset run at six workers passed, and no databases remained after
teardown.
  - Suite time was 110–133 seconds with four workers.
- CI results for the pg store job: not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com>
2026-11-10 16:32:30 -07:00
Leo
22b24cc40a
fix(core): apply semanticRecall.filter during agent runs (#29418)
## Description

`semanticRecall.filter` worked in `Memory.recall()` but not during
`agent.generate()` or `agent.stream()`. During an agent run, recall
happens in the `SemanticRecall` processor, which never read `filter` and
queried the vector store with the scope filter only. So messages from
threads that didn't match the filter ended up in the prompt.

This PR changes
`packages/core/src/processors/memory/semantic-recall.ts`:

```diff
 query filter
-  scopeFilter
+  filter ? { $and: [scopeFilter, filter] } : scopeFilter   // same as Memory.recall()

 embedding metadata written during agent runs
-  { message_id, thread_id, resource_id, ... }
+  { ...threadMetadata (only when filter is set), message_id, thread_id, resource_id, ... }
```

- The filter applies to both query paths: client-side embedder and
self-embedding stores.
- Embeddings written during agent runs had no thread metadata, so even
with the query fixed, a filter like `{ mode: { $eq: 'work' } }` could
never match messages that agents saved. When a filter is set, the
processor now reads the thread from storage and stores its metadata on
the embedding, as `Memory.saveMessages()` already does. It reads from
storage because durable and recovered runs put only the thread id in the
memory context.
- Without a filter, nothing about writes changes. Some vector stores
reject nested metadata values and this processor only logs a failed
upsert, so the metadata is written only when a filter needs it.
- Docs: added `filter` to the processor reference and a note on the
semantic recall page that messages agents saved before a filter was set
don't match it until they're saved or indexed again.

## Related issue(s)

Closes #29352

## Type of change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [x] Test update

## Checklist

- [x] I have linked the related issue(s) in the description above
- [x] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR

## Evidence

Before = `semantic-recall.ts` from `origin/main`, with the same tests.
After = this branch.

- **Core, real Agent + MockMemory + in-memory vector store that applies
filters**
(`packages/core/src/processors/memory/semantic-recall-filter.test.ts`,
new): generate and stream, filter vs no filter, self-embedding store,
thread scope, durable-shaped context (`thread: { id }` only), thread
metadata written only with a filter.
**Before:** `Tests 7 failed | 2 passed (9)` (the two no-filter cases
pass on both).
  **After:** `Tests 9 passed (9)`.
- **Memory integration, real `Memory` + LibSQL store and vector +
Agent**
(`packages/memory/integration-tests/src/semantic-recall-metadata-filter.test.ts`):
generate and stream; messages saved with `saveMessages()`, by earlier
agent runs, and in a thread the run created with metadata. Also checks
`Memory.recall()` agrees with the agent run.
**Before:** `Tests 6 failed | 2 passed (8)` (the two existing `recall()`
cases pass on both).
  **After:** `Tests 8 passed (8)`.
- **The issue's own repro script** (LibSQL, mock embedder and model),
run against local builds:
  **Before:** `agent prompt contains personal message: true`
  **After:** `agent prompt contains personal message: false`
- Neighbouring core suites (`semantic-recall.test.ts`,
`processor-caching.test.ts`, `memory-config.test.ts`,
`embedding-dimension.test.ts`): 69/69 pass. `@mastra/core` typecheck
passes; oxlint, oxfmt and lint-staged are clean on the changed files.

Not tested: a real DurableAgent run end to end. The durable context
shape is covered by calling the Memory-built processor with `thread: {
id }` only, as `durable-agent.ts` and `finalize-run.ts` set it.

## Merge Danger

**Door:** two-way. A revert restores the old behavior. Embeddings
already written with thread metadata stay in the vector store but are
harmless extra fields.

**Blast radius:** agents with `semanticRecall.filter` configured. They
now recall fewer messages (only matching threads), which is the
documented behavior. Messages saved by older versions have no thread
metadata, so they won't match a filter until re-saved or re-indexed; the
changeset and docs say so. Users without a filter see no change in
queries.

**After merging main (#29826):** main's shared embedding helper now
always stores thread metadata on agent-run embeddings, so this PR no
longer adds its own filter-only metadata read; docs, changeset and the
unit test were updated to match.

**Cloudflare Vectorize (`@mastra/vectorize`):** the scope and user
filter are combined as `{ $and: [...] }` (the same shape
`Memory.recall()` already sends on main), which the Vectorize translator
rejected, so recall with a filter returned nothing there. The translator
now merges a filter that is exactly `{ $and: [...] }` into one object.
Several conditions on one field merge only as one lower plus one upper
bound; other same-field combinations and non-object clauses throw. Not
run against a live Vectorize index; checked against Cloudflare's
metadata-filtering docs and 38 translator tests (Sol reviewed 3 rounds).

## Council review

Rounds: 4

| Seat | Round 1 | Round 2 | Round 3 | Round 4 |
| --- | --- | --- | --- | --- |
| Claude Opus 5.5 | not run | not run | 8.5/10 | 9/10 |
| GPT-6.1 Sol | 9/10 (Tests 8, Code quality 8) | 9.6/10 | 9.5/10 |
9.5/10 |
| Grok 4.7 | empty seat | empty seat | empty seat | empty seat |

Created with Claude Opus 5.5 in T3 Code.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

Agents now use the configured metadata filter when they search for
relevant messages. This keeps messages from unrelated threads out of the
agent’s prompt. Cloudflare Vectorize can also process the combined
filters used for these searches.

## Changes

- `SemanticRecall` combines the configured `VectorFilter` with the
thread or resource scope filter using `$and`.
- When a filter is configured, the processor reads thread metadata from
storage and writes it to message embeddings. Older embeddings may need
to be saved or indexed again.
- The Vectorize filter translator merges supported top-level `$and`
clauses, including repeated conditions with the same value. It rejects
unsupported combinations on the same field.
- Updated documentation and added tests for `generate()`, `stream()`,
self-embedding queries, and Vectorize filter translation.

## Verification

- Reported: 9/9 core filter tests, 8/8 integration tests, 69/69
neighboring core tests, 38 Vectorize translator tests, and the
`@mastra/core` typecheck passed.
- A real DurableAgent run was not tested end to end. A core test covers
a processor request context that contains only a thread ID.
- Live Cloudflare Vectorize testing: not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 18:19:59 -05:00
Arthur Fiorette
01b2b958af
fix(core): fail the run when a loop callback throws, instead of retrying it or leaving it running (#29850) 2026-11-10 12:53:52 -08:00
mastra-platform[bot]
b4152bd5da
fix(upstash): negate every key of multi-key $not, $nor and $and/$or vector filters (#29760) 2026-11-10 12:45:53 -08:00
NikAiyer
cf2cd73b85
Revert "fix(core): snapshot agent stream events when they are publish… (#29947)
…ed (#29803)"

This reverts commit 28ddc161a4.

## Description

<!-- Required: Provide a brief description of the changes in this PR.
-->

## Related issue(s)

<!-- Required: Link to the issue(s) this PR addresses, e.g. with: Fixes
#123. PRs without linked issues will be closed. High-quality issues with
reproductions are preferred to vague pull requests. -->

## Type of change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [ ] Test update

## Checklist

- [ ] I have linked the related issue(s) in the description above
- [ ] I have made corresponding changes to the documentation (if
applicable)
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## ELI5

This PR removes a change that copied agent stream events before
publishing them. As a result, published events can reflect later changes
to the original data.

## Changes

- Removed snapshotting and timestamp restoration from
`createAgentStreamPublisher`. Most event payloads are now passed to
event emitters without copying. Error chunks still serialize `Error`
values.
- Deleted the publisher snapshot tests and replay tests for AIMock and
Redis.
- Deleted the changeset for the snapshotting fix.
- Changed 5 files: 8 lines added and 562 removed.

## Tests

Test and CI results: not available.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-10 12:25:30 -08:00
dane-ai-mastra[bot]
779d2875b1
chore: version packages (alpha) (#29912)
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases

> The changelog information of each package has been omitted from this
message, as the content exceeds the size limit.

## @mastra/auth-studio@1.4.0-alpha.1


## @mastra/factory@0.28.0-alpha.5


## @mastra/mcp-apps-components@0.1.0-alpha.5


## @mastra/playground-ui@69.0.0-alpha.6


## @mastra/client-js@1.60.0-alpha.5


## @mastra/react@1.15.0-alpha.5


## @mastra/deployer-cloud@1.83.0-alpha.5


## @mastra/longmemeval@1.1.42-alpha.5


## @mastra/opencode@0.1.42-alpha.5


## create-factory@0.4.3-alpha.5


## @mastra/code-sdk@1.19.0-alpha.5


## mastracode@0.50.0-alpha.5


## @mastra/agent-builder@1.2.1-alpha.2


## mastra@1.39.0-alpha.5


## @mastra/core@1.83.0-alpha.5


## @mastra/deployer@1.83.0-alpha.5


## @mastra/editor@0.17.1-alpha.3


## @mastra/mcp@2.9.0-alpha.2


## @mastra/mcp-docs-server@1.3.13-alpha.8


## @mastra/memory@1.41.1-alpha.2


## @mastra/rag@2.7.3-alpha.1


## @mastra/server@1.83.0-alpha.5


## @mastra/elysia@0.1.23-alpha.5


## @mastra/express@1.5.27-alpha.5


## @mastra/fastify@1.5.27-alpha.5


## @mastra/hono@1.7.25-alpha.5


## @mastra/koa@1.7.27-alpha.5


## @mastra/nestjs@0.3.0-alpha.5


## @mastra/next@0.2.41-alpha.5


## @mastra/server-adapters-test-suite@0.3.0-alpha.2


## @mastra/tanstack-start@0.2.41-alpha.5


## @mastra/libsql@1.30.0-alpha.2


## @mastra/turso@0.1.21-alpha.2


## @mastra/inngest@1.11.2-alpha.3


## @mastra/temporal@0.6.0-alpha.5


## create-mastra@1.39.0-alpha.5


## @internal/playground@1.39.0-alpha.5

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-11-10 10:50:08 -08:00
Jamie Nichols
6563e7abfc
fix(libsql): keep vector indexes writable after truncateIndex (#26802)
## What broke

`LibSQLVector.truncateIndex()` ran `DELETE FROM <table>` with no `WHERE`
clause. On a table with a `libsql_vector_idx` index, that statement
empties the table but leaves the old rows in the vector index's shadow
table. The next insert then fails:

```
SQLITE_ERROR: vector index(insert): failed to insert shadow row
```

So once you truncated an index, you couldn't write to it again. I
reproduced this on both file-backed and `:memory:` databases with
`@libsql/client` 0.18: after the bare delete the shadow table still has
1 row and the reinsert fails. The same cycle with `WHERE rowid IS NOT
NULL` leaves 0 shadow rows and the reinsert succeeds. Upstream tracking:
tursodatabase/libsql#2289.

## The fix

`truncateIndex()` now runs `DELETE FROM <table> WHERE rowid IS NOT
NULL`. It still deletes every row, and in the repro it also removes the
shadow rows (0 left after the delete). The source change is one line in
`stores/libsql/src/vector/index.ts`, and it runs for every connection
type.

## Tests

New tests in `stores/libsql/src/vector/index.test.ts`. They check
behaviour only, using public methods:

- file database: insert, truncate, insert again, then query and get only
the new vector
- file database: truncate an empty index, then run repeated
truncate/insert cycles
- file database: single-row and filter deletes still work, and the index
still accepts inserts afterwards
- `:memory:` database: insert, truncate, insert again, then query

I ran these tests against the old `truncateIndex`. Both file truncate
tests and the `:memory:` test fail with the shadow-row error. The
partial-delete test passes there, because it guards existing behaviour.
With the fix, the whole file passes: 146 passed, 1 skipped (`npx vitest
run src/vector/index.test.ts`). oxlint and oxfmt are clean on the
changed files.

## Not tested

- Remote Turso/sqld URLs and embedded replicas (`syncUrl`). They run the
same statement, but I didn't test them against a server.
- Cost on large tables. I haven't measured the qualified delete's
performance.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## ELI5

Truncating a local LibSQL vector index could leave behind data that
stopped new vectors from being added. This change removes the vector
rows while keeping the index ready for new inserts.

## Changes

- Updated `LibSQLVector` truncation to delete rows with `rowid IS NOT
NULL`.
- Added tests for file-backed and in-memory databases. They cover
reinserting and querying vectors after truncation, repeated truncation,
and partial deletes.
- Added a patch changeset for `@mastra/libsql`.

## Validation

Test and lint results: not available.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Jamie Lee Nichols <10267569+CodingButter@users.noreply.github.com>
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-10 06:55:26 -08:00
dane-ai-mastra[bot]
7c4dd88400
chore: version packages (alpha) (#29889)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-11-10 15:48:17 +01:00
Ward Peeters
80db9c3ba2
chore: run oxfmt instead of prettier in the last six pre-commit configs (#29892)
We moved from Prettier to oxfmt a while ago and CI checks formatting
with `oxfmt --check`, but six packages still ran `prettier --write` in
their pre-commit hook: the Discord, Teams and Telegram channels, the
Turso and OracleDB stores, and the DeepEval observability package. A
commit touching those could get reformatted by Prettier and then fail
the format check in CI.

They now use the same command as every other package.

```diff
- '*.{json,md,yml,yaml}': ['prettier --write'],
+ '*.{json,md,yml,yaml}': ['oxfmt --no-error-on-unmatched-pattern'],
```

`explorations/longmemeval` still uses Prettier in its hook. I left it
alone because `explorations/` is excluded from the oxfmt check.

Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-10 12:35:37 +01:00
Eric Pinzur
832fb190f7
feat(observability): allow opting into remote storage export on Platform (#29737)
When `MASTRA_DEPLOYMENT_ID` is set, storage exporters are suppressed
even when a customer intentionally exports to both their own database
and Platform. Keep that default and add a per-config
`retainStorageExporterOnPlatform: true` opt-in for
`MastraStorageExporter`, including runtime registration.

```ts
new Observability({
  configs: {
    production: {
      serviceName: 'my-service',
      retainStorageExporterOnPlatform: true,
      exporters: [new MastraStorageExporter(), new MastraPlatformExporter()],
    },
  },
});
```

Deprecated `DefaultExporter` remains suppressed on Platform even with
the opt-in. Its implementation is unchanged.

The opt-in does not permit known local observability storage.
MastraStorageExporter warns during initialization and disables
persistence when the resolved observability domain reports local
storage. This blocks InMemoryStore (including Mastra's automatic
fallback when no storage is configured), local libSQL files, in-memory
libSQL, embedded replicas, and DuckDB while allowing remote libSQL/Turso
and PostgreSQL. Other storage domains and exporters are unaffected, and
local development retains local storage support.

Adds the optional persistenceLocation capability to observability
storage, with in-memory, libSQL, and DuckDB implementations. These
adapters must be upgraded alongside observability for the guard to
apply; older/custom adapters without location metadata retain the
existing opt-in behavior. No provider names or private connection fields
are inspected.

Constructor skip warnings are deferred until the application logger is
installed and emitted once. Docs explain the local-storage restriction
before the example, include a supported/blocked provider table, and
distinguish the implementation-specific option from the core interface.
The restriction remains explicit even for users who cannot see the
warning. Includes one minor changeset per affected package:
observability, core, libSQL, and DuckDB. Preserves the default policy
introduced in #23555.

Validation: 451 targeted observability tests pass, including regression
coverage that reproduced unwanted in-memory persistence before the fix
for both explicit InMemoryStore and the automatic fallback. The tests
also verify initialization warnings and continued local-development and
remote-storage behavior. All 756 MDX files pass formatting using the
CI-matching dependencies; focused type-aware lint, Remark, and Vale
pass. The earlier 9 adapter tests also passed. No live Platform or
PostgreSQL endpoint was exercised. Full typechecking remains limited by
workflow-helper errors previously reproduced on the base and
installed-dependency type mismatches encountered while regenerating core
declarations.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## ELI5

On Mastra Platform, storage exporters are normally turned off to prevent
observability data from going to local storage. This change lets each
observability configuration opt in to remote storage export, while still
blocking storage known to be local.

## Changes

- Added `retainStorageExporterOnPlatform` to observability
configuration. It defaults to `false` and applies to exporters
registered during setup or later.
- When enabled on Platform, the option retains `MastraStorageExporter`
but does not enable the deprecated `DefaultExporter`.
- Added `persistenceLocation` to observability storage. The base
implementation returns `undefined`; in-memory storage and DuckDB report
`local`; libSQL reports `local`, `remote`, or `undefined` based on its
client configuration.
- On Platform, `MastraStorageExporter` skips storage that reports
`local` and warns during initialization. Adapters without location
metadata retain the opt-in behavior.
- Added documentation and changesets for observability, core, libSQL,
and DuckDB.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-10 11:56:44 +01:00
dane-ai-mastra[bot]
6deb16047f
chore: version packages (alpha) (#29719)
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases

> The changelog information of each package has been omitted from this
message, as the content exceeds the size limit.

## @mastra/slack@1.9.0-alpha.0


## @mastra/factory@0.28.0-alpha.3


## @mastra/code-sdk@1.19.0-alpha.3


## mastracode@0.50.0-alpha.3


## @mastra/observability@1.24.0-alpha.2


## @mastra/core@1.83.0-alpha.3


## @mastra/mcp-apps-components@0.1.0-alpha.3


## @mastra/playground-ui@69.0.0-alpha.4


## @mastra/server@1.83.0-alpha.3


## @mastra/server-adapters-test-suite@0.3.0-alpha.1


## @mastra/platform-workspace@1.13.0-alpha.0


## @mastra/auth-better-auth@1.1.11-alpha.1


## @mastra/client-js@1.60.0-alpha.3


## @mastra/react@1.15.0-alpha.3


## @mastra/deployer-cloud@1.83.0-alpha.3


## @mastra/longmemeval@1.1.42-alpha.3


## @mastra/opencode@0.1.42-alpha.3


## create-factory@0.4.3-alpha.3


## @mastra/arize@1.3.30-alpha.2


## @mastra/arthur@0.4.30-alpha.2


## @mastra/braintrust@1.3.27-alpha.2


## @mastra/datadog@1.4.22-alpha.2


## @mastra/deepeval@0.1.24-alpha.2


## @mastra/laminar@1.3.32-alpha.2


## @mastra/langfuse@1.6.6-alpha.2


## @mastra/langsmith@1.4.3-alpha.2


## @mastra/otel-bridge@1.5.22-alpha.2


## @mastra/otel-exporter@1.5.2-alpha.2


## @mastra/posthog@1.4.7-alpha.2


## @mastra/sentry@1.2.32-alpha.2


## @mastra/agent-builder@1.2.1-alpha.1


## mastra@1.39.0-alpha.3


## @mastra/connect@1.3.2-alpha.2


## @mastra/deployer@1.83.0-alpha.3


## @mastra/editor@0.17.1-alpha.2


## @mastra/fastembed@1.3.7-alpha.0


## @mastra/mcp-docs-server@1.3.13-alpha.6


## @mastra/memory@1.41.1-alpha.1


## @mastra/elysia@0.1.23-alpha.3


## @mastra/express@1.5.27-alpha.3


## @mastra/fastify@1.5.27-alpha.3


## @mastra/hono@1.7.25-alpha.3


## @mastra/koa@1.7.27-alpha.3


## @mastra/nestjs@0.3.0-alpha.3


## @mastra/next@0.2.41-alpha.3


## @mastra/tanstack-start@0.2.41-alpha.3


## @mastra/convex@1.8.1-alpha.0


## @mastra/pg@1.36.0-alpha.2


## @mastra/inngest@1.11.2-alpha.2


## @mastra/temporal@0.6.0-alpha.3


## create-mastra@1.39.0-alpha.3


## @internal/playground@1.39.0-alpha.3

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-11-10 10:33:29 +01:00
mastra-platform[bot]
4854e36419
fix(core, server): stop a peer server from approving an aborted parked durable run (#29799)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9

## Problem
A durable agent run that is waiting for tool approval (with no thread)
can be aborted over HTTP. The abort returns `200 {"aborted":true}`, but
that was only recorded in memory on the server that handled the abort.
When the app runs on more than one server, approving the tool call on a
different server still resumed the run and **ran the tool**. When the
abort and approval happened on the same server, the approval was
rejected with a `500` rather than a clear error code. Fixes COR-1518.

## Fix
- **Abort saves the cancellation in storage before it is acknowledged**
(`packages/core/src/agent/durable/durable-agent.ts`):
- Stores that support concurrent updates: a guarded
`updateWorkflowState({ status: 'canceled', expectedStatus: 'suspended'
})`.
- Stores without `updateWorkflowState` (snapshot-only, e.g. LanceDB) and
stores without concurrent updates (e.g. Redis): reread the run, then
rewrite its snapshot as `canceled` with `persistWorkflowSnapshot(...,
expectedStatus: 'suspended')`, keeping `resourceId`.
- Storage errors propagate. The abort does not report success while the
run can still be resumed.
- If a resume has already claimed the run (status `running`), the
guarded write makes no change, and the existing abort of a running run
applies.
- **Resume on any server rejects runs canceled in storage** with
`AGENT_RESUME_ABORTED_RUN`, before it claims the run. The in-flight
resume marker is still taken synchronously before the first `await`
(#27365); the storage check runs right after it and releases the marker
when it rejects the run.
- **Observers of a canceled run are closed on every server**: the abort
that wins the `suspended → canceled` write publishes the abort event
right away on the shared pubsub, so `observe` streams on all servers end
and their history/listeners are cleaned up; the `canceled` row is kept.
If that publish fails, the abort still succeeds (the cancel is stored)
and a later refused approval/resume on that server retries the publish.
- **Docs and changeset**: `docs/.../agents/resumable-streams.mdx` and
`reference/agents/durable-agent.mdx` now point parked runs in
multi-server apps at `POST /agents/:agentId/runs/:runId/abort`, describe
the stored cancellation and the `409 Conflict`
(`AGENT_RESUME_ABORTED_RUN`) refusal, and qualify `abortRunStream` as
process-local. The changeset is scoped to
`DurableAgent`/`createDurableAgent` (evented agents inherit it; Inngest
agents are not covered) and states the best-effort limit on stores
without concurrent updates.
- **Approve route** (`packages/server/src/server/handlers/agents.ts`): a
`canceled` snapshot is treated as final instead of waiting 10s for a
suspension. The resource, thread and agent ownership checks still run
first, so other callers get `403` and learn nothing about the abort.
- **API contract** (`packages/server/src/server/handlers/error.ts`):
`AGENT_RESUME_ABORTED_RUN` now maps to **`409 Conflict`**, joining the
existing resume-claim conflict codes. It used to fall through to a
`500`.

## Proof
**Lifecycle regressions found in review and fixed in this PR** (final
head `1e0af6b5e8`):
1. *Healthy-lifecycle regression* (found by independent review at
`635c8837e3`): after a same-server abort and refused approval, an
attached observer stayed open and the run's history/listeners were
retained, because the `canceled` row skipped the old abort event. Fixed
in `c4cbdb987a`.
2. *Failed-first-publish recovery* (found by independent review at
`c4cbdb987a`): if publishing that abort event failed once, later
refusals never retried it and the observer stayed open. Fixed in
`7a856a33a3`, which also restores the #27365 marker ordering.
3. *Cross-server observer gap* (found by review at `7a856a33a3`): when
the abort and the refused approval landed on different servers,
observers stayed open because only the refusing server's local fence
published the abort event. Fixed in `6d444bee4f` by publishing at abort
time from the cancel winner.
4. *Human review (NikAiyer)*: docs, changeset scope, and a test that
pins the guarded write — addressed in `1e0af6b5e8`.

**Author: in-repo tests at final head `1e0af6b5e8`**
-
`server-adapters/hono/src/__tests__/parked-run-abort-across-servers.test.ts`:
two Hono servers over real TCP. They share a store and cache, and each
has its own pubsub and runtime state. 10/10 pass:
  - No abort, approve on the other server: `200`, tool runs once.
- Abort on the executing server, approve on the other server: `409`,
tool runs 0. **On main this returns 200 and the tool runs once (the
bug).**
- Abort on the other server, approve there: `409`, tool runs 0. Abort
and approve on the executing server: `409`, tool runs 0. On main both
return `500`.
- In all three abort rows, observers attached on **both** servers close
within 2s with exactly one abort frame each. With core from `7a856a33a3`
the executor-abort/other-approve row fails (observers stay open);
removing the abort-time publish makes it fail again.
- Another user's approval of an aborted run gets `403`, and so does
approval through another agent. In both cases the abort is not revealed.
- Race: a run that a resume has already claimed stays `running` after
the abort.
- Race inside the abort: a resume claims the run between the abort's
status read and its guarded write → the abort returns `200
{aborted:false}` and the run stays `running`. Removing `expectedStatus:
'suspended'` from the write makes this test fail.
- Observer: observe → abort → two refused approvals (409, 409) → the
observer closes within 2s, exactly one abort event is published, the row
stays `canceled`, and a peer approval still gets 409 with tool runs 0.
At `635c8837e3` the observer stays open; removing the once-guard
publishes 2.
- Failed first publish: the abort's own publish rejects, then the broker
recovers → the next refused approval on that server republishes → the
observer closes, one abort published, tool runs 0. Removing the
reservation release makes it fail (observer open, 0 published).
- `stores/lance/src/storage/parked-run-abort.test.ts`: real LanceDB with
two peers. After the abort, status is `canceled` and the peer's approval
is rejected with `AGENT_RESUME_ABORTED_RUN`, tool runs 0. Running it
with only the first commit fails with `updateWorkflowState is not
implemented`.
- `packages/server/src/server/handlers/error.test.ts`: 27/27 pass,
including the new 409 mapping.
- Revert checks: undoing the core change brings back `200`/1 tool run;
undoing the error mapping gives `500`; undoing the approve-route change
makes the approval wait out the 10s timeout.

**Author: real Redis, two processes — run on earlier heads
(`635c8837e3`/`c4cbdb987a`/`6d444bee4f`), NOT on the final head
`1e0af6b5e8`; this harness does not check observer closing** (two Node
processes running Hono, each sharing `RedisStore`, `RedisServerCache`
and `RedisStreamsPubSub` on a local Redis 7, using the public HTTP API)
| scenario | abort | approve on peer | tool commits |
|---|---|---|---|
| no abort | n/a | 200 | 1 |
| abort on executing server | 200 `{aborted:true}` | 409
`AGENT_RESUME_ABORTED_RUN` | 0 (main: 200, 1) |
| abort on peer | 200 | 409 `AGENT_RESUME_ABORTED_RUN` | 0 (main: 500,
0) |

**Independent reviewer** (separate session): approved the exact final
head `1e0af6b5e8`, including the `6d444bee4f` runtime change. Hono
10/10, HTTP ownership 4/4, Lance 1/1, resume-claim tests 37, build and
tsc. Failed-first-publish recovery and observer cleanup on both peers
were proven with real-TCP probes. Earlier rounds approved `635c8837e3`
(Lance 1/1, Hono 7/7; a LanceDB abort that returned `500` on the first
commit now returns `200` and the canceled snapshot keeps its other
fields) and `7a856a33a3` (concurrent refusals publish one terminal;
resume-marker release).

**Measured caveat (independent reviewer):** with simultaneous aborts,
the single CAS winner publishes one terminal and that closes the
observers on both servers; each original observer sees exactly one
abort, every approval gets `409`, and the tool runs 0 times. But a
**later** approval on the server whose abort lost the race can publish a
**second** terminal, because the once-guard is a per-instance set; after
cleanup this leaves 1 history entry retained. This PR does **not**
guarantee a globally-once terminal or unconditionally empty history.

Checks: tsc passes for core, server and hono. Lance oxlint is clean.
Prettier is clean on the changed lines; `agents.ts` already fails
Prettier on `main`, and that existing drift is left alone.

## Self-review (board self-review, approved at `635c8837e3`; the later
lifecycle fixes were approved by the independent re-review above)
The run abort route now records the cancellation in shared storage, and
every later resume or approval checks it on any server before anything
runs. A blocked approval returns `409 AGENT_RESUME_ABORTED_RUN`.
Adversarial check: if the resume claims first, the abort's guarded write
fails and the run stays `running` (tested). If the abort writes first,
the resume either sees `canceled` and gets 409, or loses its own claim
and gets the existing claim-lost 409. Authorization runs before the
canceled state is revealed. Nothing blocking was found.

| Check | Result | Tests |
|---|---|---|
| Build `@mastra/hono...` + `@mastra/lance...` | ✅ | — |
| Hono two-server HTTP (`parked-run-abort-across-servers.test.ts`) | ✅ |
7/7 |
| LanceDB parked abort (`parked-run-abort.test.ts`) | ✅ | 1/1, plus 8
reruns including under CPU load (1 early failure that could not be
reproduced) |
| Server error mapping (`error.test.ts`) | ✅ | 27/27 |
| Core durable suite (both storage projects, type-check on) | ❌ | 3966
passed, 1 failed, 4 skipped; no type errors |
| That failing file rerun on its own
(`durable-agent-tool-call-resumed.test.ts`) | ✅ | 30/30 |

Not run by self-review: a revert of the production files to re-run the
Hono test against the old code (Linear's main repro stands in for it,
and the author's revert checks above cover it); a real two-process Redis
run (covered by the author's harness above); repo-wide lint/format.

## Unverified
- The full Hono adapter suite was not run to completion.
- In the core durable suite, `durable-agent-tool-call-resumed.test.ts >
applies the display transform…` failed once with
`AGENT_RESUME_ALREADY_CLAIMED` and passed when run on its own. It looks
like test-isolation contention, not a confirmed cause. Separately,
`durable-agent-crash-recovery.test.ts` runs out of memory on `main` too.
- The Redis two-process harness was not run on the final head, and it
does not check observer closing; the cross-server observer tests use an
in-process shared-emitter pubsub, not Redis.
- A truly concurrent abort and approval on stores without concurrent
updates was not tested.

## Risk
- On stores without concurrent updates (Redis, Valkey, Elasticsearch,
LanceDB, ClickHouse, Cloudflare KV/DO), the canceled write is
**best-effort, not atomic**: the status is checked and then the snapshot
rewritten. An approval that lands between the two can still go ahead.
This PR does **not** guarantee that a racing abort and approval resolve
cleanly on every store; it guarantees that an abort that has already
completed is honored on every server. No new `supportsConcurrentUpdates`
requirement is added.
- The abort-event dedupe is per process and bounded (1000 run ids). It
is not global: see the measured caveat above (a later approval on the
losing-abort peer can publish a second terminal and retain one history
entry).
- A failed abort-time publish is retried only by a later refused request
on the aborting server; a peer's refusal does not retry it, and there is
no background retry.
- Aborted parked runs now keep a `canceled` row instead of a `suspended`
one.
- The thread abort route (`__abortThreadStreamAcrossProcesses`) does not
write the storage fence. That is out of COR-1518's scope.
- An abort that arrives while the run is still executing and parks just
afterwards keeps the existing behavior.
- `AGENT_RESUME_ABORTED_RUN` is also thrown by the non-durable `Agent`,
which now also gets `409` instead of `500`.



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

When a run pauses for tool approval, canceling it now records that
decision so another server cannot approve the run and start the tool.
This helps prevent canceled work from continuing elsewhere.

## Summary

- Persist cancellation for parked `DurableAgent` runs before confirming
the abort. Use a guarded status update when the storage adapter supports
concurrent updates. Otherwise, save the updated snapshot. On adapters
without concurrent updates, an approval that overlaps the save may still
proceed.
- Reject approval or resume attempts for canceled runs with
`AGENT_RESUME_ABORTED_RUN`. Check ownership before returning `409
Conflict`.
- Publish an abort event after cancellation so observers can close. If
publication fails, a later refused resume can retry it.
- Add cross-server integration tests and a LanceDB regression test.
Update the documentation to describe the behavior and clarify that it
applies to `DurableAgent`, not Inngest agents.

## Validation

Reported final-head results: the Hono two-server test passed 10/10, the
LanceDB test passed 1/1, and the server error-mapping test passed 27/27.
Core, server, and Hono type checks passed. The core durable suite
reported 3,966 passed and 1 failed; the failed test passed when rerun
alone. The full Hono suite was not completed. The Redis two-process
harness ran on earlier heads, not the final head. Repository-wide lint
and format results are not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-09 20:49:41 -08:00
mastra-platform[bot]
106c9fde0b
fix(convex): return counter values from ConvexServerCache.get so resumed streams replay correctly (#29847)
Follow-up from PLTFRM-3190 (found during #29784).

## Problem

The Convex `mastra/cache` handler's `get` op returned `null` for any
document that wasn't a plain value. Counters are stored as `kind:
'counter'` (written by `increment` and the atomic `listPushIndexed`), so
`ConvexServerCache.get(counterKey)` was always `null`. The in-memory and
Redis caches return the number.

`CachingPubSub` relies on that counter in two places, so on Convex:

- **`readLogFrom`** (#29783) couldn't work out how many holes the log
has, so every resume from an offset > 0 read the whole log instead of
the tail.
- **Restart detection in `subscribeFromOffset`** never fired. A
subscriber resuming past the end of a log that then expired or was
cleared silently dropped every frame of the new segment, because their
indices sit below its cursor.

## Change

- `stores/convex/src/server/cache.ts`: `get` returns `doc.counter` for
counter documents. Lists still read as `null`.
- Changeset: users need to redeploy their Convex functions (`npx convex
deploy`). No Mastra server restart is needed, because the client is
unchanged.

## Tests

`stores/convex/src/server/cache.test.ts`. Each new test fails with the
handler change reverted:

- `get` on a counter key returns the value after `increment` and after
`listPushIndexed`.
- `CachingPubSub` over `ConvexServerCache`: resuming at offset 4 reads
the log from position 4 (plus the single frame-0 id read), not from 0.
- `CachingPubSub` over `ConvexServerCache`: a subscriber resumed at the
log end gets the new segment's frame 0 after the log is deleted and the
run publishes again.

## Test plan

- [x] `vitest run` in `stores/convex`: 328 passed, 3 skipped
- [x] `tsc --noEmit`: no new errors (10 errors already on main in
`schema.test.ts` and `observational-memory.test.ts`)
- [x] oxlint clean on changed files
- [x] prettier clean on changed hunks (the `mastraCache` export warning
is already on main)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## ELI5

Convex could save counter values but could not read them back. This fix
lets resumed streams use those counters to find the right place in the
log and detect when a log has restarted.

## Changes

- `ConvexServerCache.get()` now returns counter values as numbers.
Missing counters and lists still return `null`.
- Added tests for counter reads, replaying only the log tail after an
offset, and receiving events after a log restarts at index 0.
- Added a patch changeset for `@mastra/convex`. It advises redeploying
Convex functions with `npx convex deploy`; a Mastra server restart is
not required.

## Checks

- Test, type-check, lint, and formatting results: not available.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-09 19:28:08 -08:00
mastra-platform[bot]
28ddc161a4
fix(core): snapshot agent stream events when they are published (#29803)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9

## Summary

Fixes COR-1517. `Agent.observe()` replays could show a later state of a
run than the one the caller saw live. After a tool approval and resume,
the replayed first `step-finish` listed 2 steps when the live one listed
1.

**Cause:** the agent stream publisher published the loop's own objects
by reference and queued the cache append. The loop keeps mutating those
objects (for example, a resumed run pushes into the same
`accumulatedSteps` array). The in-memory cache stored those references,
and a queued Redis append could serialize them after they had changed.

**Fix (`packages/core/src/agent/streams/publisher.ts` is the only
production file):** `publish()` copies every payload synchronously with
`decode(encode(value))` from `../../events/codec` before it calls
`enqueue()`. This covers step-start, chunks (including
tool-call-approval and suspended), the narrowed finish DTO `{ output,
stepResult, messages }`, and the captured suspended data. The
`producedAt` stamp is copied onto the snapshot. Each chunk is still
published once. The direct caller stream, callbacks, and processors are
unchanged.

## ⚠️ Behavior change for same-process observers

The snapshot uses the codec that cross-process pubsub already uses. It
is **not a lossless clone**. Same-process subscribers and in-memory
replays now get codec-normalized copies:

- **Kept:** `Date`, `Error`, `Map`, `Set`, `BigInt`, explicit
`undefined`, and registered classes (`DefaultStepResult`,
`DefaultGeneratedFile`, `DefaultGeneratedFileWithType`, including
generated-file bytes and media type).
- **Changed:** functions, symbols, and prototypes are dropped.
`NaN`/`Infinity` and cycles become `null`. Arbitrary `Uint8Array`s and
other unregistered class instances become plain data.
- **Fallback:** if `encode` throws (for example, a `toJSON()` that
throws), one warning is logged and *that* event is published by
reference. That event has no immutability guarantee. Other events and
the caller's stream are unaffected.
- Redis still applies its own `JSON.stringify` on top of the snapshot.

## Proof

- **Public-API repro (AIMock)**,
`observe-replay-snapshot.scenario.test.ts`: approval then resume, live
step counts `[1,2]`. On unchanged main, replay is `[2,2]` on the normal
and fs engines, for both the immediate cache append and the held
(delayed) append. With the fix, replay equals live.
- **Real Redis (author only)**,
`stores/redis/src/agent-stream-replay.integration.test.ts` against
redis:8 in Docker:
- Against core dist built from main, both the immediate and held-append
variants fail with `[2,2]` vs `[1,2]`. Real Redis is affected too.
- With the fix, both pass (3/3 runs earlier, and again 2/2 at this
head).
- **Contract test**, `publisher-snapshot.test.ts`, run over a real
`CachingPubSub` with a held append:
  - Mutating a payload after publish does not change history.
- The caller's chunk is untouched, and `producedAt` is kept on the copy.
  - The value policy above, including `DefaultStepResult` on finish.
- The fallback: a throwing `toJSON` publishes by reference with one
warning.
- **Revert proof:** with main's `publisher.ts`, all 8 new core tests
fail. With the fix, all 8 pass.
- **Author checks at this head:**
- Core streams, events, contract and scenario tests: 29/29 files, 392
tests.
  - `tsc`, `oxlint`, and `prettier` are clean.
  - The branch merges cleanly into current main.

## Self-review

Approved at `471e32c412`. The reviewer
checked for double publishing, loss of finish narrowing, and mutation of
the caller's chunk, and found no issues. Each switch case enqueues
exactly once (approval and suspend fall through to a single emit).
Finish data stays narrowed. A test asserts the caller's chunk is not the
published copy.

| Check | Result | Tests |
|---|---|---|
| publisher-snapshot.test.ts (unit, held append) | ✅ Passed | 4/4 |
| observe-replay-snapshot.scenario.test.ts (normal + fs engines,
immediate + held append) | ✅ Passed | 4/4 (durable/evented skipped by
design) |
| core src/agent/streams + src/agent/durable + src/events/codec | ✅
Passed | 396/399 files in the full parallel run; the 3 files that failed
there (execution-fence, signal-preemption, evented-execution) pass 6/6
when run on their own |
| tsc --noEmit (packages/core, stores/redis) | ✅ Passed | — |
| oxlint --type-aware (changed core files) | ✅ Passed | — |
| oxfmt --check (5 changed files) | ✅ Passed | — |

Reviewer regression proof: with base `publisher.ts` restored, all 8 new
tests fail (replay `[2,2]` vs live `[1,2]`, plus a mutated payload in
the unit test). With HEAD restored, they pass. The reviewer did not run
the Redis test (no Docker in that sandbox); it only type-checks there.

## Unverified

- Durable and evented engines: they publish `step-finish` through their
own paths without `output.steps`, so the scenario skips them. Inngest
and Valkey are also not exercised.
- A full-monorepo run at this head. In the author's broad related-suite
run, 629/631 files passed. The 2 errors were worker SIGABRTs in
`durable-agent-crash-recovery.test.ts`, which passes 20/20 on its own.
- The Redis test was run only by the author, not by the independent
reviewers.

## Risk

- Same-process consumers that relied on getting live object references,
or functions, symbols, or typed arrays, from observed or replayed events
get normalized data instead (see above).
- **Cost:** each published event now costs one synchronous `encode` +
`decode` walk of its payload. `step-finish` and finish payloads grow
with the step count, so the per-run cost grows roughly quadratically
with steps. No performance measurements were taken.

---------

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-09 17:41:15 -08:00
Tyler Barnes
5e63a3b59d
fix(pg): build indexes on newly created tables without CONCURRENTLY (#29092)
## Problem

When `@mastra/pg` initializes an empty schema, it creates its tables and
then ~23 indexes with `CREATE INDEX CONCURRENTLY`. If two or more
processes initialize the same schema at once (several replicas starting
together, or parallel test files), each concurrent index build waits for
the other process's open transaction. Postgres only clears that wait
after `deadlock_timeout` (1 s by default). That comes to ~1 s per index:

| processes cold-booting together | before | after |
|---|---:|---:|
| 2 | 20.6 s | 0.21 s |
| 4 | 29.6 s | 0.24 s |

This was also ~70% of the pg store CI job.

## Fix

If a table did not exist when this process began initializing, its
indexes are built with plain `CREATE INDEX`. Normally the table is brand
new, so the build is instant and the short lock it takes blocks nobody
(see the caveat below for when another process got there first). Tables
that already existed still get `CREATE INDEX CONCURRENTLY`, so existing
production tables are never locked for an index build.

- `schema-snapshot.ts` records which tables this init created (absent
from the schema snapshot it reads at start)
- `createTable` adds to that set; `createIndex` and the workflow
expression indexes pick plain or `CONCURRENTLY` from it

## Caveat

"Created by this init" means the table was absent when this process read
its snapshot. A slow replica could therefore plain-index a table that
another replica created moments earlier. That table may already be
taking writes from the other replica, and the plain build holds a SHARE
lock that blocks those writes for as long as the build takes
(milliseconds on a near-empty table). We think that's acceptable next to
~20 s of boot time, but the pg store owner should confirm.

## Changeset

`@mastra/pg` patch: `.changeset/pg-cold-boot-plain-indexes.md`.

## Test plan

- [x] New `concurrent-init.test.ts` case (`it.each` plain | concurrent,
both orders) converges to the same indexes
- [x] Full pg suite: 3905 passed / 20 skipped / 0 failed (86 files), 701
s → 323 s
- [x] PgBouncer transaction-pooling suite: 3 tests pass
- [x] `schema-snapshot.stress.test.ts` 355 s → 4.5 s
- [x] tsc and oxlint clean
- [ ] CI: pg store job green



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## ELI5

When the database starts, it can build indexes faster if a table is
brand new. This change uses a faster, briefly write-blocking method for
those tables, while keeping the less-blocking method for tables that
already existed.

## Changes

- Track tables created after the initialization schema snapshot was
read.
- Use ordinary `CREATE INDEX` for those tables. Keep `CREATE INDEX
CONCURRENTLY` for tables already present in the snapshot.
- Update workflow index creation to use the same rule.
- Add an integration test for both index-build orders, checking that
both builds finish and the index is valid.

## Reported validation

- The author reports that the full PostgreSQL suite passed 3,905 tests,
skipped 20, and had no failures across 86 files.
- The author reports that the PgBouncer transaction-pooling suite passed
3 tests, and that TypeScript and oxlint checks were clean.
- The author reports that `schema-snapshot.stress.test.ts` runtime
decreased from 355 seconds to 4.5 seconds.
- The PostgreSQL store CI job was still pending.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com>
2026-11-09 15:47:23 -07:00
mastra-platform[bot]
1f3035dbf5
fix(convex): stop resumed streams skipping events after a failed cache write (#29784)
## Problem

`CachingPubSub` stores each durable stream event in a cached log and
gives it an index. A client that reconnects reads the log back from an
offset, and that read assumes each event's index matches its position in
the list.

`ConvexServerCache` didn't override `listPushIndexed`, so it used the
base default: one Convex mutation to increment the counter, then a
second to append the entry. If the append failed after the increment had
committed (a timeout or a network error), the counter ended up one ahead
of the list. From then on every event sat one position below its index,
so a client resuming at offset `N` read event `N+1` first and silently
missed event `N`. The default also ignored the longer TTL requested for
durable topics. Redis, Valkey and the in-memory cache already do this
atomically (#27352). Convex was the remaining gap.

Fixes PLTFRM-3190.

## Change

- New `listPushIndexed` op in the Convex `mastra/cache` handler. One
mutation reads and increments the counter, appends `{ ...value, index }`
and gives the list and the counter the same `expiresAt`. Convex
mutations are transactional, so a failure rolls all of it back.
- `ConvexServerCache.listPushIndexed` sends that op through
`callUntilSettled`. A requested `ttlMs` can extend the default TTL but
not shorten it, and a cache with expiry disabled stays non-expiring.
Redis and the in-memory cache use the same rule.
- The handler runs in the user's own Convex deployment. If it hasn't
been redeployed yet and answers `Unsupported operation listPushIndexed`,
the cache logs one warning and falls back to the previous increment +
push path.

The core replay hardening (PLTFRM-3183) is being done separately.

## Test plan

- [x] Handler tests: indices are assigned in order, a stale `index` on
the value is replaced, the list and counter share one expiry, an expired
log restarts at index 0, and type conflicts are rejected.
- [x] End-to-end tests: real `CachingPubSub` → `ConvexServerCache` → the
real handler, through a fake client that rolls back a mutation that
throws, as Convex does.
- A publish whose cache write fails mid-mutation: resuming from an
offset delivers every cached frame, and indices stay contiguous.
- A publish that commits but whose response is lost: the frame is still
replayed at its index.
- Both tests fail when the override is disabled (old non-atomic path)
and pass with it.
- [x] Client tests: request shape, `ttlMs` extension, `ttlMs: 0` gives
`expiresAt: null`, fallback when the deployed handler lacks the op,
other errors still propagate.
- [x] `@mastra/convex` suite passes: 323 passed, 3 skipped. `oxlint` is
clean. `tsc` shows no new errors (the 10 existing errors in
`schema.test.ts` and `observational-memory.test.ts` are on main too).

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

The cache now saves a stream event and its position together. This helps
resumed streams avoid skipping events when a cache write fails. Older
Convex handlers use the previous method until they are redeployed and
the server restarts.

## Changes

- Added the `listPushIndexed` cache operation. The Convex handler
assigns an index using the larger of the list length and counter value,
then updates both records in one mutation.
- Made the list and counter share an expiry deadline. The requested TTL
can extend, but not shorten, the default TTL. If the default TTL is
disabled, entries do not expire.
- Updated `ConvexServerCache` to use the new operation. If the handler
returns `Unsupported operation listPushIndexed`, the cache warns and
uses increment plus list push for subsequent calls. This fallback is not
atomic and ignores requested retention.
- Added tests for indexed pushes, expiry, type conflicts, counter/list
drift, fallback behavior, transaction rollback, and lost responses.
- Added a changeset instructing users to redeploy Convex functions and
restart Mastra.

**CI and test results:** not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-09 12:37:46 -08:00
Damien Schneider
6bc7614a8f
perf(factory): index run binding lookups by thread (#29594)
**─────── TLDR ───────**
> Finding a Factory run binding by thread now uses an index instead of
the status index or a full scan. Postgres index DDL with an operator
class and a sort order is now valid.

In the Factory, `findActiveRunBindingByThread` and
`findRunBindingByThread` filter on org, thread, resource and session.
The only index with those columns starts with `factory_project_id`, so
they fell back to the status index or a full scan.

### Behavior changed

a run binding looked up by thread, on Postgres or LibSQL
&nbsp;&nbsp;├─ **was** — status index plus a filter, or a full scan for
any status
&nbsp;&nbsp;└─ **now** — `factory_run_bindings_org_thread_idx` on org,
thread, resource, session

a Postgres index with an operator class and `DESC` or `ASC`
&nbsp;&nbsp;├─ **was** — `createIndex` put the operator class after
`DESC`, which Postgres rejects; `exportSchemas` dropped the operator
class entirely
&nbsp;&nbsp;└─ **now** — both put it right after the column, ahead of
`DESC` or `ASC`

### Decisions

- **no default GIN index on thread metadata** — only Factory and
mastracode features filter threads by metadata, so every other Postgres
user would pay the write cost, the disk and a slower first init for
nothing. Deployments that need it can add it themselves
- **kept `factory_run_bindings_session_idx`** — `findActiveRunBinding`
and `findRunBindingBySession` still filter on `factory_project_id`

### Tradeoff

Local Postgres 16, `EXPLAIN ANALYZE` on 100k seeded bindings:

> active binding by thread: was `0.7–1.5 ms`, now `0.06 ms`
> any-status binding by thread: was `5.8 ms` (full scan), now `0.006 ms`

The index is 6.6 MB per 100k rows. It's built without `CONCURRENTLY`,
like every other Factory index, so a big `factory_run_bindings` blocks
writes while it builds (144 ms per 100k rows here).

### Not verified

- [ ] real deployment row counts. The numbers above come from seeded
local tables

---

> ██████████████████ **tests** `+188` (87% of the additions)
> ██ **source** `+19 −27`
> █ **changeset** `+10`


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5
This PR helps the database find run records faster when it searches by
thread details. It also fixes PostgreSQL index statements that combine
sort order with an operator class.

## Changes
- Added a `factory_run_bindings` index on `(org_id, thread_id,
resource_id, session_id)`. The existing project-scoped session index
remains.
- Added LibSQL and optional PostgreSQL tests for active and historical
thread lookups and their index usage.
- Updated PostgreSQL index SQL generation to place the operator class
before sort direction. Both generated schema SQL and runtime index
creation use the shared formatter.
- Added PostgreSQL integration tests for a descending column with
`text_pattern_ops`, including index creation during store
initialization.
- Added patch changesets for `@mastra/factory` and `@mastra/pg`.

## Checks
Test and CI results: not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Damien Schneider <damienschneider@damiens-MacBook-Pro.local>
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-09 10:44:58 -08:00
Eric Pinzur
7b912d293d
feat(observability): add byte-based exporter flush thresholds (#29621)
Count-based batches can retain large payloads and start many concurrent
exports during bursts. This adds opt-in byte-triggered batching to
`MastraPlatformExporter` and `MastraStorageExporter`, with an 8 MiB
default flush threshold for both.

```ts
new MastraPlatformExporter({ byteBatching: {} })
new MastraStorageExporter({ byteBatching: {} })
```

The byte threshold triggers sending; it never truncates or discards
records to fit a batch. Larger records are exported intact, and queued
data is retained when exports fall behind. Platform defaults to two
concurrent uploads and storage to one write. Existing count mode remains
the default, and existing auth, quota, and retry-exhaustion policies
still apply. CloudExporter is unchanged.

Storage ignores byte batching when the resolved strategy is `realtime`,
preserving its immediate, awaited writes. Other update-based strategies
serialize writes and alternate creates with eligible updates so
unrelated create traffic cannot starve span ends. Updates waiting for
their own create stay queued without consuming retries. Events arriving
during shutdown are accepted until the drain finishes. Failed storage
writes back off the shared connection across signals.

Includes queue diagnostics, docs, a changeset, real HTTP collector
tests, and real ClickHouse tests. The ClickHouse suite uses a declared
`@mastra/observability` dev dependency and its built public export,
defaults to the package's localhost test server, and no longer skips
without `CLICKHOUSE_URL`. CI's affected-test selector includes this
suite for storage-exporter changes, and the ClickHouse build plan
includes observability. Queues are not memory-capped or durable.

Validation: 302 focused exporter/shared-buffer tests passed. The 42-test
ClickHouse suite passed; its shutdown coverage was then strengthened to
persist actual span outputs and passed 20 repetitions. The starvation
and realtime tests failed before their fixes, and the shutdown-admission
regression failed in all 20 pre-fix runs. Package build, exporter source
type-checking, lint, formatting, and focused docs checks pass locally.

Local full-package type-checking reports four errors in
`src/test-utils/workflow-control-flow-suite.ts`. The identical four
diagnostics reproduce at the original main base commit `bb0fcd6211`
under the same local dependencies; they are not introduced by this PR.

The synthetic slow-collector benchmark retained all 800 records with
about 50 MiB additional heap in byte mode versus 104 MiB in count mode.
It uses a custom 128 KiB threshold and demonstrates
concurrency/allocation behavior, not production performance at the 8 MiB
default.

No matching GitHub issue was identified; this follows a
customer-reported exporter memory investigation.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

This PR lets the platform and storage exporters send records when their
data reaches a size limit. It keeps each record whole and retains queued
records while exports are in progress.

## Summary

- Adds opt-in byte-based batching to `MastraPlatformExporter` and
`MastraStorageExporter`, with an 8 MiB default threshold. Count-based
batching remains the default.
- Adds configurable concurrency. Platform uploads default to two
concurrent requests. Storage writes default to one. Update-based storage
strategies serialize writes; the `realtime` strategy ignores byte
batching.
- Adds byte-batching statistics, reference documentation, a changeset,
HTTP and ClickHouse tests, and a synthetic platform-exporter benchmark.
- The merge-base diff changes 15 files, with 1,574 lines added and 107
removed.
- Reported local results include 302 focused exporter/shared-buffer
tests, 42 ClickHouse tests, and 20 repeated shutdown-coverage runs
passing. The package build, exporter source type-checking, lint,
formatting, and focused documentation checks also passed. Full-package
type-checking reports four errors in
`src/test-utils/workflow-control-flow-suite.ts`; the same errors
reportedly occur at the merge base. Current CI results are not
available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-09 12:20:01 +01:00
dane-ai-mastra[bot]
f21e0742ce chore: version packages 2026-11-07 04:18:11 +00:00
Eric Pinzur
23308f9c19
feat(observability): add single score and feedback reads (#29529)
## Description

Add `GET /observability/feedback/:feedbackId` and typed client getters
for individual scores and feedback. Missing records return HTTP 200 with
`{ score: null }` or `{ feedback: null }`, preserving the existing score
endpoint contract.

```ts
const { score } = await client.getScore(scoreId);
const { feedback } = await client.getFeedback(feedbackId);
```

Client getters encode path IDs and preserve configured API prefixes,
credentials, and headers. Feedback reads use direct lookups in
in-memory, PostgreSQL, ClickHouse, and DuckDB storage. Includes
generated route types/CLI metadata, public API documentation, and
changesets.

Validation: package builds passed; 30 single-record HTTP cases passed
across five server adapters; 6 client transport tests, 341 server tests,
and 9 focused database cases passed across PostgreSQL/TimescaleDB,
ClickHouse, and DuckDB. Removing the feedback route caused the new
feedback HTTP cases to fail. Permission generation/checks, scoped
implementation lint, MDX formatting, Remark, and Vale passed.

Broader validation limitations: existing POST cases fail with
body-parser dependency errors in Express, Koa, and NestJS; the new GET
cases pass in all five adapters. The shared storage test helper also has
existing import-order lint errors outside the changed lines.

## Related issue(s)

Fixes
[OBS-790](https://linear.app/kepler-crm/issue/OBS-790/add-oss-single-score-and-feedback-reads),
under [OBS-792](https://linear.app/kepler-crm/issue/OBS-792). Platform
single-record endpoints remain scoped to OBS-791.

## Type of change

- [x] New feature (non-breaking change that adds functionality)
- [x] Documentation update
- [x] Test update

## Checklist

- [x] I have linked the related issue(s) in the description above
- [x] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

Applications can now look up a score or feedback record by its ID. If no
feedback record exists, the server returns `null`; it can also add
available author details to feedback.

## Summary

- Added `GET /observability/feedback/:feedbackId`. The route uses
existing observability authorization and returns `{ feedback: record }`
or `{ feedback: null }`.
- The feedback route adds author details when available. It returns HTTP
501 when the storage provider does not support single-feedback reads.
- Added `getScore(id)` and `getFeedback(id)` to the JavaScript client.
These methods encode IDs in the URL and use the client’s configured
request options.
- Added direct feedback lookups to in-memory, PostgreSQL, ClickHouse,
and DuckDB storage.
- Added response types, generated route metadata, API documentation,
tests, and changesets.

## Validation

Test and CI results: not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-06 10:23:52 +01:00
Anurag Ojha
f79db0de42
fix(duckdb): read payloads only for rows without previews (#29548)
## Description

DuckDB span and thread lists no longer read stored input and output when
previews are stored. This finishes the work that #29523 started. That PR
gave these lookups a time range, but a page whose rows are far apart
still read the payload columns across most of the file.

- `querySpans` and `queryThreadRows` now read only the stored previews
in their lookup. DuckDB reads every column a query names, even inside a
`CASE` that discards it.
- A further query reads `input`, `output`, and `error` only for rows
with a NULL preview, using the time range of those rows. Results do not
change.
- For spans, that query returns the whole row, so `model`, `provider`,
and the previews always come from one row.
- `error` is still read where `status` needs it: the first `querySpans`
query and the `queryThreadRows` lookup.
- Pages with rows written before previews existed now run one more
query. On a warm cache, this adds a few milliseconds.

On a 5.7 GB test file with every preview stored, a filtered `querySpans`
page of 25 rows far apart read 3006 MB from disk on `main` and 27 MB
with this change.

## Related issue(s)

Fixes OBS-645

Follow-up to #29523

## Type of change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [x] Performance improvement
- [ ] Test update

## Checklist

- [x] I have linked the related issue(s) in the description above
- [ ] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## ELI5

DuckDB was reading large input and output data even when a shorter
preview was already available. This change reads the preview first and
fetches the full data only for rows that need it, which can reduce work
on large databases.

## Changes

- Updated `querySpans` and `queryThreadRows` to fetch row metadata and
previews first. They issue a separate, bounded query for full payloads
only when a selected row lacks a required preview.
- Kept `error` in the initial thread lookup for status calculation. The
follow-up query fetches `error` only when its preview needs fallback.
- Replaced text-based query-plan inspection in DuckDB preview tests with
recursive parsing of JSON plans. Added checks that span and thread
queries avoid scanning input and output payloads when previews are
stored.
- Removed the unused `missingPreviewSql` helper and added a patch
changeset for `@mastra/duckdb`.

**Validation:** The diff adds query-plan tests. Results from completed
tests, builds, or CI checks are not available.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-06 14:35:10 +05:30
shreyas-mastra
08b25ea2df
fix(lance): escape string keys in load() (#29404) 2026-11-05 11:41:16 -08:00
Damien Schneider
defb82048f
perf(pg): skip the count query when listWorkflowRuns returns a short page (#29553)
**─────── TLDR ───────**
> `listWorkflowRuns` on pg answers a page shorter than requested with
one query instead of two, and `total` stays the same.

A short page already tells the total: the offset plus the rows it
returned. Listing an agent's suspended runs, on each tool approval and
stream resume, almost always returns a short page.

### Behavior changed

a page with fewer runs than requested
&nbsp;&nbsp;├─ **was** — a page query plus a count query
&nbsp;&nbsp;└─ **now** — the page query alone; `total` is offset plus
rows

a full page, or an empty page past the end
&nbsp;&nbsp;└─ **unchanged** — the count query still runs

### Gain

One round trip per short page: `1.13 → 0.5–0.7 ms` on local pg.

---

> ██████████████████ **tests** `+90`
> ███ **source** `+14 −11`
> █ **changeset** `+5`


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## ELI5
When a request returns fewer workflow runs than the page can hold, the
database can tell that there are no more runs to fetch. This change
avoids an extra count query in that case, while keeping the correct
total for full pages and empty pages beyond the end.

## Changes
- `WorkflowsPG.listWorkflowRuns` now fetches the page before deciding
whether to count matching runs.
- For a non-empty partial page, it sets `total` to the offset plus the
number of returned runs and skips the count query.
- For a full page or an empty page with a positive offset, it runs the
count query to determine the total.
- Added tests for partial pages, full pages, and empty pages beyond the
last run.
- Added a patch changeset for `@mastra/pg`.

## Validation
- Test and CI results: not available.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-05 20:09:38 +01:00
Anurag Ojha
83b6997a2a
fix(duckdb): bound payload lookups by time range (#29523)
## Description

DuckDB trace, span, and thread lists now stay fast on large databases.
Each read finds its page's payload rows with an OR of key tuples. DuckDB
cannot push that into the table scan, so the lookup read every row group
of `span_events`.

- Adds a `timestamp` or `endedAt` range to the lookups in
`fillMissingTracePreviews`, `fillMissingReconstructedPreviews`,
`querySpans`, and `queryThreadRows`. Each range covers every looked-up
row, so results do not change.
- `queryThreadRows` also adds an `endedAt IN` list. A chat's first and
last traces can be months apart, so the range alone could cover the
whole table.
- `querySpans` and `queryThreadRows` run this lookup on every page, so
they were slow on new data too, not only on rows without stored
previews.
- Uses time ranges rather than the `cursorId` range the issue proposes.
Rows written before `cursorId` existed hold NULL there, so their row
groups could not be pruned.
- Adds a test that counts the rows scanned on a real DuckDB table with
several row groups.

On a 9.5 GB, 1.2M-row test file with a 100 MB/s disk and an empty cache,
a `queryTraces` page of 25 rows without stored previews went from 75.6 s
to 0.8 s. Pages whose rows span a long time gain less, because more row
groups hold them. On a 3 GB file on the same disk, a thread page with an
old chat went from 25.7 s to 11 s.

## Related issue(s)

Fixes #29411

## Type of change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [x] Performance improvement
- [ ] Test update

## Checklist

- [x] I have linked the related issue(s) in the description above
- [ ] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## ELI5

When a page needs to fetch missing trace or span details, the database
could read far more rows than needed. This change limits those lookups
to the page’s time range, so large databases can avoid reading unrelated
data.

## Changes

- Added `timeRangeOf` to calculate inclusive timestamp bounds.
- Added timestamp bounds to missing-preview lookups in
`fillMissingTracePreviews`, `fillMissingReconstructedPreviews`, and
`querySpans`.
- Added timestamp bounds and an `endedAt IN` filter to
`queryThreadRows`, to narrow lookups when a thread’s first and last
traces are far apart.
- Added a large-table test that checks DuckDB’s analyzed query plan
scans fewer than two-thirds of the filler rows for four read paths.
- Added a patch changeset for `@mastra/duckdb`.

The test was added; execution results are not available.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-05 19:22:18 +05:30
dane-ai-mastra[bot]
675caf0613
chore: version packages (alpha) (#29336)
This PR was opened by the [Changesets
release](https://github.com/changesets/action) GitHub action. When
you're ready to do a release, you can merge this and publish to npm
yourself or [setup this action to publish
automatically](https://github.com/changesets/action#with-publishing). If
you're not ready to do a release yet, that's fine, whenever you add more
changesets to main, this PR will be updated.

⚠️⚠️⚠️⚠️⚠️⚠️

`main` is currently in **pre mode** so this branch has prereleases
rather than normal releases. If you want to exit prereleases, run
`changeset pre exit` on `main`.

⚠️⚠️⚠️⚠️⚠️⚠️

# Releases

> The changelog information of each package has been omitted from this
message, as the content exceeds the size limit.

## @mastra/client-js@1.60.0-alpha.0


## @mastra/react@1.15.0-alpha.0


## @mastra/factory@0.28.0-alpha.0


## @mastra/code-sdk@1.19.0-alpha.0


## mastra@1.39.0-alpha.0


## @mastra/core@1.83.0-alpha.0


## @mastra/mcp@2.9.0-alpha.0


## @mastra/mcp-apps-components@0.1.0-alpha.0


## @mastra/playground-ui@69.0.0-alpha.0


## @mastra/server@1.83.0-alpha.0


## @mastra/nestjs@0.3.0-alpha.0


## @mastra/clickhouse@1.30.0-alpha.0


## @mastra/duckdb@1.19.0-alpha.0


## @mastra/libsql@1.30.0-alpha.0


## @mastra/pg@1.36.0-alpha.0


## @mastra/temporal@0.6.0-alpha.0


## @mastra/auth-better-auth@1.1.11-alpha.0


## @mastra/deployer-cloud@1.83.0-alpha.0


## @mastra/deployer-cloudflare@1.2.42-alpha.0


## @mastra/deployer-netlify@1.2.42-alpha.0


## @mastra/deployer-sandbox@0.3.27-alpha.0


## @mastra/deployer-vercel@1.2.42-alpha.0


## @mastra/longmemeval@1.1.42-alpha.0


## @mastra/opencode@0.1.42-alpha.0


## create-factory@0.4.3-alpha.0


## mastracode@0.49.1-alpha.0


## @mastra/arize@1.3.30-alpha.0


## @mastra/arthur@0.4.30-alpha.0


## @mastra/braintrust@1.3.27-alpha.0


## @mastra/datadog@1.4.22-alpha.0


## @mastra/deepeval@0.1.24-alpha.0


## @mastra/laminar@1.3.32-alpha.0


## @mastra/langfuse@1.6.6-alpha.0


## @mastra/langsmith@1.4.3-alpha.0


## @mastra/observability@1.23.2-alpha.0


## @mastra/otel-bridge@1.5.22-alpha.0


## @mastra/otel-exporter@1.5.2-alpha.0


## @mastra/posthog@1.4.7-alpha.0


## @mastra/sentry@1.2.32-alpha.0


## @mastra/agent-builder@1.2.1-alpha.0


## @mastra/connect@1.3.2-alpha.0


## create-mastra@1.39.0-alpha.0


## @mastra/deployer@1.83.0-alpha.0


## @mastra/editor@0.17.1-alpha.0


## @mastra/mcp-docs-server@1.3.13-alpha.0


## @mastra/memory@1.41.1-alpha.0


## @mastra/rag@2.7.3-alpha.0


## @mastra/redis-streams@0.8.1-alpha.0


## @mastra/elysia@0.1.23-alpha.0


## @mastra/express@1.5.27-alpha.0


## @mastra/fastify@1.5.27-alpha.0


## @mastra/hono@1.7.25-alpha.0


## @mastra/koa@1.7.27-alpha.0


## @mastra/next@0.2.41-alpha.0


## @mastra/tanstack-start@0.2.41-alpha.0


## @mastra/spanner@1.11.2-alpha.0


## @mastra/turso@0.1.21-alpha.0


## @mastra/inngest@1.11.2-alpha.0


## @mastra/gcs@0.3.6-alpha.0


## @internal/playground@1.39.0-alpha.0

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-11-04 14:40:27 -08:00
Tony Kovanen
4afeceaf2d
feat(pg, libsql): add experimental storage v2 for workflow runs (Postgres and libSQL) (#28299)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: review <review@local>
2026-11-04 21:12:40 +01:00
Eric Pinzur
8ad7773cd4
feat(observability): align signal writes and context modes (#29289) 2026-11-04 12:00:19 -08:00