mirror of
https://github.com/mastra-ai/mastra.git
synced 2026-11-17 08:44:46 +00:00
1,827 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
d60ba71be0 | chore: version - exit prerelease mode | ||
|
|
5251e7199e |
Fix atomic pubsub replay writes on Redis Cluster (#30174)
> Opened by @abhiaiyer91
## Summary
- Preserve existing `CachingPubSub` replay keys by default, so upgrading
does not make in-flight logs unreadable for other cache backends or
Redis standalone.
- Add `keyFormat: 'redis-cluster'` as an explicit option for new Redis
Cluster deployments. It encodes the topic into a hash tag shared by the
replay list and counter, allowing the atomic Redis Lua append instead of
the `CROSSSLOT` fallback. PubSub topic names remain unchanged.
- Retain the Redis cache fallback for callers that still supply untagged
keys; add coverage for both key formats and edge-case topics.
## Usage
Pass `{ keyFormat: 'redis-cluster' }` as the third argument to `new
CachingPubSub(innerPubSub, redisServerCache, options)` on **all workers
sharing the same cache**. Roll it out for new logs, or after existing
replay logs expire. The default remains the original
`${keyPrefix}${topic}` format; opt-in keys use
`${keyPrefix}{${encodeURIComponent(topic)}}` (with `%EMPTY` for an empty
topic).
## Test plan
- Seven focused core suites: 169 tests passed, with no type errors; core
build/type generation and targeted oxlint passed.
- Convex cache suite: 21 tests passed with original key expectations.
- Redis Docker Compose integration: 14 tests passed.
- Real slot-assigned Redis Cluster: confirmed untagged two-key `EVAL`
fails with `CROSSSLOT`; opted-in replay successfully publishes and reads
indexes 0–2.
Fixes #30149
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
ELI5: Redis Cluster sometimes refuses to run one command that touches
two related cache keys. This PR puts those keys in the same place, so
the command works and event indexes stay in order.
## Summary
This PR changes `CachingPubSub` so that a topic's replay list and its
index counter share a Redis Cluster hash tag.
`RedisServerCache.listPushIndexed` can then run its Lua script on both
keys without a `CROSSSLOT` error.
## Changes
- **`packages/core/src/events/caching-pubsub.ts`**
- Adds an optional `keyFormat?: 'redis-cluster'` option. The default is
`'legacy'`.
- Legacy mode keeps the list key as `keyPrefix + topic`.
- `redis-cluster` mode builds the list key as `keyPrefix + {tag}`. The
tag is `encodeURIComponent(topic)`, or `%EMPTY` when that value is
empty.
- The counter key is now `<list key>:counter`. In `redis-cluster` mode,
it uses the same hash tag as the list key.
- **`stores/redis/src/cache.ts`**
- Updates the comment on the `CROSSSLOT` fallback. The fallback logic is
unchanged. Untagged keys can still trigger `CROSSSLOT`, so the fallback
remains for callers that do not use tagged keys.
- **`.changeset/redis-cluster-atomic-pubsub-replay.md`**
- Adds a patch changeset for `@mastra/core`.
## Behavior and compatibility
- Topic names used for publishing are unchanged.
- The default `legacy` format keeps existing replay keys. Existing logs
stay readable with the default setting.
- Opting into `keyFormat: 'redis-cluster'` changes replay key names.
Workers that use the new format cannot read logs written under the old
names until those logs expire by TTL.
## Scope limits
- The change co-locates only the replay list and counter for each topic.
It does not co-locate those keys with transport streams or other keys
for the same run.
- The change does not add a new default for `keyFormat`.
## Tests and checks
- The diff adds no test file. The PR objectives report test runs, but
this summary does not verify them.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
|
||
|
|
05a6931471 |
chore: version packages (alpha) (#30172)
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. ⚠️⚠️⚠️⚠️⚠️⚠️ `main` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `main`. ⚠️⚠️⚠️⚠️⚠️⚠️ # Releases ## @mastra/client-js@1.60.0-alpha.12 ### Minor Changes - Allow `querySpans()` to sort by `model` or `provider`. ([#30096](https://github.com/mastra-ai/mastra/pull/30096)) ```ts orderBy: [{ field: 'model', direction: 'asc' }]; ``` ### Patch Changes - The generated route types for scores now include the `SCORER` and `PROCESSOR` entity types, matching the entity types experiments can emit. ([#30145](https://github.com/mastra-ai/mastra/pull/30145)) - Add span-local error counts. ([#30088](https://github.com/mastra-ai/mastra/pull/30088)) - Updated dependencies [[`56f654f`]( |
||
|
|
f0710446e0 |
feat(observability): sort spans by model and provider (#30096)
Adds ascending/descending `model` and `provider` ordering to
`querySpans()` across Core, server, Client JS, PostgreSQL, DuckDB and
ClickHouse. No schema changes or backfill.
```ts
orderBy: [{ field: 'model', direction: 'asc' }]
```
Sorting uses the dedicated columns before pagination, with
case-sensitive UTF-8 ordering, missing values last, and identity
tie-breakers. String cursors preserve their values and remain bound to
the query and trusted scope. Legacy attribute-based display values
remain available without changing cursor boundaries. Trace/thread sort
fields stay unchanged. The server advertises
`spanQueryModelProviderSort` and rejects unsupported stores.
This PR is API-only. Studio sorting controls and live-refresh changes
are tracked in
[OBS-906](https://linear.app/kepler-crm/issue/OBS-906/add-model-and-provider-sorting-to-spans-explorer).
Enabling those controls needs raw model/provider sorting values
available to the existing head-row merge, including when polls are
ordered by end time; using display values can disagree with database
ordering. The PR has no remaining Playground UI changes or release
entry.
Validation: Core/server tests, real storage conformance across
PostgreSQL, DuckDB and ClickHouse, mixed-core checks, and
Core/server/adapter typechecks passed for the API implementation. After
removing the UI changes, both existing span component/hook suites pass
(52 tests). Changed-file lint and docs checks including CI Vale pass.
A bounded local ClickHouse 26.6.8.7 comparison used 180,000 synthetic
rows, 10,000 matches per tenant, distinct tenants across cases, both
directions and continuation pages, and disabled query/condition caches.
Median CPU was 21.9 ms for start time, 19.4 ms for model, and 21.4 ms
for provider; median query memory was about 4.6 MiB for each (maximum
6.8/8.4/7.2 MiB). These are selection-query measurements on synthetic
OSS data, not a production benchmark.
Part of OBS-896. Platform API support and package pins follow this OSS
merge and snapshot publication. UI enablement is tracked separately in
OBS-906.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
**ELI5:** Trace spans can now be sorted by their model name and provider
name, in both directions. The database does the sorting, so users can
page through the results in the right order.
## Overview
This change adds `model` and `provider` as sort fields for span queries.
It applies to the Core API, the server handlers, and the PostgreSQL,
DuckDB, and ClickHouse stores. The diff has 31 changed files, with 320
lines added and 43 removed.
## Core changes
- `SPAN_QUERY_ORDER_FIELDS` now includes `model` and `provider`.
- `SpanQueryPlanOptions` adds `allowSpanModelProviderOrdering?:
boolean`. Ordering by these fields requires this option.
- Trace and thread queries do not support these fields.
- Cursor validation accepts string or null values for `model` and
`provider`. Timestamp normalization applies only to `startedAt` and
`endedAt`.
- `SpanQuerySelectedRow` adds an optional `cursorSortValue`. Response
construction removes this field from returned spans. It uses the value
for the next-page cursor, and falls back to the row's ordering field
when the value is absent.
- The base storage feature union adds `span-query-model-provider-sort`.
## Server changes
- `assertObservabilitySpanModelProviderSortSupported` returns HTTP 501
with `SPAN_QUERY_UNSUPPORTED` when a store does not declare
`span-query-model-provider-sort`. The check runs before `querySpans` is
called.
- `ObservabilityStorageCapabilities` and its empty default add
`spanQueryModelProviderSort`, which defaults to `false`.
- The capability is reported as supported only when span queries are
supported, the core order fields include both `model` and `provider`,
and the store declares the feature.
- The system capabilities schema adds `spanQueryModelProviderSort:
z.boolean().default(false)`.
- The route handler permits model and provider ordering and performs the
capability check.
## Store changes
- **PostgreSQL:** Sorts by `model` and `provider` use `COLLATE "C"`. The
cursor bounds cast sort values to `text COLLATE "C"` when `stringSort`
is enabled. The feature is advertised only when the core order fields
include both fields.
- **DuckDB:** Sorts by `model` and `provider` use `"C"` collation.
Cursor values cast as strings with the same collation. The feature is
advertised only when the core order fields include both fields.
- **ClickHouse:** `afterCursor` treats `model` and `provider` as
strings. Null rows are included after the cursor for numeric and string
sorts. The feature is advertised only when the core order fields include
both fields.
- Returned `model` and `provider` values may still come from span
attributes. The code comments state that these fallback values are not
used for filtering or sorting.
## Tests and conformance
- Core tests cover ordering gating, rejection in trace and thread
queries, string and null cursor values, and rejection of cursors when
the order, filters, authorization, or scope change.
- Server tests cover the 501 rejection for `durationMs`, `model`, and
`provider` on older stores. They also cover capability parsing and the
capability flag in the system endpoint.
- Shared conformance tests in `stores/_test-utils` cover ascending and
descending pagination at page sizes 1, 2, and 3. They cover empty
strings, missing values, case and Unicode ordering, ties, and filter
exclusion. They also check that missing values sort last and that legacy
attribute values are not treated as dedicated fields.
- Store feature-list tests for ClickHouse, DuckDB, and PostgreSQL expect
the new feature flag.
## Changesets and documentation
- Changesets add minor release notes for `@mastra/core`,
`@mastra/server`, `@mastra/client-js`, `@mastra/duckdb`,
`@mastra/clickhouse`, and `@mastra/pg`.
- The advanced query reference adds the `spanQueryModelProviderSort`
capability row. It documents `model` and `provider` ordering. It states
that the comparison is case-sensitive and that missing values sort last
in both directions.
## Scope and limits
- This diff does not include Studio UI changes. The commit history says
UI enablement is deferred to a separate issue because it needs raw model
and provider values for head-row merging.
- This diff does not include client-js source changes. Its client-js
changeset documents the new sort fields.
- The diff does not include schema changes or backfill.
- The PR description reports these results, which this review did not
run or verify:
- Core and server tests, and real shared-storage conformance and
mixed-core checks, pass.
- Core, server, and adapter typechecks pass. Staged-file lint and docs
checks, including CI Vale, pass.
- 58 Studio and MSW tests pass.
- The full Playground UI typecheck reports four unrelated chart and form
errors, and the library declaration build fails.
- A local ClickHouse 26.6.8.7 comparison with 180,000 synthetic rows
measured median CPU of 21.9 ms for start time, 19.4 ms for model, and
21.4 ms for provider. Median query memory was about 4.6 MiB for each.
These figures are not production measurements.
- Platform backend and UI wiring, and package pins, remain under
OBS-896.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
a2998f4084 |
test(storage): verify rollback reporters settle before client release (#30160)
## Summary - Make PostgreSQL pooled/pinned and DSQL pooled async rollback reporter tests assert that reporting finishes before transaction settlement and pooled release. - Use plain async reporters to catch unhandled promise rejections rather than `vi.fn` wrappers. ## Test plan - PostgreSQL transaction tests: 18 passed; DSQL transaction tests: 9 passed. - Both package TypeScript checks, targeted Oxlint, Prettier and `git diff --check` passed. - Confirmed the changed tests fail against the parent of merged #30157: PG 2 failures, DSQL 1 failure (plus unhandled rejections). Follow-up to #30157; test-only correction to the regression coverage for its already-merged implementation. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ELI5: The change makes the database tests check the right order of events. A rollback error must be reported before the database connection is released and before the transaction ends. The tests now fail if that order is wrong. ## Summary This change updates two test files for the rollback reporter behavior in PostgreSQL and DSQL transaction clients. It changes test code only. No production code changes. Diff stats: 2 files changed, 33 insertions, 9 deletions. ### `stores/dsql/src/storage/client.tx-serialize.test.ts` - The reporter is now a plain async function, not a `vi.fn` wrapper. - The reporter stores the error it receives. - The reporter waits 10 ms, records `reported`, and then throws `async logger failed`. - The test mocks `client.release` to record `released`. - The test records `tx-settled` after the transaction call rejects with the original error. - Assertions: - The reporter received `rollbackError`. - `client.release` was called with `true`. - The event order is `['reported', 'released', 'tx-settled']`. ### `stores/pg/src/storage/client.tx-serialize.test.ts` - The same reporter change applies to the shared async rollback reporter test. - For the pooled case (`kind === 'pool'`): - `client.release` records `released`. - Assertions check that `client.release` was called with `true`. - The event order is `['reported', 'released', 'tx-settled']`. - For the pinned case: - No release tracking is added. - Assertions check that `adapter.shouldDestroyClient` is `true`. - The event order is `['reported', 'tx-settled']`. ## Not verified - The PR reports test runs, type checks, lint, and format checks. I did not run them. Their results are not available from this review. - The PR reports that the changed tests fail against the parent of the referenced prior change. I did not verify this. It is not available from this review. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com> Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
055a8c3b82 | chore: version packages | ||
|
|
b1948fb073 |
fix(storage): await rollback diagnostics before releasing clients (#30157) | ||
|
|
743a02786e |
feat(valkey): identify Mastra-created clients with a clientInfoTag (#30156) | ||
|
|
ce631e94c6 |
fix(storage): prevent poisoned PostgreSQL transactions from returning to the pool (#30144) | ||
|
|
c014d187bb |
chore(deps): update docker images (major) (#19788) | ||
|
|
9e82af0237 |
chore: version packages (alpha) (#30074)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
|
4884ee08a0 |
fix(agent-controller): answer parked tool approvals when the permission rules allow or deny them (#30114)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
1689ce38e1 |
perf(agent-controller): read only the newest page when listing threads with a limit (#30090)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com> Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com> |
||
|
|
259f142f46 |
fix(pg): support scalar and nested operators in vector elemMatch filters (#30095) | ||
|
|
cd041a5bb0 |
perf(pg): index thread metadata for listThreads metadata filters (#30086)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
c8fecd5d6f |
fix(pg): preserve all nested $elemMatch conditions in vector filters (#30091) | ||
|
|
aa49b049d0 |
feat(observability): order completed spans by duration (#30085)
Adds `durationMs` ascending/descending ordering to `querySpans()`, combined with existing duration filters and stable cursor pagination. Duration is the completed span’s elapsed time; zero is valid. Trace ordering and trace runtime semantics are unchanged. The server advertises `spanQueryDurationSort` and rejects unsupported stores before execution. A span-specific field list and default-off planner option protect old servers paired with newer Core. Updated adapters advertise support only when Core understands it. PostgreSQL and DuckDB reuse their numeric ordering paths; ClickHouse handles numeric cursor parameters and normalizes duration values returned as Int64 strings. Includes generated API types, documentation and package changesets. No schema migration. Implements the OSS portion of [OBS-899](https://linear.app/kepler-crm/issue/OBS-899). [OBS-904](https://linear.app/kepler-crm/issue/OBS-904) tracks Platform integration after merge and a snapshot from main. UI is outside this backend change. Validation: 134 Core, 112 server, 125 Client JS, 100 PostgreSQL, 94 DuckDB and 103 ClickHouse tests passed (one existing ClickHouse skip). Shared conformance exercises both directions, filters, zero durations and ties across page sizes. The new Core tests fail before the implementation; reverting the ClickHouse cursor fix makes both new pagination cases fail by skipping rows. Old-Core adapters omit the new feature; old stores receive a structured 501. Core/server/store typechecks, Core/server/SDK builds, changed-file lint/format and required docs checks passed. Vale reports six warnings in unchanged text. Local Node 23.11.0 is outside the supported 22/24 versions; CI must verify supported runtimes. A bounded local ClickHouse 26.6.8.7 experiment used 20,000 synthetic completed spans, the same duration filter and 50-row first/second pages in both directions. Duration ordering peaked at 3.43–3.67 MiB versus 3.42–3.66 MiB for timestamp ordering; both read 20,000 rows/1,488,890 bytes. Duration query CPU was 7–25 ms. These are single-run local measurements, not production capacity evidence; no ingestion or storage layout changed. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 This PR lets users sort completed spans by how long they took. It keeps pagination stable and enables the feature only when Core and the storage adapter support it. ## Changes - Added ascending and descending `durationMs` ordering to `querySpans()`. - Added duration-aware cursor validation and stable pagination. - Added `spanQueryDurationSort` capability reporting. - Added HTTP 501 handling for unsupported stores. - Added compatibility gates for older Core versions. - Updated PostgreSQL, DuckDB, and ClickHouse adapters. - Converted ClickHouse duration values to numbers. - Added Core, server, adapter, and conformance tests. - Updated generated API types, documentation, and package changesets. - No schema migration was added. ## Validation - Reported passing tests: Core 134, server 112, Client JS 125, PostgreSQL 100, DuckDB 94, and ClickHouse 103. - One existing ClickHouse test remains skipped. - Reported typechecks, builds, lint, formatting, and documentation checks passed. - Vale reported six warnings in unchanged text. - CI verification on supported Node.js versions is not available. - Diff size: 28 files changed, with 220 additions and 8 deletions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
b0ef22e7bf |
feat(schedules): Add bounded one-off schedules (#25131)
> Opened by @abhiaiyer91 Refs #24610 ## Summary Schedules can now fire once or stop after a given time, so you no longer need to pin a cron expression to a date and clean it up yourself. - **New `completed` status**: the scheduler and timing reconciliation set it when no automatic fires remain. It's written in the same atomic `nextFireAt` check-and-set used to claim a run, so a schedule still fires only once across instances. A cron with no future occurrence left also moves to `completed` instead of erroring. `pause`/`resume` on a completed schedule return 409. Changing `runAt`/`endAt` reactivates it only when the resulting timing can still fire. `list()` hides completed schedules unless you pass `status: 'completed'`. - **Declarative workflow schedules**: `createWorkflow({ schedule: { runAt } })` and `{ cron, endAt }` are supported. Past times are accepted so redeploys don't fail, and a completed schedule isn't re-armed unless its timing changes. - **Trigger kind**: one-off fires report `trigger.kind: 'once'`. - `run(id)` doesn't complete a one-off schedule. ## Surfaces - **Core**: types, validation, scheduler claim logic, create/update/pause/resume/list, declarative sync. - **Storage**: new nullable `run_at`/`end_at` columns for libsql, pg, mysql, mongodb, spanner, convex and in-memory. Existing columns are untouched: one-off schedules store `cron` as `''`. `updateScheduleNextFire` takes an optional `newStatus`. - **Server/client-js/CLI**: request bodies accept `runAt`/`endAt` (one-of rule checked in the schema). Responses include the new fields and `completed`. Route types and CLI route metadata regenerated. - **Studio**: shows the run time for one-off schedules, shows the `endAt` bound, adds a `completed` badge and hides pause/resume for completed schedules. - **Docs + changeset.** ## Not in this PR (remaining #24610 work) - ISO 8601 string inputs for `runAt`/`endAt` (numbers and `Date` are accepted today). - The template `schedule_once` tool. ## Test plan - [x] core: schedules, scheduler, cron, schedule-config and scheduler integration tests (260 passing), clean `tsc` - [x] Shared adapter suite gained tests for `runAt`/`endAt` round-trip and atomic status CAS. Passing on libsql, pg, mysql and mongodb (Docker), plus convex server tests. **Not run on Spanner (no emulator available); typecheck only.** - [x] server schedules handler tests (51 passing, including one-off/bounded creation and body validation) - [x] client-js and playground typecheck clean; playground schedules tests passing <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 Schedules can now run once at a chosen time or repeat until a chosen end time. They complete when their planned runs are over, and changing their timing can make them active again. ## Summary - Added `runAt` for one-off agent and workflow schedules. API-created schedules require a future `runAt`; declarative workflow schedules can register past times as `completed`. - Added `endAt` for cron schedules. A schedule becomes `completed` when its next run would occur after `endAt`, or when no future run remains. - Completed schedules cannot be paused or resumed. Timing changes can reactivate a completed schedule when a future run remains. `list()` hides completed schedules unless callers filter with `status: 'completed'`. - Resuming a paused one-off schedule after its `runAt` has passed allows it to fire on the next scheduler tick. Calling `run(id)` manually does not complete a one-off schedule. - Added the `once` trigger kind for scheduled one-off agent fires. Manual runs remain `manual`. - Declarative workflow schedules preserve completed status across redeploys when timing is unchanged. Timing changes can re-arm them. - Updated schedule APIs, storage adapters, Studio, and documentation. Storage persists `runAt` and `endAt` and supports atomic status updates. ## Checks Current-head CI, build, and test results: not available. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Abhi Aiyer <abhiaiyer91@gmail.com> Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com> Co-authored-by: NikAiyer <nick.aiyer@gmail.com> Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com> Co-authored-by: Ehindero Israel <iehindero419@stu.ui.edu.ng> |
||
|
|
3a6ca42696 |
chore: version packages (alpha) (#30026)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
|
c860b9bb43 |
feat(observability): support OSS feedback replacement (OBS-887) (#30059)
Implements [OBS-887](https://linear.app/kepler-crm/issue/OBS-887/add-oss-feedback-replacement-and-export). Add OSS `PUT /api/observability/feedback/:feedbackId` with the existing wrapped client request. Replacements retain the original feedback ID, trace ID, timestamp, and tenant; omitted optional fields are cleared. A full replacement can create a missing record. OSS persists locally before responding, then independently notifies supporting exporters. Resolve mode still requires the local anchor. The SDK also exposes optional `replaceFeedback()` for independent exporter delivery without a producer-side storage lookup. Replacement events use a separate optional exporter hook. `MastraPlatformExporter` sends PUT with stable identity across bounded anchor retries, and flush/shutdown drain deliveries. Older exporters skip replacements rather than replaying them as creates. Unsupported storage/package combinations return 501 or skip unavailable export support. No UI changes; review-status PATCH propagation remains tracked in OBS-785. In-memory, PostgreSQL vNext, DuckDB, and ClickHouse vNext support replacement. In-memory, PostgreSQL, and DuckDB enforce replacement identity at the write. PostgreSQL coordinates writes by feedback ID, including duplicates submitted with a different timestamp. Duplicate HTTP creates return the stored identity and timestamp and re-notify exporters with accepted values. Tenant mismatches remain hidden, and an omitted SDK replacement author stays null. HTTP/exporter creates use `onConflict: 'ignore'` so a delayed original create doesn't undo an edit; direct storage calls keep their legacy upsert default. **ClickHouse requires no migration:** it uses the existing schema and read-before-insert protection. Sequential delayed creates are covered, but overlapping writes from separate processes can race; this is not a globally ordered or exactly-once protocol. Latest review fixes: DuckDB's lazy facade delegates replacement to its guarded writer; concurrent coverage runs 20 overlapping request pairs per identity variant. Direct storage replacements normalize an omitted author to null. PostgreSQL single-feedback reads use the primary for read-after-write consistency, while list/aggregate reads retain the configured read client. Batch advisory locks use one ordered query with unchanged lock keys. Validation: the stronger DuckDB regression failed on all four identity variants before the facade fix. Afterward, the entire DuckDB package passed 718 tests across 21 files in three consecutive runs. PostgreSQL passed 57 feedback tests, including a real stale read snapshot and overlapping reversed 1,000-ID batches. Shared replacement tests passed on in-memory (12), and the new author regression passed on ClickHouse without a migration. Core/storage builds and declarations, typed lint, and formatting passed. Earlier HTTP (85 cases on each of four adapters), generic Hono routes (2,848), server/system, exporter, and docs checks remain recorded in the PR history. Local tests use Node 23.11.0; CI validates the supported runtime. The unrelated Traces UI fade assertion failure was fixed in https://github.com/mastra-ai/mastra/pull/30068; this branch includes that fix from main. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 This PR lets users replace feedback without changing its identity. It stores the replacement locally and sends it to supported exporters without allowing delayed creates to overwrite it. ## Changes - Added `PUT /api/observability/feedback/:feedbackId`. - Preserved the feedback ID, trace ID, timestamp, and tenant. - Cleared omitted optional fields. - Added capability detection and `501` responses for unsupported storage. - Added replacement event types, SDK support, client types, route metadata, and documentation. - Added replacement delivery, stable retries, flush handling, and shutdown draining for `MastraPlatformExporter`. - Added replacement support for in-memory, PostgreSQL vNext, DuckDB, and ClickHouse vNext. - Added `onConflict: 'ignore'` for HTTP and exporter creates. Direct storage calls retain their upsert default. - Older exporters skip replacements instead of replaying them as creates. - Added ClickHouse support without a migration. Concurrent writes from separate processes can still race. - No UI changes were included. ## Validation The author reports passing the HTTP route and contract suites, feedback storage suites, partition and SQL tests, server and system tests, record-builder tests, build, typecheck, lint, and documentation checks. For the current head, the available CI result is `2027dev Evals: skipped`. Other listed CI checks target different commits or remain in progress. Current-head CI results are otherwise not available. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
3b27a10322 |
fix(pg): keep thread owner and createdAt when saveThread hits an existing id (#29149) | ||
|
|
562a451e22 |
fix(pg): stop concurrent thread metadata updates from dropping keys across processes (#29150) | ||
|
|
23cc34a9c5 |
chore: version packages (alpha) (#29938)
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. ⚠️⚠️⚠️⚠️⚠️⚠️ `main` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `main`. ⚠️⚠️⚠️⚠️⚠️⚠️ # Releases > The changelog information of each package has been omitted from this message, as the content exceeds the size limit. ## @mastra/factory@0.28.0-alpha.6 ## @mastra/code-sdk@1.19.0-alpha.6 ## @mastra/playground-ui@69.0.0-alpha.7 ## @mastra/client-js@1.60.0-alpha.6 ## @mastra/react@1.15.0-alpha.6 ## @mastra/deployer-cloud@1.83.0-alpha.6 ## @mastra/longmemeval@1.1.42-alpha.6 ## @mastra/opencode@0.1.42-alpha.6 ## create-factory@0.4.3-alpha.6 ## mastracode@0.50.0-alpha.6 ## @mastra/agent-builder@1.2.1-alpha.3 ## mastra@1.39.0-alpha.6 ## @mastra/core@1.83.0-alpha.6 ## @mastra/deployer@1.83.0-alpha.6 ## @mastra/editor@0.17.1-alpha.4 ## @mastra/mcp@2.9.0-alpha.3 ## @mastra/mcp-docs-server@1.3.13-alpha.9 ## @mastra/memory@1.41.1-alpha.3 ## @mastra/rag@2.7.3-alpha.2 ## @mastra/schema-compat@1.3.18-alpha.0 ## @mastra/server@1.83.0-alpha.6 ## @mastra/elysia@0.1.23-alpha.6 ## @mastra/express@1.5.27-alpha.6 ## @mastra/fastify@1.5.27-alpha.6 ## @mastra/hono@1.7.25-alpha.6 ## @mastra/koa@1.7.27-alpha.6 ## @mastra/nestjs@0.3.0-alpha.6 ## @mastra/next@0.2.41-alpha.6 ## @mastra/server-adapters-test-suite@0.3.0-alpha.3 ## @mastra/tanstack-start@0.2.41-alpha.6 ## @mastra/upstash@1.6.1-alpha.0 ## @mastra/vectorize@1.1.5-alpha.0 ## @mastra/voice-google-gemini-live@0.14.19-alpha.0 ## @mastra/voice-openai-realtime@0.14.9-alpha.0 ## @mastra/voice-xai-realtime@0.2.18-alpha.0 ## @mastra/inngest@1.11.2-alpha.4 ## @mastra/temporal@0.6.0-alpha.6 ## create-mastra@1.39.0-alpha.6 ## @internal/playground@1.39.0-alpha.6 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
|
0af43a79d4 |
test(pg): run store test files in parallel on per-worker databases (#29093)
> **Stacked on #29092** (pg: build indexes on newly created tables
without CONCURRENTLY). The base branch is that PR's branch, so the diff
here is only this change. Merge that one first: without it, parallel
files cold-booting the same schema hit ~1 s waits per index.
## What this does
The pg store tests ran every file one after another (`fileParallelism:
false`) against a single database. This runs them on 4 workers, each
with its own database:
- `stores/pg/vitest.config.ts`: two projects. `e2e:stores/pg` runs files
in parallel; `e2e:stores/pg-sequential` runs `integration.test.ts`
afterwards on its own (`groupOrder` 1). Its delta polling reads a
cluster-wide snapshot horizon (`pg_snapshot_xmin`), so it can't share
the server with other writers.
- `vitest.global-setup.ts` creates one database per worker
(`mastra_test_w1..wN`) on both the pg and timescale containers, sized
from the resolved `maxWorkers`, so a CLI override of the worker count
can't leave a worker without a database. Teardown drops them.
- `vitest.setup.ts` points each worker at its database (`POSTGRES_DB`,
`DB_URL`, `POSTGRES_CONNECTION_STRING`, `PG_VNEXT_TIMESCALE_URL`) using
`VITEST_POOL_ID`.
- The bulk-status notification test gets a 20 s timeout. It went over 5
s under parallel load.
Test config only: no source changes, no changeset.
## Numbers
| | pg suite |
|---|---:|
| sequential, before the cold-boot fix | 701 s |
| sequential, with the cold-boot fix | ~323 s |
| this PR (4 workers) | 110–133 s |
Every run: 3905 passed / 20 skipped. CI pg job estimate: ~5.4 → ~1.9 min
(unverified until CI runs).
## Test plan
- [x] 3 full green runs at 4 workers (3905 / 20 each), plus one after
the maxWorkers change (133 s)
- [x] 6-worker subset green; no leftover databases after teardown
- [x] Root `vitest list`: 85 parallel + 1 sequential file, same as
before
- [ ] CI: pg store job green
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## ELI5
The PostgreSQL store tests now run in parallel. Each worker uses its own
database. One test file runs afterward by itself because it reads
cluster-wide state that other writers can affect.
## Changes
- Adds global setup in `stores/pg/vitest.global-setup.ts`. Before each
run, it creates one database per configured worker on the PostgreSQL and
Timescale servers. It force-drops any existing database with the same
name first. The returned teardown force-drops these databases. The
PostgreSQL server installs the `vector` extension. The Timescale server
installs no extensions.
- Adds `stores/pg/vitest.setup.ts`, which exports `workerDatabaseName`.
The name format is `mastra_test_w{worker}`. When `VITEST_POOL_ID` is
set, the setup sets `POSTGRES_DB` to the worker database. It also
updates `DB_URL`, `POSTGRES_CONNECTION_STRING`, and
`PG_VNEXT_TIMESCALE_URL` to use it. When the variable is unset, the
setup changes nothing.
- Splits `stores/pg/vitest.config.ts` into two projects:
- `e2e:stores/pg` runs in parallel with up to four workers. It excludes
performance, pooler, and cluster-wide integration tests.
- `e2e:stores/pg-sequential` runs the cluster-wide integration file with
file parallelism disabled. It uses group order 1, so it runs after the
parallel project at group order 0.
- Removes the previous `fileParallelism: false` setting and the coverage
reporter settings.
- Increases the bulk-status-update notification test timeout to 20
seconds in `stores/pg/src/storage/domains/notifications/index.test.ts`.
It also documents the load-dependent runtime of its 1,200 sequential
inserts.
- Adds `vitest.global-setup.ts` and `vitest.setup.ts` to the `include`
array in `stores/pg/tsconfig.json`.
## Test and CI results
- Author-reported, not verified in this review:
- Three full runs at four workers passed (3,905 passed, 20 skipped
each).
- A subset run at six workers passed, and no databases remained after
teardown.
- Suite time was 110–133 seconds with four workers.
- CI results for the pg store job: not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com>
|
||
|
|
22b24cc40a |
fix(core): apply semanticRecall.filter during agent runs (#29418)
## Description
`semanticRecall.filter` worked in `Memory.recall()` but not during
`agent.generate()` or `agent.stream()`. During an agent run, recall
happens in the `SemanticRecall` processor, which never read `filter` and
queried the vector store with the scope filter only. So messages from
threads that didn't match the filter ended up in the prompt.
This PR changes
`packages/core/src/processors/memory/semantic-recall.ts`:
```diff
query filter
- scopeFilter
+ filter ? { $and: [scopeFilter, filter] } : scopeFilter // same as Memory.recall()
embedding metadata written during agent runs
- { message_id, thread_id, resource_id, ... }
+ { ...threadMetadata (only when filter is set), message_id, thread_id, resource_id, ... }
```
- The filter applies to both query paths: client-side embedder and
self-embedding stores.
- Embeddings written during agent runs had no thread metadata, so even
with the query fixed, a filter like `{ mode: { $eq: 'work' } }` could
never match messages that agents saved. When a filter is set, the
processor now reads the thread from storage and stores its metadata on
the embedding, as `Memory.saveMessages()` already does. It reads from
storage because durable and recovered runs put only the thread id in the
memory context.
- Without a filter, nothing about writes changes. Some vector stores
reject nested metadata values and this processor only logs a failed
upsert, so the metadata is written only when a filter needs it.
- Docs: added `filter` to the processor reference and a note on the
semantic recall page that messages agents saved before a filter was set
don't match it until they're saved or indexed again.
## Related issue(s)
Closes #29352
## Type of change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [x] Test update
## Checklist
- [x] I have linked the related issue(s) in the description above
- [x] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR
## Evidence
Before = `semantic-recall.ts` from `origin/main`, with the same tests.
After = this branch.
- **Core, real Agent + MockMemory + in-memory vector store that applies
filters**
(`packages/core/src/processors/memory/semantic-recall-filter.test.ts`,
new): generate and stream, filter vs no filter, self-embedding store,
thread scope, durable-shaped context (`thread: { id }` only), thread
metadata written only with a filter.
**Before:** `Tests 7 failed | 2 passed (9)` (the two no-filter cases
pass on both).
**After:** `Tests 9 passed (9)`.
- **Memory integration, real `Memory` + LibSQL store and vector +
Agent**
(`packages/memory/integration-tests/src/semantic-recall-metadata-filter.test.ts`):
generate and stream; messages saved with `saveMessages()`, by earlier
agent runs, and in a thread the run created with metadata. Also checks
`Memory.recall()` agrees with the agent run.
**Before:** `Tests 6 failed | 2 passed (8)` (the two existing `recall()`
cases pass on both).
**After:** `Tests 8 passed (8)`.
- **The issue's own repro script** (LibSQL, mock embedder and model),
run against local builds:
**Before:** `agent prompt contains personal message: true`
**After:** `agent prompt contains personal message: false`
- Neighbouring core suites (`semantic-recall.test.ts`,
`processor-caching.test.ts`, `memory-config.test.ts`,
`embedding-dimension.test.ts`): 69/69 pass. `@mastra/core` typecheck
passes; oxlint, oxfmt and lint-staged are clean on the changed files.
Not tested: a real DurableAgent run end to end. The durable context
shape is covered by calling the Memory-built processor with `thread: {
id }` only, as `durable-agent.ts` and `finalize-run.ts` set it.
## Merge Danger
**Door:** two-way. A revert restores the old behavior. Embeddings
already written with thread metadata stay in the vector store but are
harmless extra fields.
**Blast radius:** agents with `semanticRecall.filter` configured. They
now recall fewer messages (only matching threads), which is the
documented behavior. Messages saved by older versions have no thread
metadata, so they won't match a filter until re-saved or re-indexed; the
changeset and docs say so. Users without a filter see no change in
queries.
**After merging main (#29826):** main's shared embedding helper now
always stores thread metadata on agent-run embeddings, so this PR no
longer adds its own filter-only metadata read; docs, changeset and the
unit test were updated to match.
**Cloudflare Vectorize (`@mastra/vectorize`):** the scope and user
filter are combined as `{ $and: [...] }` (the same shape
`Memory.recall()` already sends on main), which the Vectorize translator
rejected, so recall with a filter returned nothing there. The translator
now merges a filter that is exactly `{ $and: [...] }` into one object.
Several conditions on one field merge only as one lower plus one upper
bound; other same-field combinations and non-object clauses throw. Not
run against a live Vectorize index; checked against Cloudflare's
metadata-filtering docs and 38 translator tests (Sol reviewed 3 rounds).
## Council review
Rounds: 4
| Seat | Round 1 | Round 2 | Round 3 | Round 4 |
| --- | --- | --- | --- | --- |
| Claude Opus 5.5 | not run | not run | 8.5/10 | 9/10 |
| GPT-6.1 Sol | 9/10 (Tests 8, Code quality 8) | 9.6/10 | 9.5/10 |
9.5/10 |
| Grok 4.7 | empty seat | empty seat | empty seat | empty seat |
Created with Claude Opus 5.5 in T3 Code.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## ELI5
Agents now use the configured metadata filter when they search for
relevant messages. This keeps messages from unrelated threads out of the
agent’s prompt. Cloudflare Vectorize can also process the combined
filters used for these searches.
## Changes
- `SemanticRecall` combines the configured `VectorFilter` with the
thread or resource scope filter using `$and`.
- When a filter is configured, the processor reads thread metadata from
storage and writes it to message embeddings. Older embeddings may need
to be saved or indexed again.
- The Vectorize filter translator merges supported top-level `$and`
clauses, including repeated conditions with the same value. It rejects
unsupported combinations on the same field.
- Updated documentation and added tests for `generate()`, `stream()`,
self-embedding queries, and Vectorize filter translation.
## Verification
- Reported: 9/9 core filter tests, 8/8 integration tests, 69/69
neighboring core tests, 38 Vectorize translator tests, and the
`@mastra/core` typecheck passed.
- A real DurableAgent run was not tested end to end. A core test covers
a processor request context that contains only a thread ID.
- Live Cloudflare Vectorize testing: not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
01b2b958af |
fix(core): fail the run when a loop callback throws, instead of retrying it or leaving it running (#29850) | ||
|
|
b4152bd5da |
fix(upstash): negate every key of multi-key $not, $nor and $and/$or vector filters (#29760) | ||
|
|
cf2cd73b85 |
Revert "fix(core): snapshot agent stream events when they are publish… (#29947)
…ed (#29803)"
This reverts commit
|
||
|
|
779d2875b1 |
chore: version packages (alpha) (#29912)
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. ⚠️⚠️⚠️⚠️⚠️⚠️ `main` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `main`. ⚠️⚠️⚠️⚠️⚠️⚠️ # Releases > The changelog information of each package has been omitted from this message, as the content exceeds the size limit. ## @mastra/auth-studio@1.4.0-alpha.1 ## @mastra/factory@0.28.0-alpha.5 ## @mastra/mcp-apps-components@0.1.0-alpha.5 ## @mastra/playground-ui@69.0.0-alpha.6 ## @mastra/client-js@1.60.0-alpha.5 ## @mastra/react@1.15.0-alpha.5 ## @mastra/deployer-cloud@1.83.0-alpha.5 ## @mastra/longmemeval@1.1.42-alpha.5 ## @mastra/opencode@0.1.42-alpha.5 ## create-factory@0.4.3-alpha.5 ## @mastra/code-sdk@1.19.0-alpha.5 ## mastracode@0.50.0-alpha.5 ## @mastra/agent-builder@1.2.1-alpha.2 ## mastra@1.39.0-alpha.5 ## @mastra/core@1.83.0-alpha.5 ## @mastra/deployer@1.83.0-alpha.5 ## @mastra/editor@0.17.1-alpha.3 ## @mastra/mcp@2.9.0-alpha.2 ## @mastra/mcp-docs-server@1.3.13-alpha.8 ## @mastra/memory@1.41.1-alpha.2 ## @mastra/rag@2.7.3-alpha.1 ## @mastra/server@1.83.0-alpha.5 ## @mastra/elysia@0.1.23-alpha.5 ## @mastra/express@1.5.27-alpha.5 ## @mastra/fastify@1.5.27-alpha.5 ## @mastra/hono@1.7.25-alpha.5 ## @mastra/koa@1.7.27-alpha.5 ## @mastra/nestjs@0.3.0-alpha.5 ## @mastra/next@0.2.41-alpha.5 ## @mastra/server-adapters-test-suite@0.3.0-alpha.2 ## @mastra/tanstack-start@0.2.41-alpha.5 ## @mastra/libsql@1.30.0-alpha.2 ## @mastra/turso@0.1.21-alpha.2 ## @mastra/inngest@1.11.2-alpha.3 ## @mastra/temporal@0.6.0-alpha.5 ## create-mastra@1.39.0-alpha.5 ## @internal/playground@1.39.0-alpha.5 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
|
6563e7abfc |
fix(libsql): keep vector indexes writable after truncateIndex (#26802)
## What broke `LibSQLVector.truncateIndex()` ran `DELETE FROM <table>` with no `WHERE` clause. On a table with a `libsql_vector_idx` index, that statement empties the table but leaves the old rows in the vector index's shadow table. The next insert then fails: ``` SQLITE_ERROR: vector index(insert): failed to insert shadow row ``` So once you truncated an index, you couldn't write to it again. I reproduced this on both file-backed and `:memory:` databases with `@libsql/client` 0.18: after the bare delete the shadow table still has 1 row and the reinsert fails. The same cycle with `WHERE rowid IS NOT NULL` leaves 0 shadow rows and the reinsert succeeds. Upstream tracking: tursodatabase/libsql#2289. ## The fix `truncateIndex()` now runs `DELETE FROM <table> WHERE rowid IS NOT NULL`. It still deletes every row, and in the repro it also removes the shadow rows (0 left after the delete). The source change is one line in `stores/libsql/src/vector/index.ts`, and it runs for every connection type. ## Tests New tests in `stores/libsql/src/vector/index.test.ts`. They check behaviour only, using public methods: - file database: insert, truncate, insert again, then query and get only the new vector - file database: truncate an empty index, then run repeated truncate/insert cycles - file database: single-row and filter deletes still work, and the index still accepts inserts afterwards - `:memory:` database: insert, truncate, insert again, then query I ran these tests against the old `truncateIndex`. Both file truncate tests and the `:memory:` test fail with the shadow-row error. The partial-delete test passes there, because it guards existing behaviour. With the fix, the whole file passes: 146 passed, 1 skipped (`npx vitest run src/vector/index.test.ts`). oxlint and oxfmt are clean on the changed files. ## Not tested - Remote Turso/sqld URLs and embedded replicas (`syncUrl`). They run the same statement, but I didn't test them against a server. - Cost on large tables. I haven't measured the qualified delete's performance. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 Truncating a local LibSQL vector index could leave behind data that stopped new vectors from being added. This change removes the vector rows while keeping the index ready for new inserts. ## Changes - Updated `LibSQLVector` truncation to delete rows with `rowid IS NOT NULL`. - Added tests for file-backed and in-memory databases. They cover reinserting and querying vectors after truncation, repeated truncation, and partial deletes. - Added a patch changeset for `@mastra/libsql`. ## Validation Test and lint results: not available. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Jamie Lee Nichols <10267569+CodingButter@users.noreply.github.com> Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
7c4dd88400 |
chore: version packages (alpha) (#29889)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
|
80db9c3ba2 |
chore: run oxfmt instead of prettier in the last six pre-commit configs (#29892)
We moved from Prettier to oxfmt a while ago and CI checks formatting
with `oxfmt --check`, but six packages still ran `prettier --write` in
their pre-commit hook: the Discord, Teams and Telegram channels, the
Turso and OracleDB stores, and the DeepEval observability package. A
commit touching those could get reformatted by Prettier and then fail
the format check in CI.
They now use the same command as every other package.
```diff
- '*.{json,md,yml,yaml}': ['prettier --write'],
+ '*.{json,md,yml,yaml}': ['oxfmt --no-error-on-unmatched-pattern'],
```
`explorations/longmemeval` still uses Prettier in its hook. I left it
alone because `explorations/` is excluded from the oxfmt check.
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
|
||
|
|
832fb190f7 |
feat(observability): allow opting into remote storage export on Platform (#29737)
When `MASTRA_DEPLOYMENT_ID` is set, storage exporters are suppressed
even when a customer intentionally exports to both their own database
and Platform. Keep that default and add a per-config
`retainStorageExporterOnPlatform: true` opt-in for
`MastraStorageExporter`, including runtime registration.
```ts
new Observability({
configs: {
production: {
serviceName: 'my-service',
retainStorageExporterOnPlatform: true,
exporters: [new MastraStorageExporter(), new MastraPlatformExporter()],
},
},
});
```
Deprecated `DefaultExporter` remains suppressed on Platform even with
the opt-in. Its implementation is unchanged.
The opt-in does not permit known local observability storage.
MastraStorageExporter warns during initialization and disables
persistence when the resolved observability domain reports local
storage. This blocks InMemoryStore (including Mastra's automatic
fallback when no storage is configured), local libSQL files, in-memory
libSQL, embedded replicas, and DuckDB while allowing remote libSQL/Turso
and PostgreSQL. Other storage domains and exporters are unaffected, and
local development retains local storage support.
Adds the optional persistenceLocation capability to observability
storage, with in-memory, libSQL, and DuckDB implementations. These
adapters must be upgraded alongside observability for the guard to
apply; older/custom adapters without location metadata retain the
existing opt-in behavior. No provider names or private connection fields
are inspected.
Constructor skip warnings are deferred until the application logger is
installed and emitted once. Docs explain the local-storage restriction
before the example, include a supported/blocked provider table, and
distinguish the implementation-specific option from the core interface.
The restriction remains explicit even for users who cannot see the
warning. Includes one minor changeset per affected package:
observability, core, libSQL, and DuckDB. Preserves the default policy
introduced in #23555.
Validation: 451 targeted observability tests pass, including regression
coverage that reproduced unwanted in-memory persistence before the fix
for both explicit InMemoryStore and the automatic fallback. The tests
also verify initialization warnings and continued local-development and
remote-storage behavior. All 756 MDX files pass formatting using the
CI-matching dependencies; focused type-aware lint, Remark, and Vale
pass. The earlier 9 adapter tests also passed. No live Platform or
PostgreSQL endpoint was exercised. Full typechecking remains limited by
workflow-helper errors previously reproduced on the base and
installed-dependency type mismatches encountered while regenerating core
declarations.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## ELI5
On Mastra Platform, storage exporters are normally turned off to prevent
observability data from going to local storage. This change lets each
observability configuration opt in to remote storage export, while still
blocking storage known to be local.
## Changes
- Added `retainStorageExporterOnPlatform` to observability
configuration. It defaults to `false` and applies to exporters
registered during setup or later.
- When enabled on Platform, the option retains `MastraStorageExporter`
but does not enable the deprecated `DefaultExporter`.
- Added `persistenceLocation` to observability storage. The base
implementation returns `undefined`; in-memory storage and DuckDB report
`local`; libSQL reports `local`, `remote`, or `undefined` based on its
client configuration.
- On Platform, `MastraStorageExporter` skips storage that reports
`local` and warns during initialization. Adapters without location
metadata retain the opt-in behavior.
- Added documentation and changesets for observability, core, libSQL,
and DuckDB.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
6deb16047f |
chore: version packages (alpha) (#29719)
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. ⚠️⚠️⚠️⚠️⚠️⚠️ `main` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `main`. ⚠️⚠️⚠️⚠️⚠️⚠️ # Releases > The changelog information of each package has been omitted from this message, as the content exceeds the size limit. ## @mastra/slack@1.9.0-alpha.0 ## @mastra/factory@0.28.0-alpha.3 ## @mastra/code-sdk@1.19.0-alpha.3 ## mastracode@0.50.0-alpha.3 ## @mastra/observability@1.24.0-alpha.2 ## @mastra/core@1.83.0-alpha.3 ## @mastra/mcp-apps-components@0.1.0-alpha.3 ## @mastra/playground-ui@69.0.0-alpha.4 ## @mastra/server@1.83.0-alpha.3 ## @mastra/server-adapters-test-suite@0.3.0-alpha.1 ## @mastra/platform-workspace@1.13.0-alpha.0 ## @mastra/auth-better-auth@1.1.11-alpha.1 ## @mastra/client-js@1.60.0-alpha.3 ## @mastra/react@1.15.0-alpha.3 ## @mastra/deployer-cloud@1.83.0-alpha.3 ## @mastra/longmemeval@1.1.42-alpha.3 ## @mastra/opencode@0.1.42-alpha.3 ## create-factory@0.4.3-alpha.3 ## @mastra/arize@1.3.30-alpha.2 ## @mastra/arthur@0.4.30-alpha.2 ## @mastra/braintrust@1.3.27-alpha.2 ## @mastra/datadog@1.4.22-alpha.2 ## @mastra/deepeval@0.1.24-alpha.2 ## @mastra/laminar@1.3.32-alpha.2 ## @mastra/langfuse@1.6.6-alpha.2 ## @mastra/langsmith@1.4.3-alpha.2 ## @mastra/otel-bridge@1.5.22-alpha.2 ## @mastra/otel-exporter@1.5.2-alpha.2 ## @mastra/posthog@1.4.7-alpha.2 ## @mastra/sentry@1.2.32-alpha.2 ## @mastra/agent-builder@1.2.1-alpha.1 ## mastra@1.39.0-alpha.3 ## @mastra/connect@1.3.2-alpha.2 ## @mastra/deployer@1.83.0-alpha.3 ## @mastra/editor@0.17.1-alpha.2 ## @mastra/fastembed@1.3.7-alpha.0 ## @mastra/mcp-docs-server@1.3.13-alpha.6 ## @mastra/memory@1.41.1-alpha.1 ## @mastra/elysia@0.1.23-alpha.3 ## @mastra/express@1.5.27-alpha.3 ## @mastra/fastify@1.5.27-alpha.3 ## @mastra/hono@1.7.25-alpha.3 ## @mastra/koa@1.7.27-alpha.3 ## @mastra/nestjs@0.3.0-alpha.3 ## @mastra/next@0.2.41-alpha.3 ## @mastra/tanstack-start@0.2.41-alpha.3 ## @mastra/convex@1.8.1-alpha.0 ## @mastra/pg@1.36.0-alpha.2 ## @mastra/inngest@1.11.2-alpha.2 ## @mastra/temporal@0.6.0-alpha.3 ## create-mastra@1.39.0-alpha.3 ## @internal/playground@1.39.0-alpha.3 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
|
4854e36419 |
fix(core, server): stop a peer server from approving an aborted parked durable run (#29799)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9
## Problem
A durable agent run that is waiting for tool approval (with no thread)
can be aborted over HTTP. The abort returns `200 {"aborted":true}`, but
that was only recorded in memory on the server that handled the abort.
When the app runs on more than one server, approving the tool call on a
different server still resumed the run and **ran the tool**. When the
abort and approval happened on the same server, the approval was
rejected with a `500` rather than a clear error code. Fixes COR-1518.
## Fix
- **Abort saves the cancellation in storage before it is acknowledged**
(`packages/core/src/agent/durable/durable-agent.ts`):
- Stores that support concurrent updates: a guarded
`updateWorkflowState({ status: 'canceled', expectedStatus: 'suspended'
})`.
- Stores without `updateWorkflowState` (snapshot-only, e.g. LanceDB) and
stores without concurrent updates (e.g. Redis): reread the run, then
rewrite its snapshot as `canceled` with `persistWorkflowSnapshot(...,
expectedStatus: 'suspended')`, keeping `resourceId`.
- Storage errors propagate. The abort does not report success while the
run can still be resumed.
- If a resume has already claimed the run (status `running`), the
guarded write makes no change, and the existing abort of a running run
applies.
- **Resume on any server rejects runs canceled in storage** with
`AGENT_RESUME_ABORTED_RUN`, before it claims the run. The in-flight
resume marker is still taken synchronously before the first `await`
(#27365); the storage check runs right after it and releases the marker
when it rejects the run.
- **Observers of a canceled run are closed on every server**: the abort
that wins the `suspended → canceled` write publishes the abort event
right away on the shared pubsub, so `observe` streams on all servers end
and their history/listeners are cleaned up; the `canceled` row is kept.
If that publish fails, the abort still succeeds (the cancel is stored)
and a later refused approval/resume on that server retries the publish.
- **Docs and changeset**: `docs/.../agents/resumable-streams.mdx` and
`reference/agents/durable-agent.mdx` now point parked runs in
multi-server apps at `POST /agents/:agentId/runs/:runId/abort`, describe
the stored cancellation and the `409 Conflict`
(`AGENT_RESUME_ABORTED_RUN`) refusal, and qualify `abortRunStream` as
process-local. The changeset is scoped to
`DurableAgent`/`createDurableAgent` (evented agents inherit it; Inngest
agents are not covered) and states the best-effort limit on stores
without concurrent updates.
- **Approve route** (`packages/server/src/server/handlers/agents.ts`): a
`canceled` snapshot is treated as final instead of waiting 10s for a
suspension. The resource, thread and agent ownership checks still run
first, so other callers get `403` and learn nothing about the abort.
- **API contract** (`packages/server/src/server/handlers/error.ts`):
`AGENT_RESUME_ABORTED_RUN` now maps to **`409 Conflict`**, joining the
existing resume-claim conflict codes. It used to fall through to a
`500`.
## Proof
**Lifecycle regressions found in review and fixed in this PR** (final
head `1e0af6b5e8`):
1. *Healthy-lifecycle regression* (found by independent review at
`635c8837e3`): after a same-server abort and refused approval, an
attached observer stayed open and the run's history/listeners were
retained, because the `canceled` row skipped the old abort event. Fixed
in `
|
||
|
|
106c9fde0b |
fix(convex): return counter values from ConvexServerCache.get so resumed streams replay correctly (#29847)
Follow-up from PLTFRM-3190 (found during #29784). ## Problem The Convex `mastra/cache` handler's `get` op returned `null` for any document that wasn't a plain value. Counters are stored as `kind: 'counter'` (written by `increment` and the atomic `listPushIndexed`), so `ConvexServerCache.get(counterKey)` was always `null`. The in-memory and Redis caches return the number. `CachingPubSub` relies on that counter in two places, so on Convex: - **`readLogFrom`** (#29783) couldn't work out how many holes the log has, so every resume from an offset > 0 read the whole log instead of the tail. - **Restart detection in `subscribeFromOffset`** never fired. A subscriber resuming past the end of a log that then expired or was cleared silently dropped every frame of the new segment, because their indices sit below its cursor. ## Change - `stores/convex/src/server/cache.ts`: `get` returns `doc.counter` for counter documents. Lists still read as `null`. - Changeset: users need to redeploy their Convex functions (`npx convex deploy`). No Mastra server restart is needed, because the client is unchanged. ## Tests `stores/convex/src/server/cache.test.ts`. Each new test fails with the handler change reverted: - `get` on a counter key returns the value after `increment` and after `listPushIndexed`. - `CachingPubSub` over `ConvexServerCache`: resuming at offset 4 reads the log from position 4 (plus the single frame-0 id read), not from 0. - `CachingPubSub` over `ConvexServerCache`: a subscriber resumed at the log end gets the new segment's frame 0 after the log is deleted and the run publishes again. ## Test plan - [x] `vitest run` in `stores/convex`: 328 passed, 3 skipped - [x] `tsc --noEmit`: no new errors (10 errors already on main in `schema.test.ts` and `observational-memory.test.ts`) - [x] oxlint clean on changed files - [x] prettier clean on changed hunks (the `mastraCache` export warning is already on main) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 Convex could save counter values but could not read them back. This fix lets resumed streams use those counters to find the right place in the log and detect when a log has restarted. ## Changes - `ConvexServerCache.get()` now returns counter values as numbers. Missing counters and lists still return `null`. - Added tests for counter reads, replaying only the log tail after an offset, and receiving events after a log restarts at index 0. - Added a patch changeset for `@mastra/convex`. It advises redeploying Convex functions with `npx convex deploy`; a Mastra server restart is not required. ## Checks - Test, type-check, lint, and formatting results: not available. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com> Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
28ddc161a4 |
fix(core): snapshot agent stream events when they are published (#29803)
> Opened by @user_01KNA8J74D4RF84RXSDRX26BS9
## Summary
Fixes COR-1517. `Agent.observe()` replays could show a later state of a
run than the one the caller saw live. After a tool approval and resume,
the replayed first `step-finish` listed 2 steps when the live one listed
1.
**Cause:** the agent stream publisher published the loop's own objects
by reference and queued the cache append. The loop keeps mutating those
objects (for example, a resumed run pushes into the same
`accumulatedSteps` array). The in-memory cache stored those references,
and a queued Redis append could serialize them after they had changed.
**Fix (`packages/core/src/agent/streams/publisher.ts` is the only
production file):** `publish()` copies every payload synchronously with
`decode(encode(value))` from `../../events/codec` before it calls
`enqueue()`. This covers step-start, chunks (including
tool-call-approval and suspended), the narrowed finish DTO `{ output,
stepResult, messages }`, and the captured suspended data. The
`producedAt` stamp is copied onto the snapshot. Each chunk is still
published once. The direct caller stream, callbacks, and processors are
unchanged.
## ⚠️ Behavior change for same-process observers
The snapshot uses the codec that cross-process pubsub already uses. It
is **not a lossless clone**. Same-process subscribers and in-memory
replays now get codec-normalized copies:
- **Kept:** `Date`, `Error`, `Map`, `Set`, `BigInt`, explicit
`undefined`, and registered classes (`DefaultStepResult`,
`DefaultGeneratedFile`, `DefaultGeneratedFileWithType`, including
generated-file bytes and media type).
- **Changed:** functions, symbols, and prototypes are dropped.
`NaN`/`Infinity` and cycles become `null`. Arbitrary `Uint8Array`s and
other unregistered class instances become plain data.
- **Fallback:** if `encode` throws (for example, a `toJSON()` that
throws), one warning is logged and *that* event is published by
reference. That event has no immutability guarantee. Other events and
the caller's stream are unaffected.
- Redis still applies its own `JSON.stringify` on top of the snapshot.
## Proof
- **Public-API repro (AIMock)**,
`observe-replay-snapshot.scenario.test.ts`: approval then resume, live
step counts `[1,2]`. On unchanged main, replay is `[2,2]` on the normal
and fs engines, for both the immediate cache append and the held
(delayed) append. With the fix, replay equals live.
- **Real Redis (author only)**,
`stores/redis/src/agent-stream-replay.integration.test.ts` against
redis:8 in Docker:
- Against core dist built from main, both the immediate and held-append
variants fail with `[2,2]` vs `[1,2]`. Real Redis is affected too.
- With the fix, both pass (3/3 runs earlier, and again 2/2 at this
head).
- **Contract test**, `publisher-snapshot.test.ts`, run over a real
`CachingPubSub` with a held append:
- Mutating a payload after publish does not change history.
- The caller's chunk is untouched, and `producedAt` is kept on the copy.
- The value policy above, including `DefaultStepResult` on finish.
- The fallback: a throwing `toJSON` publishes by reference with one
warning.
- **Revert proof:** with main's `publisher.ts`, all 8 new core tests
fail. With the fix, all 8 pass.
- **Author checks at this head:**
- Core streams, events, contract and scenario tests: 29/29 files, 392
tests.
- `tsc`, `oxlint`, and `prettier` are clean.
- The branch merges cleanly into current main.
## Self-review
Approved at `
|
||
|
|
5e63a3b59d |
fix(pg): build indexes on newly created tables without CONCURRENTLY (#29092)
## Problem When `@mastra/pg` initializes an empty schema, it creates its tables and then ~23 indexes with `CREATE INDEX CONCURRENTLY`. If two or more processes initialize the same schema at once (several replicas starting together, or parallel test files), each concurrent index build waits for the other process's open transaction. Postgres only clears that wait after `deadlock_timeout` (1 s by default). That comes to ~1 s per index: | processes cold-booting together | before | after | |---|---:|---:| | 2 | 20.6 s | 0.21 s | | 4 | 29.6 s | 0.24 s | This was also ~70% of the pg store CI job. ## Fix If a table did not exist when this process began initializing, its indexes are built with plain `CREATE INDEX`. Normally the table is brand new, so the build is instant and the short lock it takes blocks nobody (see the caveat below for when another process got there first). Tables that already existed still get `CREATE INDEX CONCURRENTLY`, so existing production tables are never locked for an index build. - `schema-snapshot.ts` records which tables this init created (absent from the schema snapshot it reads at start) - `createTable` adds to that set; `createIndex` and the workflow expression indexes pick plain or `CONCURRENTLY` from it ## Caveat "Created by this init" means the table was absent when this process read its snapshot. A slow replica could therefore plain-index a table that another replica created moments earlier. That table may already be taking writes from the other replica, and the plain build holds a SHARE lock that blocks those writes for as long as the build takes (milliseconds on a near-empty table). We think that's acceptable next to ~20 s of boot time, but the pg store owner should confirm. ## Changeset `@mastra/pg` patch: `.changeset/pg-cold-boot-plain-indexes.md`. ## Test plan - [x] New `concurrent-init.test.ts` case (`it.each` plain | concurrent, both orders) converges to the same indexes - [x] Full pg suite: 3905 passed / 20 skipped / 0 failed (86 files), 701 s → 323 s - [x] PgBouncer transaction-pooling suite: 3 tests pass - [x] `schema-snapshot.stress.test.ts` 355 s → 4.5 s - [x] tsc and oxlint clean - [ ] CI: pg store job green <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 When the database starts, it can build indexes faster if a table is brand new. This change uses a faster, briefly write-blocking method for those tables, while keeping the less-blocking method for tables that already existed. ## Changes - Track tables created after the initialization schema snapshot was read. - Use ordinary `CREATE INDEX` for those tables. Keep `CREATE INDEX CONCURRENTLY` for tables already present in the snapshot. - Update workflow index creation to use the same rule. - Add an integration test for both index-build orders, checking that both builds finish and the index is valid. ## Reported validation - The author reports that the full PostgreSQL suite passed 3,905 tests, skipped 20, and had no failures across 86 files. - The author reports that the PgBouncer transaction-pooling suite passed 3 tests, and that TypeScript and oxlint checks were clean. - The author reports that `schema-snapshot.stress.test.ts` runtime decreased from 355 seconds to 4.5 seconds. - The PostgreSQL store CI job was still pending. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com> |
||
|
|
1f3035dbf5 |
fix(convex): stop resumed streams skipping events after a failed cache write (#29784)
## Problem `CachingPubSub` stores each durable stream event in a cached log and gives it an index. A client that reconnects reads the log back from an offset, and that read assumes each event's index matches its position in the list. `ConvexServerCache` didn't override `listPushIndexed`, so it used the base default: one Convex mutation to increment the counter, then a second to append the entry. If the append failed after the increment had committed (a timeout or a network error), the counter ended up one ahead of the list. From then on every event sat one position below its index, so a client resuming at offset `N` read event `N+1` first and silently missed event `N`. The default also ignored the longer TTL requested for durable topics. Redis, Valkey and the in-memory cache already do this atomically (#27352). Convex was the remaining gap. Fixes PLTFRM-3190. ## Change - New `listPushIndexed` op in the Convex `mastra/cache` handler. One mutation reads and increments the counter, appends `{ ...value, index }` and gives the list and the counter the same `expiresAt`. Convex mutations are transactional, so a failure rolls all of it back. - `ConvexServerCache.listPushIndexed` sends that op through `callUntilSettled`. A requested `ttlMs` can extend the default TTL but not shorten it, and a cache with expiry disabled stays non-expiring. Redis and the in-memory cache use the same rule. - The handler runs in the user's own Convex deployment. If it hasn't been redeployed yet and answers `Unsupported operation listPushIndexed`, the cache logs one warning and falls back to the previous increment + push path. The core replay hardening (PLTFRM-3183) is being done separately. ## Test plan - [x] Handler tests: indices are assigned in order, a stale `index` on the value is replaced, the list and counter share one expiry, an expired log restarts at index 0, and type conflicts are rejected. - [x] End-to-end tests: real `CachingPubSub` → `ConvexServerCache` → the real handler, through a fake client that rolls back a mutation that throws, as Convex does. - A publish whose cache write fails mid-mutation: resuming from an offset delivers every cached frame, and indices stay contiguous. - A publish that commits but whose response is lost: the frame is still replayed at its index. - Both tests fail when the override is disabled (old non-atomic path) and pass with it. - [x] Client tests: request shape, `ttlMs` extension, `ttlMs: 0` gives `expiresAt: null`, fallback when the deployed handler lacks the op, other errors still propagate. - [x] `@mastra/convex` suite passes: 323 passed, 3 skipped. `oxlint` is clean. `tsc` shows no new errors (the 10 existing errors in `schema.test.ts` and `observational-memory.test.ts` are on main too). <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 The cache now saves a stream event and its position together. This helps resumed streams avoid skipping events when a cache write fails. Older Convex handlers use the previous method until they are redeployed and the server restarts. ## Changes - Added the `listPushIndexed` cache operation. The Convex handler assigns an index using the larger of the list length and counter value, then updates both records in one mutation. - Made the list and counter share an expiry deadline. The requested TTL can extend, but not shorten, the default TTL. If the default TTL is disabled, entries do not expire. - Updated `ConvexServerCache` to use the new operation. If the handler returns `Unsupported operation listPushIndexed`, the cache warns and uses increment plus list push for subsequent calls. This fallback is not atomic and ignores requested retention. - Added tests for indexed pushes, expiry, type conflicts, counter/list drift, fallback behavior, transaction rollback, and lost responses. - Added a changeset instructing users to redeploy Convex functions and restart Mastra. **CI and test results:** not available. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com> Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
6bc7614a8f |
perf(factory): index run binding lookups by thread (#29594)
**─────── TLDR ───────** > Finding a Factory run binding by thread now uses an index instead of the status index or a full scan. Postgres index DDL with an operator class and a sort order is now valid. In the Factory, `findActiveRunBindingByThread` and `findRunBindingByThread` filter on org, thread, resource and session. The only index with those columns starts with `factory_project_id`, so they fell back to the status index or a full scan. ### Behavior changed a run binding looked up by thread, on Postgres or LibSQL ├─ **was** — status index plus a filter, or a full scan for any status └─ **now** — `factory_run_bindings_org_thread_idx` on org, thread, resource, session a Postgres index with an operator class and `DESC` or `ASC` ├─ **was** — `createIndex` put the operator class after `DESC`, which Postgres rejects; `exportSchemas` dropped the operator class entirely └─ **now** — both put it right after the column, ahead of `DESC` or `ASC` ### Decisions - **no default GIN index on thread metadata** — only Factory and mastracode features filter threads by metadata, so every other Postgres user would pay the write cost, the disk and a slower first init for nothing. Deployments that need it can add it themselves - **kept `factory_run_bindings_session_idx`** — `findActiveRunBinding` and `findRunBindingBySession` still filter on `factory_project_id` ### Tradeoff Local Postgres 16, `EXPLAIN ANALYZE` on 100k seeded bindings: > active binding by thread: was `0.7–1.5 ms`, now `0.06 ms` > any-status binding by thread: was `5.8 ms` (full scan), now `0.006 ms` The index is 6.6 MB per 100k rows. It's built without `CONCURRENTLY`, like every other Factory index, so a big `factory_run_bindings` blocks writes while it builds (144 ms per 100k rows here). ### Not verified - [ ] real deployment row counts. The numbers above come from seeded local tables --- > ██████████████████ **tests** `+188` (87% of the additions) > ██ **source** `+19 −27` > █ **changeset** `+10` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 This PR helps the database find run records faster when it searches by thread details. It also fixes PostgreSQL index statements that combine sort order with an operator class. ## Changes - Added a `factory_run_bindings` index on `(org_id, thread_id, resource_id, session_id)`. The existing project-scoped session index remains. - Added LibSQL and optional PostgreSQL tests for active and historical thread lookups and their index usage. - Updated PostgreSQL index SQL generation to place the operator class before sort direction. Both generated schema SQL and runtime index creation use the shared formatter. - Added PostgreSQL integration tests for a descending column with `text_pattern_ops`, including index creation during store initialization. - Added patch changesets for `@mastra/factory` and `@mastra/pg`. ## Checks Test and CI results: not available. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Damien Schneider <damienschneider@damiens-MacBook-Pro.local> Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
7b912d293d |
feat(observability): add byte-based exporter flush thresholds (#29621)
Count-based batches can retain large payloads and start many concurrent
exports during bursts. This adds opt-in byte-triggered batching to
`MastraPlatformExporter` and `MastraStorageExporter`, with an 8 MiB
default flush threshold for both.
```ts
new MastraPlatformExporter({ byteBatching: {} })
new MastraStorageExporter({ byteBatching: {} })
```
The byte threshold triggers sending; it never truncates or discards
records to fit a batch. Larger records are exported intact, and queued
data is retained when exports fall behind. Platform defaults to two
concurrent uploads and storage to one write. Existing count mode remains
the default, and existing auth, quota, and retry-exhaustion policies
still apply. CloudExporter is unchanged.
Storage ignores byte batching when the resolved strategy is `realtime`,
preserving its immediate, awaited writes. Other update-based strategies
serialize writes and alternate creates with eligible updates so
unrelated create traffic cannot starve span ends. Updates waiting for
their own create stay queued without consuming retries. Events arriving
during shutdown are accepted until the drain finishes. Failed storage
writes back off the shared connection across signals.
Includes queue diagnostics, docs, a changeset, real HTTP collector
tests, and real ClickHouse tests. The ClickHouse suite uses a declared
`@mastra/observability` dev dependency and its built public export,
defaults to the package's localhost test server, and no longer skips
without `CLICKHOUSE_URL`. CI's affected-test selector includes this
suite for storage-exporter changes, and the ClickHouse build plan
includes observability. Queues are not memory-capped or durable.
Validation: 302 focused exporter/shared-buffer tests passed. The 42-test
ClickHouse suite passed; its shutdown coverage was then strengthened to
persist actual span outputs and passed 20 repetitions. The starvation
and realtime tests failed before their fixes, and the shutdown-admission
regression failed in all 20 pre-fix runs. Package build, exporter source
type-checking, lint, formatting, and focused docs checks pass locally.
Local full-package type-checking reports four errors in
`src/test-utils/workflow-control-flow-suite.ts`. The identical four
diagnostics reproduce at the original main base commit `
|
||
|
|
f21e0742ce | chore: version packages | ||
|
|
23308f9c19 |
feat(observability): add single score and feedback reads (#29529)
## Description
Add `GET /observability/feedback/:feedbackId` and typed client getters
for individual scores and feedback. Missing records return HTTP 200 with
`{ score: null }` or `{ feedback: null }`, preserving the existing score
endpoint contract.
```ts
const { score } = await client.getScore(scoreId);
const { feedback } = await client.getFeedback(feedbackId);
```
Client getters encode path IDs and preserve configured API prefixes,
credentials, and headers. Feedback reads use direct lookups in
in-memory, PostgreSQL, ClickHouse, and DuckDB storage. Includes
generated route types/CLI metadata, public API documentation, and
changesets.
Validation: package builds passed; 30 single-record HTTP cases passed
across five server adapters; 6 client transport tests, 341 server tests,
and 9 focused database cases passed across PostgreSQL/TimescaleDB,
ClickHouse, and DuckDB. Removing the feedback route caused the new
feedback HTTP cases to fail. Permission generation/checks, scoped
implementation lint, MDX formatting, Remark, and Vale passed.
Broader validation limitations: existing POST cases fail with
body-parser dependency errors in Express, Koa, and NestJS; the new GET
cases pass in all five adapters. The shared storage test helper also has
existing import-order lint errors outside the changed lines.
## Related issue(s)
Fixes
[OBS-790](https://linear.app/kepler-crm/issue/OBS-790/add-oss-single-score-and-feedback-reads),
under [OBS-792](https://linear.app/kepler-crm/issue/OBS-792). Platform
single-record endpoints remain scoped to OBS-791.
## Type of change
- [x] New feature (non-breaking change that adds functionality)
- [x] Documentation update
- [x] Test update
## Checklist
- [x] I have linked the related issue(s) in the description above
- [x] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## ELI5
Applications can now look up a score or feedback record by its ID. If no
feedback record exists, the server returns `null`; it can also add
available author details to feedback.
## Summary
- Added `GET /observability/feedback/:feedbackId`. The route uses
existing observability authorization and returns `{ feedback: record }`
or `{ feedback: null }`.
- The feedback route adds author details when available. It returns HTTP
501 when the storage provider does not support single-feedback reads.
- Added `getScore(id)` and `getFeedback(id)` to the JavaScript client.
These methods encode IDs in the URL and use the client’s configured
request options.
- Added direct feedback lookups to in-memory, PostgreSQL, ClickHouse,
and DuckDB storage.
- Added response types, generated route metadata, API documentation,
tests, and changesets.
## Validation
Test and CI results: not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
f79db0de42 |
fix(duckdb): read payloads only for rows without previews (#29548)
## Description DuckDB span and thread lists no longer read stored input and output when previews are stored. This finishes the work that #29523 started. That PR gave these lookups a time range, but a page whose rows are far apart still read the payload columns across most of the file. - `querySpans` and `queryThreadRows` now read only the stored previews in their lookup. DuckDB reads every column a query names, even inside a `CASE` that discards it. - A further query reads `input`, `output`, and `error` only for rows with a NULL preview, using the time range of those rows. Results do not change. - For spans, that query returns the whole row, so `model`, `provider`, and the previews always come from one row. - `error` is still read where `status` needs it: the first `querySpans` query and the `queryThreadRows` lookup. - Pages with rows written before previews existed now run one more query. On a warm cache, this adds a few milliseconds. On a 5.7 GB test file with every preview stored, a filtered `querySpans` page of 25 rows far apart read 3006 MB from disk on `main` and 27 MB with this change. ## Related issue(s) Fixes OBS-645 Follow-up to #29523 ## Type of change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Code refactoring - [x] Performance improvement - [ ] Test update ## Checklist - [x] I have linked the related issue(s) in the description above - [ ] I have made corresponding changes to the documentation (if applicable) - [x] I have added tests that prove my fix is effective or that my feature works - [ ] I have addressed all Coderabbit comments on this PR <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 DuckDB was reading large input and output data even when a shorter preview was already available. This change reads the preview first and fetches the full data only for rows that need it, which can reduce work on large databases. ## Changes - Updated `querySpans` and `queryThreadRows` to fetch row metadata and previews first. They issue a separate, bounded query for full payloads only when a selected row lacks a required preview. - Kept `error` in the initial thread lookup for status calculation. The follow-up query fetches `error` only when its preview needs fallback. - Replaced text-based query-plan inspection in DuckDB preview tests with recursive parsing of JSON plans. Added checks that span and thread queries avoid scanning input and output payloads when previews are stored. - Removed the unused `missingPreviewSql` helper and added a patch changeset for `@mastra/duckdb`. **Validation:** The diff adds query-plan tests. Results from completed tests, builds, or CI checks are not available. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
08b25ea2df |
fix(lance): escape string keys in load() (#29404) | ||
|
|
defb82048f |
perf(pg): skip the count query when listWorkflowRuns returns a short page (#29553)
**─────── TLDR ───────** > `listWorkflowRuns` on pg answers a page shorter than requested with one query instead of two, and `total` stays the same. A short page already tells the total: the offset plus the rows it returned. Listing an agent's suspended runs, on each tool approval and stream resume, almost always returns a short page. ### Behavior changed a page with fewer runs than requested ├─ **was** — a page query plus a count query └─ **now** — the page query alone; `total` is offset plus rows a full page, or an empty page past the end └─ **unchanged** — the count query still runs ### Gain One round trip per short page: `1.13 → 0.5–0.7 ms` on local pg. --- > ██████████████████ **tests** `+90` > ███ **source** `+14 −11` > █ **changeset** `+5` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 When a request returns fewer workflow runs than the page can hold, the database can tell that there are no more runs to fetch. This change avoids an extra count query in that case, while keeping the correct total for full pages and empty pages beyond the end. ## Changes - `WorkflowsPG.listWorkflowRuns` now fetches the page before deciding whether to count matching runs. - For a non-empty partial page, it sets `total` to the offset plus the number of returned runs and skips the count query. - For a full page or an empty page with a positive offset, it runs the count query to determine the total. - Added tests for partial pages, full pages, and empty pages beyond the last run. - Added a patch changeset for `@mastra/pg`. ## Validation - Test and CI results: not available. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
83b6997a2a |
fix(duckdb): bound payload lookups by time range (#29523)
## Description DuckDB trace, span, and thread lists now stay fast on large databases. Each read finds its page's payload rows with an OR of key tuples. DuckDB cannot push that into the table scan, so the lookup read every row group of `span_events`. - Adds a `timestamp` or `endedAt` range to the lookups in `fillMissingTracePreviews`, `fillMissingReconstructedPreviews`, `querySpans`, and `queryThreadRows`. Each range covers every looked-up row, so results do not change. - `queryThreadRows` also adds an `endedAt IN` list. A chat's first and last traces can be months apart, so the range alone could cover the whole table. - `querySpans` and `queryThreadRows` run this lookup on every page, so they were slow on new data too, not only on rows without stored previews. - Uses time ranges rather than the `cursorId` range the issue proposes. Rows written before `cursorId` existed hold NULL there, so their row groups could not be pruned. - Adds a test that counts the rows scanned on a real DuckDB table with several row groups. On a 9.5 GB, 1.2M-row test file with a 100 MB/s disk and an empty cache, a `queryTraces` page of 25 rows without stored previews went from 75.6 s to 0.8 s. Pages whose rows span a long time gain less, because more row groups hold them. On a 3 GB file on the same disk, a thread page with an old chat went from 25.7 s to 11 s. ## Related issue(s) Fixes #29411 ## Type of change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Code refactoring - [x] Performance improvement - [ ] Test update ## Checklist - [x] I have linked the related issue(s) in the description above - [ ] I have made corresponding changes to the documentation (if applicable) - [x] I have added tests that prove my fix is effective or that my feature works - [ ] I have addressed all Coderabbit comments on this PR <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 When a page needs to fetch missing trace or span details, the database could read far more rows than needed. This change limits those lookups to the page’s time range, so large databases can avoid reading unrelated data. ## Changes - Added `timeRangeOf` to calculate inclusive timestamp bounds. - Added timestamp bounds to missing-preview lookups in `fillMissingTracePreviews`, `fillMissingReconstructedPreviews`, and `querySpans`. - Added timestamp bounds and an `endedAt IN` filter to `queryThreadRows`, to narrow lookups when a thread’s first and last traces are far apart. - Added a large-table test that checks DuckDB’s analyzed query plan scans fewer than two-thirds of the filler rows for four read paths. - Added a patch changeset for `@mastra/duckdb`. The test was added; execution results are not available. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
675caf0613 |
chore: version packages (alpha) (#29336)
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. ⚠️⚠️⚠️⚠️⚠️⚠️ `main` is currently in **pre mode** so this branch has prereleases rather than normal releases. If you want to exit prereleases, run `changeset pre exit` on `main`. ⚠️⚠️⚠️⚠️⚠️⚠️ # Releases > The changelog information of each package has been omitted from this message, as the content exceeds the size limit. ## @mastra/client-js@1.60.0-alpha.0 ## @mastra/react@1.15.0-alpha.0 ## @mastra/factory@0.28.0-alpha.0 ## @mastra/code-sdk@1.19.0-alpha.0 ## mastra@1.39.0-alpha.0 ## @mastra/core@1.83.0-alpha.0 ## @mastra/mcp@2.9.0-alpha.0 ## @mastra/mcp-apps-components@0.1.0-alpha.0 ## @mastra/playground-ui@69.0.0-alpha.0 ## @mastra/server@1.83.0-alpha.0 ## @mastra/nestjs@0.3.0-alpha.0 ## @mastra/clickhouse@1.30.0-alpha.0 ## @mastra/duckdb@1.19.0-alpha.0 ## @mastra/libsql@1.30.0-alpha.0 ## @mastra/pg@1.36.0-alpha.0 ## @mastra/temporal@0.6.0-alpha.0 ## @mastra/auth-better-auth@1.1.11-alpha.0 ## @mastra/deployer-cloud@1.83.0-alpha.0 ## @mastra/deployer-cloudflare@1.2.42-alpha.0 ## @mastra/deployer-netlify@1.2.42-alpha.0 ## @mastra/deployer-sandbox@0.3.27-alpha.0 ## @mastra/deployer-vercel@1.2.42-alpha.0 ## @mastra/longmemeval@1.1.42-alpha.0 ## @mastra/opencode@0.1.42-alpha.0 ## create-factory@0.4.3-alpha.0 ## mastracode@0.49.1-alpha.0 ## @mastra/arize@1.3.30-alpha.0 ## @mastra/arthur@0.4.30-alpha.0 ## @mastra/braintrust@1.3.27-alpha.0 ## @mastra/datadog@1.4.22-alpha.0 ## @mastra/deepeval@0.1.24-alpha.0 ## @mastra/laminar@1.3.32-alpha.0 ## @mastra/langfuse@1.6.6-alpha.0 ## @mastra/langsmith@1.4.3-alpha.0 ## @mastra/observability@1.23.2-alpha.0 ## @mastra/otel-bridge@1.5.22-alpha.0 ## @mastra/otel-exporter@1.5.2-alpha.0 ## @mastra/posthog@1.4.7-alpha.0 ## @mastra/sentry@1.2.32-alpha.0 ## @mastra/agent-builder@1.2.1-alpha.0 ## @mastra/connect@1.3.2-alpha.0 ## create-mastra@1.39.0-alpha.0 ## @mastra/deployer@1.83.0-alpha.0 ## @mastra/editor@0.17.1-alpha.0 ## @mastra/mcp-docs-server@1.3.13-alpha.0 ## @mastra/memory@1.41.1-alpha.0 ## @mastra/rag@2.7.3-alpha.0 ## @mastra/redis-streams@0.8.1-alpha.0 ## @mastra/elysia@0.1.23-alpha.0 ## @mastra/express@1.5.27-alpha.0 ## @mastra/fastify@1.5.27-alpha.0 ## @mastra/hono@1.7.25-alpha.0 ## @mastra/koa@1.7.27-alpha.0 ## @mastra/next@0.2.41-alpha.0 ## @mastra/tanstack-start@0.2.41-alpha.0 ## @mastra/spanner@1.11.2-alpha.0 ## @mastra/turso@0.1.21-alpha.0 ## @mastra/inngest@1.11.2-alpha.0 ## @mastra/gcs@0.3.6-alpha.0 ## @internal/playground@1.39.0-alpha.0 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
|
4afeceaf2d |
feat(pg, libsql): add experimental storage v2 for workflow runs (Postgres and libSQL) (#28299)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com> Co-authored-by: review <review@local> |
||
|
|
8ad7773cd4 |
feat(observability): align signal writes and context modes (#29289) |