mirror of
https://github.com/mastra-ai/mastra.git
synced 2026-11-17 08:44:46 +00:00
512 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
cd2a326988 |
fix(libsql): stop background task writes stalling shutdown, and fix backend timing flakes (#30504)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
adc0a70bf8 |
fix(memory, couchbase): check semantic recall scope in Memory and warn on ignored Couchbase filters (#30446) | ||
|
|
d1d736db29 |
test(memory): pin attachment token counts in the #29851 image test (#30415) | ||
|
|
c8a35ac624 |
fix(memory): keep Observational Memory local when the gateway has no memory API (#30414)
> Opened by @abhiaiyer91 > Opened by @factory-project:7d5175c3-3a20-42ac-aba9-756f183136f4 ## Summary Keep Observational Memory local for agents using `mastra/<provider>/<model>` with gateway organization keys (`msk-gateway-*` and `mskv2_*`), because those keys do not provide gateway-managed memory. Gateway project keys (`msk_*`) continue to use gateway-managed memory. Finish an open local turn before switching to gateway-managed memory. Credential ownership checks report local memory only for models that are not on the Mastra gateway. A credential the gateway cannot resolve — including one it reports as missing — is thrown instead of read as local ownership: guessing local ownership can cause both the gateway and local store to persist the same thread, because the model call resolves the same credential again. The Observational Memory processor also propagates ownership-check errors. Older models that cannot report gateway memory ownership retain their previous behavior. Docs explain the gateway-key distinction and the paired `@mastra/core` and `@mastra/memory` upgrade. Project keys routed BYOK or with gateway memory disabled remain outside this change's scope. ## Verification (head ` |
||
|
|
1072f7c626 |
fix(memory): reuse message embeddings across saves (#30429)
## Summary Saving messages with semantic recall embeds each message's text and writes a vector row for it. Since the message write path moved onto the shared row builder, Memory stopped reusing embeddings for text it had already embedded. The same text saved again (a repeated reply, a re-saved message, a cloned thread) went back to the embedder every time: three saves of the same 50 texts made 150 embedding calls instead of 50. This restores bounded reuse on the write path without changing how rows are written: - Vectors are cached per Memory instance in a bounded LRU, keyed by each message's exact prepared chunks, so chunk-size limits still apply and a different chunking is a miss. - The cache is emptied when the embedder or its options change. - Failed calls, and responses without exactly one embedding per chunk, are never kept. Concurrent writes of the same chunks share one call. - Reused vectors are not counted again in reported embedding usage. - Only vectors are cached. Every message still gets its own row with its own id and metadata, so there are no duplicate recall rows. The memory performance benchmark previously timed recalls without checking that anything had been indexed, so a writer that skipped embedding would pass it. It now checks that every message has a vector row and that each thread's semantic recall returns its own messages before timing. ## Test plan - New `semantic-recall-embedding-reuse.test.ts` (10 tests): repeated text, re-saved ids, usage on reuse, long-text reuse, invalid responses and changed embedder options fail before this change. Edited text, failed embeds, embedder swaps and raw-string content are covered as controls. - `@mastra/memory` unit suite: 2,689 pass + 1 todo. Typecheck, lint and format clean. - Real file-backed LibSQL, 3 saves × 50 messages: repeated text 150 → 50 embedding calls, re-saved ids 150 → 50. Stored rows unchanged (151 / 51 / 451 for long chunked text). Unique, raw-string and long-text call counts unchanged. - LibSQL performance benchmark (2,500 messages, FastEmbed) passes with the new check. The check fails when vector writes are disabled or recall returns nothing. - Local FastEmbed: repeated text ~7.6s → ~2.6s; unique text unchanged. Ref #29826 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ELI5: When a message is saved, Memory turns its text into numbers (an embedding). If the same text was already turned into numbers, this change reuses them instead of asking the embedding service again. Each message still gets its own stored row. ## Overview This change adds a bounded cache of message-chunk embeddings to `Memory`. The cache applies only to the message-vector write path. The change also adds checks to the LibSQL performance test, a patch changeset for `@mastra/memory`, and a new test file. Diff scope: 4 files changed, 350 lines added, 11 removed. ## Changes **`packages/memory/src/index.ts`** (+55 / -11) - Adds `chunkEmbeddingCache`, an LRU cache bounded by `DEFAULT_EMBEDDING_CACHE_MAX_SIZE`. It holds promises of vectors only. Keys are a hash of `JSON.stringify(chunks)`. - Adds `chunkEmbeddingCacheOwner`. It records the embedder and options that produced the cached vectors. - Adds `embedMessageChunks(chunks)`, which: - Snapshots `this.embedder` and `this.embedderOptions` before hashing. - Calls `embedValues` with the snapshot and skips the cache if the embedder or options change during hashing. - Clears the cache when the owner record differs from the snapshot. The clear happens at the next miss, not when the embedder changes. - Returns cached vectors on a hit. A hit returns no usage data. - On a miss, calls `embedValues` and stores the pending promise. Concurrent writes with the same chunks share that promise. - Evicts the entry when the call fails or returns a number of embeddings that differs from the number of chunks. - Returns usage only for the call that reached the embedder. - Changes `embedValues` to accept an optional `{ embedder, embedderOptions }` argument. The default is `this`, so other callers keep the current behavior. - Changes the message-vector row builder to call `embedMessageChunks` instead of `embedValues`. The diff for this file shows no change to the query or content embedding paths. **`packages/memory/src/semantic-recall-embedding-reuse.test.ts`** (+267) - Adds a new test file. It covers reuse across saves, one row per message on resave, re-embedding of edited or distinct text, usage counting, non-reuse of failed or mis-sized results, invalidation on embedder or option change, an embedder change during an in-flight save, raw-string content, and chunking within the embedder input limit. - The number of tests in this file is not verified from the diff. The PR reports 10. **`packages/memory/integration-tests/src/performance-testing/performance-tests.ts`** (+23) - Imports the `MastraVector` type. - Before timing, checks that indexed vector rows are at least `threadCount * messagesPerThread`. - Before timing, runs a thread-scoped semantic recall for each thread. Checks that results are nonempty and belong to that thread. - The timed query measurements and p95 threshold are not changed. **`.changeset/quiet-owls-reuse.md`** (+5) - Adds a patch changeset for `@mastra/memory`. It describes reuse of vectors for repeated or re-saved text and the exclusion of reused vectors from reported usage. ## Reported results These results come from the PR description. They are not verified by completed checks on the current head. - Three saves of 50 messages with repeated text: embedding calls fell from 150 to 50. Stored row counts are reported unchanged. - Repeated-text FastEmbed time: about 7.6 s before, 2.6 s after. - Unique, raw-string, and long-text call counts: reported unchanged. - `@mastra/memory` unit tests: 2,689 passing, 1 todo. - Typecheck, lint, and format: reported clean. - LibSQL benchmark with 2,500 messages: reported passing. ## Not verified - Test, typecheck, lint, format, and benchmark results. No completed check on the current head was available. - The test count in the new test file. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com> |
||
|
|
9eae9680d2 |
fix(memory): serialize tool results half as often in long Observational Memory runs (#30400) | ||
|
|
8ac88228d6 |
fix(core): remove the 10 ms pause after memory saves and keep same-millisecond turns in order (#29148)
## What Removes the 10 ms pause `MessageHistory` waited after every memory save, and fixes the ordering bug that pause had been hiding. Stacked on #29142 (keep a forwarded queued message until the owning run accepts it). Retarget to `main` once that merges. ## Why The pause was added so the next message's timestamp would always land after the saved one. Without it, a new turn's input can be stamped in the same millisecond as the previous turn's stored answer. Sorting on equal timestamps then puts the stored answer last, and the new turn's response merges into that old answer. Under load, `signals-pending-cancellation` "answers the follow-up after upstream/thread cancellation" failed 2 of 12 runs this way. Every memory-backed call also paid at least 10 ms per save. ## How - `message-history.ts`: drop the post-save `setTimeout`. Stored history is now added in a single `add(array, 'memory')` call, as semantic recall already does. - `MessageList`: after a memory add, any live message whose timestamp equals a stored one moves 1 ms later, skipping other stored times, and its signal `createdAt` is kept in sync. Live messages that tie only with each other keep their timestamps. The pass runs once per add, in O(n log n). ## Test changes (setup only, no assertion changes) - New `input-after-stored-history-tie.test.ts`: a tied new turn stays after the stored answer (fails without the fix), and same-ms live messages with no stored tie are left untouched. - `signals-pending-cancellation` lease-loss test now waits for the follow-up run to register before it vanishes. Without the pause, the run could vanish before registering, and the message was delayed past the test window. The test still fails against a runtime without the lease watch. - `provider-history-compat` fixture: explicit ascending `createdAt` values in place of four `new Date()` calls that landed in the same ms. - `background-task-call-id-reuse` scenario: waits for the last background result instead of relying on the pause. ## Testing - Full core unit lane (`--project unit:packages/core`, Node 24, 5 CPUs): 1,560/1,560 files, 25,295 passed, 0 failed. - Signals, message-list and call-id-reuse suites: 3 repeated runs, 1,169 passed each. - `tsc --noEmit` and oxlint are clean. Not verified: real multi-process pubsub (Unix socket/Redis), and stores other than core's in-memory and test stores. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> **ELI5:** When a chat app saves a reply, it used to wait 10 ms. Two messages could also get the same timestamp and sort in the wrong order. This change removes the wait and keeps messages in the correct order, even when timestamps match. ## Overview This change removes the 10 ms pause after each memory save in `MessageHistory`. It also changes how `MessageList` orders live messages whose timestamps match stored history. Without this change, a new turn's response could merge into the previous turn's stored answer. The change also covers a forwarded-idle case, where a run can start after another process's lease disappears. The runtime clears the forwarded run's active IDs only when the stream ID and run ID still match the values it read before the lease check. ## Changes by area **Memory save path** - `packages/core/src/processors/memory/message-history.ts`: `processInput` adds all chronological history in one `add(array, 'memory')` call. It previously added each message separately. `processOutputResult` no longer waits 10 ms after it persists messages. **Message ordering** - `packages/core/src/agent/message-list/message-list.ts`: `MessageList.add` runs `moveLiveMessagesPastStoredTies` after it processes inputs. The pass runs once per add. - When memory messages exist, the pass marks them as stored. - It moves live messages that share a stored timestamp. It also moves live messages that would no longer follow a moved live message. - It skips other stored times when it chooses new timestamps. - It orders placements by each message's own timestamp and arrival order. - It preserves anchored signals. It updates matching signal `createdAt` values and the ordering watermark. - It re-queues moved persisted messages, so the moved timestamps are saved again. - It sorts the list only when a message moved. - Live messages that tie only with each other keep their timestamps. **Remote run forwarding** - `packages/core/src/agent/thread-stream-runtime.ts`: The forwarded idle-signal lease checker snapshots the active stream ID before it reads the lease. - After consecutive checks show that the target run no longer holds the lease, and its unadmitted signals are requeued, the runtime clears that run's active run ID and stream ID. - It clears them only when the run has no local record, the active stream ID is unchanged from the snapshot, and the active run ID still matches. - It then drains pending idle signals. **Tests** - `packages/core/src/agent/__tests__/signals-pending-cancellation.test.ts`: Adds and updates coverage for follower idle messages. The cases cover runs that end, are superseded, or lose their lease before delivery. The vanished-owner tests wait for the follow-up run's registration to reach subscribers before they end the run. A new test checks that a follower with queued work does not start local execution after it unsubscribes and sees the lease disappear. Another test checks that a run that takes the thread during an in-flight lease read stays active. A further test checks that a remote run that starts and ends during that read does not block the drain of queued work. - `packages/core/src/agent/message-list/tests/input-after-stored-history-tie.test.ts` (new): Covers ties between stored history and live input, response, and context. It checks reminder placement, deduplication under turn sources, and turn order when stored history arrives in batches. A save test checks that a moved input is returned for saving with its updated timestamp. - `packages/core/src/loop/test-utils/aimock/scenarios/background-task-call-id-reuse.scenario.test.ts`: Waits after each turn until that turn's background result appears in recalled memory. - `packages/core/src/processors/provider-history-compat.test.ts`: Gives transcript messages explicit, sequential timestamps. The test still checks that API-error repair removes the earlier empty signed block. - `packages/memory/src/processors/observational-memory/__tests__/message-utils.test.ts`: Gives stored messages timestamps earlier than the live input. It expects observable order `memory-1`, `input-1`, `response-1`. It still checks that the synthetic continuation is excluded from observable messages. **Release notes** - `.changeset/memory-save-no-pause.md`: Adds a patch changeset for `@mastra/core`. It documents the removed pause, the same-millisecond ordering fix, and the queued-message case. ## Public API No exported or public declarations change. ## Scope 9 files changed. 571 lines added and 26 removed. ## Validation The PR author reports these results. This summary does not verify them. - Core unit lane (`--project unit:packages/core`, Node 24, 5 CPUs): 1,560 files and 25,295 tests passed. - Signals, message-list, and call-ID-reuse suites: 1,169 tests each passed across three runs. - `tsc --noEmit` and oxlint: clean. - Not verified: real multi-process pubsub (Unix socket or Redis), and stores other than core's in-memory and test stores. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com> Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
3c1b528b6e |
fix(memory): block irrelevant replace of shared working memory at default scope (#30306)
## Summary
With working memory `scope` left at its default (`'resource'`), the
experimental vNext update-working-memory tool let one thread erase a
line another thread wrote to shared memory by calling it with
`updateReason: 'replace-irrelevant-memory'`. The tool resolved `scope ||
'resource'` for its ID checks, but the guard compared the raw config
value, which is `undefined` by default. The guard now uses the resolved
scope, so default and explicit `'resource'` behave the same, and the
refusal says the memory is shared instead of the misleading "no
searchString was provided".
**Why this way:** reusing the `scope` already resolved a few lines above
keeps one source of truth, rather than repeating `|| 'resource'` in the
guard (the issue's preferred option too).
Door: two-way · Risk: low (composes `@mastra/memory` vNext working
memory tool) · Cleanup: none
## Evidence
New test file
`packages/memory/src/tools/working-memory-vnext-scope.test.ts` drives
the real tool from `Memory.listTools()` with a real `Memory` and
`InMemoryStore`, writing from two threads and reading back from both.
- **Before** (origin/main's `working-memory.ts`):
```
× another thread can't erase shared resource memory as irrelevant (scope
left at its default)
expected { success: true, … } to match object { success: false, … } //
thread B replaced Apollo
× another thread can't erase shared resource memory as irrelevant
(scope: 'resource' set explicitly)
// refused, but with "no searchString was provided"
✓ still lets a thread clarify an existing resource-scoped line when
scope is left at its default
✓ with scope: 'thread', a thread can replace its own irrelevant memory
and other threads are untouched
Tests 2 failed | 2 passed (4)
```
**After:**
```
✓ another thread can't erase shared resource memory as irrelevant (scope
left at its default)
✓ another thread can't erase shared resource memory as irrelevant
(scope: 'resource' set explicitly)
✓ still lets a thread clarify an existing resource-scoped line when
scope is left at its default
✓ with scope: 'thread', a thread can replace its own irrelevant memory
and other threads are untouched
Tests 4 passed (4)
```
- Neighboring suites `src/tools/working-memory.test.ts` and
`src/index.test.ts` with the new file: 215 passed. `tsc --noEmit` for
`@mastra/memory` clean; `oxfmt --check` and `oxlint --deny-warnings`
clean on changed files.
One behavior note: the issue describes the guard as turning the update
into an append. On main, explicit `scope: 'resource'` actually refuses
the call (the guard cleared `searchString`, then the next check rejected
it) and nothing is written. This PR keeps that refusal and makes the
default match it; only the reason text changes.
## Merge Danger
Two-way: a one-condition change plus a clearer refusal message in the
experimental vNext tool; the riskiest part is that default-scope agents
that relied on `replace-irrelevant-memory` succeeding will now get a
refusal, which is the documented intent.
**Blast radius:** agents using working memory `version: 'vnext'` with
resource scope (default or explicit). The stable tool and thread scope
are unchanged. Open PR #29209 also edits `working-memory.ts`, in
different hunks (lines 1-90; this change is at line 426).
Closes #30287
## Council review
Rounds: 3
| Seat | Round 1 | Round 2 | Round 3 |
| --- | --- | --- | --- |
| Claude Opus 5.5 | not run | 9/10 | 10/10 |
| GPT-6.1 Sol | 9.8/10 | 10/10 | 10/10 |
| Grok 4.7 | empty seat | empty seat | empty seat |
| Kimi K3.1 | not run | not run | 10/10 |
| DeepSeek V4.1 Flash | not run | not run | 9/10 |
Created with Claude Opus 5.5 in T3 Code.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
**ELI5:** Working memory can be shared by every chat that one user has.
Before this fix, if the scope was not set, one chat could delete a line
that another chat had saved. Now the default scope gets the same
protection as the explicit shared setting.
## Summary
This change fixes the guard for `replace-irrelevant-memory` in the
experimental vNext working-memory tool. The guard read the raw `scope`
config value. When `scope` was unset, that value was `undefined`, so the
guard did not run. The guard now uses the resolved `scope` value, which
includes the `resource` default.
## Behavior
- **Default or explicit `resource` scope:** A
`replace-irrelevant-memory` call that includes a `searchString` is
refused. The tool returns a specific failure. The failure tells the
agent to use `clarify-existing-memory` or `append-new-memory`. Before
this change, the tool cleared the `searchString` and continued to later
validation.
- **`thread` scope:** No change.
- **Stable (v1) working-memory tool:** No change.
- **`clarify-existing-memory`:** No change. The linked issue excludes
this tool.
The linked issue expected the replacement to become an append. This
change keeps the refusal for `resource` scope. It makes the default
scope match explicit `resource` scope.
## Changed files
- `packages/memory/src/tools/working-memory.ts` (+6/-3): Uses the
resolved scope in the resource-scope check. Returns the specific failure
for `replace-irrelevant-memory` with a `searchString`.
- `packages/memory/src/tools/working-memory-vnext-scope.test.ts` (+95):
Adds tests for:
- refusal under default and explicit `resource` scope;
- clarification of existing shared resource memory;
- replacement isolation under `thread` scope.
- `.changeset/nine-boxes-pump.md` (+5): Adds a patch changeset for
`@mastra/memory`.
No exported or public declarations change.
## Verification
The PR author reports the following results. This summary did not rerun
them.
- Four new tests pass.
- 215 tests pass across the new test file and the neighboring
`src/tools/working-memory.test.ts` and `src/index.test.ts` suites.
- `tsc --noEmit`, `oxfmt --check`, and `oxlint --deny-warnings` report
no issues.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
a21d692444 |
fix(memory): stop the Observer crashing on message parts without text (#30272) | ||
|
|
56f654fc0d |
fix(core): name the agent in the memory missing-storage error (#30164)
Co-authored-by: Abhi Aiyer <abhiaiyer91@gmail.com> |
||
|
|
ebbea50d38 |
fix(memory): clarify missing storage for unregistered agents (#29972) | ||
|
|
08e677fdbe |
fix(memory): keep observation groups whole in buffered reflection (#29412)
## Description With observational memory's `retrieval` on, observations are wrapped in `<observation-group>` tags. Buffered reflection chose how many leading lines to reflect from a token estimate alone, so the cut could land inside a group. The reflector then got the group's opening tag, and activation kept the rest of its bullets plus a `</observation-group>` with no opening tag. The agent saw a stray closing tag, and those observations lost the `range` recall uses to find their source messages. `doAsyncBufferedReflection` now passes the cut through a new `alignLineCutToObservationGroups` helper before anything uses it, so the stored `reflectedObservationLineCount`, the token estimate and the compression target all agree on a group boundary. Storage adapters slice on that stored count, so none of them change. ```diff cut = floor(activationPointTokens / avgTokensPerLine) +if cut lands inside a group + move it back to before that group's opening tag (and its blank separator) + if nothing precedes that group, move it forward past the group's closing tag reflect lines[0..cut), keep lines[cut..] verbatim on activation ``` Observations without group tags, and cuts that already fall between groups, are unchanged. ## Evidence New test `buffered-reflection-group-boundary.test.ts` drives the real path: `Memory` with `InMemoryStore`, a turn that buffers a reflection, then a turn that activates it. Only the reflector model is mocked. It checks what the reflector received, the stored cut, and the activated observations (balanced tags, each kept group's id, range and bullets). - **Before** (fix reverted, `reflector-runner.ts` from `main`): `Tests 2 failed | 1 passed (3)` - the reflector received `group 3 observation 0-3`, the first part of a group whose closing tag stayed active - a first group bigger than the budget was cut at line 24 instead of after its closing tag at 31 - the no-group-tags case passes both before and after, which is expected: plain observations keep their cut - **After:** `Tests 36 passed (36)` across the new file and `observation-groups.test.ts` (which gains unit cases for the helper: cuts between groups, ungrouped lines before the first group, indented tags, an unclosed last group, no tags). - Guards: 10 related observational-memory test files, `Tests 668 passed | 1 todo`; `tsc --noEmit` and `oxlint --type-aware` on `@mastra/memory` pass; Prettier clean. Not run: storage adapter integration suites (pg, libsql, etc.). They only consume the stored line count, which this change fixes at the source. ## Merge Danger **Door:** two-way. Reverting the commit restores the old cut; nothing persisted changes shape. **Blast radius:** `@mastra/memory` buffered reflection with `retrieval` on. The cut can now reflect slightly fewer lines than the estimate (back to a group boundary), or a whole first group that is larger than the estimate. Records already damaged by this bug are not repaired. Overlap: #28293 (`fix/27262-buffered-reflection-values`) edits the same `doAsyncBufferedReflection` function. This PR touches different lines (one import and the `linesToReflect` assignment); a local `git merge-tree` of #28293 with this branch adds no conflict beyond the two that #28293 already has with `main` in the activation code. ## Related issue(s) Fixes #28445 ## Type of change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Code refactoring - [ ] Performance improvement - [x] Test update ## Checklist - [x] I have linked the related issue(s) in the description above - [ ] I have made corresponding changes to the documentation (if applicable) - [x] I have added tests that prove my fix is effective or that my feature works - [ ] I have addressed all Coderabbit comments on this PR ## Council review Rounds: 3 | Seat | Round 1 | Round 2 | Round 3 | | --- | --- | --- | --- | | Claude Opus 5.5 | not run | 9/10 | 9/10 | | GPT-6.1 Sol | 10/10 | 10/10 | 10/10 | | Grok 4.7 | empty seat | empty seat | empty seat | Round 2 findings applied: opening tags are matched after trimming, like closing tags; a long test comment was rewrapped. Created with Claude Opus 5.5 in T3 Code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 Buffered reflection could split a group of related observations in half. This change keeps each group together, so the remaining observations stay properly organized and linked to their source messages. ## Summary - Aligns the buffered reflection cut to an `<observation-group>` boundary. - Moves a cut back before the group when earlier content exists. If the group starts the observations, moves the cut forward past that group. - Leaves cuts unchanged when they are outside a group or when observations have no group tags. - Adds regression tests for boundary alignment, activation, preserved group ranges, and ungrouped observations. - Adds a patch changeset for `@mastra/memory`. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6644a82748 |
fix(memory): lock vNext working memory updates by resolved scope (#29477) | ||
|
|
ec8d9b8e7b |
fix(memory): require the working memory section in single-thread observer calls for Markdown working memory (#26203)
Fixes #25350 ## Problem When observational memory manages Markdown working memory (`observation.manageWorkingMemory: true` without a working memory schema), the Observer updates working memory by writing a `<working-memory>` section in its own reply. The prompt called that section optional. In the issue's repro the Observer noticed the user's preference change every time but often left the section out, so working memory kept the old value. Nothing recorded that the update was missing, so a dropped update looked the same as "nothing changed". ## Fix - The working memory extractor is now marked as required for single-thread Observer calls. The prompt tells the Observer to always write `<working-memory>`, and to write `UNCHANGED` inside it when nothing changed. - `UNCHANGED` is skipped rather than stored, in both the tag form and the `<extracted-values>` JSON form. Small variations like `Unchanged.` or `` `UNCHANGED` `` are skipped too. - If the section is missing, an entry is added to `extractionFailures` and the Observer logs a warning naming the section. - If a reply has nothing else in it, such as a refusal ("I'm sorry, I can't help with that"), it is still retried. The check that decides whether a reply has content ignores the missing-section failure. Without that, every refusal would have counted as content and been committed, and the observation cursor would have moved past messages nobody observed. - Batched multi-thread Observer calls keep the section optional. One reply covers several threads there, so requiring a per-thread section would record false failures. - Working memory with a schema doesn't change. It uses a separate structured extraction call. `required` is not an `Extractor` config option. Only the working memory extractor turns it on, through a constructor argument marked `@internal`, the same way the existing `internal` flag works. ## Testing - `extractor.test.ts` has tests for the required prompt wording, a failure recorded when the section is missing, `UNCHANGED` and close variants skipped in the tag form, and exact `UNCHANGED` skipped in the JSON form, and the section staying optional for batched calls and for other extractors. `required-section-logging.test.ts` checks the warning. Run against main's source, the prompt test fails because main still calls the section optional, and the logging test fails with `expected undefined to deeply equal [ 'mood' ]`. - `refusal-observer-output.test.ts` adds three end-to-end cases with Markdown working memory turned on, using a real Agent and Memory against a mock model server. A refusal doesn't advance `lastObservedAt`. A refusal is retried and the next good reply is observed. A reply that has observations but no `<working-memory>` commits without a retry. Without the content-check change the first two fail: `lastObservedAt` is set after the refusal, and the retry test sees one Observer call instead of two. Both pass with the change. - The `@mastra/memory` unit suite passes (2645 passed, 1 todo), and `tsc --noEmit` and `oxlint` are clean. I didn't re-run the issue's live-model repro, because it needs a real model. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 The Observer can notice a change to Markdown working memory but fail to save it if it leaves out that section. This change requires the section in single-thread calls and gives the Observer a clear way to say that no update is needed. ## Changes - Require the `<working-memory>` section in single-thread Observer calls. The prompt instructs the Observer to write `UNCHANGED` when no update is needed. - Treat `UNCHANGED` and common formatting variants as no update, rather than storing them. - Record a missing required section as an extraction failure and log a warning with the failed section slugs and thread ID. - Keep required sections optional in batched multi-thread calls. - Exclude missing-required-section failures from the check that decides whether an Observer refusal should be retried. - Add tests for extraction, logging, refusal handling, and batched calls. **Validation:** Current-head test, build, and CI results are not available. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Jamie Lee Nichols <10267569+CodingButter@users.noreply.github.com> Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
c9140d2d4d |
fix(memory): write one semantic recall row per message on every agent engine (#29826)
## Description
Durable and evented agents wrote two or three semantic recall vector
rows for each message. Two writers embedded the same messages:
- the `SemanticRecall` output processor, which keys rows by message id;
- `Memory.saveMessages`, which the save queue calls from the durable
input save and the finalize flush, and which wrote rows without ids, so
the store generated random ones.
The duplicates took up `topK` slots, and the embedder ran twice per
message.
The two writers also built rows differently. Only `saveMessages` chunked
long text, stripped working-memory tags and stored thread metadata.
Neither writer can simply be removed: the processor serves agent runs,
while `saveMessages` serves direct callers (the HTTP route, the client
SDK, user code) and the save queue. So this PR gives both writers one
row builder and one id scheme.
Changes:
- Core: `buildMessageEmbeddingRows` and `upsertMessageVectors`
(internal, in `@mastra/core/memory`) now decide the row format: text
extraction, chunking, metadata and row ids. A message's first chunk
keeps its message id, which is the id the processor already wrote, so
existing rows get replaced rather than duplicated. Later chunks get a
UUID derived from the message id and chunk index. Every message-vector
upsert now sends ids. Each writer passes its own embed function, so
embedding calls, retries and caches work as before.
- Core: `MessageList` keeps a per-run record of embedded messages (index
and message id to text hash) and serializes it with the list. The save
queue passes the record to `saveMessages`, and the durable input save
copies it back from its cloned list, so the second writer skips what the
first one embedded into the same index. If a message's text has changed
since, it is embedded again. A processor writing to a custom
`semanticRecall.indexName` and `Memory` writing to its default index
each still get every message.
- Core: new `semanticRecall.maxChunkChars` option, default 16,384
characters (about 4,096 English tokens). The limit is in characters
because each provider uses its own tokenizer. The processor now chunks
long messages the way `saveMessages` already did, instead of embedding
them as one input, which failed past the embedder's limit.
- Memory: `saveMessages`, `updateMessages`, cloned-message embedding and
`updateThreadResourceId` all write through the builder. Edits and
resource moves embed before deleting the old rows, so a failed embedding
call leaves them in place. Thread metadata is read only when rows will
be written. The `@mastra/core` peer range is now `>=1.83.0-0`.
- FastEmbed: models declare `maxInputTokens: 512`, so recall chunks fit
their window.
Existing rows are not rewritten. Rows `saveMessages` wrote before this
change keep their random ids. Rows the processor wrote already use the
message id, so new writes replace them. An evented or Inngest finalize
that runs on another worker doesn't see the input save's record, so it
embeds the user message once more into the same row.
```ts
const memory = new Memory({
options: {
semanticRecall: { topK: 3, messageRange: 1, maxChunkChars: 6000 },
},
})
```
Tests:
`packages/memory/integration-tests/src/semantic-recall-embedding-rows.test.ts`
runs plain, durable and evented agents against a LibSQL vector index,
with thread scope, resource scope and a custom index. It checks:
- each persisted message has exactly one row under its own id;
- no row is written twice in a run;
- the fact is recalled once.
On `main` the durable and evented cases fail with 30 rows for 12
messages. With the record not passed to `saveMessages`, the write-once
check fails on durable and evented. Memory cases cover:
- record skips;
- chunk size taken from the embedder's input limit;
- an edit that shortens a long message;
- an embedder failure during an edit.
Unit tests cover the builder and the `MessageList` record.
Six cases in `agent-memory.test.ts` fail because their recorded fixtures
no longer match the requests. They fail the same way on `main`.
## Related issue(s)
Related to #29352. Agent runs now store thread metadata on vector rows,
which the read-side fix in that issue needs. This PR does not change the
read side.
## Type of change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [x] Test update
## Checklist
- [x] I have linked the related issue(s) in the description above
- [x] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## ELI5
The PR helps agents avoid saving the same message to search more than
once. It also splits long messages into smaller pieces so search can use
more of their content.
## Summary
- Shared helpers extract message text, split it into chunks, build
vector rows, and upsert them. The first chunk uses the message ID when
it is a UUID or small integer; other IDs and later chunks use stable
derived IDs.
- `SemanticRecall` and `Memory.saveMessages()` use the shared
row-building path. Agent runs track embedded text by message and index,
so writers can skip unchanged messages.
- `semanticRecall.maxChunkChars` sets the chunk limit. It defaults to
16,384 characters and can use the embedder’s declared `maxInputTokens`
when unset. FastEmbed models declare a 512-token input limit.
- Memory updates and thread-resource moves build replacement rows before
deleting existing vectors. Re-saving shorter text can leave older chunk
rows searchable; the documentation describes this behavior.
- Tests cover row counts, scopes, custom indexes, chunking, repeated
saves, edits, and embedding failures. The supplied description reports
six fixture-based `agent-memory.test.ts` failures that also occur on the
base branch. Completed check results are not available.
- Documentation and peer dependency minimum versions were updated.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
791cf606e9 |
fix(core): restore lease process marker removed by temp cleaners (#29723)
Co-authored-by: mastracode Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
36bbc29129 |
fix(core): stop re-adding sealed messages reloaded from memory (#29537)
## Description When an agent with memory resumed a run, a sealed assistant message (for example one observational memory had observed) could appear twice in the prompt. Memory cleans a message when it saves it: it trims text, strips `<working_memory>` tags and removes `updateWorkingMemory` tool steps. `MessageList.addOne` compared the recalled copy with the live sealed message, saw a difference, and appended the copy under a new Mastra id while it kept the same OpenAI item ids. The next Responses request then failed with `Duplicate item found with id rs_...`. A sealed message whose text ends in a newline was enough to trigger it. The memory dedupe check in `MessageList.addOne` now also treats a stored copy as the same message when it matches the live one after that same save cleanup (`isPersistedCopy` in `packages/core/src/agent/message-list/conversion/utils.ts`, which reuses core's existing `working-memory-utils` helpers). Observational memory's buffering saves sealed messages through `MessageHistory`, which first runs core's `filterMessagesForPersistence` and drops streaming `partial-call` tool parts (they can arrive in client-sent history, such as an AI SDK `dynamic-tool` part still in `input-streaming`). So `isPersistedCopy` also compares the live message after that filter. Memory's direct `saveMessages` keeps partial calls, and those copies still match too. ```diff addOne(message, 'memory') for each existing message with the same id - if messagesAreEqual(existing, incoming) -> skip + if messagesAreEqual(existing, incoming) + or isPersistedCopy(existing, incoming) // same id, equal after memory's save cleanup + -> skip sealed existing + "different" copy -> append under new id // was reached by the trimmed copy ``` Text that really changed after the seal still differs after the cleanup, so it is still added as a new message (covered by a test). ## Evidence - **Before (origin/main core):** `packages/memory/src/sealed-message-reload.test.ts` (the issue's `repro.mts` as a test, using real `Memory`, `InMemoryStore`, `recall()` and `MessageList`): `Tests 5 failed | 3 passed (8)`. The list held 2 messages with `rs_example` twice, and the LLM prompt carried `rs_example` twice. **After:** `Tests 9 passed (9)`. The ninth case seals a client-sent message with a streaming tool call through the real OM seal and `MessageHistory` save. Before the `partial-call` follow-up it failed with a second message id next to `assistant-message`. - **Before:** `sealed-reload-duplicate-reasoning.issue-29380.test.ts`, the issue's agent-only fixture ported to a test: a real Agent with observational memory, suspended twice in `ask_user`, against a scripted Responses server over real HTTP that returns 400 on duplicate input ids. It failed 5/5 runs with `expected [ 'rs_script_1', 'msg_script_1', … ] to deeply equal []`. **After:** passed 10/10 runs. - **Before:** the issue's original `agent-only-repro.mts`, run against the local build with `@ai-sdk/openai` 3.0.135, got `HTTP 400 Duplicate item found with id rs_script_1`. **After:** no duplicate ids, and the `no-newline`, `no-buffer` and `one-tool` controls stayed clean. - New core unit test `conversion/is-persisted-copy.test.ts` (12 cases): it matches trimmed, whitespace-only and working-memory-stripped copies, and copies saved with or without a streaming `partial-call` part. It rejects a copy missing a tool call that had finished streaming, and copies that differ by an inner space, a different id, a different reasoning or text item id, or an extra part. - Neighboring suites: core `src/agent/message-list` 979 passed, 2 skipped. Memory unit `src` 2620 passed, 1 todo; after the `partial-call` follow-up, memory `src/processors/observational-memory` 2012 passed, 1 todo. `tsc --noEmit` passed for core (`tsconfig.build.json`) and memory, and `oxlint --type-aware --deny-warnings` and `oxfmt --check` were clean on the changed files. Not tested: live OpenAI (no key used); the HTTP fixture mirrors the API's duplicate-item 400. ## Merge Danger **Door:** two-way. Code-only change in `@mastra/core`'s `MessageList`; reverting the commit restores the old behavior, and no stored data changes. **Blast radius:** every agent that loads history from memory. The new check only runs for a memory-loaded message that shares an id with a message already in the list, and it only drops the stored copy when the two match after memory's own save cleanup, so the live message is kept. Risk: if a stored copy really differs from the live message only by whitespace or working memory, the live version now wins. That matches how the unsealed path already handles trimmed text. ## Related issue(s) Fixes #29380 The issue also suggests a request-wide guard that removes repeated Responses item ids. This PR leaves that out: as the issue notes, it would hide the 400 but not fix the duplicated history. ## Type of change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Code refactoring - [ ] Performance improvement - [ ] Test update ## Checklist - [x] I have linked the related issue(s) in the description above - [ ] I have made corresponding changes to the documentation (if applicable) - [x] I have added tests that prove my fix is effective or that my feature works - [ ] I have addressed all Coderabbit comments on this PR ## Council review Rounds: 4 | Seat | Round 1 | Round 2 | Round 3 | Round 4 | | --- | --- | --- | --- | --- | | Claude Opus 5.5 | not run | 8.5/10 | 8.5/10 | 9/10 | | GPT-6.1 Sol | 9/10 | 9/10 | 9/10 | 9/10 | | Grok 4.7 | empty seat | empty seat | empty seat | empty seat | Created with Claude Opus 5.5 in T3 Code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 Memory can change a message when it saves it, such as by trimming spaces or removing working-memory details. This change recognizes the saved copy when it is loaded again, so it is not added a second time with the same provider item IDs. ## Changes - Added `isPersistedCopy` to compare messages with the same ID after applying persistence cleanup, including trimming text and removing working-memory content. - Updated `MessageList.addOne` to skip a recalled message when it matches either the existing equality check or the persisted-copy check. - Added tests for persistence cleanup, streaming tool calls, repeated reloads, and duplicate provider item IDs. - Added a patch changeset for `@mastra/core`. - Did not add a request-wide safeguard against duplicate Responses item IDs. **Checks:** not available. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d7d18072ba |
fix(memory): omit stored reasoning from recall results (#28810)
Fixes https://github.com/mastra-ai/mastra/issues/28809. Recall returns stored reasoning as ordinary tool text. Save an assistant message with reasoning at part 0 and visible text at part 1, then page its messages or request part 0 directly: the reasoning is exposed. Exclude reasoning/redacted-reasoning in the shared formatter, use rendered visibility for paging/fallback, and stop advertising a reasoning filter. Visible text/tool evidence keeps its original part indices. Stale helper calls with a reasoning filter return no reasoning. Tests: `PR_ROOT=$PWD PR_PACKAGE=packages/memory PR_TESTS=src/tools/om-tools.test.ts node node_modules/vitest/vitest.mjs run --config /tmp/m18-vitest.config.mjs --reporter=dot` passed 146/146. Eleven regression cases failed before the fix. Tests use real memory/storage and cover low/high paging, direct indices, filters, tool schema and next-message fallback. The temporary source config stubs unused catalog/embedding imports. Full builds/typechecks and external database suites were not run. Stored messages, provider history replay, and the existing recall lookahead limits retain their behavior. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 Recall results should show useful message content, not stored reasoning that looks like ordinary text. This change hides reasoning while keeping the original positions of visible text and tool results. ## Changes - Excludes stored `reasoning` and `redacted-reasoning` parts from recall output. - Uses renderable content to decide which messages count toward paging and fallback. - Removes `reasoning` from the advertised `partType` schema. Stale calls that still request it return no matching parts. - Preserves stored part indices for visible text and tool results. - Adds regression tests for hidden parts, paging, filters, schema, and fallback. - Adds a patch changeset for `@mastra/memory`. ## Validation The PR description reports 146 passing tests. It also states that full builds, typechecks, and external database suites were not run. The supplied repository output did not include test execution results. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Aamish <your@email.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
35ab0c807b |
feat(memory): observational memory search without a client-side embedder (#27005)
## Description Follow-up to #25009. Observational Memory's `retrieval: { vector: true }` still required an `embedder` even when the vector store generates embeddings itself, so it didn't work with MongoDB Automated Embedding. Now it does: ```ts const memory = new Memory({ storage, vector: new MongoDBVector({ id: 'vec', uri, dbName, autoEmbed: { model: 'voyage-4' } }), options: { observationalMemory: { retrieval: { vector: true } } }, }) await memory.searchMessages({ query: 'What deadline did we agree on?', resourceId: 'user-123' }) ``` This builds on the `isSelfEmbedding` flag from #25009. When the store reports it and there's no embedder, `Memory` sends text instead of vectors. Four spots needed it: the constructor check, `hasRetrievalSearch` (which also turns on the `recall` tool's search mode), `indexObservation`, and `searchMessages`. Observations go into an index called `memory_observations_selfembed`, without a dimension. Setups that pass an `embedder` behave exactly as before. `MongoDBVector` needed no changes. I added a live test for it in `stores/mongodb` that indexes observations, searches them, checks the thread and time filters, and checks that deleting a thread removes its rows. It's opt-in (`TEST_MONGODB_AUTOEMBEDDING=1`) and I ran it against an Atlas cluster. The rest is covered by unit tests with a mocked store. Docs aren't in this PR. The issue asks for a separate docs ticket, so I'll follow up there. The experimental Subconscious knowledge memory still requires an embedder and is out of scope. ## Related issue(s) Fixes #27002 ## Type of change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Code refactoring - [ ] Performance improvement - [ ] Test update ## Checklist - [x] I have linked the related issue(s) in the description above - [ ] I have made corresponding changes to the documentation (if applicable) - [x] I have added tests that prove my fix is effective or that my feature works - [ ] I have addressed all Coderabbit comments on this PR <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 Observational Memory can now search using a vector store that creates embeddings for it. Users can enable semantic recall without setting up a separate embedding service. ## Changes - `@mastra/memory` accepts `retrieval: { vector: true }` without a client-side embedder when the vector store reports `isSelfEmbedding`. - Memory indexes observation text under `memory_observations_selfembed` without passing a dimension. It sends search text to the store. - Observation-vector migration reads self-embedded rows with text queries and updates their metadata without requesting vectors. - Configurations with a client-side embedder keep the existing vector-based behavior. - The recall tool’s setup message now includes stores that generate embeddings themselves. - Added mocked-store unit tests and an opt-in MongoDB integration suite. The live suite requires `TEST_MONGODB_AUTOEMBEDDING=1` and either `MONGODB_AUTOEMBED_URL` or `VOYAGE_API_KEY`. Test execution results were not provided. - Documentation is not included; it is planned for a separate follow-up. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Ehindero Israel <iehindero419@stu.ui.edu.ng> |
||
|
|
642c62fee3 |
feat(core,server,client-js): answer a suspended background task from a chat (#28998)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
ea8fefcd83 |
fix(memory): close the observations block in Observational Memory context (#29224) | ||
|
|
e1e64f072c |
fix(memory): keep user signal attributes in Observer labels (#29176)
## Description
In a thread several people write to, Observational Memory's Observer saw
every user message as plain `User`, so observations could not say who
said what. User signals carry attributes (for example `attributes: {
name: 'Jim' }` passed to `agent.sendMessage`), and those were dropped
when the Observer's transcript was built.
The Observer now labels user signals with their attributes, e.g. `User
(name="Jim")`. Messages without attributes keep the plain `User` label.
Attribute handling is deliberately bounded so a value cannot break or
inject lines into the transcript: keys must be identifier-like, values
must be strings, numbers or booleans, keys are sorted, at most 8
attributes are shown, values are capped at 80 characters, and values are
JSON-quoted.
Changed:
`packages/memory/src/processors/observational-memory/observer-agent.ts`
(label formatting), with tests and a patch changeset for
`@mastra/memory`.
## Test plan
Regression tests in `observer-agent.test.ts` (labels each signal with
its attributes; quotes and bounds values; keeps plain `User` with no
usable attributes).
- Without the fix (`observer-agent.ts` from `origin/main`): 2 failed, 25
passed.
- With the fix: 27 passed.
- Observational Memory suite: 89 files, 2007 passed, 1 todo.
- `tsc --noEmit` exit 0; oxlint 0 warnings, 0 errors.
Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com>
|
||
|
|
0a553f0f6f |
Knowledge alpha: standalone Knowledge for agents, Subconscious removed (#28353)
## What this is
**Knowledge** is Mastra's store for durable knowledge that agents share
across conversations. Until now its agent-facing parts lived in
`@mastra/memory`, inside an experimental feature called
**Subconscious**, a background curator that wrote to Knowledge after
observational-memory reflections.
This PR makes Knowledge stand on its own as an experimental alpha,
rebuilt around pages and declared scopes.
- **Subconscious is removed** from `@mastra/memory` and Mastra Code.
Setting `experimental_subconscious` or the Memory `knowledge` option
throws, saying to attach Knowledge to the agent instead. Old
Subconscious data is ignored.
- **Knowledge attaches to an agent directly:** `new Agent({ knowledge,
scopes })`. It brings its own tools, instructions and pinned context.
Tools work without Memory; pinned context needs Memory.
- **Knowledge is a tree of pages.** Each page lives in one scope and
holds a body plus short notes. Notes can mention other pages with
`[[wikilinks]]`. A page can have a key, unique among its siblings, and
is addressed by key path: `<scope>#<key>/<child key>`. Pages declared in
a scope's config are created when missing and can be edited but not
moved, deleted or merged away by tools.
- **Access comes from declared scopes.** Agent `scopes` are plain
`"<type>:<value>"` strings that say who the run is, set by the host and
never by the model. Knowledge's required `scopes` option declares which
scope types exist and which roles (viewer, contributor, editor, owner)
holders get on which scopes. Rules reach one hop with no inheritance. A
run holding an undeclared scope fails. `knowledgeMemoryScopes()`
declares Memory's `thread:`/`resource:` scopes and an `org:` scope.
- **Five tools:** `knowledge_search`, `knowledge_read`, `knowledge_page`
(write, move, merge, delete), `knowledge_note` (add, restamp, remove)
and `knowledge_pin` (pin, unpin). Tools are off by default: `tools:
true` enables read and write, `tools: { read: true }` read only, and
`pinnedContext` adds pins. Deleting a page needs owner on the page's
scope and is refused for pages with child pages, pages declared in a
scope's config, and scope pages. On a scope page, write accepts the body
only: its name and description come from config.
- **Limits are in tokens:** about 250 per note and 100 per page
description. Pinned context defaults to 500 tokens, with a maximum of
2,000.
- **Search:** keyword search by default: every word of the query must
appear in the page (or note), and name matches rank first, the same in
every store. With a `vector` store and an `embedder`, search also
returns semantic matches. Every semantic hit is re-checked against what
the run can read before ranking, so hidden content never takes a result
slot. `knowledge.reindex()` re-embeds everything.
- **Host APIs:** `knowledge.readScopeFor(principals)` returns what
holders of those scopes can read. `knowledge.explainAccess(runScopes,
target)` says why a run can or can't reach a scope or page.
`explainAccess` is host-only.
- **Renamed scope declarations:** if stored scopes use a declaration
name the config no longer declares, Knowledge fails loudly on first use,
naming each orphaned name and how many scopes use it.
`knowledge.renameScopeDeclaration(from, to)` renames them. It is
host-only and atomic, except on MongoDB without a replica set.
- **Per-message scopes:** `session.sendMessage({ content, scopes })`,
plus queue, steer and notifications. A message whose scopes differ from
the running run's waits for that run to end and then starts its own run.
Scopes are set in-process only.
- **Storage:** LibSQL, PostgreSQL, MySQL and MongoDB store Knowledge as
schema 4 (pages, notes, mentions, pins). On startup, the tables of the
earlier experimental Knowledge node store (shipped in 1.79.0) are
dropped with any rows in them, and the new tables are created. If an
older process recreates them during a rolling deploy, the next startup
drops them again. Schema-4 tables and non-Knowledge tables are never
touched. On MongoDB, Knowledge collections found without a schema
version are adopted as schema 4 rather than refused. Don't run
`dangerouslyReset()` from an older version against a database the new
version already uses. Development databases created from earlier commits
of this branch also need a reset, since sibling key uniqueness changed
within schema 4. `storage.stores?.knowledge?.dangerouslyReset()` clears
Knowledge rows only, never Memory. Adapter peer floors are unchanged: on
an older `@mastra/core` the adapters still load and the other storage
domains keep working.
- **Mastra Code:** Knowledge is opt-in through **Experimental
Knowledge** in `/settings`. The `/knowledge` browser and the SDK
inspector read pages. Factory's Knowledge routes are disabled until the
Studio page readers land.
## Test plan
- Shared schema-4 storage contract suite on real stores (as of
|
||
|
|
95ddbb7fc3 |
fix(memory): avoid reindexing unchanged observation groups (#28150)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com> Co-authored-by: Tyler Barnes <tylerdbarnes@gmail.com> |
||
|
|
f241b9d595 |
fix(pg): use HNSW index for filtered vector queries (#29058) | ||
|
|
ba8b955b1a |
fix(redis-streams,valkey-streams): expire idle streams by default and clear them on thread delete (#28968)
A production Factory deployment ran out of Redis memory. Every key was a
`mastra:topic:*` stream, and almost all of them (3,673 thread streams)
had no expiry. The deployment built `new RedisStreamsPubSub({ url })`
with defaults, and the default `streamIdleTtlMs` was `0`, meaning
streams never expire. Thread streams are open-ended and never get
`clearTopic()`, so they piled up forever. The Redis instance used
`volatile-lru`, which can't evict keys without a TTL, so writes just
started failing.
This PR makes `@mastra/redis-streams` and `@mastra/valkey-streams` give
every stream a sliding 7-day idle TTL by default. Writes refresh it, and
live subscribers keep receiving events after a stream expires because
the read loop recreates it. Streams are an event bus, not storage:
replay history lives in the caching layer and storage, so the backlog
only matters for grouped workers that are offline longer than the TTL.
```ts
// Before: streams never expire unless you opt in
new RedisStreamsPubSub({ url });
// After: 7-day idle TTL by default; opt out explicitly
new RedisStreamsPubSub({ url });
new RedisStreamsPubSub({ url, streamIdleTtlMs: 0 });
```
Both pubsubs now log one warning after they first connect if Redis has
`maxmemory` set with any policy other than `noeviction`. A TTL makes
streams, consumer groups and lease keys evictable under `volatile-*`,
which would silently drop events and leases (ElastiCache defaults to
`volatile-lru`). The check uses `INFO memory` because managed Redis
often blocks `CONFIG GET`.
`memory.deleteThread()` now also deletes the thread's event stream
through the registered Mastra pubsub. This runs fire-and-forget, so
neither deleting a thread nor `memory.settled()` waits on Redis during
an outage. It's skipped on older core versions. Agents configured with
their own pubsub still rely on the TTL.
The docs now require `noeviction`; the Helm guide used to say a
`volatile-*` policy was fine. They also add a runbook for reclaiming
streams created before the TTL: switch to `noeviction`, upgrade the
writers, then expire each stream relative to its last write. Tested
against real Redis/Valkey containers, including a real TTL expiry while
a subscriber is listening and a thread delete with Redis unreachable.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## ELI5
Redis and Valkey can keep old event streams forever, which uses storage.
This change removes streams after seven days without writes by default
and cleans up a thread’s stream when the thread is deleted.
## What changed
- Redis and Valkey streams now use a sliding seven-day idle TTL by
default. Writes refresh the TTL; reads do not. Set `streamIdleTtlMs: 0`
to disable expiry.
- The read loop recreates an expired stream so a live subscriber can
continue receiving events.
- After connecting, each pubsub warns once when `maxmemory` is enabled
with a policy other than `noeviction`. The check uses `INFO memory`.
- Thread deletion starts best-effort stream cleanup through the
registered Mastra pubsub without waiting for Redis. Older core versions
skip this cleanup.
- Documentation and the local Docker configuration now recommend
`noeviction`. The documentation also explains how to expire existing
streams that lack a TTL.
## Tests
Tests cover TTL behavior, delivery after expiry for live subscribers,
eviction-policy warnings, and thread-stream cleanup. Test execution
results were not provided.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
|
||
|
|
9e36f35910 |
feat(observability): capture and export native execution outcomes (#28703)
Adds native `outcome`, `stopReason`, and `stopReasonContext` to agent, model, tool, and workflow tracing and carries them through `MastraStorageExporter` and `MastraPlatformExporter`. Implements [OBS-724](https://linear.app/kepler-crm/issue/OBS-724). Outcomes describe execution meaning. Guardrails record `blocked / guardrail`; failed observers retain their own result while the affected execution records `interrupted / observer_failed`. Caller cancellation records `abort` on its target and the agent run; interrupted descendants record `interrupted`. Run deadlines record `limit / timeout`, with `interrupted / timeout` on affected descendants. Completed spans retain their results during cleanup, and recovered retries can finish successfully. Framework stop-reason context is a flat summary of at most three scalar facts, with strings bounded to 128 characters and span references for further diagnostics. Custom outcome strings and context objects remain supported and are safely serialized. ```ts span.end({ endTree: true, outcome: 'abort', stopReason: 'caller_cancelled' }); ``` Processed `MODEL_CHUNK` spans now consistently belong to `MODEL_STEP`. Provider-stream instrumentation records timing and usage without capturing raw output before processors run. Agent-stream `MODEL_INFERENCE` spans no longer include response output; consumers should use processed chunks or generation output. Error finalization preserves completed-request usage and output. When the inner workflow fails without a finish callback, an internal tracing-only notification closes the generation and agent spans, including when the user error callback throws. Public error callback arguments and stream payloads are unchanged. Compatibility changes are explicit: public native fields now also allow `null`, which TypeScript consumers must handle. A null or absent outcome means no outcome was recorded; endTime determines whether the span has finished, including with older observability releases. Memory failure markers in streams and persisted messages, and tripwire metadata, include `observerFailure` with span references and classification. Certain workflow events and durable snapshots gain tripwire details, including processor-supplied metadata. The compact-context restriction applies to `stopReasonContext`, not all existing diagnostic payloads. Runtime continuation decisions and return statuses are retained. TripWire now uses the name `TripWire` instead of `Error`, affecting string representations, stacks, and serialized error names; thrown error objects otherwise retain their identity. Continuation causes remain in internal run state and are not added to public finish callback context, stream chunks, or snapshots. Older package combinations tolerate missing helpers without dependency or peer-range bumps, including a cleanup fallback for older model trackers. Cross-worker tracing remains follow-up work in [OBS-754](https://linear.app/kepler-crm/issue/OBS-754/preserve-tracing-outcomes-and-resumed-span-identity-across-workers). Without the local registry, ordinary completion and max-step exits can report `unknown`, and observer/delegation or provider-failure detail can be lost. The new caller-owned Inngest resume spans have been removed: separate resumed-segment tracing needs a worker handoff protocol. Inngest still reuses the original span identity on resume, and core durable remote resume can leave fresh local spans open or finalize original snapshots. A caller-side Inngest deadline may also be missing from worker-owned outcomes rather than appearing as `limit / timeout`. Process-loss recovery and these identity/finalization gaps are not solved here; the Inngest regression remains an explicit expected failure. No storage/query implementation, checkpoint/recovery infrastructure, or new evented control-flow/nested-run/sleep instrumentation is included. Native storage support comes from [#28389](https://github.com/mastra-ai/mastra/pull/28389). Validation: 216 tests passed across the core native-outcome, workflow-result, and classification suites, followed by five final focused checks including an added nested-tripwire result case. All 116 observability tracker, span-lifecycle, and ESM/CommonJS compatibility tests passed. Real Inngest verification passed 12 workflow TripWire cases and two tool suspension/error cases. Core, observability, and Inngest builds (including type generation), affected-file lint/formatting, and focused documentation checks passed. The status and fatal-bail tests were also verified to fail when their fix/fallback was disabled. The deferred Inngest resume-identity regression remains an explicit expected failure under OBS-754. Review threads are left for their authors/tools to resolve. |
||
|
|
976fde7416 |
refactor(lint): enable no-else-return at root (#28601)
Enables `no-else-return: error` in the root Oxlint config and removes 186 redundant `else` branches across 111 source/test files. Existing rule defaults are retained. Reviewed every code diff for behavior preservation. An independent TypeScript AST comparison confirms that changes are limited to removing `else` after returning branches, with the same expressions, statement order, return values, and `try`/`catch`/`finally` boundaries. All 181,158 identifier occurrences retain their bindings. The recorder replay path keeps an explicit block: flattening it would shadow the earlier request `body` with the response `body` and introduce a temporal-dead-zone error. Extends the existing monorepo optimized-dependency test to assert that a raw TypeScript workspace package, including its enum and transitive dependency, still executes in the production bundle. Validation: the root rule-only scan passes on 15,273 files with zero diagnostics; formatting and `git diff --check` pass. Schema compatibility tests: 1,060 passed, 9 skipped. Recorder runtime tests: 21 passed. The core selection initially passed 3,513 tests; rerunning the suites requiring local server/cache access passed 185 tests, with one remaining cache-file assertion also reproduced on unchanged `main`. Remaining validation limits: four asset-download cases cannot reach `placehold.co`; the changed-file lint run reports six import-order errors reproduced on unchanged files from `main`. Deployer unit tests and the monorepo test could not run because this checkout lacks required package dependencies (`fs-extra`/workspace links and `get-port`, respectively). These limits are left visible for CI. No changeset is included because this is a behavior-preserving lint cleanup. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 This PR makes the code easier to read by removing `else` branches that are not needed after a return. It also enables a rule that catches those branches and adds a test for loading a raw TypeScript workspace package in a production bundle. ## Summary - Enables Oxlint’s `no-else-return` rule at error severity and removes 186 redundant `else` branches across 111 files. - Extends the monorepo optimized-dependencies test to check a raw TypeScript workspace package, its enum, and a transitive dependency. - Updates the deployer’s transpile-package path pattern. - In workflow event processing, a truthy branch index with no matching child now falls through to publish a skipped-step end event. Previously, this case published no event. - Keeps the recorder replay block explicit to avoid request/response `body` shadowing and a temporal-dead-zone error. ## Validation Reported checks passed: the rule-only scan across 15,273 files, formatting, `git diff --check`, schema compatibility tests (1,060 passed, 9 skipped), recorder runtime tests (21 passed), and an initial core test selection (3,513 passed). Some test limits remain. Four asset-download cases could not reach `placehold.co`. Changed-file lint reported six import-order errors that also occur on unchanged files from `main`. Deployer unit tests and the monorepo test could not run because dependencies were unavailable. On rerun, suites requiring local server/cache access had 185 passes; one cache-file assertion also reproduced on unchanged `main`. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
1fc075ab7f |
refactor(core): one provider-failure reader for retries, memory, rotation and Factory (#28496)
**─────── TLDR ───────** > Retries, memory, account rotation, the TUI and Factory now read a failed model call through one reader in `@mastra/core/error`, so one rule change fixes them all. Follows #27811, which put this reader in Factory first. Before this PR, seven hand-written status, code and wording tables disagreed. The retry processor retried an exhausted quota, because the AI SDK marks every 429 retryable. Rotation only read quota wording on a 400. The run report said `Bad Request`. Now `classifyProviderError` reads the HTTP status, the provider codes, the body and the wording, and every site asks it. Each site keeps only the signals the reader can't see: a numeric `status`, network errors, Retry-After and `ProviderAuthRequiredError`. ### Behavior changed a 429 whose body says the quota is exhausted (`insufficient_quota`) ├─ **was** — retried, and the TUI says to wait a moment └─ **now** — never retried; the TUI names the quota and points to billing a credential or usage failure named only in the provider's body ├─ **was** — no account rotation; Factory reports a plain run failure └─ **now** — rotates to the next account; Factory reports bad credentials a Factory run report for a failed provider call ├─ **was** — quotes the provider for usage limits only, else `Bad Request` └─ **now** — quotes the provider's own reason, cut at 400 characters a mid-stream provider code, such as a Codex `server_error`, on the web card ├─ **was** — `Run stopped`, live and after a reload └─ **now** — named as an outage; the saved part keeps the codes a stream error that your own retry matcher and the built-in OpenAI matcher both accept ├─ **was** — the built-in policy wins: one immediate retry, no callback └─ **now** — your matcher's `maxRetries`, `delayMs` and `onRetry` apply a tool's `EACCES: permission denied` ending a run ├─ **was** — the reader called it a provider credential failure └─ **now** — unknown; only the code form `permission_denied` reads as auth ### Shape changed | | | | |---|---|---| | **−** | `mastracode/factory/src/provider-error.ts` | its home in #27811, back in core | | **−** | memory `context-overflow.ts` | replaced by the reader's `context_overflow` | | **−** | `USAGE_LIMIT_MESSAGE_PATTERN` ×2 | rotation and run report each kept a copy | | **−** | `RETRYABLE_OPENAI_ERROR_CODES` · `TERMINAL_AUTHORIZATION_ERROR_CODES` | the reader's kinds | | **+** | `quota` in code-sdk `ErrorType` | a quota can't be waited out | | **~** | `MastraErrorPart['error']` | keeps `codes` and the provider's `reason` for a reload | what callers now import from `@mastra/core/error` `classifyProviderError(error)` · `providerErrorReason(error)` · `providerRequestId(error)` · `providerModelId(error)` · `toPersistedErrorIdentity(error)` · `PROVIDER_CALL_ERROR_NAME` · `type ProviderFailureKind` ### Decisions - **overflow wording outranks a 5xx** — a gateway relaying `prompt is too long` gets compacted, not retried; one rule in `provider-error.ts` to drop. - **a caller's numeric `status` outranks wording** — the reader ignores `status`, which Octokit and MCP errors also carry, so rotation and `parseError` read it first; a 503 saying `weekly limit` retries instead of burning an account. - **only a provider's own sentence is saved** — the saved `reason` comes from an HTTP response or a stream frame, never from another error's `detail`, which can name a database row. ### Not verified - [ ] an app on `@mastra/core` 1.80.0-alpha.0 to alpha.4: the repo's `-0` floor admits them and they predate the reader, as with `@mastra/pg` on main; `check:core-imports` passes for server, memory and code-sdk against this branch's core packed as 1.80.0 - [ ] `changed-tests` stays red: the deployer's core floor moves with no monorepo E2E change, as on #28417 - [ ] account rotation against real provider accounts: the `account-rotation`, `account-routing-targeted` and `pack-fallback` TUI E2E pass with HTTP 429s through the real AI SDK, not a live account - [ ] the web card after a reload with a real Codex `server_error` frame: fixtures only --- > ██████████████████ **source** `+304 −455` — net −151 > █████████████ **tests** `+399 −140` > ███ **changeset** `+112` > █ **package.json** `+20 −20` — core peer floor 1.80: server, memory, agent-builder and editor import the reader > █ **docs** `+10 −6` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 Different parts of the project could interpret the same provider error in different ways. This change gives them one shared reader so retries, account rotation, saved errors, and user-facing reports handle those errors more consistently. ## What changed - Added provider-error classification and extraction helpers to `@mastra/core/error`. They read provider codes, reasons, request IDs, and model IDs from responses and wrapped errors. - Updated retry handling, account rotation, observational-memory context-overflow recovery, server error serialization, Factory reports, and TUI error mapping to use the shared reader. - Saved errors can include provider codes and reasons. Persisted provider text is capped at 2,000 characters. Factory failed-run reports use the provider reason when available and cap the report text at 400 characters. - Configured retry matchers now run before built-in matchers. Built-in retry handling excludes credential and exhausted-quota failures. - Raised the minimum `@mastra/core` peer dependency to 1.80.0 across affected packages and deployers. ## Verification and limitations The supplied notes report that `check:core-imports` at the 1.80 floor is unverified. They also report that `changed-tests` remains red after the deployer core-floor change. Account rotation against real provider accounts and web-card behavior after reloading a real Codex `server_error` frame were not verified; unit tests and fixtures were used instead. Review finding counts are unavailable because no current review findings were supplied. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
0b7e61996e |
fix(memory): retry Observer replies that contain no observations (#28456) | ||
|
|
f02d3b711e |
refactor(lint): remove redundant returns while preserving exits (#28448)
Enable `no-useless-return` in the shared configuration. Remove 28 terminal bare returns, use 31 explicit switch breaks, and retain 12 value-returning contracts with explicit `undefined`. Preserve 13 intentional guards with local explanations, including index-skipping property reads and the platform availability probe. Reviewed every edit and checked that removed returns and new switch breaks have no effectful continuation before function exit, including cleanup through `finally`. All other source and emitted JavaScript syntax is preserved apart from formatting, and all 2,020 function return types and 3,280 variable binding types in the 59 affected source files are unchanged. The new rule reports zero violations across 15,106 files. CI lint and build, the local 57-task build, Factory build, core typecheck, formatting, and whitespace checks pass. All 756 focused tests pass across stream conversions, idle-loop cleanup, workflow validation, Factory, Factory UI, Playground UI, Datadog, Sentry, Cloudflare, and E2B. Both selected published-package monorepo E2E cases pass. The extended build case covers a missing application `.npmrc` and verifies successful builds with dependency installation both skipped and enabled. The DuckDB CI job also fails two existing capability-list assertions because their expected arrays omit `span-aggregate`. Both failures were reproduced on unchanged base ` |
||
|
|
ffc663d2ba |
fix(memory): treat empty Observer output as an observation failure (#27294)
Co-authored-by: Abhi Aiyer <abhiaiyer91@gmail.com> |
||
|
|
0f82b38ed4 |
refactor(lint): flatten lone if statements in else branches (#28431)
Enable `no-lonely-if` in the shared Oxlint config and flatten 38 lone
`if` statements in `else` blocks across 35 files. For example, `else {
if (condition) { ... } }` becomes `else if (condition) { ... }`.
Conditions, evaluation order, statements, comments, and local bindings
are preserved. No package filters or lint exceptions are added.
Strengthen the existing monorepo test to execute another named export
from an optimized dependency. Comparisons of TypeScript syntax, emitted
JavaScript, raw template text, comments, and local bindings pass for all
35 migrated files after normalizing only the redundant else blocks. The
rule clears all 15,094 scanned files.
[CI lint and formatting
pass](https://github.com/mastra-ai/mastra/actions/runs/40344286057/job/121647174331)
for `
|
||
|
|
c73d201c62 |
fix(memory): don't inject the continuation hint on threads with no observations (#26826)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
4303017f39 |
fix(memory): don't tell agents app-managed working memory can't be updated (#28441) | ||
|
|
68f311e0da |
fix(memory): stop Observational Memory from crashing on object-valued tool errorText (#28440) | ||
|
|
501b207b1b |
fix(memory): cancel observational memory with its owning run (#28158)
## Description Aborting an agent run left Observational Memory's Observer and Reflector calls running. They could keep step preparation blocked and write messages or thread metadata after the application handled the abort. The run's signal now reaches the work the run is waiting on: synchronous observation, synchronous reflection, buffered-observation and buffered-reflection activation, and context-overflow recovery in `processAPIError`. That work stops with the run. A cancelled synchronous observation leaves the run's messages unobserved, and the next cycle observes them. Async buffering and background reflection are detached work and keep running after the run aborts. Only the run's waits on them (`waitForBuffering`, activation waits, the buffered-reflection wait) stop. The jobs finish, commit, and stay tracked by `settled()`. A storage commit that has already started also finishes for an aborted caller, so the bookkeeping after it still runs; only a commit still waiting in the queue is skipped. Extracted thread values from a buffered reflection are written only after that reflection is stored. In `@mastra/core`, a run aborted while an error processor handles an API error now ends as an abort. Previously the processor's cancellation was swallowed as "no retry" and the step reported the original API error. ```diff run aborts - observer / reflector / waits keep running, then write + sync observation, sync reflection, activation: stop, no late writes + run's waits on background work: stop + async buffer, background reflection: keep running, tracked by settled() + queued commit: skipped; started commit: finishes before the caller returns ``` **Evidence.** `abort-signal.test.ts` holds provider streams and the commit queue open with latches, then aborts a real `Agent` run. Before the fix, the original cases failed (12 of the initial 15). Three new cases cover yoko's review: async buffer and background reflection keep running after the last run aborts, and `settled()` resolves only after they finish; a background reflection queued behind a held commit still stores its reflection and extracted values. All three fail on the previous head and pass now. A further case shows a cancelled sync observation's messages being observed by the next cycle. In `commit-queue.test.ts`, "waits for an op that already started when its caller aborts" fails on the previous `runOMCommit` and passes now. The core test `llm-execution-step.api-error-abort.test.ts` failed before its fix (`onAbort` never ran). After: the abort suite passes all 27 cases across 8 repeated runs. All 2,102 Observational Memory tests and the other 568 memory unit tests pass. Package lint passes, and the root TypeScript 7 compiler passes for `@mastra/memory`. Not checked: external storage-adapter integration, whose separate harness is not installed. Tests replace model responses at the provider boundary. ## Related issue(s) Fixes #27792. The issue also asked for background buffering to cancel; per review, async buffering and background reflection now deliberately keep running after an abort. Related: #28084 changes processor input-failure handling; this PR's processor signal forwarding stays outside those lines. #27868 adds the Slack stop-button trigger. ## Type of change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Code refactoring - [ ] Performance improvement - [x] Test update ## Checklist - [x] I have linked the related issue(s) in the description above - [ ] I have made corresponding changes to the documentation (if applicable) - [x] I have added tests that prove my fix is effective or that my feature works - [x] I have addressed all Coderabbit comments on this PR Created with GPT-6.1 Sol and Claude Opus 5.5 in T3 Code. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 When someone cancels an agent run, its memory work should stop holding up that run or saving unfinished results. Shared background work can continue for other runs, and the agent now reports cancellation correctly if it happens during API-error handling. ## What changed - Passed the run’s `AbortSignal` through Observational Memory observation, reflection, buffering, activation, and related waits. - Added cancellation checks around markers, metadata updates, buffering state, and queued commits. A queued commit can be skipped after cancellation; a commit that has started can finish. - Let canceled callers stop waiting for shared background work without stopping that work. `settled()` continues to track it. - Delayed persistence of extracted thread values until buffered reflection is saved. - Updated agent execution so cancellation during API-error processing ends the run as an abort. ## Validation The PR reports that `abort-signal.test.ts` passed all 27 cases across eight repeated runs. It also reports 2,102 Observational Memory tests and 568 other memory unit tests passed, along with package lint and the root TypeScript 7 compiler for `@mastra/memory`. The external storage-adapter integration harness was unavailable. Tests replaced model responses at the provider boundary. These results were not independently verified here. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5bd713b4f2 |
fix(core): hand each input-step processor the messages as they are now (#28369)
## What was wrong An agent runs its input processors as steps of one processor workflow. In `processInputStep`, each step handed the next processor the `messages` array it had been given, not the list as it stood after the processor ran. Observational memory removes observed messages from the list in `processInputStep` once a run is past its first step, and returns the `MessageList`. Every processor after it was therefore handed the array from before the cleanup. A processor that returned that array put every removed message back through `applyMessagesToMessageList`, which re-adds any returned message missing from the list. Returning the array is natural for a processor that only inspects or rewrites messages. The result: observational memory reported the messages removed and its own count dropped, while the prompt kept growing until the provider rejected it. ## Fix The `inputStep` step in `createStepFromProcessor` now hands on `messageList.get.all.db()`, unless the processor returned its own array. A returned array is still handed on as before. This matches the `input` and `outputStep` phases, which already hand on the current list when a processor returns it. The same change is made in the evented workflow's copy. ## Tests - **`input-step-removed-messages.scenario.test.ts`:** an aimock scenario on all four engines. One processor removes an earlier user message from step 1 on and returns the list, as compaction does. A second returns the `messages` it was handed. - The second request must not contain the removed message. - The second processor must be handed the list without it. - A control runs the first processor alone. - On `main`, the main case fails on all four engines and the control passes. - **`processor-step.test.ts`:** runs the `inputStep` step from both the workflow and the evented workflow builders against a real `MessageList`. The removed message must be gone from the step's output. On `main`, both fail. - **`second-input-processor-long-run.test.ts`** (`@mastra/memory`): a real `Agent` with real `Memory` and observational memory over 60 tool steps of about 2,800-character results, with a pass-through input processor after observational memory. The largest of the last 20 requests must stay under 80,000 characters. | Core build | Largest of last 20 requests | |---|---| | `main` | 281,356 chars | | This PR | 40,105 chars | 40,105 is the same as the run without the pass-through processor. - **Break check:** reverting the change in `workflow.ts` fails the scenario on all four engines. Reverting only the evented copy fails its `processor-step` case. - **Suites:** core processor tests, the input-processor, input-step, guardrail, memory and parity-processor scenarios, and `src/agent/__tests__`: 4,147 passed, 67 skipped. `@mastra/memory` unit tests against the rebuilt core: 2,669 passed, 1 skipped, 1 todo. Core and memory type checks pass. ## Not verified - **Other phases:** `outputStep` and `toolResult` were not changed. Their list-returning branches already hand on the current list. - **Memory integration tests:** the `packages/memory/integration-tests` suite was not run. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 This change makes each input processor see the messages that remain after earlier processors finish. That stops removed messages from coming back when a later processor passes along an older list. ## What changed - Updated both workflow builders to pass the current `MessageList` contents to the next processor when the current processor does not return its own `messages` array. - Preserved processor-returned arrays unchanged. - Added regression tests for removed messages and a long-run observational memory case. ## Validation - The PR reports 4,147 tests passed and 67 skipped. Memory unit tests report 2,669 passed, 1 skipped, and 1 todo. - Core and memory type checks passed. - The long-run test reports that the largest of the last 20 requests decreased from 281,356 characters on `main` to 40,105 with this change. - The `packages/memory/integration-tests` suite was not run. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
a282b3c70c |
fix(memory): stop keeping a run's observed request in the prompt (#28364) | ||
|
|
01e060c971 |
fix(memory): keep context-overflow recovery working for the rest of a long run (#28358) | ||
|
|
fe8c11d479 |
fix(memory): hide working-memory data on raw writes and edits (#28149)
## Description OM buffer sealing and raw message writes could retain working-memory tool arguments and tags that normal saves hide. Message edits bypassed the same filter. Apply the existing memory persistence projection to raw writes and the fields explicitly replaced by message edits. Use effective content for filtering decisions while preserving the caller's patch shape to avoid overwriting concurrent edits. Keep omitted content fields, observation markers, seal timestamps, ordinary tools (including ordinary partial calls), and state-signal audit calls. Only working-memory tool calls and tags are removed. No data migration. Semantic recall indexes each edited message from its projected effective text. Fields the edit sets come first; stored text is used only when the projection hid the edited text, so an edit that leaves only a `<working_memory>` block keeps the message searchable, while an edit that clears text still removes its vectors. Edits that touch neither `content.content` nor `parts` leave vectors alone (metadata-only edits used to delete them), and repeated ids in one call are indexed once. ```diff raw upsert / message edit - write unfiltered content + derive effective content for filtering + write sanitized fields from the original patch ``` Before/after: `timeout 5m pnpm --filter @mastra/memory exec vitest run src/persistence-projection.test.ts --maxWorkers=4 --bail=0` fails 5 cases on the original implementation and passes all 9 with the fix. The failures expose leaked raw/edit content and an unfiltered working-memory-only row. The shared suite also passes all 9 cases against LibSQL, including real Agent `generate` and `stream` turns crossing OM buffer seals. Agent end-of-turn histories were already clean on the base revision; raw writes and edits provide the failing regressions. The surrounding memory and OM files pass 733 tests (1 existing todo). The vector-index cases (working-memory-only edit, edited parts over a stale mirror, explicit clear, repeated id, metadata-only edit) run against a self-embedding mock vector store; the working-memory-only, clear, repeated-id, and metadata-only cases failed before their fixes. The full `@mastra/memory` unit suite passes (2664 tests). Independent GPT-6.1 Sol review finished clean in round 2 after fixing a concurrent-edit regression and legacy partial-call mirrors. A later Sol pass on the indexing change found the clear and repeated-id cases, both fixed. Left as is: when one call repeats an id, LibSQL and PostgreSQL keep the first patch while the index follows the last. That adapter inconsistency exists on main too. Package lint passes. The installed TypeScript 6.0.3 compiler crashes before checking source; a fresh copy of the same compiler passes both package configurations, as does workspace TypeScript 7. The changeset CLI failed in its installed fast-glob dependency, so the patch changeset was written manually. Not tested: other storage adapters and durable/evented/Inngest engines. The focused LibSQL run skips the harness's unrelated worker compilation and fastembed model warmup. The dependency build emitted runtime files but its core declaration step hit an existing `globalThis.crypto` error. ## Related issue(s) Fixes #27946. Related: #28143 changes the same memory source file for per-request OM opt-out; this PR keeps its implementation limited to persistence filtering. ## Type of change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Code refactoring - [ ] Performance improvement - [x] Test update ## Checklist - [x] I have linked the related issue(s) in the description above - [ ] I have made corresponding changes to the documentation (if applicable) - [x] I have added tests that prove my fix is effective or that my feature works - [ ] I have addressed all Coderabbit comments on this PR Created with GPT-6.1 Sol in T3 Code. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 Some message paths already hid working-memory details, but raw writes and edits could still save them. This change applies the same cleanup to those paths so recalled history stays consistent. ## Changes - Apply the persistence projection in `saveMessages` and `persistMessages`. - In `updateMessages`, project the stored message together with the patch. Preserve the patch shape so omitted fields are not sent in the update. - Preserve observation markers and `mastra.sealedAt` where required. Keep ordinary partial tool calls. - Use projected visible text for semantic-recall vector updates, including when edits change only working-memory content or metadata. - Add shared regression tests for writes, edits, recalled history, and Agent generation and streaming. The reported suite passes with in-memory storage and LibSQL. - Add a patch changeset for `@mastra/memory`. No data migration is included. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
75ffb145b4 |
test(memory): revise an activated assistant message in the activated-id test (#27893)
Co-authored-by: TylerBarnes <14190743+TylerBarnes@users.noreply.github.com> |
||
|
|
35806cb9b6 |
fix(memory): retain active user request when OM resumes on another worker (#28045)
> Opened by @abhiaiyer91 ## Summary Preserve the active user request when Observational Memory recreates an in-flight turn on another worker. Reconcile forward-progressed stored tool results with the recovered snapshot without replacing newer live background results. Shared-LibSQL two-process AIMock coverage exercises durable and evented recovery. Fixes #27614; response sealing in #27789 is separate. ## Recovery mechanism `ObservationTurn.toJSON()` omits `drainedPartKeys`. At step > 0 the processor restores request IDs from `messageList.getPersisted.input.db()`. When storage has a completed tool result missing from the snapshot, or a background task advances from snapshot `running` to stored `completed`/`failed`, it seeds `drainedPartKeys` with snapshot part keys and merges the stored response. The subsequent OM step recognizes the new result as unread. `MessageList` preserves an independently completed live background result when another tool in the same stored response advanced. This derives relevant part-level state; it does not serialize the original turn map. ## Proof On head ` |
||
|
|
12f5050847 |
fix(memory): honor per-request observational memory opt-out (#28143)
## Description
With observational memory enabled on a Memory instance, a per-request
`observationalMemory: false` or `{ enabled: false }` still attached its
processor. Normalization erased the override and restored the instance
default.
Preserve override presence when selecting the request config. Disabled
runs use message history without calling the observer or injecting
observations. The shared engine remains available for later and
concurrent enabled runs.
```diff
- select request config only when normalized override is truthy
+ select request config whenever an override was supplied
+ preserve a normalized disabled override as false
```
Before/after, from `packages/memory`:
`timeout 15m pnpm exec vitest run
src/processors/observational-memory/__tests__/mock-om-agent.test.ts
--maxWorkers=4 -t "Per-request observational memory options"`
- Before: failed because the observer was called despite an explicit
`false`.
- After: all 10 cases passed through real Agent.generate/stream, Memory,
and InMemoryStore. Cases cover both disable forms, omitted/true/object
options, history recall and persistence, and subsequent/concurrent
enabled runs.
Additional checks passed in a Boat Linux sandbox:
- `timeout 15m pnpm exec vitest run src/index.test.ts
src/processors/observational-memory/__tests__/mock-om-agent.test.ts
src/processors/observational-memory/__tests__/observational-memory.test.ts
--maxWorkers=4`: 737 passed, 1 existing todo.
- `timeout 15m pnpm check` and `timeout 15m pnpm lint` in
`packages/memory`.
- From `packages/memory/integration-tests`: `timeout 15m pnpm exec
vitest run src/with-libsql-storage.test.ts --maxWorkers=4 -t "persists
user then assistant with final text.*no bufferTokens"`: 1 passed.
Paid providers and the full external-storage/engine suites were not
tested; the regression substitutes only the model boundary and targets
this memory processor-selection path.
## Related issue(s)
Fixes #27814
## Type of change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [x] Test update
## Checklist
- [x] I have linked the related issue(s) in the description above
- [ ] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR
Created with GPT-6.1 Sol in T3 Code.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## ELI5
A request can turn off observational memory even when the `Memory`
instance has it enabled. The request still uses message history, and
later requests can use observational memory as usual.
## Changes
- `Memory.createOMProcessor` now distinguishes an omitted
`observationalMemory` option from an explicit override.
- Per-request `false` and `{ enabled: false }` disable observational
memory instead of falling back to the instance setting.
- Added `generate` and `stream` tests for disabled, omitted, and enabled
options, including later and concurrent requests.
- Added a patch changeset for `@mastra/memory`.
## Validation
The author reports that all 10 focused regression cases passed. The
author also reports 737 passed and 1 existing todo in the memory test
run, successful `pnpm check` and `pnpm lint`, and 1 passing libSQL
integration test.
Paid-provider tests and the full external-storage and engine suites were
not run.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
e05356995e |
fix(core): retain output processors on empty final steps with observational memory (#28046)
> Opened by @abhiaiyer91
Fixes #27920
## Outcome
When an agent uses observational memory and its final model step is
empty after a tool call, output processors can still append text. That
text is visible in the stream and saved in the assistant message for
durable and evented runs, matching the normal agent loop.
## Change
- Call
``messageList.enrichLastStepStart(`${currentModel.provider}/${currentModel.modelId}`)``
in the durable LLM path after materializing streamed messages and before
`processOutputStep`, guarded on both fields as
`llm-execution-step.ts:1762` does. That call stamps the trailing
unsealed assistant `step-start` with the executed model and moves the
message back into the `response` bucket when observational memory
drained it to `memory`, so output processors see the current step's
response.
- Add a real AIMock + Memory regression across normal, durable and
evented engines asserting processor response visibility, streamed text,
persisted assistant parts, and the persisted `step-start.model`.
## Proof
- Reverted the fix and rebuilt core: durable and evented each failed
with `processOutputStep` response count 0; normal passed. Restored fix
and rebuilt: all three passed.
- Reran the exact approved head after rebuilding core on October 22: the
three-engine regression passed 3/3.
- At `
|
||
|
|
461ee61237 |
fix: repair build and test failures surfaced by the release-PR run (#28139)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com> |
||
|
|
be5c9292ac |
fix(core): persist sub-agent suspensions after OM cold resume (#28050) | ||
|
|
db2a1a659e |
fix(core): keep durable memory refs when token budgets annotate history (#28049) | ||
|
|
5b936b65d2 |
fix(memory): keep observed messages sealed when another worker recovers an OM turn (#27887)
## What When a worker recovers an agent turn that another worker started (for example after a durable agent's worker crashes), Observational Memory could merge the rest of the reply into a message that had already been observed. The stored conversation then differed from a run that never crashed. This relands #27800 (reverted in #27856) as a complete fix. When a turn starts, OM checks whether the response message was already observed. If it was, OM seals that message and starts a new one for the rest of the reply. A message counts as observed when any of these holds: - Its id is in the record's observed-message list (sync observation). - It carries OM's seal. Buffering persists the seal before the Observer runs, so this still holds after the buffered chunk is activated. - It has a completed observation end marker. Sync observation persists that marker before it may reflect, so this still holds after a reflection starts a new record. Storage is unchanged. The check also requires the stored observed list to be an array; a non-array value used to crash the turn with `includes is not a function`. ## Tests - `packages/memory/integration-tests/src/cross-process/om-resume-prompt.test.ts`: "stores the same messages as the single-process run" flips from `it.fails` to `it` (durable and evented). - New `recovered-turn-observed-ids.test.ts`: - recovery after buffered activation; - recovery after a real sync observation that reflected; - non-array stored values (no crash, no substring match); - an unobserved response is left alone. - `ai-sdk-usage.test.ts`: `observe()` still picks up a later revision saved under an activated id, and content appended to an observed message after a reflection. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 When one worker continues a turn started by another worker, this change prevents new text from being added to a response that was already saved. It starts a new response message when the existing response was already observed. ## Changes - Detects observed responses by message ID, OM seal, or completed observation marker. - Validates that stored observed-message data is an array. - Persists the OM seal when recovery identifies a message by ID. - Preserves unloaded message parts when persisting the seal. - Fails the step when seal persistence fails so a retry can attempt sealing again. - Rotates the response ID before continuing an observed response. - Adds regression tests for recovery, buffering, reflection, malformed values, message revisions, appended content, and unobserved responses. - Enables the durable and evented cross-process recovery tests. - Adds a patch changeset for `@mastra/memory`. ## Validation The supplied context reports coverage for cross-process recovery, recovered-turn observed-message handling, message revisions, and content appended after reflection. Test execution results were not provided. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com> |
||
|
|
0e5d045be4 |
fix(core): stop the missing-memory warning for Observational Memory's internal agents (#27865) |