0
Fork 0
mirror of https://github.com/mastra-ai/mastra.git synced 2026-11-17 08:44:46 +00:00
Commit graph

512 commits

Author SHA1 Message Date
Ward Peeters
cd2a326988
fix(libsql): stop background task writes stalling shutdown, and fix backend timing flakes (#30504)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-16 13:18:29 +01:00
Tyler Barnes
adc0a70bf8
fix(memory, couchbase): check semantic recall scope in Memory and warn on ignored Couchbase filters (#30446) 2026-11-15 14:29:24 -07:00
Tyler Barnes
d1d736db29
test(memory): pin attachment token counts in the #29851 image test (#30415) 2026-11-15 14:26:54 -07:00
mastra-platform[bot]
c8a35ac624
fix(memory): keep Observational Memory local when the gateway has no memory API (#30414)
> Opened by @abhiaiyer91

> Opened by @factory-project:7d5175c3-3a20-42ac-aba9-756f183136f4

## Summary

Keep Observational Memory local for agents using
`mastra/<provider>/<model>` with gateway organization keys
(`msk-gateway-*` and `mskv2_*`), because those keys do not provide
gateway-managed memory. Gateway project keys (`msk_*`) continue to use
gateway-managed memory. Finish an open local turn before switching to
gateway-managed memory.

Credential ownership checks report local memory only for models that are
not on the Mastra gateway. A credential the gateway cannot resolve —
including one it reports as missing — is thrown instead of read as local
ownership: guessing local ownership can cause both the gateway and local
store to persist the same thread, because the model call resolves the
same credential again. The Observational Memory processor also
propagates ownership-check errors. Older models that cannot report
gateway memory ownership retain their previous behavior.

Docs explain the gateway-key distinction and the paired `@mastra/core`
and `@mastra/memory` upgrade. Project keys routed BYOK or with gateway
memory disabled remain outside this change's scope.

## Verification (head `67ca1ea9c4`)

- Core gateway-memory predicate tests: 11/11; memory gateway ownership
and persistence tests: 14/14.
- Full `@mastra/memory` suite: 119 files, 2,680 passed, 1 todo; core LLM
suite: 45 files, 550 passed, 2 skipped.
- Core and memory TypeScript checks passed.
- Reverting only the two behavior files makes the new regression tests
fail (after `a0a4d3f6`: core 2 failed/8 passed, memory 2 failed/11
passed; after `1981f9ca`: core 2 failed/9 passed, memory 1 failed/13
passed), confirming the tests detect the bug.

Fixes #30391

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

ELI5: Mastra has two kinds of gateway keys. One kind lets the gateway
save chat memory. The other kind does not. This change makes Mastra save
and recall chat messages locally when the gateway does not save them.

## Summary

This change fixes memory loss for agents that use a
`mastra/<provider>/<model>` model. Before this change, `@mastra/memory`
skipped local Observational Memory (OM) and message persistence for
every `mastra/` model. Messages were then not saved, and the agent could
not recall earlier turns.

The change decides gateway memory ownership by key type:

- Gateway project keys (`msk_*`) use gateway-managed memory.
- Gateway organization keys (`msk-gateway-*` and `mskv2_*`) use local
OM.
- Models that cannot report ownership keep their previous behavior.

Gateway-managed memory is not active when a project key routes to a BYOK
provider or when gateway memory is disabled. The docs describe this
distinction.

## Changes by area

**`@mastra/core`**
- `packages/core/src/llm/model/gateways/mastra.ts`: adds
`gatewayServesObservationalMemory(apiKey)`. It returns `false` for
missing, non-string, or empty keys and for `msk-gateway-` and `mskv2_`
keys. It returns `true` only for `msk_` keys.
- `packages/core/src/llm/model/gateways/index.ts`: re-exports
`gatewayServesObservationalMemory`.
- `packages/core/src/llm/model/router.ts`: adds
`ModelRouterLanguageModel.isGatewayMemoryManaged(): Promise<boolean>`.
It returns `false` for non-Mastra gateways. For Mastra gateways, it
resolves the route and auth, then checks the API key.
Credential-resolution errors propagate to the caller.

**`@mastra/memory`**
- `packages/memory/src/processors/observational-memory/processor.ts`:
uses `isGatewayMemoryManaged()` when the model provides it. If the model
does not provide it, the code falls back to the `mastra/` prefix check.
The result is stored per step.
- For gateway-managed input, the processor ends any open local turn and
skips local processing.
- For gateway-managed output, the processor ends any open local turn
before it returns.
- If ending a turn fails, the processor persists the unsaved messages
directly when possible. It then rethrows the original error.
- Context-overflow recovery remains skipped for gateway-managed memory.
- Ownership-check errors propagate. The processor does not treat them as
`false`.

**Documentation and changesets**
- `docs/src/content/en/docs/memory/observational-memory.mdx`: adds a
"Gateway models" section.
- `docs/src/content/en/models/gateways/mastra.mdx`: explains the key and
routing conditions for OM.
- `.changeset/tidy-geese-report.md`: minor bump for `@mastra/core`.
- `.changeset/wild-pumas-recall.md`: patch bump for `@mastra/memory`. It
tells users to upgrade `@mastra/core` and `@mastra/memory` together.
Older `@mastra/core` versions do not report ownership, so
`@mastra/memory` keeps its previous behavior for them.

**Tests**
- `packages/core/src/llm/model/router-gateway-memory.test.ts`: covers
project and organization keys, explicit-key precedence over the
environment key, models served by another gateway, credential lookup
errors, and lookups that fail once and then succeed.
- `packages/memory/src/gateway-memory-persistence.test.ts`: end-to-end
tests with real agent, router, memory, and in-memory storage.
Organization keys store messages and recall them. Project keys store no
messages locally. A failed ownership lookup rejects generation and
stores no assistant text.
-
`packages/memory/src/processors/observational-memory/__tests__/gateway-skip.test.ts`:
covers ownership results, legacy `gatewayId: 'mastra'` behavior,
rejected checks, and ownership changes during a run. It also covers
drain storage failures.

## Compatibility

- The change needs paired `@mastra/core` and `@mastra/memory` releases.
An older `@mastra/core` cannot report gateway memory ownership.
- Models that cannot report ownership keep the previous behavior.

## Testing

The following results are reported by the PR author. They were not
verified in this review, and no CI results were available.

- Core gateway-memory predicate tests: 10 of 10 passing.
- Memory gateway ownership and persistence tests: 13 of 13 passing.
- Full `@mastra/memory` suite: 119 files, 2,679 passed, 1 todo.
- Core LLM suite: 45 files, 549 passed, 2 skipped.
- Core and memory TypeScript checks: passing.
- The author reports that reverting the two behavior files makes the new
regression tests fail.

## Linked issue

Issue `#30391` reports that gateway-model messages are not saved or
recalled. The issue asks that messages be saved and recalled, or that
local processing be skipped only when gateway-side memory is active.
This change follows the second option for organization keys. It keeps
local processing for those keys. It skips local processing only for
project keys that the gateway manages.

The issue also reports a separate 422 error on the OpenAI-compatible URL
workaround. This change does not address that error.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform <284800079+mastra-platform@users.noreply.github.com>
2026-11-15 12:00:06 -08:00
Tyler Barnes
1072f7c626
fix(memory): reuse message embeddings across saves (#30429)
## Summary

Saving messages with semantic recall embeds each message's text and
writes a vector row for it. Since the message write path moved onto the
shared row builder, Memory stopped reusing embeddings for text it had
already embedded. The same text saved again (a repeated reply, a
re-saved message, a cloned thread) went back to the embedder every time:
three saves of the same 50 texts made 150 embedding calls instead of 50.

This restores bounded reuse on the write path without changing how rows
are written:

- Vectors are cached per Memory instance in a bounded LRU, keyed by each
message's exact prepared chunks, so chunk-size limits still apply and a
different chunking is a miss.
- The cache is emptied when the embedder or its options change.
- Failed calls, and responses without exactly one embedding per chunk,
are never kept. Concurrent writes of the same chunks share one call.
- Reused vectors are not counted again in reported embedding usage.
- Only vectors are cached. Every message still gets its own row with its
own id and metadata, so there are no duplicate recall rows.

The memory performance benchmark previously timed recalls without
checking that anything had been indexed, so a writer that skipped
embedding would pass it. It now checks that every message has a vector
row and that each thread's semantic recall returns its own messages
before timing.

## Test plan

- New `semantic-recall-embedding-reuse.test.ts` (10 tests): repeated
text, re-saved ids, usage on reuse, long-text reuse, invalid responses
and changed embedder options fail before this change. Edited text,
failed embeds, embedder swaps and raw-string content are covered as
controls.
- `@mastra/memory` unit suite: 2,689 pass + 1 todo. Typecheck, lint and
format clean.
- Real file-backed LibSQL, 3 saves × 50 messages: repeated text 150 → 50
embedding calls, re-saved ids 150 → 50. Stored rows unchanged (151 / 51
/ 451 for long chunked text). Unique, raw-string and long-text call
counts unchanged.
- LibSQL performance benchmark (2,500 messages, FastEmbed) passes with
the new check. The check fails when vector writes are disabled or recall
returns nothing.
- Local FastEmbed: repeated text ~7.6s → ~2.6s; unique text unchanged.

Ref #29826


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

ELI5: When a message is saved, Memory turns its text into numbers (an
embedding). If the same text was already turned into numbers, this
change reuses them instead of asking the embedding service again. Each
message still gets its own stored row.

## Overview

This change adds a bounded cache of message-chunk embeddings to
`Memory`. The cache applies only to the message-vector write path. The
change also adds checks to the LibSQL performance test, a patch
changeset for `@mastra/memory`, and a new test file.

Diff scope: 4 files changed, 350 lines added, 11 removed.

## Changes

**`packages/memory/src/index.ts`** (+55 / -11)

- Adds `chunkEmbeddingCache`, an LRU cache bounded by
`DEFAULT_EMBEDDING_CACHE_MAX_SIZE`. It holds promises of vectors only.
Keys are a hash of `JSON.stringify(chunks)`.
- Adds `chunkEmbeddingCacheOwner`. It records the embedder and options
that produced the cached vectors.
- Adds `embedMessageChunks(chunks)`, which:
  - Snapshots `this.embedder` and `this.embedderOptions` before hashing.
- Calls `embedValues` with the snapshot and skips the cache if the
embedder or options change during hashing.
- Clears the cache when the owner record differs from the snapshot. The
clear happens at the next miss, not when the embedder changes.
  - Returns cached vectors on a hit. A hit returns no usage data.
- On a miss, calls `embedValues` and stores the pending promise.
Concurrent writes with the same chunks share that promise.
- Evicts the entry when the call fails or returns a number of embeddings
that differs from the number of chunks.
  - Returns usage only for the call that reached the embedder.
- Changes `embedValues` to accept an optional `{ embedder,
embedderOptions }` argument. The default is `this`, so other callers
keep the current behavior.
- Changes the message-vector row builder to call `embedMessageChunks`
instead of `embedValues`. The diff for this file shows no change to the
query or content embedding paths.

**`packages/memory/src/semantic-recall-embedding-reuse.test.ts`** (+267)

- Adds a new test file. It covers reuse across saves, one row per
message on resave, re-embedding of edited or distinct text, usage
counting, non-reuse of failed or mis-sized results, invalidation on
embedder or option change, an embedder change during an in-flight save,
raw-string content, and chunking within the embedder input limit.
- The number of tests in this file is not verified from the diff. The PR
reports 10.


**`packages/memory/integration-tests/src/performance-testing/performance-tests.ts`**
(+23)

- Imports the `MastraVector` type.
- Before timing, checks that indexed vector rows are at least
`threadCount * messagesPerThread`.
- Before timing, runs a thread-scoped semantic recall for each thread.
Checks that results are nonempty and belong to that thread.
- The timed query measurements and p95 threshold are not changed.

**`.changeset/quiet-owls-reuse.md`** (+5)

- Adds a patch changeset for `@mastra/memory`. It describes reuse of
vectors for repeated or re-saved text and the exclusion of reused
vectors from reported usage.

## Reported results

These results come from the PR description. They are not verified by
completed checks on the current head.

- Three saves of 50 messages with repeated text: embedding calls fell
from 150 to 50. Stored row counts are reported unchanged.
- Repeated-text FastEmbed time: about 7.6 s before, 2.6 s after.
- Unique, raw-string, and long-text call counts: reported unchanged.
- `@mastra/memory` unit tests: 2,689 passing, 1 todo.
- Typecheck, lint, and format: reported clean.
- LibSQL benchmark with 2,500 messages: reported passing.

## Not verified

- Test, typecheck, lint, format, and benchmark results. No completed
check on the current head was available.
- The test count in the new test file.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com>
2026-11-14 18:29:53 -08:00
Tyler Barnes
9eae9680d2
fix(memory): serialize tool results half as often in long Observational Memory runs (#30400) 2026-11-14 09:36:02 -07:00
Tyler Barnes
8ac88228d6
fix(core): remove the 10 ms pause after memory saves and keep same-millisecond turns in order (#29148)
## What

Removes the 10 ms pause `MessageHistory` waited after every memory save,
and fixes the ordering bug that pause had been hiding.

Stacked on #29142 (keep a forwarded queued message until the owning run
accepts it). Retarget to `main` once that merges.

## Why

The pause was added so the next message's timestamp would always land
after the saved one. Without it, a new turn's input can be stamped in
the same millisecond as the previous turn's stored answer. Sorting on
equal timestamps then puts the stored answer last, and the new turn's
response merges into that old answer. Under load,
`signals-pending-cancellation` "answers the follow-up after
upstream/thread cancellation" failed 2 of 12 runs this way.

Every memory-backed call also paid at least 10 ms per save.

## How

- `message-history.ts`: drop the post-save `setTimeout`. Stored history
is now added in a single `add(array, 'memory')` call, as semantic recall
already does.
- `MessageList`: after a memory add, any live message whose timestamp
equals a stored one moves 1 ms later, skipping other stored times, and
its signal `createdAt` is kept in sync. Live messages that tie only with
each other keep their timestamps. The pass runs once per add, in O(n log
n).

## Test changes (setup only, no assertion changes)

- New `input-after-stored-history-tie.test.ts`: a tied new turn stays
after the stored answer (fails without the fix), and same-ms live
messages with no stored tie are left untouched.
- `signals-pending-cancellation` lease-loss test now waits for the
follow-up run to register before it vanishes. Without the pause, the run
could vanish before registering, and the message was delayed past the
test window. The test still fails against a runtime without the lease
watch.
- `provider-history-compat` fixture: explicit ascending `createdAt`
values in place of four `new Date()` calls that landed in the same ms.
- `background-task-call-id-reuse` scenario: waits for the last
background result instead of relying on the pause.

## Testing

- Full core unit lane (`--project unit:packages/core`, Node 24, 5 CPUs):
1,560/1,560 files, 25,295 passed, 0 failed.
- Signals, message-list and call-id-reuse suites: 3 repeated runs, 1,169
passed each.
- `tsc --noEmit` and oxlint are clean.

Not verified: real multi-process pubsub (Unix socket/Redis), and stores
other than core's in-memory and test stores.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

**ELI5:** When a chat app saves a reply, it used to wait 10 ms. Two
messages could also get the same timestamp and sort in the wrong order.
This change removes the wait and keeps messages in the correct order,
even when timestamps match.

## Overview

This change removes the 10 ms pause after each memory save in
`MessageHistory`. It also changes how `MessageList` orders live messages
whose timestamps match stored history. Without this change, a new turn's
response could merge into the previous turn's stored answer. The change
also covers a forwarded-idle case, where a run can start after another
process's lease disappears. The runtime clears the forwarded run's
active IDs only when the stream ID and run ID still match the values it
read before the lease check.

## Changes by area

**Memory save path**
- `packages/core/src/processors/memory/message-history.ts`:
`processInput` adds all chronological history in one `add(array,
'memory')` call. It previously added each message separately.
`processOutputResult` no longer waits 10 ms after it persists messages.

**Message ordering**
- `packages/core/src/agent/message-list/message-list.ts`:
`MessageList.add` runs `moveLiveMessagesPastStoredTies` after it
processes inputs. The pass runs once per add.
- When memory messages exist, the pass marks them as stored.
- It moves live messages that share a stored timestamp. It also moves
live messages that would no longer follow a moved live message.
- It skips other stored times when it chooses new timestamps.
- It orders placements by each message's own timestamp and arrival
order.
- It preserves anchored signals. It updates matching signal `createdAt`
values and the ordering watermark.
- It re-queues moved persisted messages, so the moved timestamps are
saved again.
- It sorts the list only when a message moved.
- Live messages that tie only with each other keep their timestamps.

**Remote run forwarding**
- `packages/core/src/agent/thread-stream-runtime.ts`: The forwarded
idle-signal lease checker snapshots the active stream ID before it reads
the lease.
- After consecutive checks show that the target run no longer holds the
lease, and its unadmitted signals are requeued, the runtime clears that
run's active run ID and stream ID.
- It clears them only when the run has no local record, the active
stream ID is unchanged from the snapshot, and the active run ID still
matches.
- It then drains pending idle signals.

**Tests**
-
`packages/core/src/agent/__tests__/signals-pending-cancellation.test.ts`:
Adds and updates coverage for follower idle messages. The cases cover
runs that end, are superseded, or lose their lease before delivery. The
vanished-owner tests wait for the follow-up run's registration to reach
subscribers before they end the run. A new test checks that a follower
with queued work does not start local execution after it unsubscribes
and sees the lease disappear. Another test checks that a run that takes
the thread during an in-flight lease read stays active. A further test
checks that a remote run that starts and ends during that read does not
block the drain of queued work.
-
`packages/core/src/agent/message-list/tests/input-after-stored-history-tie.test.ts`
(new): Covers ties between stored history and live input, response, and
context. It checks reminder placement, deduplication under turn sources,
and turn order when stored history arrives in batches. A save test
checks that a moved input is returned for saving with its updated
timestamp.
-
`packages/core/src/loop/test-utils/aimock/scenarios/background-task-call-id-reuse.scenario.test.ts`:
Waits after each turn until that turn's background result appears in
recalled memory.
- `packages/core/src/processors/provider-history-compat.test.ts`: Gives
transcript messages explicit, sequential timestamps. The test still
checks that API-error repair removes the earlier empty signed block.
-
`packages/memory/src/processors/observational-memory/__tests__/message-utils.test.ts`:
Gives stored messages timestamps earlier than the live input. It expects
observable order `memory-1`, `input-1`, `response-1`. It still checks
that the synthetic continuation is excluded from observable messages.

**Release notes**
- `.changeset/memory-save-no-pause.md`: Adds a patch changeset for
`@mastra/core`. It documents the removed pause, the same-millisecond
ordering fix, and the queued-message case.

## Public API

No exported or public declarations change.

## Scope

9 files changed. 571 lines added and 26 removed.

## Validation

The PR author reports these results. This summary does not verify them.
- Core unit lane (`--project unit:packages/core`, Node 24, 5 CPUs):
1,560 files and 25,295 tests passed.
- Signals, message-list, and call-ID-reuse suites: 1,169 tests each
passed across three runs.
- `tsc --noEmit` and oxlint: clean.
- Not verified: real multi-process pubsub (Unix socket or Redis), and
stores other than core's in-memory and test stores.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com>
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-13 14:33:04 -07:00
Leo
3c1b528b6e
fix(memory): block irrelevant replace of shared working memory at default scope (#30306)
## Summary

With working memory `scope` left at its default (`'resource'`), the
experimental vNext update-working-memory tool let one thread erase a
line another thread wrote to shared memory by calling it with
`updateReason: 'replace-irrelevant-memory'`. The tool resolved `scope ||
'resource'` for its ID checks, but the guard compared the raw config
value, which is `undefined` by default. The guard now uses the resolved
scope, so default and explicit `'resource'` behave the same, and the
refusal says the memory is shared instead of the misleading "no
searchString was provided".

**Why this way:** reusing the `scope` already resolved a few lines above
keeps one source of truth, rather than repeating `|| 'resource'` in the
guard (the issue's preferred option too).

Door: two-way · Risk: low (composes `@mastra/memory` vNext working
memory tool) · Cleanup: none

## Evidence

New test file
`packages/memory/src/tools/working-memory-vnext-scope.test.ts` drives
the real tool from `Memory.listTools()` with a real `Memory` and
`InMemoryStore`, writing from two threads and reading back from both.

- **Before** (origin/main's `working-memory.ts`):
  ```
× another thread can't erase shared resource memory as irrelevant (scope
left at its default)
expected { success: true, … } to match object { success: false, … } //
thread B replaced Apollo
× another thread can't erase shared resource memory as irrelevant
(scope: 'resource' set explicitly)
      // refused, but with "no searchString was provided"
✓ still lets a thread clarify an existing resource-scoped line when
scope is left at its default
✓ with scope: 'thread', a thread can replace its own irrelevant memory
and other threads are untouched
  Tests  2 failed | 2 passed (4)
  ```
  **After:**
  ```
✓ another thread can't erase shared resource memory as irrelevant (scope
left at its default)
✓ another thread can't erase shared resource memory as irrelevant
(scope: 'resource' set explicitly)
✓ still lets a thread clarify an existing resource-scoped line when
scope is left at its default
✓ with scope: 'thread', a thread can replace its own irrelevant memory
and other threads are untouched
  Tests  4 passed (4)
  ```
- Neighboring suites `src/tools/working-memory.test.ts` and
`src/index.test.ts` with the new file: 215 passed. `tsc --noEmit` for
`@mastra/memory` clean; `oxfmt --check` and `oxlint --deny-warnings`
clean on changed files.

One behavior note: the issue describes the guard as turning the update
into an append. On main, explicit `scope: 'resource'` actually refuses
the call (the guard cleared `searchString`, then the next check rejected
it) and nothing is written. This PR keeps that refusal and makes the
default match it; only the reason text changes.

## Merge Danger

Two-way: a one-condition change plus a clearer refusal message in the
experimental vNext tool; the riskiest part is that default-scope agents
that relied on `replace-irrelevant-memory` succeeding will now get a
refusal, which is the documented intent.

**Blast radius:** agents using working memory `version: 'vnext'` with
resource scope (default or explicit). The stable tool and thread scope
are unchanged. Open PR #29209 also edits `working-memory.ts`, in
different hunks (lines 1-90; this change is at line 426).

Closes #30287

## Council review

Rounds: 3

| Seat | Round 1 | Round 2 | Round 3 |
| --- | --- | --- | --- |
| Claude Opus 5.5 | not run | 9/10 | 10/10 |
| GPT-6.1 Sol | 9.8/10 | 10/10 | 10/10 |
| Grok 4.7 | empty seat | empty seat | empty seat |
| Kimi K3.1 | not run | not run | 10/10 |
| DeepSeek V4.1 Flash | not run | not run | 9/10 |

Created with Claude Opus 5.5 in T3 Code.
🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

**ELI5:** Working memory can be shared by every chat that one user has.
Before this fix, if the scope was not set, one chat could delete a line
that another chat had saved. Now the default scope gets the same
protection as the explicit shared setting.

## Summary

This change fixes the guard for `replace-irrelevant-memory` in the
experimental vNext working-memory tool. The guard read the raw `scope`
config value. When `scope` was unset, that value was `undefined`, so the
guard did not run. The guard now uses the resolved `scope` value, which
includes the `resource` default.

## Behavior

- **Default or explicit `resource` scope:** A
`replace-irrelevant-memory` call that includes a `searchString` is
refused. The tool returns a specific failure. The failure tells the
agent to use `clarify-existing-memory` or `append-new-memory`. Before
this change, the tool cleared the `searchString` and continued to later
validation.
- **`thread` scope:** No change.
- **Stable (v1) working-memory tool:** No change.
- **`clarify-existing-memory`:** No change. The linked issue excludes
this tool.

The linked issue expected the replacement to become an append. This
change keeps the refusal for `resource` scope. It makes the default
scope match explicit `resource` scope.

## Changed files

- `packages/memory/src/tools/working-memory.ts` (+6/-3): Uses the
resolved scope in the resource-scope check. Returns the specific failure
for `replace-irrelevant-memory` with a `searchString`.
- `packages/memory/src/tools/working-memory-vnext-scope.test.ts` (+95):
Adds tests for:
  - refusal under default and explicit `resource` scope;
  - clarification of existing shared resource memory;
  - replacement isolation under `thread` scope.
- `.changeset/nine-boxes-pump.md` (+5): Adds a patch changeset for
`@mastra/memory`.

No exported or public declarations change.

## Verification

The PR author reports the following results. This summary did not rerun
them.

- Four new tests pass.
- 215 tests pass across the new test file and the neighboring
`src/tools/working-memory.test.ts` and `src/index.test.ts` suites.
- `tsc --noEmit`, `oxfmt --check`, and `oxlint --deny-warnings` report
no issues.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:54:36 -05:00
Tyler Barnes
a21d692444
fix(memory): stop the Observer crashing on message parts without text (#30272) 2026-11-12 21:08:55 -07:00
mastra-platform[bot]
56f654fc0d
fix(core): name the agent in the memory missing-storage error (#30164)
Co-authored-by: Abhi Aiyer <abhiaiyer91@gmail.com>
2026-11-11 22:50:45 -08:00
mastra-platform[bot]
ebbea50d38
fix(memory): clarify missing storage for unregistered agents (#29972) 2026-11-10 16:15:37 -08:00
Leo
08e677fdbe
fix(memory): keep observation groups whole in buffered reflection (#29412)
## Description

With observational memory's `retrieval` on, observations are wrapped in
`<observation-group>` tags. Buffered reflection chose how many leading
lines to reflect from a token estimate alone, so the cut could land
inside a group. The reflector then got the group's opening tag, and
activation kept the rest of its bullets plus a `</observation-group>`
with no opening tag. The agent saw a stray closing tag, and those
observations lost the `range` recall uses to find their source messages.

`doAsyncBufferedReflection` now passes the cut through a new
`alignLineCutToObservationGroups` helper before anything uses it, so the
stored `reflectedObservationLineCount`, the token estimate and the
compression target all agree on a group boundary. Storage adapters slice
on that stored count, so none of them change.

```diff
 cut = floor(activationPointTokens / avgTokensPerLine)
+if cut lands inside a group
+  move it back to before that group's opening tag (and its blank separator)
+  if nothing precedes that group, move it forward past the group's closing tag
 reflect lines[0..cut), keep lines[cut..] verbatim on activation
```

Observations without group tags, and cuts that already fall between
groups, are unchanged.

## Evidence

New test `buffered-reflection-group-boundary.test.ts` drives the real
path: `Memory` with `InMemoryStore`, a turn that buffers a reflection,
then a turn that activates it. Only the reflector model is mocked. It
checks what the reflector received, the stored cut, and the activated
observations (balanced tags, each kept group's id, range and bullets).

- **Before** (fix reverted, `reflector-runner.ts` from `main`): `Tests 2
failed | 1 passed (3)`
- the reflector received `group 3 observation 0-3`, the first part of a
group whose closing tag stayed active
- a first group bigger than the budget was cut at line 24 instead of
after its closing tag at 31
- the no-group-tags case passes both before and after, which is
expected: plain observations keep their cut
- **After:** `Tests 36 passed (36)` across the new file and
`observation-groups.test.ts` (which gains unit cases for the helper:
cuts between groups, ungrouped lines before the first group, indented
tags, an unclosed last group, no tags).
- Guards: 10 related observational-memory test files, `Tests 668 passed
| 1 todo`; `tsc --noEmit` and `oxlint --type-aware` on `@mastra/memory`
pass; Prettier clean.

Not run: storage adapter integration suites (pg, libsql, etc.). They
only consume the stored line count, which this change fixes at the
source.

## Merge Danger

**Door:** two-way. Reverting the commit restores the old cut; nothing
persisted changes shape.

**Blast radius:** `@mastra/memory` buffered reflection with `retrieval`
on. The cut can now reflect slightly fewer lines than the estimate (back
to a group boundary), or a whole first group that is larger than the
estimate. Records already damaged by this bug are not repaired.

Overlap: #28293 (`fix/27262-buffered-reflection-values`) edits the same
`doAsyncBufferedReflection` function. This PR touches different lines
(one import and the `linesToReflect` assignment); a local `git
merge-tree` of #28293 with this branch adds no conflict beyond the two
that #28293 already has with `main` in the activation code.

## Related issue(s)

Fixes #28445

## Type of change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [x] Test update

## Checklist

- [x] I have linked the related issue(s) in the description above
- [ ] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR

## Council review

Rounds: 3

| Seat | Round 1 | Round 2 | Round 3 |
| --- | --- | --- | --- |
| Claude Opus 5.5 | not run | 9/10 | 9/10 |
| GPT-6.1 Sol | 10/10 | 10/10 | 10/10 |
| Grok 4.7 | empty seat | empty seat | empty seat |

Round 2 findings applied: opening tags are matched after trimming, like
closing tags; a long test comment was rewrapped.

Created with Claude Opus 5.5 in T3 Code.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## ELI5

Buffered reflection could split a group of related observations in half.
This change keeps each group together, so the remaining observations
stay properly organized and linked to their source messages.

## Summary

- Aligns the buffered reflection cut to an `<observation-group>`
boundary.
- Moves a cut back before the group when earlier content exists. If the
group starts the observations, moves the cut forward past that group.
- Leaves cuts unchanged when they are outside a group or when
observations have no group tags.
- Adds regression tests for boundary alignment, activation, preserved
group ranges, and ungrouped observations.
- Adds a patch changeset for `@mastra/memory`.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 18:14:13 -05:00
shreyas-mastra
6644a82748
fix(memory): lock vNext working memory updates by resolved scope (#29477) 2026-11-10 08:51:39 -08:00
Jamie Nichols
ec8d9b8e7b
fix(memory): require the working memory section in single-thread observer calls for Markdown working memory (#26203)
Fixes #25350

## Problem

When observational memory manages Markdown working memory
(`observation.manageWorkingMemory: true` without a working memory
schema), the Observer updates working memory by writing a
`<working-memory>` section in its own reply. The prompt called that
section optional. In the issue's repro the Observer noticed the user's
preference change every time but often left the section out, so working
memory kept the old value. Nothing recorded that the update was missing,
so a dropped update looked the same as "nothing changed".

## Fix

- The working memory extractor is now marked as required for
single-thread Observer calls. The prompt tells the Observer to always
write `<working-memory>`, and to write `UNCHANGED` inside it when
nothing changed.
- `UNCHANGED` is skipped rather than stored, in both the tag form and
the `<extracted-values>` JSON form. Small variations like `Unchanged.`
or `` `UNCHANGED` `` are skipped too.
- If the section is missing, an entry is added to `extractionFailures`
and the Observer logs a warning naming the section.
- If a reply has nothing else in it, such as a refusal ("I'm sorry, I
can't help with that"), it is still retried. The check that decides
whether a reply has content ignores the missing-section failure. Without
that, every refusal would have counted as content and been committed,
and the observation cursor would have moved past messages nobody
observed.
- Batched multi-thread Observer calls keep the section optional. One
reply covers several threads there, so requiring a per-thread section
would record false failures.
- Working memory with a schema doesn't change. It uses a separate
structured extraction call.

`required` is not an `Extractor` config option. Only the working memory
extractor turns it on, through a constructor argument marked
`@internal`, the same way the existing `internal` flag works.

## Testing

- `extractor.test.ts` has tests for the required prompt wording, a
failure recorded when the section is missing, `UNCHANGED` and close
variants skipped in the tag form, and exact `UNCHANGED` skipped in the
JSON form, and the section staying optional for batched calls and for
other extractors. `required-section-logging.test.ts` checks the warning.
Run against main's source, the prompt test fails because main still
calls the section optional, and the logging test fails with `expected
undefined to deeply equal [ 'mood' ]`.
- `refusal-observer-output.test.ts` adds three end-to-end cases with
Markdown working memory turned on, using a real Agent and Memory against
a mock model server. A refusal doesn't advance `lastObservedAt`. A
refusal is retried and the next good reply is observed. A reply that has
observations but no `<working-memory>` commits without a retry. Without
the content-check change the first two fail: `lastObservedAt` is set
after the refusal, and the retry test sees one Observer call instead of
two. Both pass with the change.
- The `@mastra/memory` unit suite passes (2645 passed, 1 todo), and `tsc
--noEmit` and `oxlint` are clean.

I didn't re-run the issue's live-model repro, because it needs a real
model.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## ELI5

The Observer can notice a change to Markdown working memory but fail to
save it if it leaves out that section. This change requires the section
in single-thread calls and gives the Observer a clear way to say that no
update is needed.

## Changes

- Require the `<working-memory>` section in single-thread Observer
calls. The prompt instructs the Observer to write `UNCHANGED` when no
update is needed.
- Treat `UNCHANGED` and common formatting variants as no update, rather
than storing them.
- Record a missing required section as an extraction failure and log a
warning with the failed section slugs and thread ID.
- Keep required sections optional in batched multi-thread calls.
- Exclude missing-required-section failures from the check that decides
whether an Observer refusal should be retried.
- Add tests for extraction, logging, refusal handling, and batched
calls.

**Validation:** Current-head test, build, and CI results are not
available.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Jamie Lee Nichols <10267569+CodingButter@users.noreply.github.com>
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-10 06:56:14 -08:00
Arthur Fiorette
c9140d2d4d
fix(memory): write one semantic recall row per message on every agent engine (#29826)
## Description

Durable and evented agents wrote two or three semantic recall vector
rows for each message. Two writers embedded the same messages:

- the `SemanticRecall` output processor, which keys rows by message id;
- `Memory.saveMessages`, which the save queue calls from the durable
input save and the finalize flush, and which wrote rows without ids, so
the store generated random ones.

The duplicates took up `topK` slots, and the embedder ran twice per
message.

The two writers also built rows differently. Only `saveMessages` chunked
long text, stripped working-memory tags and stored thread metadata.
Neither writer can simply be removed: the processor serves agent runs,
while `saveMessages` serves direct callers (the HTTP route, the client
SDK, user code) and the save queue. So this PR gives both writers one
row builder and one id scheme.

Changes:

- Core: `buildMessageEmbeddingRows` and `upsertMessageVectors`
(internal, in `@mastra/core/memory`) now decide the row format: text
extraction, chunking, metadata and row ids. A message's first chunk
keeps its message id, which is the id the processor already wrote, so
existing rows get replaced rather than duplicated. Later chunks get a
UUID derived from the message id and chunk index. Every message-vector
upsert now sends ids. Each writer passes its own embed function, so
embedding calls, retries and caches work as before.
- Core: `MessageList` keeps a per-run record of embedded messages (index
and message id to text hash) and serializes it with the list. The save
queue passes the record to `saveMessages`, and the durable input save
copies it back from its cloned list, so the second writer skips what the
first one embedded into the same index. If a message's text has changed
since, it is embedded again. A processor writing to a custom
`semanticRecall.indexName` and `Memory` writing to its default index
each still get every message.
- Core: new `semanticRecall.maxChunkChars` option, default 16,384
characters (about 4,096 English tokens). The limit is in characters
because each provider uses its own tokenizer. The processor now chunks
long messages the way `saveMessages` already did, instead of embedding
them as one input, which failed past the embedder's limit.
- Memory: `saveMessages`, `updateMessages`, cloned-message embedding and
`updateThreadResourceId` all write through the builder. Edits and
resource moves embed before deleting the old rows, so a failed embedding
call leaves them in place. Thread metadata is read only when rows will
be written. The `@mastra/core` peer range is now `>=1.83.0-0`.
- FastEmbed: models declare `maxInputTokens: 512`, so recall chunks fit
their window.

Existing rows are not rewritten. Rows `saveMessages` wrote before this
change keep their random ids. Rows the processor wrote already use the
message id, so new writes replace them. An evented or Inngest finalize
that runs on another worker doesn't see the input save's record, so it
embeds the user message once more into the same row.

```ts
const memory = new Memory({
  options: {
    semanticRecall: { topK: 3, messageRange: 1, maxChunkChars: 6000 },
  },
})
```

Tests:
`packages/memory/integration-tests/src/semantic-recall-embedding-rows.test.ts`
runs plain, durable and evented agents against a LibSQL vector index,
with thread scope, resource scope and a custom index. It checks:

- each persisted message has exactly one row under its own id;
- no row is written twice in a run;
- the fact is recalled once.

On `main` the durable and evented cases fail with 30 rows for 12
messages. With the record not passed to `saveMessages`, the write-once
check fails on durable and evented. Memory cases cover:

- record skips;
- chunk size taken from the embedder's input limit;
- an edit that shortens a long message;
- an embedder failure during an edit.

Unit tests cover the builder and the `MessageList` record.

Six cases in `agent-memory.test.ts` fail because their recorded fixtures
no longer match the requests. They fail the same way on `main`.

## Related issue(s)

Related to #29352. Agent runs now store thread metadata on vector rows,
which the read-side fix in that issue needs. This PR does not change the
read side.

## Type of change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [x] Test update

## Checklist

- [x] I have linked the related issue(s) in the description above
- [x] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

The PR helps agents avoid saving the same message to search more than
once. It also splits long messages into smaller pieces so search can use
more of their content.

## Summary

- Shared helpers extract message text, split it into chunks, build
vector rows, and upsert them. The first chunk uses the message ID when
it is a UUID or small integer; other IDs and later chunks use stable
derived IDs.
- `SemanticRecall` and `Memory.saveMessages()` use the shared
row-building path. Agent runs track embedded text by message and index,
so writers can skip unchanged messages.
- `semanticRecall.maxChunkChars` sets the chunk limit. It defaults to
16,384 characters and can use the embedder’s declared `maxInputTokens`
when unset. FastEmbed models declare a 512-token input limit.
- Memory updates and thread-resource moves build replacement rows before
deleting existing vectors. Re-saving shorter text can leave older chunk
rows searchable; the documentation describes this behavior.
- Tests cover row counts, scopes, custom indexes, chunking, repeated
saves, edits, and embedding failures. The supplied description reports
six fixture-based `agent-memory.test.ts` failures that also occur on the
base branch. Completed check results are not available.
- Documentation and peer dependency minimum versions were updated.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-10 00:03:33 -03:00
Ward Peeters
791cf606e9
fix(core): restore lease process marker removed by temp cleaners (#29723)
Co-authored-by: mastracode
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-09 15:49:40 +01:00
Leo
36bbc29129
fix(core): stop re-adding sealed messages reloaded from memory (#29537)
## Description

When an agent with memory resumed a run, a sealed assistant message (for
example one observational memory had observed) could appear twice in the
prompt. Memory cleans a message when it saves it: it trims text, strips
`<working_memory>` tags and removes `updateWorkingMemory` tool steps.
`MessageList.addOne` compared the recalled copy with the live sealed
message, saw a difference, and appended the copy under a new Mastra id
while it kept the same OpenAI item ids. The next Responses request then
failed with `Duplicate item found with id rs_...`. A sealed message
whose text ends in a newline was enough to trigger it.

The memory dedupe check in `MessageList.addOne` now also treats a stored
copy as the same message when it matches the live one after that same
save cleanup (`isPersistedCopy` in
`packages/core/src/agent/message-list/conversion/utils.ts`, which reuses
core's existing `working-memory-utils` helpers). Observational memory's
buffering saves sealed messages through `MessageHistory`, which first
runs core's `filterMessagesForPersistence` and drops streaming
`partial-call` tool parts (they can arrive in client-sent history, such
as an AI SDK `dynamic-tool` part still in `input-streaming`). So
`isPersistedCopy` also compares the live message after that filter.
Memory's direct `saveMessages` keeps partial calls, and those copies
still match too.

```diff
 addOne(message, 'memory')
   for each existing message with the same id
-    if messagesAreEqual(existing, incoming) -> skip
+    if messagesAreEqual(existing, incoming)
+       or isPersistedCopy(existing, incoming)   // same id, equal after memory's save cleanup
+      -> skip
   sealed existing + "different" copy -> append under new id   // was reached by the trimmed copy
```

Text that really changed after the seal still differs after the cleanup,
so it is still added as a new message (covered by a test).

## Evidence

- **Before (origin/main core):**
`packages/memory/src/sealed-message-reload.test.ts` (the issue's
`repro.mts` as a test, using real `Memory`, `InMemoryStore`, `recall()`
and `MessageList`): `Tests 5 failed | 3 passed (8)`. The list held 2
messages with `rs_example` twice, and the LLM prompt carried
`rs_example` twice.
**After:** `Tests 9 passed (9)`. The ninth case seals a client-sent
message with a streaming tool call through the real OM seal and
`MessageHistory` save. Before the `partial-call` follow-up it failed
with a second message id next to `assistant-message`.
- **Before:** `sealed-reload-duplicate-reasoning.issue-29380.test.ts`,
the issue's agent-only fixture ported to a test: a real Agent with
observational memory, suspended twice in `ask_user`, against a scripted
Responses server over real HTTP that returns 400 on duplicate input ids.
It failed 5/5 runs with `expected [ 'rs_script_1', 'msg_script_1', … ]
to deeply equal []`.
  **After:** passed 10/10 runs.
- **Before:** the issue's original `agent-only-repro.mts`, run against
the local build with `@ai-sdk/openai` 3.0.135, got `HTTP 400 Duplicate
item found with id rs_script_1`.
**After:** no duplicate ids, and the `no-newline`, `no-buffer` and
`one-tool` controls stayed clean.
- New core unit test `conversion/is-persisted-copy.test.ts` (12 cases):
it matches trimmed, whitespace-only and working-memory-stripped copies,
and copies saved with or without a streaming `partial-call` part. It
rejects a copy missing a tool call that had finished streaming, and
copies that differ by an inner space, a different id, a different
reasoning or text item id, or an extra part.
- Neighboring suites: core `src/agent/message-list` 979 passed, 2
skipped. Memory unit `src` 2620 passed, 1 todo; after the `partial-call`
follow-up, memory `src/processors/observational-memory` 2012 passed, 1
todo. `tsc --noEmit` passed for core (`tsconfig.build.json`) and memory,
and `oxlint --type-aware --deny-warnings` and `oxfmt --check` were clean
on the changed files.

Not tested: live OpenAI (no key used); the HTTP fixture mirrors the
API's duplicate-item 400.

## Merge Danger

**Door:** two-way. Code-only change in `@mastra/core`'s `MessageList`;
reverting the commit restores the old behavior, and no stored data
changes.

**Blast radius:** every agent that loads history from memory. The new
check only runs for a memory-loaded message that shares an id with a
message already in the list, and it only drops the stored copy when the
two match after memory's own save cleanup, so the live message is kept.
Risk: if a stored copy really differs from the live message only by
whitespace or working memory, the live version now wins. That matches
how the unsealed path already handles trimmed text.

## Related issue(s)

Fixes #29380

The issue also suggests a request-wide guard that removes repeated
Responses item ids. This PR leaves that out: as the issue notes, it
would hide the 400 but not fix the duplicated history.

## Type of change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [ ] Test update

## Checklist

- [x] I have linked the related issue(s) in the description above
- [ ] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR

## Council review

Rounds: 4

| Seat | Round 1 | Round 2 | Round 3 | Round 4 |
| --- | --- | --- | --- | --- |
| Claude Opus 5.5 | not run | 8.5/10 | 8.5/10 | 9/10 |
| GPT-6.1 Sol | 9/10 | 9/10 | 9/10 | 9/10 |
| Grok 4.7 | empty seat | empty seat | empty seat | empty seat |

Created with Claude Opus 5.5 in T3 Code.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

Memory can change a message when it saves it, such as by trimming spaces
or removing working-memory details. This change recognizes the saved
copy when it is loaded again, so it is not added a second time with the
same provider item IDs.

## Changes

- Added `isPersistedCopy` to compare messages with the same ID after
applying persistence cleanup, including trimming text and removing
working-memory content.
- Updated `MessageList.addOne` to skip a recalled message when it
matches either the existing equality check or the persisted-copy check.
- Added tests for persistence cleanup, streaming tool calls, repeated
reloads, and duplicate provider item IDs.
- Added a patch changeset for `@mastra/core`.
- Did not add a request-wide safeguard against duplicate Responses item
IDs.

**Checks:** not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 21:14:50 -05:00
Aamish
d7d18072ba
fix(memory): omit stored reasoning from recall results (#28810)
Fixes https://github.com/mastra-ai/mastra/issues/28809.

Recall returns stored reasoning as ordinary tool text. Save an assistant
message with reasoning at part 0 and visible text at part 1, then page
its messages or request part 0 directly: the reasoning is exposed.

Exclude reasoning/redacted-reasoning in the shared formatter, use
rendered visibility for paging/fallback, and stop advertising a
reasoning filter. Visible text/tool evidence keeps its original part
indices. Stale helper calls with a reasoning filter return no reasoning.

Tests: `PR_ROOT=$PWD PR_PACKAGE=packages/memory
PR_TESTS=src/tools/om-tools.test.ts node node_modules/vitest/vitest.mjs
run --config /tmp/m18-vitest.config.mjs --reporter=dot` passed 146/146.
Eleven regression cases failed before the fix. Tests use real
memory/storage and cover low/high paging, direct indices, filters, tool
schema and next-message fallback.

The temporary source config stubs unused catalog/embedding imports. Full
builds/typechecks and external database suites were not run. Stored
messages, provider history replay, and the existing recall lookahead
limits retain their behavior.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## ELI5

Recall results should show useful message content, not stored reasoning
that looks like ordinary text. This change hides reasoning while keeping
the original positions of visible text and tool results.

## Changes

- Excludes stored `reasoning` and `redacted-reasoning` parts from recall
output.
- Uses renderable content to decide which messages count toward paging
and fallback.
- Removes `reasoning` from the advertised `partType` schema. Stale calls
that still request it return no matching parts.
- Preserves stored part indices for visible text and tool results.
- Adds regression tests for hidden parts, paging, filters, schema, and
fallback.
- Adds a patch changeset for `@mastra/memory`.

## Validation

The PR description reports 146 passing tests. It also states that full
builds, typechecks, and external database suites were not run. The
supplied repository output did not include test execution results.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Aamish <your@email.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 12:21:07 -08:00
Sergey Zelenov
35ab0c807b
feat(memory): observational memory search without a client-side embedder (#27005)
## Description

Follow-up to #25009. Observational Memory's `retrieval: { vector: true
}` still required an `embedder` even when the vector store generates
embeddings itself, so it didn't work with MongoDB Automated Embedding.
Now it does:

```ts
const memory = new Memory({
  storage,
  vector: new MongoDBVector({ id: 'vec', uri, dbName, autoEmbed: { model: 'voyage-4' } }),
  options: { observationalMemory: { retrieval: { vector: true } } },
})

await memory.searchMessages({ query: 'What deadline did we agree on?', resourceId: 'user-123' })
```

This builds on the `isSelfEmbedding` flag from #25009. When the store
reports it and there's no embedder, `Memory` sends text instead of
vectors. Four spots needed it: the constructor check,
`hasRetrievalSearch` (which also turns on the `recall` tool's search
mode), `indexObservation`, and `searchMessages`. Observations go into an
index called `memory_observations_selfembed`, without a dimension.
Setups that pass an `embedder` behave exactly as before.

`MongoDBVector` needed no changes. I added a live test for it in
`stores/mongodb` that indexes observations, searches them, checks the
thread and time filters, and checks that deleting a thread removes its
rows. It's opt-in (`TEST_MONGODB_AUTOEMBEDDING=1`) and I ran it against
an Atlas cluster. The rest is covered by unit tests with a mocked store.

Docs aren't in this PR. The issue asks for a separate docs ticket, so
I'll follow up there. The experimental Subconscious knowledge memory
still requires an embedder and is out of scope.

## Related issue(s)

Fixes #27002

## Type of change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [ ] Test update

## Checklist

- [x] I have linked the related issue(s) in the description above
- [ ] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

Observational Memory can now search using a vector store that creates
embeddings for it. Users can enable semantic recall without setting up a
separate embedding service.

## Changes

- `@mastra/memory` accepts `retrieval: { vector: true }` without a
client-side embedder when the vector store reports `isSelfEmbedding`.
- Memory indexes observation text under `memory_observations_selfembed`
without passing a dimension. It sends search text to the store.
- Observation-vector migration reads self-embedded rows with text
queries and updates their metadata without requesting vectors.
- Configurations with a client-side embedder keep the existing
vector-based behavior.
- The recall tool’s setup message now includes stores that generate
embeddings themselves.
- Added mocked-store unit tests and an opt-in MongoDB integration suite.
The live suite requires `TEST_MONGODB_AUTOEMBEDDING=1` and either
`MONGODB_AUTOEMBED_URL` or `VOYAGE_API_KEY`. Test execution results were
not provided.
- Documentation is not included; it is planned for a separate follow-up.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Ehindero Israel <iehindero419@stu.ui.edu.ng>
2026-11-03 15:19:19 +01:00
Daniel Lew
642c62fee3
feat(core,server,client-js): answer a suspended background task from a chat (#28998)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-11-02 14:54:42 -05:00
mastra-platform[bot]
ea8fefcd83
fix(memory): close the observations block in Observational Memory context (#29224) 2026-11-02 11:44:05 -08:00
Tyler Barnes
e1e64f072c
fix(memory): keep user signal attributes in Observer labels (#29176)
## Description

In a thread several people write to, Observational Memory's Observer saw
every user message as plain `User`, so observations could not say who
said what. User signals carry attributes (for example `attributes: {
name: 'Jim' }` passed to `agent.sendMessage`), and those were dropped
when the Observer's transcript was built.

The Observer now labels user signals with their attributes, e.g. `User
(name="Jim")`. Messages without attributes keep the plain `User` label.

Attribute handling is deliberately bounded so a value cannot break or
inject lines into the transcript: keys must be identifier-like, values
must be strings, numbers or booleans, keys are sorted, at most 8
attributes are shown, values are capped at 80 characters, and values are
JSON-quoted.

Changed:
`packages/memory/src/processors/observational-memory/observer-agent.ts`
(label formatting), with tests and a patch changeset for
`@mastra/memory`.

## Test plan

Regression tests in `observer-agent.test.ts` (labels each signal with
its attributes; quotes and bounds values; keeps plain `User` with no
usable attributes).

- Without the fix (`observer-agent.ts` from `origin/main`): 2 failed, 25
passed.
- With the fix: 27 passed.
- Observational Memory suite: 89 files, 2007 passed, 1 todo.
- `tsc --noEmit` exit 0; oxlint 0 warnings, 0 errors.

Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com>
2026-11-01 22:25:12 -07:00
Tyler Barnes
0a553f0f6f
Knowledge alpha: standalone Knowledge for agents, Subconscious removed (#28353)
## What this is

**Knowledge** is Mastra's store for durable knowledge that agents share
across conversations. Until now its agent-facing parts lived in
`@mastra/memory`, inside an experimental feature called
**Subconscious**, a background curator that wrote to Knowledge after
observational-memory reflections.

This PR makes Knowledge stand on its own as an experimental alpha,
rebuilt around pages and declared scopes.

- **Subconscious is removed** from `@mastra/memory` and Mastra Code.
Setting `experimental_subconscious` or the Memory `knowledge` option
throws, saying to attach Knowledge to the agent instead. Old
Subconscious data is ignored.
- **Knowledge attaches to an agent directly:** `new Agent({ knowledge,
scopes })`. It brings its own tools, instructions and pinned context.
Tools work without Memory; pinned context needs Memory.
- **Knowledge is a tree of pages.** Each page lives in one scope and
holds a body plus short notes. Notes can mention other pages with
`[[wikilinks]]`. A page can have a key, unique among its siblings, and
is addressed by key path: `<scope>#<key>/<child key>`. Pages declared in
a scope's config are created when missing and can be edited but not
moved, deleted or merged away by tools.
- **Access comes from declared scopes.** Agent `scopes` are plain
`"<type>:<value>"` strings that say who the run is, set by the host and
never by the model. Knowledge's required `scopes` option declares which
scope types exist and which roles (viewer, contributor, editor, owner)
holders get on which scopes. Rules reach one hop with no inheritance. A
run holding an undeclared scope fails. `knowledgeMemoryScopes()`
declares Memory's `thread:`/`resource:` scopes and an `org:` scope.
- **Five tools:** `knowledge_search`, `knowledge_read`, `knowledge_page`
(write, move, merge, delete), `knowledge_note` (add, restamp, remove)
and `knowledge_pin` (pin, unpin). Tools are off by default: `tools:
true` enables read and write, `tools: { read: true }` read only, and
`pinnedContext` adds pins. Deleting a page needs owner on the page's
scope and is refused for pages with child pages, pages declared in a
scope's config, and scope pages. On a scope page, write accepts the body
only: its name and description come from config.
- **Limits are in tokens:** about 250 per note and 100 per page
description. Pinned context defaults to 500 tokens, with a maximum of
2,000.
- **Search:** keyword search by default: every word of the query must
appear in the page (or note), and name matches rank first, the same in
every store. With a `vector` store and an `embedder`, search also
returns semantic matches. Every semantic hit is re-checked against what
the run can read before ranking, so hidden content never takes a result
slot. `knowledge.reindex()` re-embeds everything.
- **Host APIs:** `knowledge.readScopeFor(principals)` returns what
holders of those scopes can read. `knowledge.explainAccess(runScopes,
target)` says why a run can or can't reach a scope or page.
`explainAccess` is host-only.
- **Renamed scope declarations:** if stored scopes use a declaration
name the config no longer declares, Knowledge fails loudly on first use,
naming each orphaned name and how many scopes use it.
`knowledge.renameScopeDeclaration(from, to)` renames them. It is
host-only and atomic, except on MongoDB without a replica set.
- **Per-message scopes:** `session.sendMessage({ content, scopes })`,
plus queue, steer and notifications. A message whose scopes differ from
the running run's waits for that run to end and then starts its own run.
Scopes are set in-process only.
- **Storage:** LibSQL, PostgreSQL, MySQL and MongoDB store Knowledge as
schema 4 (pages, notes, mentions, pins). On startup, the tables of the
earlier experimental Knowledge node store (shipped in 1.79.0) are
dropped with any rows in them, and the new tables are created. If an
older process recreates them during a rolling deploy, the next startup
drops them again. Schema-4 tables and non-Knowledge tables are never
touched. On MongoDB, Knowledge collections found without a schema
version are adopted as schema 4 rather than refused. Don't run
`dangerouslyReset()` from an older version against a database the new
version already uses. Development databases created from earlier commits
of this branch also need a reset, since sibling key uniqueness changed
within schema 4. `storage.stores?.knowledge?.dangerouslyReset()` clears
Knowledge rows only, never Memory. Adapter peer floors are unchanged: on
an older `@mastra/core` the adapters still load and the other storage
domains keep working.
- **Mastra Code:** Knowledge is opt-in through **Experimental
Knowledge** in `/settings`. The `/knowledge` browser and the SDK
inspector read pages. Factory's Knowledge routes are disabled until the
Studio page readers land.

## Test plan

- Shared schema-4 storage contract suite on real stores (as of
d5b72de125): LibSQL 59/59, MySQL 60/60, PostgreSQL 69/69, MongoDB
61/61. LibSQL Knowledge index suite 58/58 at 5b137a7677, including
startup over a published 1.79.0 layout (empty and with rows), old tables
recreated under a schema-4 database, and four concurrent startups. On
MySQL, concurrent scope materialization can deadlock on InnoDB gap
locks; it retries up to 5 times on duplicate-key or deadlock errors (31
consecutive passing runs of the concurrency test).
- Core Knowledge suites: declarations, access, page tools for the spec's
scenarios 1–13, pins, token limits, `explainAccess` agreement with the
tools (8 actors × 12 pages), and semantic search (25 hidden
higher-scoring pages never hide the visible result).
- Integration on real LibSQL and PostgreSQL: an agent writes a page,
note and pin. An outsider can't read them. They persist across a restart
and pinned context is delivered. On startup the 1.79.0 node-store tables
are dropped, rows included, while schema-4 and non-Knowledge tables are
left alone.
- Independent validation on real LibSQL: scenarios 1–13 including 6 and
11 with a real agent, and live semantic search through OpenRouter
`openai/text-embedding-3-small` with LibSQLVector, isolated per tenant.
- Mastra Code: inspector, TUI browser and Knowledge-browser E2E.
- `tsc`, `oxlint` and `oxfmt` clean on touched packages. Each milestone
was reviewed independently.

---------

Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com>
Co-authored-by: avis-reader[bot] <332681745+avis-reader[bot]@users.noreply.github.com>
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-30 22:24:20 -07:00
Leo
95ddbb7fc3
fix(memory): avoid reindexing unchanged observation groups (#28150)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com>
Co-authored-by: Tyler Barnes <tylerdbarnes@gmail.com>
2026-10-30 15:34:16 -07:00
Arthur Fiorette
f241b9d595
fix(pg): use HNSW index for filtered vector queries (#29058) 2026-10-30 15:29:41 -07:00
Ward Peeters
ba8b955b1a
fix(redis-streams,valkey-streams): expire idle streams by default and clear them on thread delete (#28968)
A production Factory deployment ran out of Redis memory. Every key was a
`mastra:topic:*` stream, and almost all of them (3,673 thread streams)
had no expiry. The deployment built `new RedisStreamsPubSub({ url })`
with defaults, and the default `streamIdleTtlMs` was `0`, meaning
streams never expire. Thread streams are open-ended and never get
`clearTopic()`, so they piled up forever. The Redis instance used
`volatile-lru`, which can't evict keys without a TTL, so writes just
started failing.

This PR makes `@mastra/redis-streams` and `@mastra/valkey-streams` give
every stream a sliding 7-day idle TTL by default. Writes refresh it, and
live subscribers keep receiving events after a stream expires because
the read loop recreates it. Streams are an event bus, not storage:
replay history lives in the caching layer and storage, so the backlog
only matters for grouped workers that are offline longer than the TTL.

```ts
// Before: streams never expire unless you opt in
new RedisStreamsPubSub({ url });

// After: 7-day idle TTL by default; opt out explicitly
new RedisStreamsPubSub({ url });
new RedisStreamsPubSub({ url, streamIdleTtlMs: 0 });
```

Both pubsubs now log one warning after they first connect if Redis has
`maxmemory` set with any policy other than `noeviction`. A TTL makes
streams, consumer groups and lease keys evictable under `volatile-*`,
which would silently drop events and leases (ElastiCache defaults to
`volatile-lru`). The check uses `INFO memory` because managed Redis
often blocks `CONFIG GET`.

`memory.deleteThread()` now also deletes the thread's event stream
through the registered Mastra pubsub. This runs fire-and-forget, so
neither deleting a thread nor `memory.settled()` waits on Redis during
an outage. It's skipped on older core versions. Agents configured with
their own pubsub still rely on the TTL.

The docs now require `noeviction`; the Helm guide used to say a
`volatile-*` policy was fine. They also add a runbook for reclaiming
streams created before the TTL: switch to `noeviction`, upgrade the
writers, then expire each stream relative to its last write. Tested
against real Redis/Valkey containers, including a real TTL expiry while
a subscriber is listening and a thread delete with Redis unreachable.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

Redis and Valkey can keep old event streams forever, which uses storage.
This change removes streams after seven days without writes by default
and cleans up a thread’s stream when the thread is deleted.

## What changed

- Redis and Valkey streams now use a sliding seven-day idle TTL by
default. Writes refresh the TTL; reads do not. Set `streamIdleTtlMs: 0`
to disable expiry.
- The read loop recreates an expired stream so a live subscriber can
continue receiving events.
- After connecting, each pubsub warns once when `maxmemory` is enabled
with a policy other than `noeviction`. The check uses `INFO memory`.
- Thread deletion starts best-effort stream cleanup through the
registered Mastra pubsub without waiting for Redis. Older core versions
skip this cleanup.
- Documentation and the local Docker configuration now recommend
`noeviction`. The documentation also explains how to expire existing
streams that lack a TTL.

## Tests

Tests cover TTL behavior, delivery after expiry for live subscribers,
eviction-policy warnings, and thread-stream cleanup. Test execution
results were not provided.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-30 11:51:23 +01:00
Eric Pinzur
9e36f35910
feat(observability): capture and export native execution outcomes (#28703)
Adds native `outcome`, `stopReason`, and `stopReasonContext` to agent,
model, tool, and workflow tracing and carries them through
`MastraStorageExporter` and `MastraPlatformExporter`. Implements
[OBS-724](https://linear.app/kepler-crm/issue/OBS-724).

Outcomes describe execution meaning. Guardrails record `blocked /
guardrail`; failed observers retain their own result while the affected
execution records `interrupted / observer_failed`. Caller cancellation
records `abort` on its target and the agent run; interrupted descendants
record `interrupted`. Run deadlines record `limit / timeout`, with
`interrupted / timeout` on affected descendants. Completed spans retain
their results during cleanup, and recovered retries can finish
successfully. Framework stop-reason context is a flat summary of at most
three scalar facts, with strings bounded to 128 characters and span
references for further diagnostics. Custom outcome strings and context
objects remain supported and are safely serialized.

```ts
span.end({ endTree: true, outcome: 'abort', stopReason: 'caller_cancelled' });
```

Processed `MODEL_CHUNK` spans now consistently belong to `MODEL_STEP`.
Provider-stream instrumentation records timing and usage without
capturing raw output before processors run. Agent-stream
`MODEL_INFERENCE` spans no longer include response output; consumers
should use processed chunks or generation output. Error finalization
preserves completed-request usage and output. When the inner workflow
fails without a finish callback, an internal tracing-only notification
closes the generation and agent spans, including when the user error
callback throws. Public error callback arguments and stream payloads are
unchanged.

Compatibility changes are explicit: public native fields now also allow
`null`, which TypeScript consumers must handle. A null or absent outcome
means no outcome was recorded; endTime determines whether the span has
finished, including with older observability releases. Memory failure
markers in streams and persisted messages, and tripwire metadata,
include `observerFailure` with span references and classification.
Certain workflow events and durable snapshots gain tripwire details,
including processor-supplied metadata. The compact-context restriction
applies to `stopReasonContext`, not all existing diagnostic payloads.
Runtime continuation decisions and return statuses are retained.
TripWire now uses the name `TripWire` instead of `Error`, affecting
string representations, stacks, and serialized error names; thrown error
objects otherwise retain their identity. Continuation causes remain in
internal run state and are not added to public finish callback context,
stream chunks, or snapshots. Older package combinations tolerate missing
helpers without dependency or peer-range bumps, including a cleanup
fallback for older model trackers.

Cross-worker tracing remains follow-up work in
[OBS-754](https://linear.app/kepler-crm/issue/OBS-754/preserve-tracing-outcomes-and-resumed-span-identity-across-workers).
Without the local registry, ordinary completion and max-step exits can
report `unknown`, and observer/delegation or provider-failure detail can
be lost. The new caller-owned Inngest resume spans have been removed:
separate resumed-segment tracing needs a worker handoff protocol.
Inngest still reuses the original span identity on resume, and core
durable remote resume can leave fresh local spans open or finalize
original snapshots. A caller-side Inngest deadline may also be missing
from worker-owned outcomes rather than appearing as `limit / timeout`.
Process-loss recovery and these identity/finalization gaps are not
solved here; the Inngest regression remains an explicit expected
failure. No storage/query implementation, checkpoint/recovery
infrastructure, or new evented control-flow/nested-run/sleep
instrumentation is included. Native storage support comes from
[#28389](https://github.com/mastra-ai/mastra/pull/28389).

Validation: 216 tests passed across the core native-outcome,
workflow-result, and classification suites, followed by five final
focused checks including an added nested-tripwire result case. All 116
observability tracker, span-lifecycle, and ESM/CommonJS compatibility
tests passed. Real Inngest verification passed 12 workflow TripWire
cases and two tool suspension/error cases. Core, observability, and
Inngest builds (including type generation), affected-file
lint/formatting, and focused documentation checks passed. The status and
fatal-bail tests were also verified to fail when their fix/fallback was
disabled. The deferred Inngest resume-identity regression remains an
explicit expected failure under OBS-754. Review threads are left for
their authors/tools to resolve.
2026-10-29 13:43:35 +01:00
Damien Schneider
976fde7416
refactor(lint): enable no-else-return at root (#28601)
Enables `no-else-return: error` in the root Oxlint config and removes
186 redundant `else` branches across 111 source/test files. Existing
rule defaults are retained.

Reviewed every code diff for behavior preservation. An independent
TypeScript AST comparison confirms that changes are limited to removing
`else` after returning branches, with the same expressions, statement
order, return values, and `try`/`catch`/`finally` boundaries. All
181,158 identifier occurrences retain their bindings. The recorder
replay path keeps an explicit block: flattening it would shadow the
earlier request `body` with the response `body` and introduce a
temporal-dead-zone error.

Extends the existing monorepo optimized-dependency test to assert that a
raw TypeScript workspace package, including its enum and transitive
dependency, still executes in the production bundle.

Validation: the root rule-only scan passes on 15,273 files with zero
diagnostics; formatting and `git diff --check` pass. Schema
compatibility tests: 1,060 passed, 9 skipped. Recorder runtime tests: 21
passed. The core selection initially passed 3,513 tests; rerunning the
suites requiring local server/cache access passed 185 tests, with one
remaining cache-file assertion also reproduced on unchanged `main`.

Remaining validation limits: four asset-download cases cannot reach
`placehold.co`; the changed-file lint run reports six import-order
errors reproduced on unchanged files from `main`. Deployer unit tests
and the monorepo test could not run because this checkout lacks required
package dependencies (`fs-extra`/workspace links and `get-port`,
respectively). These limits are left visible for CI. No changeset is
included because this is a behavior-preserving lint cleanup.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## ELI5
This PR makes the code easier to read by removing `else` branches that
are not needed after a return. It also enables a rule that catches those
branches and adds a test for loading a raw TypeScript workspace package
in a production bundle.

## Summary
- Enables Oxlint’s `no-else-return` rule at error severity and removes
186 redundant `else` branches across 111 files.
- Extends the monorepo optimized-dependencies test to check a raw
TypeScript workspace package, its enum, and a transitive dependency.
- Updates the deployer’s transpile-package path pattern.
- In workflow event processing, a truthy branch index with no matching
child now falls through to publish a skipped-step end event. Previously,
this case published no event.
- Keeps the recorder replay block explicit to avoid request/response
`body` shadowing and a temporal-dead-zone error.

## Validation
Reported checks passed: the rule-only scan across 15,273 files,
formatting, `git diff --check`, schema compatibility tests (1,060
passed, 9 skipped), recorder runtime tests (21 passed), and an initial
core test selection (3,513 passed).

Some test limits remain. Four asset-download cases could not reach
`placehold.co`. Changed-file lint reported six import-order errors that
also occur on unchanged files from `main`. Deployer unit tests and the
monorepo test could not run because dependencies were unavailable. On
rerun, suites requiring local server/cache access had 185 passes; one
cache-file assertion also reproduced on unchanged `main`.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-28 12:30:27 +01:00
Damien Schneider
1fc075ab7f
refactor(core): one provider-failure reader for retries, memory, rotation and Factory (#28496)
**─────── TLDR ───────**
> Retries, memory, account rotation, the TUI and Factory now read a
failed model call through one reader in `@mastra/core/error`, so one
rule change fixes them all.

Follows #27811, which put this reader in Factory first. Before this PR,
seven hand-written status, code and wording tables disagreed. The retry
processor retried an exhausted quota, because the AI SDK marks every 429
retryable. Rotation only read quota wording on a 400. The run report
said `Bad Request`.

Now `classifyProviderError` reads the HTTP status, the provider codes,
the body and the wording, and every site asks it. Each site keeps only
the signals the reader can't see: a numeric `status`, network errors,
Retry-After and `ProviderAuthRequiredError`.

### Behavior changed

a 429 whose body says the quota is exhausted (`insufficient_quota`)
&nbsp;&nbsp;├─ **was** — retried, and the TUI says to wait a moment
&nbsp;&nbsp;└─ **now** — never retried; the TUI names the quota and
points to billing

a credential or usage failure named only in the provider's body
&nbsp;&nbsp;├─ **was** — no account rotation; Factory reports a plain
run failure
&nbsp;&nbsp;└─ **now** — rotates to the next account; Factory reports
bad credentials

a Factory run report for a failed provider call
&nbsp;&nbsp;├─ **was** — quotes the provider for usage limits only, else
`Bad Request`
&nbsp;&nbsp;└─ **now** — quotes the provider's own reason, cut at 400
characters

a mid-stream provider code, such as a Codex `server_error`, on the web
card
&nbsp;&nbsp;├─ **was** — `Run stopped`, live and after a reload
&nbsp;&nbsp;└─ **now** — named as an outage; the saved part keeps the
codes

a stream error that your own retry matcher and the built-in OpenAI
matcher both accept
&nbsp;&nbsp;├─ **was** — the built-in policy wins: one immediate retry,
no callback
&nbsp;&nbsp;└─ **now** — your matcher's `maxRetries`, `delayMs` and
`onRetry` apply

a tool's `EACCES: permission denied` ending a run
&nbsp;&nbsp;├─ **was** — the reader called it a provider credential
failure
&nbsp;&nbsp;└─ **now** — unknown; only the code form `permission_denied`
reads as auth

### Shape changed

| | | |
|---|---|---|
| **−** | `mastracode/factory/src/provider-error.ts` | its home in
#27811, back in core |
| **−** | memory `context-overflow.ts` | replaced by the reader's
`context_overflow` |
| **−** | `USAGE_LIMIT_MESSAGE_PATTERN` ×2 | rotation and run report
each kept a copy |
| **−** | `RETRYABLE_OPENAI_ERROR_CODES` ·
`TERMINAL_AUTHORIZATION_ERROR_CODES` | the reader's kinds |
| **+** | `quota` in code-sdk `ErrorType` | a quota can't be waited out
|
| **~** | `MastraErrorPart['error']` | keeps `codes` and the provider's
`reason` for a reload |

what callers now import from `@mastra/core/error`
&nbsp;&nbsp;&nbsp;&nbsp;`classifyProviderError(error)` ·
`providerErrorReason(error)` · `providerRequestId(error)` ·
`providerModelId(error)` · `toPersistedErrorIdentity(error)` ·
`PROVIDER_CALL_ERROR_NAME` · `type ProviderFailureKind`

### Decisions

- **overflow wording outranks a 5xx** — a gateway relaying `prompt is
too long` gets compacted, not retried; one rule in `provider-error.ts`
to drop.
- **a caller's numeric `status` outranks wording** — the reader ignores
`status`, which Octokit and MCP errors also carry, so rotation and
`parseError` read it first; a 503 saying `weekly limit` retries instead
of burning an account.
- **only a provider's own sentence is saved** — the saved `reason` comes
from an HTTP response or a stream frame, never from another error's
`detail`, which can name a database row.

### Not verified

- [ ] an app on `@mastra/core` 1.80.0-alpha.0 to alpha.4: the repo's
`-0` floor admits them and they predate the reader, as with `@mastra/pg`
on main; `check:core-imports` passes for server, memory and code-sdk
against this branch's core packed as 1.80.0
- [ ] `changed-tests` stays red: the deployer's core floor moves with no
monorepo E2E change, as on #28417
- [ ] account rotation against real provider accounts: the
`account-rotation`, `account-routing-targeted` and `pack-fallback` TUI
E2E pass with HTTP 429s through the real AI SDK, not a live account
- [ ] the web card after a reload with a real Codex `server_error`
frame: fixtures only

---

> ██████████████████ **source** `+304 −455` — net −151
> █████████████ **tests** `+399 −140`
> ███ **changeset** `+112`
> █ **package.json** `+20 −20` — core peer floor 1.80: server, memory,
agent-builder and editor import the reader
> █ **docs** `+10 −6`


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## ELI5

Different parts of the project could interpret the same provider error
in different ways. This change gives them one shared reader so retries,
account rotation, saved errors, and user-facing reports handle those
errors more consistently.

## What changed

- Added provider-error classification and extraction helpers to
`@mastra/core/error`. They read provider codes, reasons, request IDs,
and model IDs from responses and wrapped errors.
- Updated retry handling, account rotation, observational-memory
context-overflow recovery, server error serialization, Factory reports,
and TUI error mapping to use the shared reader.
- Saved errors can include provider codes and reasons. Persisted
provider text is capped at 2,000 characters. Factory failed-run reports
use the provider reason when available and cap the report text at 400
characters.
- Configured retry matchers now run before built-in matchers. Built-in
retry handling excludes credential and exhausted-quota failures.
- Raised the minimum `@mastra/core` peer dependency to 1.80.0 across
affected packages and deployers.

## Verification and limitations

The supplied notes report that `check:core-imports` at the 1.80 floor is
unverified. They also report that `changed-tests` remains red after the
deployer core-floor change. Account rotation against real provider
accounts and web-card behavior after reloading a real Codex
`server_error` frame were not verified; unit tests and fixtures were
used instead.

Review finding counts are unavailable because no current review findings
were supplied.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-27 14:04:37 +01:00
mastra-platform[bot]
0b7e61996e
fix(memory): retry Observer replies that contain no observations (#28456) 2026-10-26 16:02:20 -05:00
Damien Schneider
f02d3b711e
refactor(lint): remove redundant returns while preserving exits (#28448)
Enable `no-useless-return` in the shared configuration. Remove 28
terminal bare returns, use 31 explicit switch breaks, and retain 12
value-returning contracts with explicit `undefined`. Preserve 13
intentional guards with local explanations, including index-skipping
property reads and the platform availability probe.

Reviewed every edit and checked that removed returns and new switch
breaks have no effectful continuation before function exit, including
cleanup through `finally`. All other source and emitted JavaScript
syntax is preserved apart from formatting, and all 2,020 function return
types and 3,280 variable binding types in the 59 affected source files
are unchanged. The new rule reports zero violations across 15,106 files.
CI lint and build, the local 57-task build, Factory build, core
typecheck, formatting, and whitespace checks pass.

All 756 focused tests pass across stream conversions, idle-loop cleanup,
workflow validation, Factory, Factory UI, Playground UI, Datadog,
Sentry, Cloudflare, and E2B. Both selected published-package monorepo
E2E cases pass. The extended build case covers a missing application
`.npmrc` and verifies successful builds with dependency installation
both skipped and enabled.

The DuckDB CI job also fails two existing capability-list assertions
because their expected arrays omit `span-aggregate`. Both failures were
reproduced on unchanged base `f4cbdb4d78`; DuckDB and core storage files
are unchanged by this PR.

No changeset is included for this lint maintenance, as requested; the
changeset check is expected to fail. Keep this PR as a draft.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## ELI5

This PR teaches the shared lint configuration to flag returns that do
not change what a function does. It updates the affected code so it
still exits at the right point, while keeping returns that are needed to
return a value or protect an intentional guard.

## Summary

- Enables `no-useless-return` as an error in the shared Oxlint
configuration.
- Removes redundant terminal returns, replaces switch-case returns with
`break` where execution continues after the switch, and keeps
value-returning contracts explicit.
- Adds local explanations or lint suppressions for intentional guards.
- Adds assertions that the monorepo build test leaves `.npmrc` files
absent when installation is skipped.

The author reports that source and emitted JavaScript syntax, apart from
formatting, and the return types and variable binding types across all
59 affected files remain unchanged. The author also reports reviewing
changes for effects before function exit, including cleanup through
`finally`.

## Validation and status

The author reports zero violations across 15,106 files and passing CI
lint and build, the local 57-task build, Factory build, core typecheck,
formatting, whitespace checks, 756 focused tests, and two selected
published-package monorepo E2E cases.

The author reports two DuckDB CI failures in existing capability-list
assertions because the expected arrays omit `span-aggregate`. The author
reports reproducing both failures on unchanged base `f4cbdb4d78`; DuckDB
and core storage files were not changed.

No changeset is included, as requested. The author expects the changeset
check to fail and requests that the PR remain a draft.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-26 21:03:00 +01:00
mastra-platform[bot]
ffc663d2ba
fix(memory): treat empty Observer output as an observation failure (#27294)
Co-authored-by: Abhi Aiyer <abhiaiyer91@gmail.com>
2026-10-26 13:02:14 -05:00
Damien Schneider
0f82b38ed4
refactor(lint): flatten lone if statements in else branches (#28431)
Enable `no-lonely-if` in the shared Oxlint config and flatten 38 lone
`if` statements in `else` blocks across 35 files. For example, `else {
if (condition) { ... } }` becomes `else if (condition) { ... }`.
Conditions, evaluation order, statements, comments, and local bindings
are preserved. No package filters or lint exceptions are added.

Strengthen the existing monorepo test to execute another named export
from an optimized dependency. Comparisons of TypeScript syntax, emitted
JavaScript, raw template text, comments, and local bindings pass for all
35 migrated files after normalizing only the redundant else blocks. The
rule clears all 15,094 scanned files.

[CI lint and formatting
pass](https://github.com/mastra-ai/mastra/actions/runs/40344286057/job/121647174331)
for `36ca4471f9`. All 634 focused tests, four package typechecks (core,
playground, connect, and deployer), 57 packaged build tasks, and two
selected published-package monorepo E2E cases pass. The broader local
lint run hit a Tailwind design-system precomputation timeout; CI lint
succeeds. Live vendor and external-database suites were not run locally.

[DuckDB CI has two existing capability-list assertion
failures](https://github.com/mastra-ai/mastra/actions/runs/40344286246/job/121649643109):
its implementation advertises `span-aggregate`, while the expected lists
omit it. The failures reproduce locally, and the relevant implementation
and tests are unchanged from the base commit.

This maintenance PR intentionally contains no changeset and requests no
package releases. The missing-changeset check may fail.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## ELI5
This PR turns on a lint rule that asks for simpler branching in code. It
updates the affected code and adds a leap-year check to an existing
test.

## Changes
- Enables Oxlint’s `no-lonely-if` rule in the shared configuration.
- Flattens 38 lone `if` statements across 35 files. The PR description
reports that the changes preserve behavior.
- Extends the optimized-dependency monorepo test to check `isLeapYear`.

## Validation
- The PR description reports that CI lint and formatting pass.
- It also reports 634 focused tests, four package typechecks, 57
packaged build tasks, and two selected monorepo E2E cases pass.
- The PR description notes two existing DuckDB CI capability-list
assertion failures. It reports that the related implementation and tests
are unchanged from the base commit.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-26 18:44:48 +01:00
Jamie Nichols
c73d201c62
fix(memory): don't inject the continuation hint on threads with no observations (#26826)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-26 11:31:17 -05:00
mastra-platform[bot]
4303017f39
fix(memory): don't tell agents app-managed working memory can't be updated (#28441) 2026-10-26 11:27:16 -05:00
mastra-platform[bot]
68f311e0da
fix(memory): stop Observational Memory from crashing on object-valued tool errorText (#28440) 2026-10-26 11:24:20 -05:00
Leo
501b207b1b
fix(memory): cancel observational memory with its owning run (#28158)
## Description

Aborting an agent run left Observational Memory's Observer and Reflector
calls running. They could keep step preparation blocked and write
messages or thread metadata after the application handled the abort.

The run's signal now reaches the work the run is waiting on: synchronous
observation, synchronous reflection, buffered-observation and
buffered-reflection activation, and context-overflow recovery in
`processAPIError`. That work stops with the run. A cancelled synchronous
observation leaves the run's messages unobserved, and the next cycle
observes them.

Async buffering and background reflection are detached work and keep
running after the run aborts. Only the run's waits on them
(`waitForBuffering`, activation waits, the buffered-reflection wait)
stop. The jobs finish, commit, and stay tracked by `settled()`. A
storage commit that has already started also finishes for an aborted
caller, so the bookkeeping after it still runs; only a commit still
waiting in the queue is skipped. Extracted thread values from a buffered
reflection are written only after that reflection is stored.

In `@mastra/core`, a run aborted while an error processor handles an API
error now ends as an abort. Previously the processor's cancellation was
swallowed as "no retry" and the step reported the original API error.

```diff
 run aborts
-  observer / reflector / waits keep running, then write
+  sync observation, sync reflection, activation: stop, no late writes
+  run's waits on background work: stop
+  async buffer, background reflection: keep running, tracked by settled()
+  queued commit: skipped; started commit: finishes before the caller returns
```

**Evidence.** `abort-signal.test.ts` holds provider streams and the
commit queue open with latches, then aborts a real `Agent` run. Before
the fix, the original cases failed (12 of the initial 15). Three new
cases cover yoko's review: async buffer and background reflection keep
running after the last run aborts, and `settled()` resolves only after
they finish; a background reflection queued behind a held commit still
stores its reflection and extracted values. All three fail on the
previous head and pass now. A further case shows a cancelled sync
observation's messages being observed by the next cycle. In
`commit-queue.test.ts`, "waits for an op that already started when its
caller aborts" fails on the previous `runOMCommit` and passes now. The
core test `llm-execution-step.api-error-abort.test.ts` failed before its
fix (`onAbort` never ran).

After: the abort suite passes all 27 cases across 8 repeated runs. All
2,102 Observational Memory tests and the other 568 memory unit tests
pass. Package lint passes, and the root TypeScript 7 compiler passes for
`@mastra/memory`.

Not checked: external storage-adapter integration, whose separate
harness is not installed. Tests replace model responses at the provider
boundary.

## Related issue(s)

Fixes #27792. The issue also asked for background buffering to cancel;
per review, async buffering and background reflection now deliberately
keep running after an abort.

Related: #28084 changes processor input-failure handling; this PR's
processor signal forwarding stays outside those lines. #27868 adds the
Slack stop-button trigger.

## Type of change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [x] Test update

## Checklist

- [x] I have linked the related issue(s) in the description above
- [ ] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] I have addressed all Coderabbit comments on this PR

Created with GPT-6.1 Sol and Claude Opus 5.5 in T3 Code.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

When someone cancels an agent run, its memory work should stop holding
up that run or saving unfinished results. Shared background work can
continue for other runs, and the agent now reports cancellation
correctly if it happens during API-error handling.

## What changed

- Passed the run’s `AbortSignal` through Observational Memory
observation, reflection, buffering, activation, and related waits.
- Added cancellation checks around markers, metadata updates, buffering
state, and queued commits. A queued commit can be skipped after
cancellation; a commit that has started can finish.
- Let canceled callers stop waiting for shared background work without
stopping that work. `settled()` continues to track it.
- Delayed persistence of extracted thread values until buffered
reflection is saved.
- Updated agent execution so cancellation during API-error processing
ends the run as an abort.

## Validation

The PR reports that `abort-signal.test.ts` passed all 27 cases across
eight repeated runs. It also reports 2,102 Observational Memory tests
and 568 other memory unit tests passed, along with package lint and the
root TypeScript 7 compiler for `@mastra/memory`.

The external storage-adapter integration harness was unavailable. Tests
replaced model responses at the provider boundary. These results were
not independently verified here.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 10:34:50 -04:00
Tyler Barnes
5bd713b4f2
fix(core): hand each input-step processor the messages as they are now (#28369)
## What was wrong

An agent runs its input processors as steps of one processor workflow.
In `processInputStep`, each step handed the next processor the
`messages` array it had been given, not the list as it stood after the
processor ran.

Observational memory removes observed messages from the list in
`processInputStep` once a run is past its first step, and returns the
`MessageList`. Every processor after it was therefore handed the array
from before the cleanup. A processor that returned that array put every
removed message back through `applyMessagesToMessageList`, which re-adds
any returned message missing from the list. Returning the array is
natural for a processor that only inspects or rewrites messages.

The result: observational memory reported the messages removed and its
own count dropped, while the prompt kept growing until the provider
rejected it.

## Fix

The `inputStep` step in `createStepFromProcessor` now hands on
`messageList.get.all.db()`, unless the processor returned its own array.
A returned array is still handed on as before. This matches the `input`
and `outputStep` phases, which already hand on the current list when a
processor returns it. The same change is made in the evented workflow's
copy.

## Tests

- **`input-step-removed-messages.scenario.test.ts`:** an aimock scenario
on all four engines. One processor removes an earlier user message from
step 1 on and returns the list, as compaction does. A second returns the
`messages` it was handed.
  - The second request must not contain the removed message.
  - The second processor must be handed the list without it.
  - A control runs the first processor alone.
- On `main`, the main case fails on all four engines and the control
passes.
- **`processor-step.test.ts`:** runs the `inputStep` step from both the
workflow and the evented workflow builders against a real `MessageList`.
The removed message must be gone from the step's output. On `main`, both
fail.
- **`second-input-processor-long-run.test.ts`** (`@mastra/memory`): a
real `Agent` with real `Memory` and observational memory over 60 tool
steps of about 2,800-character results, with a pass-through input
processor after observational memory. The largest of the last 20
requests must stay under 80,000 characters.

  | Core build | Largest of last 20 requests |
  |---|---|
  | `main` | 281,356 chars |
  | This PR | 40,105 chars |

  40,105 is the same as the run without the pass-through processor.

- **Break check:** reverting the change in `workflow.ts` fails the
scenario on all four engines. Reverting only the evented copy fails its
`processor-step` case.
- **Suites:** core processor tests, the input-processor, input-step,
guardrail, memory and parity-processor scenarios, and
`src/agent/__tests__`: 4,147 passed, 67 skipped. `@mastra/memory` unit
tests against the rebuilt core: 2,669 passed, 1 skipped, 1 todo. Core
and memory type checks pass.

## Not verified

- **Other phases:** `outputStep` and `toolResult` were not changed.
Their list-returning branches already hand on the current list.
- **Memory integration tests:** the `packages/memory/integration-tests`
suite was not run.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## ELI5

This change makes each input processor see the messages that remain
after earlier processors finish. That stops removed messages from coming
back when a later processor passes along an older list.

## What changed

- Updated both workflow builders to pass the current `MessageList`
contents to the next processor when the current processor does not
return its own `messages` array.
- Preserved processor-returned arrays unchanged.
- Added regression tests for removed messages and a long-run
observational memory case.

## Validation

- The PR reports 4,147 tests passed and 67 skipped. Memory unit tests
report 2,669 passed, 1 skipped, and 1 todo.
- Core and memory type checks passed.
- The long-run test reports that the largest of the last 20 requests
decreased from 281,356 characters on `main` to 40,105 with this change.
- The `packages/memory/integration-tests` suite was not run.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-24 16:06:48 -07:00
Tyler Barnes
a282b3c70c
fix(memory): stop keeping a run's observed request in the prompt (#28364) 2026-10-23 23:13:20 -07:00
Tyler Barnes
01e060c971
fix(memory): keep context-overflow recovery working for the rest of a long run (#28358) 2026-10-23 19:48:04 -07:00
Leo
fe8c11d479
fix(memory): hide working-memory data on raw writes and edits (#28149)
## Description

OM buffer sealing and raw message writes could retain working-memory
tool arguments and tags that normal saves hide. Message edits bypassed
the same filter.

Apply the existing memory persistence projection to raw writes and the
fields explicitly replaced by message edits. Use effective content for
filtering decisions while preserving the caller's patch shape to avoid
overwriting concurrent edits. Keep omitted content fields, observation
markers, seal timestamps, ordinary tools (including ordinary partial
calls), and state-signal audit calls. Only working-memory tool calls and
tags are removed. No data migration.

Semantic recall indexes each edited message from its projected effective
text. Fields the edit sets come first; stored text is used only when the
projection hid the edited text, so an edit that leaves only a
`<working_memory>` block keeps the message searchable, while an edit
that clears text still removes its vectors. Edits that touch neither
`content.content` nor `parts` leave vectors alone (metadata-only edits
used to delete them), and repeated ids in one call are indexed once.

```diff
 raw upsert / message edit
-  write unfiltered content
+  derive effective content for filtering
+  write sanitized fields from the original patch
```

Before/after: `timeout 5m pnpm --filter @mastra/memory exec vitest run
src/persistence-projection.test.ts --maxWorkers=4 --bail=0` fails 5
cases on the original implementation and passes all 9 with the fix. The
failures expose leaked raw/edit content and an unfiltered
working-memory-only row.

The shared suite also passes all 9 cases against LibSQL, including real
Agent `generate` and `stream` turns crossing OM buffer seals. Agent
end-of-turn histories were already clean on the base revision; raw
writes and edits provide the failing regressions. The surrounding memory
and OM files pass 733 tests (1 existing todo).

The vector-index cases (working-memory-only edit, edited parts over a
stale mirror, explicit clear, repeated id, metadata-only edit) run
against a self-embedding mock vector store; the working-memory-only,
clear, repeated-id, and metadata-only cases failed before their fixes.
The full `@mastra/memory` unit suite passes (2664 tests).

Independent GPT-6.1 Sol review finished clean in round 2 after fixing a
concurrent-edit regression and legacy partial-call mirrors. A later Sol
pass on the indexing change found the clear and repeated-id cases, both
fixed. Left as is: when one call repeats an id, LibSQL and PostgreSQL
keep the first patch while the index follows the last. That adapter
inconsistency exists on main too. Package lint passes. The installed
TypeScript 6.0.3 compiler crashes before checking source; a fresh copy
of the same compiler passes both package configurations, as does
workspace TypeScript 7. The changeset CLI failed in its installed
fast-glob dependency, so the patch changeset was written manually.

Not tested: other storage adapters and durable/evented/Inngest engines.
The focused LibSQL run skips the harness's unrelated worker compilation
and fastembed model warmup. The dependency build emitted runtime files
but its core declaration step hit an existing `globalThis.crypto` error.

## Related issue(s)

Fixes #27946.

Related: #28143 changes the same memory source file for per-request OM
opt-out; this PR keeps its implementation limited to persistence
filtering.

## Type of change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [x] Test update

## Checklist

- [x] I have linked the related issue(s) in the description above
- [ ] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR

Created with GPT-6.1 Sol in T3 Code.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

Some message paths already hid working-memory details, but raw writes
and edits could still save them. This change applies the same cleanup to
those paths so recalled history stays consistent.

## Changes

- Apply the persistence projection in `saveMessages` and
`persistMessages`.
- In `updateMessages`, project the stored message together with the
patch. Preserve the patch shape so omitted fields are not sent in the
update.
- Preserve observation markers and `mastra.sealedAt` where required.
Keep ordinary partial tool calls.
- Use projected visible text for semantic-recall vector updates,
including when edits change only working-memory content or metadata.
- Add shared regression tests for writes, edits, recalled history, and
Agent generation and streaming. The reported suite passes with in-memory
storage and LibSQL.
- Add a patch changeset for `@mastra/memory`.

No data migration is included.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 13:13:34 -07:00
mastra-platform[bot]
75ffb145b4
test(memory): revise an activated assistant message in the activated-id test (#27893)
Co-authored-by: TylerBarnes <14190743+TylerBarnes@users.noreply.github.com>
2026-10-22 23:02:47 -04:00
mastra-platform[bot]
35806cb9b6
fix(memory): retain active user request when OM resumes on another worker (#28045)
> Opened by @abhiaiyer91

## Summary
Preserve the active user request when Observational Memory recreates an
in-flight turn on another worker. Reconcile forward-progressed stored
tool results with the recovered snapshot without replacing newer live
background results. Shared-LibSQL two-process AIMock coverage exercises
durable and evented recovery. Fixes #27614; response sealing in #27789
is separate.

## Recovery mechanism
`ObservationTurn.toJSON()` omits `drainedPartKeys`. At step > 0 the
processor restores request IDs from
`messageList.getPersisted.input.db()`. When storage has a completed tool
result missing from the snapshot, or a background task advances from
snapshot `running` to stored `completed`/`failed`, it seeds
`drainedPartKeys` with snapshot part keys and merges the stored
response. The subsequent OM step recognizes the new result as unread.
`MessageList` preserves an independently completed live background
result when another tool in the same stored response advanced. This
derives relevant part-level state; it does not serialize the original
turn map.

## Proof
On head `e26ee8588c`:
- Rebuilt `packages/core`; shared-LibSQL
`om-resume-aimock.scenario.test.ts` and `om-resume-prompt.test.ts`:
**18/18 passed**, including durable and evented
delayed-background-result requests at worker B's model boundary.
- Core MessageList suites: **440 passed, 2 skipped** across 29 files;
five adjacent OM suites, including `scorers-resume.test.ts`: **25/25
passed**.
- The mixed-progress same-ID response regression in
`stored-history-layering.test.ts` failed on source revert (stored
`running` instead of live `completed` at line 193), then passed with the
correction (**24/24** file tests). It covers stored tool A completed / B
running and snapshot A running / B completed, including background-task
metadata.
- Core `tsc --noEmit`, changed-file type-aware `oxlint` (0
errors/warnings), `oxfmt --check`, `git diff --check`, and core build
passed. On the preceding `7d23593f` head, the full memory unit suite
passed **2,629 tests** and memory typecheck/lint passed; these
full-suite checks were not rerun on this head.

## Unverified
Natural lease expiration (tests release ownership explicitly),
non-LibSQL adapters, evented pubsub worker affinity, full
integration-tests typecheck, and new-head CI/human review. The
mixed-progress case is verified through a real MessageList merge
regression, not a new two-process model-boundary scenario.

## Risk: medium
The forward-only recovery path reconciles completed tool outcomes, not
arbitrary changed parts. **Assumption:** keep the existing `stepNumber >
0` condition without an execution-mode discriminator; ordinary mid-run
turn recreation reaches this path, but no harm was reproduced after the
forward-progress gate. Persistence format is unchanged.

---------

Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-22 21:34:08 -05:00
Leo
12f5050847
fix(memory): honor per-request observational memory opt-out (#28143)
## Description

With observational memory enabled on a Memory instance, a per-request
`observationalMemory: false` or `{ enabled: false }` still attached its
processor. Normalization erased the override and restored the instance
default.

Preserve override presence when selecting the request config. Disabled
runs use message history without calling the observer or injecting
observations. The shared engine remains available for later and
concurrent enabled runs.

```diff
- select request config only when normalized override is truthy
+ select request config whenever an override was supplied
+ preserve a normalized disabled override as false
```

Before/after, from `packages/memory`:

`timeout 15m pnpm exec vitest run
src/processors/observational-memory/__tests__/mock-om-agent.test.ts
--maxWorkers=4 -t "Per-request observational memory options"`

- Before: failed because the observer was called despite an explicit
`false`.
- After: all 10 cases passed through real Agent.generate/stream, Memory,
and InMemoryStore. Cases cover both disable forms, omitted/true/object
options, history recall and persistence, and subsequent/concurrent
enabled runs.

Additional checks passed in a Boat Linux sandbox:

- `timeout 15m pnpm exec vitest run src/index.test.ts
src/processors/observational-memory/__tests__/mock-om-agent.test.ts
src/processors/observational-memory/__tests__/observational-memory.test.ts
--maxWorkers=4`: 737 passed, 1 existing todo.
- `timeout 15m pnpm check` and `timeout 15m pnpm lint` in
`packages/memory`.
- From `packages/memory/integration-tests`: `timeout 15m pnpm exec
vitest run src/with-libsql-storage.test.ts --maxWorkers=4 -t "persists
user then assistant with final text.*no bufferTokens"`: 1 passed.

Paid providers and the full external-storage/engine suites were not
tested; the regression substitutes only the model boundary and targets
this memory processor-selection path.

## Related issue(s)

Fixes #27814

## Type of change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [x] Test update

## Checklist

- [x] I have linked the related issue(s) in the description above
- [ ] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR

Created with GPT-6.1 Sol in T3 Code.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

A request can turn off observational memory even when the `Memory`
instance has it enabled. The request still uses message history, and
later requests can use observational memory as usual.

## Changes

- `Memory.createOMProcessor` now distinguishes an omitted
`observationalMemory` option from an explicit override.
- Per-request `false` and `{ enabled: false }` disable observational
memory instead of falling back to the instance setting.
- Added `generate` and `stream` tests for disabled, omitted, and enabled
options, including later and concurrent requests.
- Added a patch changeset for `@mastra/memory`.

## Validation

The author reports that all 10 focused regression cases passed. The
author also reports 737 passed and 1 existing todo in the memory test
run, successful `pnpm check` and `pnpm lint`, and 1 passing libSQL
integration test.

Paid-provider tests and the full external-storage and engine suites were
not run.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-22 12:41:46 -07:00
mastra-platform[bot]
e05356995e
fix(core): retain output processors on empty final steps with observational memory (#28046)
> Opened by @abhiaiyer91

Fixes #27920

## Outcome
When an agent uses observational memory and its final model step is
empty after a tool call, output processors can still append text. That
text is visible in the stream and saved in the assistant message for
durable and evented runs, matching the normal agent loop.

## Change
- Call
``messageList.enrichLastStepStart(`${currentModel.provider}/${currentModel.modelId}`)``
in the durable LLM path after materializing streamed messages and before
`processOutputStep`, guarded on both fields as
`llm-execution-step.ts:1762` does. That call stamps the trailing
unsealed assistant `step-start` with the executed model and moves the
message back into the `response` bucket when observational memory
drained it to `memory`, so output processors see the current step's
response.
- Add a real AIMock + Memory regression across normal, durable and
evented engines asserting processor response visibility, streamed text,
persisted assistant parts, and the persisted `step-start.model`.

## Proof
- Reverted the fix and rebuilt core: durable and evented each failed
with `processOutputStep` response count 0; normal passed. Restored fix
and rebuilt: all three passed.
- Reran the exact approved head after rebuilding core on October 22: the
three-engine regression passed 3/3.
- At `c7d1caeff`: removing the fix (test kept) fails durable and evented
with `responseCounts` `[0]` vs `[1]` while normal passes; reverting the
stamp to a bare `modelId` fails the persisted `model` assertion (`actor`
vs `openai.chat/actor`).
- Memory regression plus inflight OM tests: 11 passed; core adjacent
output-step/empty-turn scenarios: 58 passed, 2 existing expected-fail.
- Core and memory build TypeScript checks, changed-file oxlint and
oxfmt: pass.

## Unverified
- No external-provider/Inngest or full-repository test suite run; local
tests use real AIMock, Memory, and InMemoryStore.

## Risk: low
- Single durable-path call mirrors the normal loop: it stamps the
`step-start` model and re-buckets the message to `response` when
observational memory moved it, then output processors run against the
current step as they do in the default loop.

---------

Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-22 13:49:08 -05:00
Ward Peeters
461ee61237
fix: repair build and test failures surfaced by the release-PR run (#28139)
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
2026-10-22 16:48:29 +02:00
mastra-platform[bot]
be5c9292ac
fix(core): persist sub-agent suspensions after OM cold resume (#28050) 2026-10-21 22:57:01 -04:00
mastra-platform[bot]
db2a1a659e
fix(core): keep durable memory refs when token budgets annotate history (#28049) 2026-10-21 22:22:01 -04:00
Tyler Barnes
5b936b65d2
fix(memory): keep observed messages sealed when another worker recovers an OM turn (#27887)
## What

When a worker recovers an agent turn that another worker started (for
example after a durable agent's worker crashes), Observational Memory
could merge the rest of the reply into a message that had already been
observed. The stored conversation then differed from a run that never
crashed.

This relands #27800 (reverted in #27856) as a complete fix. When a turn
starts, OM checks whether the response message was already observed. If
it was, OM seals that message and starts a new one for the rest of the
reply.

A message counts as observed when any of these holds:
- Its id is in the record's observed-message list (sync observation).
- It carries OM's seal. Buffering persists the seal before the Observer
runs, so this still holds after the buffered chunk is activated.
- It has a completed observation end marker. Sync observation persists
that marker before it may reflect, so this still holds after a
reflection starts a new record.

Storage is unchanged. The check also requires the stored observed list
to be an array; a non-array value used to crash the turn with `includes
is not a function`.

## Tests

-
`packages/memory/integration-tests/src/cross-process/om-resume-prompt.test.ts`:
"stores the same messages as the single-process run" flips from
`it.fails` to `it` (durable and evented).
- New `recovered-turn-observed-ids.test.ts`:
  - recovery after buffered activation;
  - recovery after a real sync observation that reflected;
  - non-array stored values (no crash, no substring match);
  - an unobserved response is left alone.
- `ai-sdk-usage.test.ts`: `observe()` still picks up a later revision
saved under an activated id, and content appended to an observed message
after a reflection.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

When one worker continues a turn started by another worker, this change
prevents new text from being added to a response that was already saved.
It starts a new response message when the existing response was already
observed.

## Changes

- Detects observed responses by message ID, OM seal, or completed
observation marker.
- Validates that stored observed-message data is an array.
- Persists the OM seal when recovery identifies a message by ID.
- Preserves unloaded message parts when persisting the seal.
- Fails the step when seal persistence fails so a retry can attempt
sealing again.
- Rotates the response ID before continuing an observed response.
- Adds regression tests for recovery, buffering, reflection, malformed
values, message revisions, appended content, and unobserved responses.
- Enables the durable and evented cross-process recovery tests.
- Adds a patch changeset for `@mastra/memory`.

## Validation

The supplied context reports coverage for cross-process recovery,
recovered-turn observed-message handling, message revisions, and content
appended after reflection. Test execution results were not provided.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com>
2026-10-20 14:12:55 -07:00
mastra-platform[bot]
0e5d045be4
fix(core): stop the missing-memory warning for Observational Memory's internal agents (#27865) 2026-10-20 11:18:58 -07:00