0
Fork 0
mirror of https://github.com/mastra-ai/mastra.git synced 2026-11-17 08:44:46 +00:00
mastra/packages/memory/scripts/simulate/reconstruct.ts
Jamie Nichols b40fa91496
refactor(memory): curate observations directly in Subconscious (#22738)
## Summary

Simplify experimental Subconscious knowledge ingestion around two
built-ins, `remind` and `curate`, both configured in the observation
agent list:

- keep `remind` unchanged for selective retrieval of relevant durable
knowledge;
- make `curate` the sole experimental Subconscious ingestion and
reconciliation agent, receiving each successfully committed observation
directly;
- remove the separate `capture` and `learn` agents, reflection-time
Subconscious execution, and Subconscious-owned worklist/cursor/cadence
orchestration.

This is intentionally deletion-heavy to reduce two ingestion prompts and
quality surfaces to one: 40 files changed, with 1,047 additions and
4,032 deletions overall; Subconscious production source has 80 additions
and 718 deletions.

This is plan-driven work with no linked GitHub issue.

## How direct curation works

Observation strategies preserve the raw per-cycle observation delta
separately from cumulative observational memory. After persistence and
observation end markers succeed, the shared strategy lifecycle awaits
one failure-isolated curator invocation per nonblank source-thread delta
before normal reflection, so curation model latency is now part of every
observation-cycle path that produces a delta. Resource-scoped cycles
retain per-thread attribution and additionally run one curator call per
contributing thread delta, serially.

The curator receives the current observation delta, searches relevant
existing knowledge through constrained read tools, and creates or
reconciles durable knowledge through code-governed write tools.
Provenance, timestamps, source identity, scope ceilings, versioning,
activity, mention indexing, and semantic indexing remain code-owned
rather than model-supplied.

Reminder execution remains independent: reminder failure or no-signal
cannot suppress curator delivery. Curator failure is reported and
isolated after observation commit; it does not fail the committed
observation or prevent normal reflection.

## Deliberate reliability boundaries

Direct curator delivery is best-effort in this PR. Removing the durable
worklist/cursor frontier means a failed or interrupted curator run is
reported but is not automatically replayed. This PR does not introduce a
replacement queue, scheduler, retry, cadence, or backoff mechanism.

Core storage worklist/cursor APIs, including `knowledgeBySource`, remain
for other consumers such as Factory's knowledge route; the Subconscious
runtime stops using them for curation orchestration.

The curator continues to require `organizationId` in request context,
matching the prior capture path; a missing value is reported as a
curator error after observation commit.

The new `knowledge_create` tool preserves the existing node-first
storage sequence: if the initial record append fails after node
creation, the node may remain. No new cross-adapter transaction contract
is introduced.

## Compatibility and cross-package cleanup

- `@mastra/memory`: removes `capture`, `learn`, reflection-time
Subconscious execution, worklist/cursor/cadence orchestration, and the
public `Memory.runCuration()` method. There is no replacement manual
curation entry point; Subconscious curation now runs only from
successfully committed observations.
- `@mastra/code-sdk`: removes Factory-only capture-pinning and cadence
configuration.
- `@mastra/factory`: removes phase-exit curation triggers that depended
on `Memory.runCuration()`.
- Simulator: removes capture arms, cadence/tail flushing, cursor
accounting, and A/B comparison in favor of one direct-curator replay
path.

A minimal patch changeset is included for `@mastra/memory`,
`@mastra/code-sdk`, and `@mastra/factory`.

## Verification

- Focused Subconscious, lifecycle, replay, reminder, scope, activity,
read-tool, and knowledge-write coverage: 11 files, 59 tests passed.
- Full `@mastra/memory` unit suite: 63 files, 1,497 tests passed, 1
todo.
- Memory simulator suite: 3 files, 46 tests passed.
- `pnpm build:memory`: 15/15 tasks passed.
- Memory typecheck and lint passed; lint reported 0 warnings and 0
errors.
- `@mastra/factory` build, typecheck, full tests, and lint passed: 103
files, 1,980 tests passed, 6 skipped.
- `@mastra/code-sdk` build, typecheck, and lint passed; the changed
memory-construction suite passed 15/15 tests.
- Red/green lifecycle coverage and a deterministic two-cycle replay over
real in-memory storage verify direct delivery, reconciliation to one
active fact, later cross-thread reminder retrieval, and no Subconscious
worklist/cursor calls in the exercised paths. No live hosted-model run
is claimed.
- Removal greps, `git diff --check`, and clean-status checks passed.

Two broader gates could not complete cleanly; details follow.

## Verification limitations

The isolated `packages/memory/integration-tests` bootstrap could not
install dependencies in this environment: the frozen install reports an
override mismatch, and the required non-frozen retry cannot fetch
private `@internal/ai-v6` from the npm registry. The integration tests
therefore did not execute.

The full Code SDK suite completed with 105 files and 1,524 tests
passing, while four tests failed from startup timeout, `spawn E2BIG`,
callback-port `EADDRINUSE`, and an MCP OAuth callback assertion. These
failures were not used as evidence for this change; the changed SDK
memory-construction suite passes independently.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## ELI5

The PR makes `curate` the single agent that turns new observations into
knowledge. It removes older capture, learn, and automatic scheduling
paths so curation runs independently after observations are saved.

## Changes

- Refactor Subconscious ingestion around `remind` and `curate`.
- Dispatch committed observation deltas directly to
`SubconsciousCurateExtractor`.
- Make curation asynchronous and failure-isolated.
- Remove reflection-time execution, capture and learn agents, curation
cadence, worklist orchestration, and cursor management.
- Preserve reminder execution, provenance, scope enforcement, and
code-owned indexing.
- Add `knowledge_create` with scoped writes and provenance metadata.
- Validate RFC 3339 timestamps for knowledge writes.
- Harden observation framing against untrusted input.
- Remove related public APIs and types from Memory, Code SDK, Factory,
and simulator integrations.
- Replace the simulator A/B workflow with direct observation replay.
- Add tests for replay isolation, curation behavior, scope enforcement,
provenance, failure handling, and input validation.
- Add a patch changeset for `@mastra/memory`, `@mastra/code-sdk`, and
`@mastra/factory`.

## Validation

Focused tests, memory and simulator suites, builds, typechecks, linting,
and relevant Factory and Code SDK checks passed. Memory integration
tests were blocked by dependency installation issues. Four unrelated
failures remained in the full Code SDK suite; changed SDK tests passed
independently.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Jamie Nichols <CodingButter@users.noreply.github.com>
Co-authored-by: Mastra Code (openai/gpt-5.6-sol) <noreply@mastra.ai>
Co-authored-by: Jamie Nichols <10267569+CodingButter@users.noreply.github.com>
Co-authored-by: Tyler Barnes <tylerdbarnes@gmail.com>
2026-09-01 23:03:56 -07:00

191 lines
6.5 KiB