mirror of
https://github.com/mastra-ai/mastra.git
synced 2026-11-17 08:44:46 +00:00
## What this is
**Knowledge** is Mastra's store for durable knowledge that agents share
across conversations. Until now its agent-facing parts lived in
`@mastra/memory`, inside an experimental feature called
**Subconscious**, a background curator that wrote to Knowledge after
observational-memory reflections.
This PR makes Knowledge stand on its own as an experimental alpha,
rebuilt around pages and declared scopes.
- **Subconscious is removed** from `@mastra/memory` and Mastra Code.
Setting `experimental_subconscious` or the Memory `knowledge` option
throws, saying to attach Knowledge to the agent instead. Old
Subconscious data is ignored.
- **Knowledge attaches to an agent directly:** `new Agent({ knowledge,
scopes })`. It brings its own tools, instructions and pinned context.
Tools work without Memory; pinned context needs Memory.
- **Knowledge is a tree of pages.** Each page lives in one scope and
holds a body plus short notes. Notes can mention other pages with
`[[wikilinks]]`. A page can have a key, unique among its siblings, and
is addressed by key path: `<scope>#<key>/<child key>`. Pages declared in
a scope's config are created when missing and can be edited but not
moved, deleted or merged away by tools.
- **Access comes from declared scopes.** Agent `scopes` are plain
`"<type>:<value>"` strings that say who the run is, set by the host and
never by the model. Knowledge's required `scopes` option declares which
scope types exist and which roles (viewer, contributor, editor, owner)
holders get on which scopes. Rules reach one hop with no inheritance. A
run holding an undeclared scope fails. `knowledgeMemoryScopes()`
declares Memory's `thread:`/`resource:` scopes and an `org:` scope.
- **Five tools:** `knowledge_search`, `knowledge_read`, `knowledge_page`
(write, move, merge, delete), `knowledge_note` (add, restamp, remove)
and `knowledge_pin` (pin, unpin). Tools are off by default: `tools:
true` enables read and write, `tools: { read: true }` read only, and
`pinnedContext` adds pins. Deleting a page needs owner on the page's
scope and is refused for pages with child pages, pages declared in a
scope's config, and scope pages. On a scope page, write accepts the body
only: its name and description come from config.
- **Limits are in tokens:** about 250 per note and 100 per page
description. Pinned context defaults to 500 tokens, with a maximum of
2,000.
- **Search:** keyword search by default: every word of the query must
appear in the page (or note), and name matches rank first, the same in
every store. With a `vector` store and an `embedder`, search also
returns semantic matches. Every semantic hit is re-checked against what
the run can read before ranking, so hidden content never takes a result
slot. `knowledge.reindex()` re-embeds everything.
- **Host APIs:** `knowledge.readScopeFor(principals)` returns what
holders of those scopes can read. `knowledge.explainAccess(runScopes,
target)` says why a run can or can't reach a scope or page.
`explainAccess` is host-only.
- **Renamed scope declarations:** if stored scopes use a declaration
name the config no longer declares, Knowledge fails loudly on first use,
naming each orphaned name and how many scopes use it.
`knowledge.renameScopeDeclaration(from, to)` renames them. It is
host-only and atomic, except on MongoDB without a replica set.
- **Per-message scopes:** `session.sendMessage({ content, scopes })`,
plus queue, steer and notifications. A message whose scopes differ from
the running run's waits for that run to end and then starts its own run.
Scopes are set in-process only.
- **Storage:** LibSQL, PostgreSQL, MySQL and MongoDB store Knowledge as
schema 4 (pages, notes, mentions, pins). On startup, the tables of the
earlier experimental Knowledge node store (shipped in 1.79.0) are
dropped with any rows in them, and the new tables are created. If an
older process recreates them during a rolling deploy, the next startup
drops them again. Schema-4 tables and non-Knowledge tables are never
touched. On MongoDB, Knowledge collections found without a schema
version are adopted as schema 4 rather than refused. Don't run
`dangerouslyReset()` from an older version against a database the new
version already uses. Development databases created from earlier commits
of this branch also need a reset, since sibling key uniqueness changed
within schema 4. `storage.stores?.knowledge?.dangerouslyReset()` clears
Knowledge rows only, never Memory. Adapter peer floors are unchanged: on
an older `@mastra/core` the adapters still load and the other storage
domains keep working.
- **Mastra Code:** Knowledge is opt-in through **Experimental
Knowledge** in `/settings`. The `/knowledge` browser and the SDK
inspector read pages. Factory's Knowledge routes are disabled until the
Studio page readers land.
## Test plan
- Shared schema-4 storage contract suite on real stores (as of
d5b72de125): LibSQL 59/59, MySQL 60/60, PostgreSQL 69/69, MongoDB
61/61. LibSQL Knowledge index suite 58/58 at 5b137a7677, including
startup over a published 1.79.0 layout (empty and with rows), old tables
recreated under a schema-4 database, and four concurrent startups. On
MySQL, concurrent scope materialization can deadlock on InnoDB gap
locks; it retries up to 5 times on duplicate-key or deadlock errors (31
consecutive passing runs of the concurrency test).
- Core Knowledge suites: declarations, access, page tools for the spec's
scenarios 1–13, pins, token limits, `explainAccess` agreement with the
tools (8 actors × 12 pages), and semantic search (25 hidden
higher-scoring pages never hide the visible result).
- Integration on real LibSQL and PostgreSQL: an agent writes a page,
note and pin. An outsider can't read them. They persist across a restart
and pinned context is delivered. On startup the 1.79.0 node-store tables
are dropped, rows included, while schema-4 and non-Knowledge tables are
left alone.
- Independent validation on real LibSQL: scenarios 1–13 including 6 and
11 with a real agent, and live semantic search through OpenRouter
`openai/text-embedding-3-small` with LibSQLVector, isolated per tenant.
- Mastra Code: inspector, TUI browser and Knowledge-browser E2E.
- `tsc`, `oxlint` and `oxfmt` clean on touched packages. Each milestone
was reviewed independently.
---------
Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com>
Co-authored-by: avis-reader[bot] <332681745+avis-reader[bot]@users.noreply.github.com>
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
1.8 KiB
1.8 KiB
Conversation simulation
Extract recorded Observational Memory generations into a local input database and reconstruct their observation-cycle boundaries.
Flow
- Extract copies a bounded set of threads, messages, and Observational Memory generations from a Mastra-schema Postgres database into a local input database. The source is opened read-only.
- Reconstruct rebuilds the recorded observation-cycle boundaries. It does not rerun the observer, so prompt changes cannot silently alter those boundaries.
Lifecycle ordering across sync, buffered, idle, and resource-scoped observation strategies is covered by the observation-strategy tests. This simulator starts at the already-completed observation boundary.
Extract
pnpm simulate:extract \
--source "$SIMULATE_SOURCE_URL" \
--target "postgres://user@127.0.0.1:55432/simulate_input" \
--threads 5
| Flag | Meaning |
|---|---|
--source <url> |
Required. Opened read-only; never written. |
--target <url> |
Required. Must use a literal loopback address. |
--threads <n> |
Most-recent N threads with an OM record. |
--thread-id <id> |
Repeatable explicit IDs; mutually exclusive with --threads. |
The final lines report EXTRACTED_THREADS=, EXTRACTED_MESSAGES=, and EXTRACTED_OM_RECORDS=.
Reconstruction boundaries
Generation-zero leading observation text is kept. Leading text on later generations is a reflection head and is excluded. Duplicate generations are rejected; missing generations, empty chunks, and unparseable dates are reported as warnings. Input reconstruction currently supports thread-scoped OM records only.