mirror of
https://github.com/mastra-ai/mastra.git
synced 2026-11-17 08:44:46 +00:00
## Summary Add local conversation simulation tooling to `@mastra/memory` for comparing experimental Subconscious capture and curation prompt changes against reconstructed real observation cycles. ## Root cause Evaluating prompt changes through deployments mixes prompt behavior with newly derived cycle boundaries and requires days of organic usage. Re-observing raw messages would also let the tested prompt move segmentation boundaries, making A/B results look meaningful while changing more than one variable. The simulator tests are colocated with the implementation they cover under `packages/memory/scripts/simulate`, which sits outside the package's original `src/**/*.test.ts` Vitest include. The package include is widened to `['src/**/*.test.ts', 'scripts/**/*.test.ts']` so they are discovered and run as part of the normal unit suite. ## What changed - Add local extract, replay, and A/B commands under `packages/memory/scripts/simulate`. - Reconstruct persisted observation cycles while excluding reflection heads from later generations. - Reject duplicate `generationCount` records as ambiguous instead of replaying them in unstable order. - Preserve deterministic ordering and report missing-generation warnings. - Replay through the real capture and curation path with optional production knowledge-resource scoping. - Isolate every arm in a fresh local database and refuse remote write targets. - Add an A/A control to expose model sampling noise. - Keep the simulator suites beside the implementation under `packages/memory/scripts/simulate`, named after the module each covers (`diff.test.ts`, `extract.test.ts`, `replay.test.ts`, `reconstruct.test.ts`, `drive.test.ts`, `drive.isolation.test.ts`), and widen the package Vitest include so they are still collected by the unit run. - Add simulator TypeScript checking to package verification. - Export the existing observation-boundary regex for simulator reuse without changing its pattern or runtime behavior. ## Design notes This is a bounded simulator, not a general PostgreSQL cloning tool. Generated target DDL does not preserve every PostgreSQL type, constraint, default, or precision detail. Worklist counting is capped, missing generations produce incomplete comparisons with warnings, and the content differ remains aggregate and positional in several cases. The replay command dynamically imports the built PostgreSQL adapter, so linked workspace packages must be built before running the simulator from a fresh checkout. The README documents the local PostgreSQL, pgvector, credentials, and build prerequisites. The widened Vitest include makes the simulator suites part of the default `@mastra/memory` unit run while leaving them colocated with the tooling. Normal production TypeScript scope remains unchanged because package TypeScript excludes `**/*.test.ts`; simulator checking remains owned by `tsconfig.scripts.json`. ## Safety - Source access is read-only. - Replay targets must use a literal IPv4 or IPv6 loopback address; hostnames and routing query parameters such as `host`, `hostaddr`, and `service` are rejected. - Extraction, standalone replay, and database recreation attest the live PostgreSQL server address as loopback before target initialization, writes, or destructive operations. - Extraction compares connected database identity and port, treating IPv4 and IPv6 loopback as the same local server, and refuses to overwrite the source database. - Each arm uses a separate fresh database. - No conversation or knowledge payload is committed. Tracking: COR-1167 ## Test plan From `packages/memory`: ```bash pnpm vitest run scripts/simulate --reporter=dot pnpm test:unit pnpm lint git diff --check ``` Results on the reviewed local head: - Simulator selection: 78 passed across 6 files. - Full memory unit suite: 1,602 passed and 1 todo across 68 files. - Lint: 0 warnings and 0 errors across 207 files. - Diff check: passed. - Simulator suites are discovered at their colocated path by the widened include; the full unit run collects 69 files with the suites in place. The production-source blast radius is limited to exporting the existing observation-boundary regex. Its pattern and runtime behavior are unchanged. No changeset is included: this adds repo-local tooling under `scripts/`, not a shipped package feature. --------- Co-authored-by: Jamie Nichols <CodingButter@users.noreply.github.com> Co-authored-by: Jamie Nichols <10267569+CodingButter@users.noreply.github.com> Co-authored-by: Mastra Code (openai/gpt-5.6-sol) <noreply@mastra.ai>
194 lines
5.7 KiB
194 lines
5.7 KiB