## What Core's unit tests run in two vitest projects: - `unit:packages/core`: 1,418 files with `isolate: false`. Each worker loads core once instead of re-importing it for every test file. - `unit:packages/core-isolated`: 121 files that still run isolated, each listed in `packages/core/vitest.isolated-tests.ts` with its reason (`vi.mock`, env read at import, global stubs such as `fetch` or `process.on`, `vi.resetModules`, or a heap assertion). No test file or assertion changes. ## Why Most of core's unit-test worker time went to importing core again for every file. Sharing the module cache cuts import time from ~1,050s to ~210s of worker time, and the full core lane from 780s to 433–505s (median ~465s, −40%), measured on CI's machine shape (5 vCPU, Node 24.22.1). ## How leaks are kept out The shared test setup restarts its deterministic UUID counter for every test, so every file creates runs with ids like `…0001`. Core keeps a few process-wide maps keyed by run id (run registry, thread-stream runtime, run lease fences). With isolation off, a later file could find an earlier file's entry for the same id. - `packages/core/vitest.no-isolate-setup.ts` runs after each file in the shared project and clears those maps through existing reset hooks, plus a new test-only `__resetRunLeaseFencesForTests`. Imports are lazy so files using `vi.mock` keep their load order. - `packages/_test-utils/src/setup.ts` creates one `AsyncLocalStorage` per worker instead of one per file. - `packages/core/vitest.config.ts` points anyone adding a mock or env-setting test at the isolated list, and says to rerun a shared-project failure alone or with `--sequence.shuffle.files` before blaming the file. ## Testing - All 1,539 core files under 9 shuffled file orders: only failures are the 3 durable-agent tests fixed on main by #29081 (fix for #29065 stubs), one known heap-assertion file (now isolated) and one load-sensitive real-timer test (`signals-claimed-ownership`, passes 15/15 alone). - Live ref'd timers per worker stay bounded (max 132, no monotonic growth). - `tsc --noEmit` clean. This is a one-week CI trial: retry counts on core unit tests are watched for a week before calling it done. Stacked on #29100 (stop sleeping through retry backoff and lease waits in unit tests), which it builds on. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 Most core unit tests now share one worker, so the code they load is reused between files. This makes the core test lane faster. Tests that change global state, mock modules, or depend on timing still run in their own isolated project. ## Changes - Splits core unit tests into two Vitest projects in `packages/core/vitest.config.ts`: - `unit:packages/core` sets `isolate: false`. It excludes end-to-end tests and isolated tests. It uses `vitest.no-isolate-setup.ts`. - `unit:packages/core-isolated` runs the files in `isolatedTestFiles` with the standard setup. - Defines shared empty API-key environment values in `vitest.config.ts` for both projects. - Adds `isolatedTestFiles` in `packages/core/vitest.isolated-tests.ts`. The file contains 140 lines that match a `.test.ts` search. The PR objectives state that 121 files stay isolated. The raw summary states 141 paths. This count does not match the objectives and is not reconciled. - Adds `vitest.no-isolate-setup.ts` for the shared project. Before each test, it sets the UUID store counter to a range assigned to the current file. After each file, it resets the global run registry, registry activity, execution fences, thread-stream runtime, and run-lease fences. - Changes `uuidStore` in `packages/_test-utils/src/setup.ts`. It reuses one `AsyncLocalStorage` per worker. The instance is stored on `globalThis.__mastraTestUuidStore` and created only when that property is nullish. - Adds `__resetRunLeaseFencesForTests()` in `packages/core/src/agent/run-lease-fence.ts`. It clears all registered run lease checks. The function is for tests only. The diff changes 5 files, with 252 lines added and 8 removed. ## Verification - The PR objectives report timing results and a nine-order shuffled run of all core files. These results are not verified in this review. - No CI results are available for the current head. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com> |
History
|
|
|---|---|---|
| .. | ||
| __recordings__ | ||
| src | ||
| CHANGELOG.md | ||
| lint-staged.config.js | ||
| package.json | ||
| README.md | ||
| tsconfig.json | ||
| tsdown.config.ts | ||
| vitest.config.ts | ||
@internal/test-utils
Mastra-specific test helpers. Provides version-agnostic agent wrappers, API key management, and provider-scoped LLM mocking for integration tests.
Note
: This is an internal package. Not for public consumption.
For LLM recording/replay, see
@internal/llm-recorder.
Installation
{
"devDependencies": {
"@internal/test-utils": "workspace:*"
}
}
API Reference
Version-Agnostic Agent Helpers
Helper functions for writing tests that work with both AI SDK v4 and v5+ models:
import { agentGenerate, agentStream, isV5PlusModel, getModelRecordingName } from '@internal/test-utils';
agentGenerate(agent, message, options, model)
Calls agent.generate() (v5+) or agent.generateLegacy() (v4), transforming threadId/resourceId to the memory: { thread, resource } format for v5+. Also maps output → structuredOutput for v5+.
const result = await agentGenerate(agent, 'Hello', { threadId, resourceId }, model);
// With structured output (v5+ uses structuredOutput, v4 uses output)
const result = await agentGenerate(agent, 'Extract data', { threadId, output: mySchema }, model);
agentStream(agent, message, options, model)
Calls agent.stream() (v5+) or agent.streamLegacy() (v4), transforming parameters the same way as agentGenerate.
const stream = await agentStream(agent, 'Count to 5', { threadId }, model);
isV5PlusModel(model)
Check if a model uses the v5+ API:
isV5PlusModel('openai/gpt-4o'); // true (string models)
isV5PlusModel({ specificationVersion: 'v2' }); // true
isV5PlusModel({ specificationVersion: 'v1' }); // false
getModelRecordingName(model)
Convert a model config to a recording-safe filename:
getModelRecordingName('openai/gpt-4o-mini'); // "openai-gpt-4o-mini"
getModelRecordingName({ modelId: 'gpt-4o' }); // "gpt-4o"
API Key Management
setupDummyApiKeys(mode, providers?)
Set placeholder API keys for replay mode so agent validation passes without real credentials:
import { setupDummyApiKeys } from '@internal/test-utils';
import { getLLMTestMode } from '@internal/llm-recorder';
setupDummyApiKeys(getLLMTestMode()); // All providers
setupDummyApiKeys(getLLMTestMode(), ['openai']); // Just OpenAI
setupDummyApiKeys('live'); // No-op in live/record mode
hasApiKey(provider)
Check if an API key is set:
import { hasApiKey } from '@internal/test-utils';
hasApiKey('openai'); // checks OPENAI_API_KEY
hasApiKey('anthropic'); // checks ANTHROPIC_API_KEY
hasApiKey('google'); // checks GOOGLE_API_KEY
hasApiKey('openrouter'); // checks OPENROUTER_API_KEY
Provider-Scoped LLM Mocking
Mock specific LLM providers while leaving others live. Built on @internal/llm-recorder for recording/replay.
Returns a self-contained instance — no global state. You control the lifecycle.
import { createLLMMock } from '@internal/test-utils';
createLLMMock(model, options?)
Create a mock by wrapping a real AI SDK model instance. The mock reads its provider and modelId for naming the recording file. MSW intercepts all LLM API traffic.
import { openai } from '@ai-sdk/openai';
describe('OpenAI agent', () => {
const mock = createLLMMock(openai('gpt-4o'));
beforeAll(() => mock.start());
afterAll(() => mock.saveAndStop());
it('generates a response', async () => {
const result = await agent.generate('Hello');
expect(result.text).toBeDefined();
});
});
Works with any AI SDK model instance:
createLLMMock(openai('gpt-4o')); // recording tagged with openai.chat + gpt-4o
createLLMMock(anthropic('claude-3')); // recording tagged with anthropic + claude-3
Note
: For gateway/string models like
'openai/gpt-4o', usecreateGatewayMock()instead.
The returned LLMMock instance has:
| Property / Method | Description |
|---|---|
providerId |
Extracted provider (e.g. "openai") |
modelId |
Extracted model if present (e.g. "gpt-4o") |
recordingName |
Name used for the recording file |
mode |
Current test mode (record, replay, auto, live) |
start() |
Start intercepting requests |
saveAndStop() |
Save recordings and stop intercepting |
recorder |
Underlying LLMRecorderInstance for advanced use |
Options:
name— Explicit recording name (auto-derived from test file path if omitted)recordingsDir— Directory for recording files (default:__recordings__in cwd)forceRecord— Force re-record even if recording existsreplayWithTiming— Replay with original chunk timingmaxChunkDelay— Max delay between chunks in replay, ms (default: 10)transformRequest— Transform requests before hashingextraHosts— Additional API hosts to intercept (merged with auto-detected ones)debug— Enable verbose debug logging
Utility Functions
extractProviderId(modelRouterId)
Extract the provider from a model router ID:
extractProviderId('openai/gpt-4o'); // 'openai'
extractProviderId('netlify/anthropic/claude-3'); // 'anthropic'
extractProviderId('azure-openai/my-deployment'); // 'azure-openai'
extractModelId(modelRouterId)
Extract the model from a model router ID:
extractModelId('openai/gpt-4o'); // 'gpt-4o'
extractModelId('netlify/anthropic/claude-3'); // 'claude-3'
extractModelId('openai'); // undefined
PROVIDER_HOSTS
Mapping of known provider IDs to their API hosts. Exported for advanced use cases.