0
Fork 0
mirror of https://github.com/mastra-ai/mastra.git synced 2026-11-17 08:44:46 +00:00
mastra/packages/_test-utils
Tyler Barnes ce3be915c9
test(core): share the module cache across core unit test files (#29131)
## What

Core's unit tests run in two vitest projects:

- `unit:packages/core`: 1,418 files with `isolate: false`. Each worker
loads core once instead of re-importing it for every test file.
- `unit:packages/core-isolated`: 121 files that still run isolated, each
listed in `packages/core/vitest.isolated-tests.ts` with its reason
(`vi.mock`, env read at import, global stubs such as `fetch` or
`process.on`, `vi.resetModules`, or a heap assertion).

No test file or assertion changes.

## Why

Most of core's unit-test worker time went to importing core again for
every file. Sharing the module cache cuts import time from ~1,050s to
~210s of worker time, and the full core lane from 780s to 433–505s
(median ~465s, −40%), measured on CI's machine shape (5 vCPU, Node
24.22.1).

## How leaks are kept out

The shared test setup restarts its deterministic UUID counter for every
test, so every file creates runs with ids like `…0001`. Core keeps a few
process-wide maps keyed by run id (run registry, thread-stream runtime,
run lease fences). With isolation off, a later file could find an
earlier file's entry for the same id.

- `packages/core/vitest.no-isolate-setup.ts` runs after each file in the
shared project and clears those maps through existing reset hooks, plus
a new test-only `__resetRunLeaseFencesForTests`. Imports are lazy so
files using `vi.mock` keep their load order.
- `packages/_test-utils/src/setup.ts` creates one `AsyncLocalStorage`
per worker instead of one per file.
- `packages/core/vitest.config.ts` points anyone adding a mock or
env-setting test at the isolated list, and says to rerun a
shared-project failure alone or with `--sequence.shuffle.files` before
blaming the file.

## Testing

- All 1,539 core files under 9 shuffled file orders: only failures are
the 3 durable-agent tests fixed on main by #29081 (fix for #29065
stubs), one known heap-assertion file (now isolated) and one
load-sensitive real-timer test (`signals-claimed-ownership`, passes
15/15 alone).
- Live ref'd timers per worker stay bounded (max 132, no monotonic
growth).
- `tsc --noEmit` clean.

This is a one-week CI trial: retry counts on core unit tests are watched
for a week before calling it done.

Stacked on #29100 (stop sleeping through retry backoff and lease waits
in unit tests), which it builds on.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

Most core unit tests now share one worker, so the code they load is
reused between files. This makes the core test lane faster. Tests that
change global state, mock modules, or depend on timing still run in
their own isolated project.

## Changes

- Splits core unit tests into two Vitest projects in
`packages/core/vitest.config.ts`:
- `unit:packages/core` sets `isolate: false`. It excludes end-to-end
tests and isolated tests. It uses `vitest.no-isolate-setup.ts`.
- `unit:packages/core-isolated` runs the files in `isolatedTestFiles`
with the standard setup.
- Defines shared empty API-key environment values in `vitest.config.ts`
for both projects.
- Adds `isolatedTestFiles` in `packages/core/vitest.isolated-tests.ts`.
The file contains 140 lines that match a `.test.ts` search. The PR
objectives state that 121 files stay isolated. The raw summary states
141 paths. This count does not match the objectives and is not
reconciled.
- Adds `vitest.no-isolate-setup.ts` for the shared project. Before each
test, it sets the UUID store counter to a range assigned to the current
file. After each file, it resets the global run registry, registry
activity, execution fences, thread-stream runtime, and run-lease fences.
- Changes `uuidStore` in `packages/_test-utils/src/setup.ts`. It reuses
one `AsyncLocalStorage` per worker. The instance is stored on
`globalThis.__mastraTestUuidStore` and created only when that property
is nullish.
- Adds `__resetRunLeaseFencesForTests()` in
`packages/core/src/agent/run-lease-fence.ts`. It clears all registered
run lease checks. The function is for tests only.

The diff changes 5 files, with 252 lines added and 8 removed.

## Verification

- The PR objectives report timing results and a nine-order shuffled run
of all core files. These results are not verified in this review.
- No CI results are available for the current head.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com>
History 2026-11-12 21:05:07 -07:00
..
__recordings__ feat(llm-recorder): add transformRequest option and useLiveMode() per-test helper (#13298) 2026-02-23 01:50:42 +01:00
src test(core): share the module cache across core unit test files (#29131) 2026-11-12 21:05:07 -07:00
CHANGELOG.md chore: version - exit prerelease mode 2026-11-12 21:12:29 +00:00
lint-staged.config.js refactor(lint): replace ESLint with type-aware oxlint (#27476) 2026-10-18 10:38:35 -07:00
package.json chore: version - exit prerelease mode 2026-11-12 21:12:29 +00:00
README.md feat(test-utils): createLLMMock/createGatewayMock for recording/replaying LLM API calls (#13923) 2026-03-09 11:33:12 +01:00
tsconfig.json chore: migrate package builds to tsdown (#19829) 2026-07-28 12:32:21 +02:00
tsdown.config.ts test(core): consolidate agent one-off tests into cross-engine aimock scenarios (#27493) 2026-10-18 12:47:10 -07:00
vitest.config.ts feat(llm-recorder): add transformRequest option and useLiveMode() per-test helper (#13298) 2026-02-23 01:50:42 +01:00

@internal/test-utils

Mastra-specific test helpers. Provides version-agnostic agent wrappers, API key management, and provider-scoped LLM mocking for integration tests.

Note

: This is an internal package. Not for public consumption.

For LLM recording/replay, see @internal/llm-recorder.

Installation

{
  "devDependencies": {
    "@internal/test-utils": "workspace:*"
  }
}

API Reference

Version-Agnostic Agent Helpers

Helper functions for writing tests that work with both AI SDK v4 and v5+ models:

import { agentGenerate, agentStream, isV5PlusModel, getModelRecordingName } from '@internal/test-utils';

agentGenerate(agent, message, options, model)

Calls agent.generate() (v5+) or agent.generateLegacy() (v4), transforming threadId/resourceId to the memory: { thread, resource } format for v5+. Also maps output → structuredOutput for v5+.

const result = await agentGenerate(agent, 'Hello', { threadId, resourceId }, model);

// With structured output (v5+ uses structuredOutput, v4 uses output)
const result = await agentGenerate(agent, 'Extract data', { threadId, output: mySchema }, model);

agentStream(agent, message, options, model)

Calls agent.stream() (v5+) or agent.streamLegacy() (v4), transforming parameters the same way as agentGenerate.

const stream = await agentStream(agent, 'Count to 5', { threadId }, model);

isV5PlusModel(model)

Check if a model uses the v5+ API:

isV5PlusModel('openai/gpt-4o'); // true (string models)
isV5PlusModel({ specificationVersion: 'v2' }); // true
isV5PlusModel({ specificationVersion: 'v1' }); // false

getModelRecordingName(model)

Convert a model config to a recording-safe filename:

getModelRecordingName('openai/gpt-4o-mini'); // "openai-gpt-4o-mini"
getModelRecordingName({ modelId: 'gpt-4o' }); // "gpt-4o"

API Key Management

setupDummyApiKeys(mode, providers?)

Set placeholder API keys for replay mode so agent validation passes without real credentials:

import { setupDummyApiKeys } from '@internal/test-utils';
import { getLLMTestMode } from '@internal/llm-recorder';

setupDummyApiKeys(getLLMTestMode()); // All providers
setupDummyApiKeys(getLLMTestMode(), ['openai']); // Just OpenAI
setupDummyApiKeys('live'); // No-op in live/record mode

hasApiKey(provider)

Check if an API key is set:

import { hasApiKey } from '@internal/test-utils';

hasApiKey('openai'); // checks OPENAI_API_KEY
hasApiKey('anthropic'); // checks ANTHROPIC_API_KEY
hasApiKey('google'); // checks GOOGLE_API_KEY
hasApiKey('openrouter'); // checks OPENROUTER_API_KEY

Provider-Scoped LLM Mocking

Mock specific LLM providers while leaving others live. Built on @internal/llm-recorder for recording/replay.

Returns a self-contained instance — no global state. You control the lifecycle.

import { createLLMMock } from '@internal/test-utils';

createLLMMock(model, options?)

Create a mock by wrapping a real AI SDK model instance. The mock reads its provider and modelId for naming the recording file. MSW intercepts all LLM API traffic.

import { openai } from '@ai-sdk/openai';

describe('OpenAI agent', () => {
  const mock = createLLMMock(openai('gpt-4o'));

  beforeAll(() => mock.start());
  afterAll(() => mock.saveAndStop());

  it('generates a response', async () => {
    const result = await agent.generate('Hello');
    expect(result.text).toBeDefined();
  });
});

Works with any AI SDK model instance:

createLLMMock(openai('gpt-4o')); // recording tagged with openai.chat + gpt-4o
createLLMMock(anthropic('claude-3')); // recording tagged with anthropic + claude-3

Note

: For gateway/string models like 'openai/gpt-4o', use createGatewayMock() instead.

The returned LLMMock instance has:

Property / Method Description
providerId Extracted provider (e.g. "openai")
modelId Extracted model if present (e.g. "gpt-4o")
recordingName Name used for the recording file
mode Current test mode (record, replay, auto, live)
start() Start intercepting requests
saveAndStop() Save recordings and stop intercepting
recorder Underlying LLMRecorderInstance for advanced use

Options:

  • name — Explicit recording name (auto-derived from test file path if omitted)
  • recordingsDir — Directory for recording files (default: __recordings__ in cwd)
  • forceRecord — Force re-record even if recording exists
  • replayWithTiming — Replay with original chunk timing
  • maxChunkDelay — Max delay between chunks in replay, ms (default: 10)
  • transformRequest — Transform requests before hashing
  • extraHosts — Additional API hosts to intercept (merged with auto-detected ones)
  • debug — Enable verbose debug logging

Utility Functions

extractProviderId(modelRouterId)

Extract the provider from a model router ID:

extractProviderId('openai/gpt-4o'); // 'openai'
extractProviderId('netlify/anthropic/claude-3'); // 'anthropic'
extractProviderId('azure-openai/my-deployment'); // 'azure-openai'

extractModelId(modelRouterId)

Extract the model from a model router ID:

extractModelId('openai/gpt-4o'); // 'gpt-4o'
extractModelId('netlify/anthropic/claude-3'); // 'claude-3'
extractModelId('openai'); // undefined

PROVIDER_HOSTS

Mapping of known provider IDs to their API hosts. Exported for advanced use cases.