0
Fork 0
mirror of https://github.com/mastra-ai/mastra.git synced 2026-11-17 08:44:46 +00:00
mastra/packages/evals
Tyler Barnes 51c340a782
test(core,mcp): stop sleeping through retry backoff and lease waits in unit tests (#29100)
## What this does

About 390 s of core unit-test time was spent really sleeping through
retry backoff: provider retry-after waits (up to 30 s each), `waitDelay`
backoff, `p-retry` delays, and `fetchWithRetry`'s own `setTimeout`. This
makes those waits instant in tests, while the files that test the delays
themselves keep real timing.

- `packages/_test-utils/src/setup.ts` (the shared core test setup) mocks
`waitDelay` globally with a version that yields one microtask and still
honours `abortSignal`. It also sets `p-retry`'s
`minTimeout`/`maxTimeout` to 0, because the AI SDK v5 stream path
retries through `p-retry` and never calls `waitDelay`.
- Files that assert on the delay itself opt out at the top of the file.
For `waitDelay`, that's `vi.unmock('<path>/utils/retry-after')`, in all
seven: model-retry-after, stream-error-retry-processor,
llm-execution-step, thread-lease-loss-abort, fetchWithRetry, utils, and
download-assets. For `p-retry`, it's
`vi.mock('<path>/core/node_modules/p-retry', importOriginal =>
importOriginal())`, in the two that also need p-retry's real backoff
(model-retry-after, thread-lease-loss-abort); `vi.unmock` on that path
leaves p-retry's lazy import unresolved under fake timers. The comments
in `packages/_test-utils/src/setup.ts` say the same.
- `signal-messages.test.ts` advances fake timers through its post-abort
teardown wait instead of waiting 30 s.
- **Product change:** `fetchWithRetry`
(`packages/core/src/utils/fetchWithRetry.ts`) now waits through the
shared `waitDelay` helper instead of its own `setTimeout`. Retry timing
and abort behaviour are unchanged, and it lets the global test mock
reach attachment downloads (`attachment-download-recovery` 60 s → 3.1
s). Changeset: `@mastra/core` patch.

No test is removed or skipped, and no assertion changes.

## Numbers (local, the 127–128 core files touching retry/backoff)

| | wall | test time | passed |
|---|---:|---:|---:|
| before | 792 s | 686.8 s | 3413 |
| after | 661 s → 153 s on the full lever stack | 546.1 s | 3413 (3419
with download-assets' opt-out tests counted) |

Largest single wins: an evented test 60 s → 0.75 s (two 30 s retry-after
waits), signal-messages 30 s → 1 s, per-file `waitDelay` users 123 s →
12 s. Total ≈ 390 s of test time, about 8.5% of Affected UNIT worker
time on a core PR.

## Background-task lease tests

`BackgroundTaskManager` rejects leases under 3s and floors its heartbeat
and stale-recovery delays at 1s, so tests that prove a lease expires
wait several real seconds per case. An internal, test-only env var
`MASTRA_BACKGROUND_TASKS_MIN_LEASE_MS` can only lower that minimum. It's
read when each manager is constructed, not at import, and the 1s floor
scales down with it (minimum / 3). With the env unset, every value is
the same as before: a 3000ms minimum, a 1000ms floor, and the same error
text. `manager-owners.test.ts` and the mcp
`server-tasks-background.test.ts` (plus the child server it spawns) run
on a 600ms lease. `manager-lifecycle.test.ts` still asserts the real
3000ms rejection.

- mcp `server-tasks-background.test.ts`: 198.7s → 81.8 / 79.8 / 78.6s,
48/48 each run
- core `manager-owners.test.ts`: 44s → ~11s, 83/83 (with lifecycle) over
3 runs

## Test plan

- [x] Retry/backoff file set green before and after (above)
- [x] Opt-out files still see real delays (their delay assertions pass)
- [x] fetchWithRetry tests pass with real timing
- [ ] CI: Affected UNIT green




<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

This PR makes unit tests run faster. Tests skip most real waiting, and
tests that check real delays still use them. Background-task tests can
use a shorter lease through a test-only setting. The production defaults
do not change.

## Changes

- `fetchWithRetry` (`packages/core/src/utils/fetchWithRetry.ts`) uses
the shared `waitDelay` helper for retry backoff. After the delay, it
checks the abort signal and throws the abort reason if the signal was
aborted.
- The shared Vitest setup (`packages/_test-utils/src/setup.ts`) mocks
core's `waitDelay` to yield one microtask. The mock still honors an
already-aborted signal. It also sets core's `p-retry` `minTimeout` and
`maxTimeout` to `0` and `randomize` to `false`.
- Setting `MASTRA_TEST_REAL_RETRY_DELAYS=1` makes the setup use the
original modules. The `e2e:packages/core` and `e2e:packages/evals`
Vitest projects set this variable.
- Tests that assert real delay behavior unmock `retry-after`. These
files include `model-retry-after.test.ts`,
`thread-lease-loss-abort.test.ts`, `download-assets.test.ts`,
`llm-execution-step.test.ts`, `stream-error-retry-processor.test.ts`,
`utils.test.ts`, and `fetchWithRetry.test.ts`.
- `signal-messages.test.ts` uses fake timers to advance through the
30-second post-abort teardown wait. It restores the timer substitution
in a `finally` block.
- `retry-after.e2e.test.ts` adds checks that `waitDelay(50)` and a
one-retry `p-retry` run take at least 45 ms.
- `BackgroundTaskManager`
(`packages/core/src/background-tasks/manager.ts`) reads the internal
`MASTRA_BACKGROUND_TASKS_MIN_LEASE_MS` override when a manager is
constructed. The override applies only when the value is finite,
positive, and below 3000 ms. The minimum lease defaults to 3000 ms. The
heartbeat and stale-recovery floor is one third of the minimum, which is
1000 ms by default.
- `manager-owners.test.ts` sets the minimum lease to 600 ms in
`beforeAll`. It derives the heartbeat interval from that value.
- The MCP background-task tests (`server-tasks-background.test.ts` and
`background-tasks-child.ts`) use a 600 ms lease. The lease reaches child
servers as a command-line argument. The `Probe` fixture gains a `hold`
promise. The open-task limit test uses it to keep tasks open until the
limit check completes.
- Changesets:
- `fetch-with-retry-wait-delay.md` declares a patch for `@mastra/core`.
- `full-humans-glow.md` declares a patch for `@mastra/core` only. The
commit message says `@mastra/mcp` was removed from this changeset.

## Validation

- The author reports 3,413 tests passed before and after the change in
the measured core file set. The count is 3,419 when `download-assets`
opt-out tests are included.
- The author reports these local timings:
  - `signal-messages.test.ts`: 30 seconds to 1 second.
  - `manager-owners.test.ts`: 44 seconds to about 11 seconds.
- The author reports that `manager-lifecycle.test.ts` still checks the
real 3000 ms rejection.
- Current-head test results: not available.
- CI status: not available. The "Affected UNIT green" check is unchecked
in the PR objectives.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com>
History 2026-11-12 21:05:06 -07:00
..
__recordings__ fix(core): make the native Agent goal mechanism robust (tools, waiting, budget, judge-failure) (#18016) 2026-06-17 20:14:02 +02:00
src chore(deps): update build tools (major) (#29026) 2026-11-03 20:18:21 +01:00
.gitignore Mastra Evals Package Suite (#1412) 2025-01-16 19:19:30 -08:00
CHANGELOG.md chore: version - exit prerelease mode 2026-11-03 23:45:58 +00:00
lint-staged.config.js refactor(lint): replace ESLint with type-aware oxlint (#27476) 2026-10-18 10:38:35 -07:00
package.json chore: version - exit prerelease mode 2026-11-03 23:45:58 +00:00
README.md chore(*): Update/unify/add READMEs (#22858) 2026-09-02 14:36:59 +02:00
tsconfig.build.json chore: fix go-to-definition using tsc declaration maps (#6089) 2025-07-30 15:41:00 -07:00
tsconfig.json chore: migrate package builds to tsdown (#19829) 2026-07-28 12:32:21 +02:00
tsdown.config.ts feat(evals): Vitest integration for runEvals (@mastra/evals/vitest) (#22665) 2026-09-01 11:31:30 +02:00
turbo.json chore: migrate package builds to tsdown (#19829) 2026-07-28 12:32:21 +02:00
vitest.config.ts test(core,mcp): stop sleeping through retry backoff and lease waits in unit tests (#29100) 2026-11-12 21:05:06 -07:00

@mastra/evals

@mastra/evals ships a collection of scoring utilities you can run locally or inside your own evaluation pipelines. These scorers come in two flavors:

  • LLM scorers – leverage a judge model (e.g. OpenAI, Anthropic) to rate responses for qualities such as faithfulness or toxicity.
  • Code/NLP scorers – deterministic heuristics (keyword coverage, similarity, etc.) that do not require an external model.

Installation

npm install @mastra/evals

Usage

Import prebuilt scorers from the package subpath.

import { createToolCallAccuracyScorerCode } from '@mastra/evals/scorers/prebuilt';

const scorer = createToolCallAccuracyScorerCode({ expectedTool: 'weather' });

Documentation

Changelog

See the package changelog for version history and release notes.

Support

We have an open community Discord. Come and say hello and let us know if you have any questions or need any help getting things running.