0
Fork 0
mirror of https://github.com/mastra-ai/mastra.git synced 2026-11-17 08:44:46 +00:00
mastra/packages/evals
History 2026-11-03 23:45:58 +00:00
..
__recordings__ fix(core): make the native Agent goal mechanism robust (tools, waiting, budget, judge-failure) (#18016) 2026-06-17 20:14:02 +02:00
src chore(deps): update build tools (major) (#29026) 2026-11-03 20:18:21 +01:00
.gitignore Mastra Evals Package Suite (#1412) 2025-01-16 19:19:30 -08:00
CHANGELOG.md chore: version - exit prerelease mode 2026-11-03 23:45:58 +00:00
lint-staged.config.js refactor(lint): replace ESLint with type-aware oxlint (#27476) 2026-10-18 10:38:35 -07:00
package.json chore: version - exit prerelease mode 2026-11-03 23:45:58 +00:00
README.md chore(*): Update/unify/add READMEs (#22858) 2026-09-02 14:36:59 +02:00
tsconfig.build.json chore: fix go-to-definition using tsc declaration maps (#6089) 2025-07-30 15:41:00 -07:00
tsconfig.json chore: migrate package builds to tsdown (#19829) 2026-07-28 12:32:21 +02:00
tsdown.config.ts feat(evals): Vitest integration for runEvals (@mastra/evals/vitest) (#22665) 2026-09-01 11:31:30 +02:00
turbo.json chore: migrate package builds to tsdown (#19829) 2026-07-28 12:32:21 +02:00
vitest.config.ts fix(core,evals): add LLM recording to evals, fix autoResume and Gemini e2e tests (#14054) 2026-03-10 10:33:39 +01:00

@mastra/evals

@mastra/evals ships a collection of scoring utilities you can run locally or inside your own evaluation pipelines. These scorers come in two flavors:

  • LLM scorers – leverage a judge model (e.g. OpenAI, Anthropic) to rate responses for qualities such as faithfulness or toxicity.
  • Code/NLP scorers – deterministic heuristics (keyword coverage, similarity, etc.) that do not require an external model.

Installation

npm install @mastra/evals

Usage

Import prebuilt scorers from the package subpath.

import { createToolCallAccuracyScorerCode } from '@mastra/evals/scorers/prebuilt';

const scorer = createToolCallAccuracyScorerCode({ expectedTool: 'weather' });

Documentation

Changelog

See the package changelog for version history and release notes.

Support

We have an open community Discord. Come and say hello and let us know if you have any questions or need any help getting things running.