0
Fork 0
mirror of https://github.com/mastra-ai/mastra.git synced 2026-11-17 08:44:46 +00:00
mastra/scripts/test-tiers/seed-data
Tyler Barnes add6f402d6
ci: skip test files and jobs that have not failed in 30 days (test tiers) (#30374)
## Summary

On pull requests, CI now skips test files and whole CI jobs that haven't
failed in the last 30 days. Main still runs everything, so a quiet test
that breaks is still caught on main.

- `scripts/test-tiers/` builds `tiers.json` from the last 30 days of CI
failures on main and PRs. A file or job goes back "in" as soon as it
fails, and drops "out" after 30 days without a failure. The scheduled
`test-tiers.yml` workflow keeps that file current on the `ci/test-tiers`
branch.
- `select-affected-tests.cjs` removes "out" unit and package E2E files
before the files are split across shards. Prebuild exposes
`tier_skip_jobs` so the store, workspace, E2E and mastracode workflows
can skip "out" jobs.
- Everything runs when: the PR has the `full-tests` label, the PR
changes the tier files or a job's own workflow, or `tiers.json` or the
PR's labels can't be read.

## Measurement

One-line change to `packages/core/src/agent/agent.ts`, with Affected
UNIT run as CI runs it (3 shards, 4 CPUs):

| | Unit files | Slowest shard |
|---|---|---|
| Before | 3,208 | 639 s |
| After | 1,118 | 361 s |

Package E2E files go from 49 to 3, and 9 whole jobs are skipped.

## Test plan

- 110 routing tests, including 7 new tier-routing cases: dropped files
and shard count, job gates, the edited-path override, workspace and
cloud package lists, the `full-tests` label, and no tiers file.
- 38 tier script tests.
- 18 parity scenarios against the previous selection.
- actionlint and oxfmt; the one actionlint finding is pre-existing on
main.
- Timing measured on 4-CPU runners: the numbers above.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

This change makes pull request CI run fewer tests. Tests and CI jobs
that have not failed in the last 30 days are skipped. If a skipped test
or job fails again, it returns to the normal set.

## Overview

This change adds test-tier filtering to pull request CI. A tier file,
`tiers.json` on the `ci/test-tiers` branch, lists unit test files,
package E2E files, and CI jobs as `in` (run) or `out` (skip). A
scheduled workflow updates the file from recent failures.

Rules:

- Full-suite runs, the `full-tests` label, and missing or invalid tier
data run everything.
- Changing a tier file, or changing a job's own workflow, runs all
tests.
- An `out` file is dropped only when the pull request does not change
it.
- An `out` job is skipped only when the pull request does not change
that job's owned paths.
- Filtering runs before unit sharding.
- The workspace artifact keeps the full workspace selection.

## Changes by area

**Routing and selection**

- `.github/scripts/select-affected-tests.cjs`: `routeTests` accepts
optional `tiers` and `storeNames`. Its result includes retained tests,
dropped files, and skipped jobs. It also emits workspace packages for
unit and Docker runs, omitting quiet packages. Store, memory, E2E,
experiment-worker E2E, MastraCode E2E, and workspace flags depend on
whether their jobs are skipped.
- `scripts/test-tiers/select.mjs` (new): loads tier configuration,
classifies E2E files, selects unit tests and jobs, and provides a
`skip-jobs` CLI.
- `.github/workflows/prebuild.yml`: the `affected-tests` job exposes
`tier_skip_jobs`. It reads tier configuration from `ci/test-tiers` and
reads PR labels. A label-read failure runs all tests. It passes
`skip_jobs` to the reusable E2E, Mastra Code E2E, and combined-store
workflows. It adds `issues: read` permission.

**Workflow inputs and job gates**

- `.github/workflows/e2e-tests.yml`: new optional input `skip_jobs`
(default `'[]'`). Nine E2E jobs run only when their `job:e2e/...` key is
not listed. The E2E registry publisher and the E2E workflow gate stay
enabled when tiered E2E jobs are quiet, because npm import needs the
registry.
- `.github/workflows/mastracode-e2e.yml`: new optional input
`skip_jobs`. Skips `mastracode-e2e` when the list contains
`job:mastracode-e2e`. Skips `mastracode-web-e2e` when the list contains
`job:mastracode-om`.
- `.github/workflows/test-combined-stores.yml`: new optional input
`skip_jobs`. Removes stores listed as `job:store/<name>` before
splitting outputs. The `test` job runs only when its store output is
nonempty.
- `.github/workflows/lint.yml`: the `changeset-check` job applies tier
selection to its validator tests. It forces full tests if the
changed-file diff fails. It skips nothing if tier selection fails. It
adds a step that runs the test-tier script tests.
- `.github/workflows/secrets.test-workspaces.yml`: removes cloud-test
packages whose `job:workspace-cloud/<package>` tier is `out`.
Changeset-release branches, the `full-tests` label, and failures to read
PR labels or changed files select full tests. Invalid tier data skips no
packages. If no packages remain, `workspaces-changed` becomes `false`.

**Tier data maintenance**

- `scripts/test-tiers/update.mjs` (new): CLI with `versioning` and
`daily` commands. It records failures and settles whole-job failures. It
removes deleted files, adds new files and job keys in `all` scope, and
moves entries that stay clean for `WINDOW_DAYS` to `out`. It requires
`--tiers`. `versioning` also requires `--run-id`. `daily` defaults the
lookback to 14 days.
- `scripts/test-tiers/seed.mjs` (new): generates initial tier data from
file ages, qualification records, failure indexes, and rename mappings.
It validates job keys and tier values.
- `.github/workflows/test-tiers.yml` (new): runs after completed
Prebuild workflows, on a daily schedule, or manually. It runs only in
`mastra-ai/mastra` and serializes runs. It commits and pushes changed
tiers to `ci/test-tiers`. A manual run with `seed=true` creates the
branch.

**Seed data**

- `scripts/test-tiers/seed-data/fail-index.json`, `jobs-seed.json`, and
`renames.json`: seed inputs. `jobs-seed.json` covers 54 jobs.
`renames.json` contains 669 path mappings.

**Tests**

- `.github/scripts/select-affected-tests.test.ts`,
`scripts/test-tiers/select.test.mjs`,
`scripts/test-tiers/seed.test.mjs`, and
`scripts/test-tiers/update.test.mjs`: cover routing, tier selection,
seeding, and tier updates.

## Reported results (author-reported, not verified here)

- One-line change to `packages/core/src/agent/agent.ts`, measured on
4-CPU runners:
  - Affected unit files: 3,208 to 1,118.
  - Slowest shard: 639 to 361 seconds.
  - Package E2E files: 49 to 3. Nine whole jobs skipped.
- Reported test plan: 110 routing tests (7 new tier-routing cases), 38
tier-script tests, 18 parity scenarios, actionlint, and oxfmt.
- Reported: one actionlint finding exists on main before this change.
- CI status on the current head: not available.
- Review severity counts: not available.

## Review notes

- The diff has 20 files, with 12,163 lines added and 12 removed. Most
added lines are seed data (`fail-index.json`, `renames.json`,
`jobs-seed.json`).
- Reviewers should check the paths that decide whether tests are skipped
first. These are tier loading failures, label-read failures, full-suite
mode, and the changed-file rules.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com>
History 2026-11-13 16:42:42 -08:00
..
fail-index.json ci: skip test files and jobs that have not failed in 30 days (test tiers) (#30374) 2026-11-13 16:42:42 -08:00
file-ages.tsv ci: skip test files and jobs that have not failed in 30 days (test tiers) (#30374) 2026-11-13 16:42:42 -08:00
jobs-seed.json ci: skip test files and jobs that have not failed in 30 days (test tiers) (#30374) 2026-11-13 16:42:42 -08:00
qualify.tsv ci: skip test files and jobs that have not failed in 30 days (test tiers) (#30374) 2026-11-13 16:42:42 -08:00
renames.json ci: skip test files and jobs that have not failed in 30 days (test tiers) (#30374) 2026-11-13 16:42:42 -08:00