mirror of
https://github.com/mastra-ai/mastra.git
synced 2026-11-17 08:44:46 +00:00
## Summary On pull requests, CI now skips test files and whole CI jobs that haven't failed in the last 30 days. Main still runs everything, so a quiet test that breaks is still caught on main. - `scripts/test-tiers/` builds `tiers.json` from the last 30 days of CI failures on main and PRs. A file or job goes back "in" as soon as it fails, and drops "out" after 30 days without a failure. The scheduled `test-tiers.yml` workflow keeps that file current on the `ci/test-tiers` branch. - `select-affected-tests.cjs` removes "out" unit and package E2E files before the files are split across shards. Prebuild exposes `tier_skip_jobs` so the store, workspace, E2E and mastracode workflows can skip "out" jobs. - Everything runs when: the PR has the `full-tests` label, the PR changes the tier files or a job's own workflow, or `tiers.json` or the PR's labels can't be read. ## Measurement One-line change to `packages/core/src/agent/agent.ts`, with Affected UNIT run as CI runs it (3 shards, 4 CPUs): | | Unit files | Slowest shard | |---|---|---| | Before | 3,208 | 639 s | | After | 1,118 | 361 s | Package E2E files go from 49 to 3, and 9 whole jobs are skipped. ## Test plan - 110 routing tests, including 7 new tier-routing cases: dropped files and shard count, job gates, the edited-path override, workspace and cloud package lists, the `full-tests` label, and no tiers file. - 38 tier script tests. - 18 parity scenarios against the previous selection. - actionlint and oxfmt; the one actionlint finding is pre-existing on main. - Timing measured on 4-CPU runners: the numbers above. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 This change makes pull request CI run fewer tests. Tests and CI jobs that have not failed in the last 30 days are skipped. If a skipped test or job fails again, it returns to the normal set. ## Overview This change adds test-tier filtering to pull request CI. A tier file, `tiers.json` on the `ci/test-tiers` branch, lists unit test files, package E2E files, and CI jobs as `in` (run) or `out` (skip). A scheduled workflow updates the file from recent failures. Rules: - Full-suite runs, the `full-tests` label, and missing or invalid tier data run everything. - Changing a tier file, or changing a job's own workflow, runs all tests. - An `out` file is dropped only when the pull request does not change it. - An `out` job is skipped only when the pull request does not change that job's owned paths. - Filtering runs before unit sharding. - The workspace artifact keeps the full workspace selection. ## Changes by area **Routing and selection** - `.github/scripts/select-affected-tests.cjs`: `routeTests` accepts optional `tiers` and `storeNames`. Its result includes retained tests, dropped files, and skipped jobs. It also emits workspace packages for unit and Docker runs, omitting quiet packages. Store, memory, E2E, experiment-worker E2E, MastraCode E2E, and workspace flags depend on whether their jobs are skipped. - `scripts/test-tiers/select.mjs` (new): loads tier configuration, classifies E2E files, selects unit tests and jobs, and provides a `skip-jobs` CLI. - `.github/workflows/prebuild.yml`: the `affected-tests` job exposes `tier_skip_jobs`. It reads tier configuration from `ci/test-tiers` and reads PR labels. A label-read failure runs all tests. It passes `skip_jobs` to the reusable E2E, Mastra Code E2E, and combined-store workflows. It adds `issues: read` permission. **Workflow inputs and job gates** - `.github/workflows/e2e-tests.yml`: new optional input `skip_jobs` (default `'[]'`). Nine E2E jobs run only when their `job:e2e/...` key is not listed. The E2E registry publisher and the E2E workflow gate stay enabled when tiered E2E jobs are quiet, because npm import needs the registry. - `.github/workflows/mastracode-e2e.yml`: new optional input `skip_jobs`. Skips `mastracode-e2e` when the list contains `job:mastracode-e2e`. Skips `mastracode-web-e2e` when the list contains `job:mastracode-om`. - `.github/workflows/test-combined-stores.yml`: new optional input `skip_jobs`. Removes stores listed as `job:store/<name>` before splitting outputs. The `test` job runs only when its store output is nonempty. - `.github/workflows/lint.yml`: the `changeset-check` job applies tier selection to its validator tests. It forces full tests if the changed-file diff fails. It skips nothing if tier selection fails. It adds a step that runs the test-tier script tests. - `.github/workflows/secrets.test-workspaces.yml`: removes cloud-test packages whose `job:workspace-cloud/<package>` tier is `out`. Changeset-release branches, the `full-tests` label, and failures to read PR labels or changed files select full tests. Invalid tier data skips no packages. If no packages remain, `workspaces-changed` becomes `false`. **Tier data maintenance** - `scripts/test-tiers/update.mjs` (new): CLI with `versioning` and `daily` commands. It records failures and settles whole-job failures. It removes deleted files, adds new files and job keys in `all` scope, and moves entries that stay clean for `WINDOW_DAYS` to `out`. It requires `--tiers`. `versioning` also requires `--run-id`. `daily` defaults the lookback to 14 days. - `scripts/test-tiers/seed.mjs` (new): generates initial tier data from file ages, qualification records, failure indexes, and rename mappings. It validates job keys and tier values. - `.github/workflows/test-tiers.yml` (new): runs after completed Prebuild workflows, on a daily schedule, or manually. It runs only in `mastra-ai/mastra` and serializes runs. It commits and pushes changed tiers to `ci/test-tiers`. A manual run with `seed=true` creates the branch. **Seed data** - `scripts/test-tiers/seed-data/fail-index.json`, `jobs-seed.json`, and `renames.json`: seed inputs. `jobs-seed.json` covers 54 jobs. `renames.json` contains 669 path mappings. **Tests** - `.github/scripts/select-affected-tests.test.ts`, `scripts/test-tiers/select.test.mjs`, `scripts/test-tiers/seed.test.mjs`, and `scripts/test-tiers/update.test.mjs`: cover routing, tier selection, seeding, and tier updates. ## Reported results (author-reported, not verified here) - One-line change to `packages/core/src/agent/agent.ts`, measured on 4-CPU runners: - Affected unit files: 3,208 to 1,118. - Slowest shard: 639 to 361 seconds. - Package E2E files: 49 to 3. Nine whole jobs skipped. - Reported test plan: 110 routing tests (7 new tier-routing cases), 38 tier-script tests, 18 parity scenarios, actionlint, and oxfmt. - Reported: one actionlint finding exists on main before this change. - CI status on the current head: not available. - Review severity counts: not available. ## Review notes - The diff has 20 files, with 12,163 lines added and 12 removed. Most added lines are seed data (`fail-index.json`, `renames.json`, `jobs-seed.json`). - Reviewers should check the paths that decide whether tests are skipped first. These are tier loading failures, label-read failures, full-suite mode, and the changed-file rules. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com> |
History
|
|
|---|---|---|
| .. | ||
| fail-index.json | ||
| file-ages.tsv | ||
| jobs-seed.json | ||
| qualify.tsv | ||
| renames.json | ||