0
Fork 0
mirror of https://github.com/mastra-ai/mastra.git synced 2026-11-17 08:44:46 +00:00
mastra/scripts
Tyler Barnes 5d2be9b551
ci(test-tiers): cover the full 14-day window, remember settled runs, and stop at 45 minutes (#30554)
## What

The daily test-tier update (`scripts/test-tiers/update.mjs daily`) reads
failed PR runs from the last 14 days to find catches: failures a PR then
fixed in source. A catch keeps the test file running on every PR. This
PR makes the update see the whole window while staying inside the
workflow's 60-minute job timeout.

- **Full window.** GitHub's workflow-run listing returns at most 1,000
runs per query, so the single 14-day query only reached back about 3.5
days. Runs are now listed one UTC day at a time, and a day with more
than 1,000 runs is split until each part fits.
- **Remember settled runs.** A run whose failures all have a final
verdict is recorded in `tiers.json` under `inspected_runs` (run id → `{
day, attempt }`) and its jobs are never fetched again. A rerun bumps
`run_attempt`, so it is inspected again. Entries leave with the window.
- **Time budget.** Runs are inspected newest first. The update stops
after 45 minutes (`--budget-minutes`) and saves what it has; the next
day picks up the rest. It also stops and saves when a GitHub error is
still failing after the retries; a permanent error still fails the run.
- **No move-outs on a partial day.** Files only move out on a day when
every run was inspected, because a run left for tomorrow may hold the
catch that keeps a file in.

## Rollout

The full window has about 3,200 failed or cancelled runs, and each takes
3–8 s to inspect. A 45-minute day covers roughly 300–700 runs, against
about 220 new ones a day. Each new day's runs are covered straight away,
and the backlog drains in about a week. Until it has, no files move out.

## Testing

- `node --test scripts/test-tiers/*.test.mjs`: 63/63 pass. The new tests
cover day-by-day listing over the 1,000 cap, splitting a busy day,
settled vs open runs, remembered runs not being fetched again, the time
budget, no move-out on a partial day, a GitHub outage, newest-first
ordering, a rerun of a remembered run, and entries saved before attempt
numbers were recorded. Each new test failed on its parent.
- Real dry runs against GitHub, writing only to a copy of `tiers.json`:
- Listing found 10.5k runs back to Nov 2 in about 8 minutes, where the
old query stopped at 1,000 Prebuild runs reaching back to Nov 12.
- Three budgeted runs (15, 12 and 11 minutes) each stopped on time, kept
every earlier entry without fetching it again, and moved nothing out.
  - The final-code run started on today's runs.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

**ELI5:** The test-tier job checks recent failed pull request runs to
find unstable test files. It used to check every run each day, which
took too long. Now it works through runs newest first, remembers the
runs it already settled, and stops after a time limit. It saves its
progress and continues on later runs.

## Overview

The change updates the daily test-tier update in
`scripts/test-tiers/update.mjs`. It lists PR runs one UTC day at a time
across the 14-day window. It records settled runs in `tiers.json` so it
does not fetch them again. It stops at a time budget or when transient
GitHub errors persist after retries. It moves test files out of the tier
only after every run for that day is inspected.

## Changed files

- `.github/workflows/test-tiers.yml` (+2/-1): Updates the job comment to
describe the new behavior. The diff changes only the comment.
- `scripts/test-tiers/update.mjs` (+221/-26): Adds day-based run
listing, budgeted inspection, settlement tracking, and deadline-bounded
GitHub and Git calls. Adds the `--budget-minutes` option, default 45.
- `scripts/test-tiers/update.test.mjs` (+368/-0): Adds tests for the new
functions. I did not read this file. The contents come from the
AI-generated summary only.

Totals for the PR: 3 files changed, 591 lines added, 27 lines removed.

## Behavior

- **Listing by day:** `listRunsByDay` lists runs for each inclusive UTC
day. If a day exceeds the 1,000-run cap (`RUN_LIST_CAP`), the range
splits recursively. A one-second range that is still over the cap logs a
warning and uses the runs it listed.
- **Time budget:** `DAILY_BUDGET_MINUTES` is 45. `inspectRuns` stops at
its deadline. GitHub and Git calls stop at the budget plus an 8-minute
grace period.
- **Order:** `inspectRuns` inspects the newest runs first.
- **Skipping runs:** A run with the same recorded attempt is skipped. A
rerun has a newer attempt, so it is inspected again.
- **Waiting runs:** Runs that are not yet settled are remembered as
waiting. `waitingRunReady` makes a waiting run eligible for recheck when
its recheck time arrives, or when a later successful run appears on the
same workflow, branch, and repository.
- **GitHub errors:** If a transient error persists after retries,
inspection ends for that day. Results gathered so far are kept. A
permanent error still fails the run.
- **Settlement and pruning:** `settleDay` records the day's results.
Legacy records without attempt data are inspected again.
- **Moving files out:** Files move out only when no runs remain for that
day.

## Verification status

- The author reports that `node --test scripts/test-tiers/*.test.mjs`
passed 63 of 63 tests. I did not run the tests. This result is not
verified.
- The author reports dry runs against GitHub. Listing found about 10.5k
runs back to Nov 2. Budgeted runs stopped on time and kept earlier
results. I did not verify these results.
- The author estimates that the backlog drains in about a week. Until
then, no test files move out.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com>
History 2026-11-16 11:14:20 -07:00
..
gh-dane-command refactor(mastracode): split into core, TUI, and standalone web packages (#18986) 2026-07-08 14:59:43 +02:00
gh-issue-triage chore(deps): use the latest @ai-sdk/provider-utils and @ai-sdk/provider patch in each major (#30518) 2026-11-16 15:17:56 +01:00
release-issue-notification chore(deps): update build tools (#27997) 2026-10-22 14:33:21 +02:00
test-tiers ci(test-tiers): cover the full 14-day window, remember settled runs, and stop at 45 minutes (#30554) 2026-11-16 11:14:20 -07:00
affected-tests-cache.mjs ci: cache what affected-test detection parses, so PR test selection takes ~4s instead of ~40s (#29133) 2026-11-05 20:02:44 +00:00
affected-tests.mjs ci: cache what affected-test detection parses, so PR test selection takes ~4s instead of ~40s (#29133) 2026-11-05 20:02:44 +00:00
check-core-imports.ts test(libsql): port T81 recovered-history cross-process coverage (#27945) 2026-10-21 10:20:41 -05:00
commonjs-tsc-fixer.js fix: resolve CodeQL sanitization and error-exposure alerts (#18805) 2026-07-02 15:26:54 +02:00
EMBEDDED_DOCS.md docs: make packages frontmatter more precise (#29499) 2026-11-04 17:49:54 -08:00
generate-package-docs.ts fix(docs): generate SOURCE_MAP.json from public export names (#29328) 2026-11-04 09:58:46 -08:00
ignore-example.js chore: migrate linting to oxlint (#20173) 2026-07-25 01:04:02 +02:00
install-example.js chore: migrate linting to oxlint (#20173) 2026-07-25 01:04:02 +02:00
madge.webpack.config.cjs Make the affected-test detector see packages that import with .js specifiers (#24488) 2026-09-21 14:58:17 -07:00
npm-placeholder-publish.mjs feat(ci): fail PRs and publishes on packages missing from npm (#26721) 2026-10-14 16:28:27 -07:00
npm-placeholder-publish.test.mjs feat(ci): fail PRs and publishes on packages missing from npm (#26721) 2026-10-14 16:28:27 -07:00
package-docs-source-map.test.ts fix(docs): generate SOURCE_MAP.json from public export names (#29328) 2026-11-04 09:58:46 -08:00
package-docs-source-map.ts fix(docs): generate SOURCE_MAP.json from public export names (#29328) 2026-11-04 09:58:46 -08:00
README-template.md chore(*): Update/unify/add READMEs (#22858) 2026-09-02 14:36:59 +02:00
test-migration-e2e.sh fix(storage): handle duplicate spans during migration (#12073) 2026-01-20 13:32:49 +01:00
test-observability.cjs chore: migrate linting to oxlint (#20173) 2026-07-25 01:04:02 +02:00
validate-peerdeps.mjs chore: remove stale scripts, workspace entries and package metadata (#29891) 2026-11-10 14:09:52 +01:00
workspace-source-aliases.cjs fix(ci): include exported server subpaths in affected tests (#21140) 2026-08-10 10:53:46 +02:00