mirror of
https://github.com/mastra-ai/mastra.git
synced 2026-11-17 08:44:46 +00:00
## Why Test tiers skip test files and jobs on PRs when they haven't failed in 30 days. The full suite still runs on the alpha versioning PR (`changeset-release/main`), so a merge that breaks a skipped test goes green on its PR. The failure then turns up later on the versioning run, with nobody pointed at it. That's what happened with #29370, which broke `goal-step.test.ts`, and #30434, which broke the Playground autosave test. ## What this does When a versioning run fails in a file or job that tiers had skipped, the tier update now passes those escapes to a new `attribute` job. That job: 1. Finds the last green versioning run (up to 72 hours back) and lists the first-parent merges on `main` between it and the red head (at most 60). 2. Bisects those merges, running only the escaped test file at each step. It checks out the commit, installs, builds the package and its dependencies from the read-only Turbo cache, and runs Vitest across the `unit`, `typecheck`, `e2e:packages/*` and `e2e:client-sdks/*` projects, the same ones the PR jobs use. 3. Names a merge only when the file fails there twice and passes at its parent. A file that passes at the red head is reported as not reproduced (a flake). A file that also fails before the range, or can't be run, is unconfirmed. Nobody is tagged in either case. When a file is reported by more than one project, any failure wins. The file passes only if every result passed, and a file with no result is unconfirmed. A separate `notify` job holds the Slack token and sends the messages. If `attribute` fails or times out (45 minutes), `notify` sends the plain fallback alert that `update` prepared. The escaped file goes back into the PR tier for 30 days, as before. ### Who gets tagged The message names the PR, its title and its author, tagged through `.github/slack-users.json`. A login that isn't in the map is printed as a plain `@login`. When the PR was opened by a bot, the bot can't answer for it, so the message tags a person instead and says who they are to the PR: 1. the human who merged it ("merged by"); otherwise 2. every human whose latest review approved it ("approved by"); otherwise 3. the human(s) credited on the most commits, counting commit authors and `Co-authored-by` trailers with GitHub noreply addresses ("co-authored by"); otherwise 4. `@here`. `@here` is for when we can't tell who the person is. It is not for a person we know but who is missing from the Slack map: they still get a plain `@login`. Reviews and commits are read across every page. A lookup that fails, or a commit list at GitHub's 250-commit cap (which may be cut off), counts as unknown, and that rung is skipped rather than guessed from. PR titles are escaped, so a title can't inject `<!here>` or other mentions. ## Testing - `node --test scripts/test-tiers/*.test.mjs`: 96 pass on Node 22 and 24. The new tests fail with the attribution module removed. Each stand-in rung, the title escaping, any-failure-wins aggregation, E2E project selection and pagination fails its own test when that rule is removed. - Replays against real versioning runs, at the real commits: - #29370 (`goal-step.test.ts`, run 43907413456): found merge `929c8ab` over 3 merges in 182 s. - #30434 (autosave test, run 43750070686): found merge `b666fdb` over 2 merges in 495 s. The PR was opened by a bot, so the message reads "by mastra-platform[bot] (bot), merged by @abhiaiyer91". - A flaky `execute-tool-core.test.ts` failure (run 43848445791): passed at the red head, so not reproduced and no alert, in 64 s. - The runner, run for real on `main`, passed a unit file and a package E2E file and returned an error for a missing file. That took 51 s. - `actionlint` reports only the custom runner label. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ELI5: When a versioning test fails in a file that the test tiers had skipped, this PR finds the code change that caused it. It then posts a Slack message that names the likely author. If that search cannot finish, it posts a simpler alert instead. ## Overview This PR adds attribution for test failures from versioning runs. The failing file or job was skipped by the test tiers. The PR adds a new `attribute` job. The job searches for the failing commit range and reports the merge that introduced the failure. A separate `notify` job posts a fallback alert if attribution fails or times out. ## Changes - **`scripts/test-tiers/attribute.mjs`** (new, +455): Main attribution logic. - Finds files and jobs that moved from tier `out` to `in`. - Selects the latest successful versioning run from the preceding 72 hours. - Considers at most 60 first-parent merges between the green run and the failing run. - Bisects those merges by running the escaped file. - Reports a merge only when the file fails there twice and passes at its parent. - Reports a file as not reproduced when it passes at the failing head. - Reports a file as unconfirmed when it also fails before the range or cannot run. - Resolves culprit PRs and selects stand-ins for bot-authored PRs. - Generates Slack messages with escaped mrkdwn. - Runs the `unit`, `typecheck`, `e2e:packages/*`, and `e2e:client-sdks/*` projects. - Passes a remaining-time timeout to each command. The script defaults `DEADLINE_MINUTES` to 38. The PR description states a 45-minute job deadline. The source does not yet confirm how these two values relate. - **`scripts/test-tiers/attribute.test.mjs`** (new, +518): Tests for the functions above. Covers bisect steps, verdicts, stand-ins, alert grouping, paginated lookups, and deadline-aware runner timeouts. - **`scripts/test-tiers/update.mjs`** (+9/-1): Imports `escapedFiles`. Adds an optional `--escapes-out=<file>` argument to the versioning command. Snapshots tier data before applying failures. Writes the run ID and escape details to the output file when the argument is set. - **`.github/workflows/test-tiers.yml`** (+106/-2): Records escapes in job outputs. Runs the `attribute` job when escapes exist. Posts attribution results or the fallback `notify` alert. Documents the new behavior. - **`.github/slack-users.json`** (+1): Adds an empty JSON object. ## Alert behavior - For human-authored PRs, the alert names the PR, title, and author. - For bot-authored PRs, the alert names, in order: 1. The human who merged the PR. 2. If unavailable, the humans with the latest approving reviews. 3. If unavailable, the most-credited commit authors. 4. If none are available, `@here`. - Unknown Slack mappings appear as plain `@login`. - Titles are escaped to prevent injected mentions. ## Result semantics - A file that fails in any project result is a failure. - A file passes only when every project result passes. - A file with no project result is unconfirmed. - An escaped file returns to the PR tier for 30 days. ## Verification - The PR author reports 96 passing tests on Node 22 and 24, using `node --test scripts/test-tiers/*.test.mjs`. This was not verified in this review. - The PR author reports replays against two real versioning runs that found the breaking merges. This was not verified in this review. - The PR author reports a flaky failure that was not reproduced and did not alert. This was not verified in this review. - The PR author reports a real `main` runner check that passed unit and package E2E files and errored on a missing file. This was not verified in this review. ## Scope - Total: 5 files changed, +1089 / -3 lines. - No exported or public declarations were removed. - New exported functions and constants are listed in the `attribute.mjs` source. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com>
455 lines
20 KiB
455 lines
20 KiB