0
Fork 0
mirror of https://github.com/mastra-ai/mastra.git synced 2026-11-17 08:44:46 +00:00
mastra/scripts
Tyler Barnes 8847e06005
ci(test-tiers): name the merge that broke a skipped test and alert its author (#30606)
## Why

Test tiers skip test files and jobs on PRs when they haven't failed in
30 days. The full suite still runs on the alpha versioning PR
(`changeset-release/main`), so a merge that breaks a skipped test goes
green on its PR. The failure then turns up later on the versioning run,
with nobody pointed at it. That's what happened with #29370, which broke
`goal-step.test.ts`, and #30434, which broke the Playground autosave
test.

## What this does

When a versioning run fails in a file or job that tiers had skipped, the
tier update now passes those escapes to a new `attribute` job. That job:

1. Finds the last green versioning run (up to 72 hours back) and lists
the first-parent merges on `main` between it and the red head (at most
60).
2. Bisects those merges, running only the escaped test file at each
step. It checks out the commit, installs, builds the package and its
dependencies from the read-only Turbo cache, and runs Vitest across the
`unit`, `typecheck`, `e2e:packages/*` and `e2e:client-sdks/*` projects,
the same ones the PR jobs use.
3. Names a merge only when the file fails there twice and passes at its
parent. A file that passes at the red head is reported as not reproduced
(a flake). A file that also fails before the range, or can't be run, is
unconfirmed. Nobody is tagged in either case.

When a file is reported by more than one project, any failure wins. The
file passes only if every result passed, and a file with no result is
unconfirmed.

A separate `notify` job holds the Slack token and sends the messages. If
`attribute` fails or times out (45 minutes), `notify` sends the plain
fallback alert that `update` prepared. The escaped file goes back into
the PR tier for 30 days, as before.

### Who gets tagged

The message names the PR, its title and its author, tagged through
`.github/slack-users.json`. A login that isn't in the map is printed as
a plain `@login`.

When the PR was opened by a bot, the bot can't answer for it, so the
message tags a person instead and says who they are to the PR:

1. the human who merged it ("merged by"); otherwise
2. every human whose latest review approved it ("approved by");
otherwise
3. the human(s) credited on the most commits, counting commit authors
and `Co-authored-by` trailers with GitHub noreply addresses
("co-authored by"); otherwise
4. `@here`.

`@here` is for when we can't tell who the person is. It is not for a
person we know but who is missing from the Slack map: they still get a
plain `@login`. Reviews and commits are read across every page. A lookup
that fails, or a commit list at GitHub's 250-commit cap (which may be
cut off), counts as unknown, and that rung is skipped rather than
guessed from. PR titles are escaped, so a title can't inject `<!here>`
or other mentions.

## Testing

- `node --test scripts/test-tiers/*.test.mjs`: 96 pass on Node 22 and
24. The new tests fail with the attribution module removed. Each
stand-in rung, the title escaping, any-failure-wins aggregation, E2E
project selection and pagination fails its own test when that rule is
removed.
- Replays against real versioning runs, at the real commits:
- #29370 (`goal-step.test.ts`, run 43907413456): found merge `929c8ab`
over 3 merges in 182 s.
- #30434 (autosave test, run 43750070686): found merge `b666fdb` over 2
merges in 495 s. The PR was opened by a bot, so the message reads "by
mastra-platform[bot] (bot), merged by @abhiaiyer91".
- A flaky `execute-tool-core.test.ts` failure (run 43848445791): passed
at the red head, so not reproduced and no alert, in 64 s.
- The runner, run for real on `main`, passed a unit file and a package
E2E file and returned an error for a missing file. That took 51 s.
- `actionlint` reports only the custom runner label.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

ELI5: When a versioning test fails in a file that the test tiers had
skipped, this PR finds the code change that caused it. It then posts a
Slack message that names the likely author. If that search cannot
finish, it posts a simpler alert instead.

## Overview

This PR adds attribution for test failures from versioning runs. The
failing file or job was skipped by the test tiers. The PR adds a new
`attribute` job. The job searches for the failing commit range and
reports the merge that introduced the failure. A separate `notify` job
posts a fallback alert if attribution fails or times out.

## Changes

- **`scripts/test-tiers/attribute.mjs`** (new, +455): Main attribution
logic.
  - Finds files and jobs that moved from tier `out` to `in`.
- Selects the latest successful versioning run from the preceding 72
hours.
- Considers at most 60 first-parent merges between the green run and the
failing run.
  - Bisects those merges by running the escaped file.
- Reports a merge only when the file fails there twice and passes at its
parent.
  - Reports a file as not reproduced when it passes at the failing head.
- Reports a file as unconfirmed when it also fails before the range or
cannot run.
  - Resolves culprit PRs and selects stand-ins for bot-authored PRs.
  - Generates Slack messages with escaped mrkdwn.
- Runs the `unit`, `typecheck`, `e2e:packages/*`, and
`e2e:client-sdks/*` projects.
- Passes a remaining-time timeout to each command. The script defaults
`DEADLINE_MINUTES` to 38. The PR description states a 45-minute job
deadline. The source does not yet confirm how these two values relate.
- **`scripts/test-tiers/attribute.test.mjs`** (new, +518): Tests for the
functions above. Covers bisect steps, verdicts, stand-ins, alert
grouping, paginated lookups, and deadline-aware runner timeouts.
- **`scripts/test-tiers/update.mjs`** (+9/-1): Imports `escapedFiles`.
Adds an optional `--escapes-out=<file>` argument to the versioning
command. Snapshots tier data before applying failures. Writes the run ID
and escape details to the output file when the argument is set.
- **`.github/workflows/test-tiers.yml`** (+106/-2): Records escapes in
job outputs. Runs the `attribute` job when escapes exist. Posts
attribution results or the fallback `notify` alert. Documents the new
behavior.
- **`.github/slack-users.json`** (+1): Adds an empty JSON object.

## Alert behavior

- For human-authored PRs, the alert names the PR, title, and author.
- For bot-authored PRs, the alert names, in order:
  1. The human who merged the PR.
  2. If unavailable, the humans with the latest approving reviews.
  3. If unavailable, the most-credited commit authors.
  4. If none are available, `@here`.
- Unknown Slack mappings appear as plain `@login`.
- Titles are escaped to prevent injected mentions.

## Result semantics

- A file that fails in any project result is a failure.
- A file passes only when every project result passes.
- A file with no project result is unconfirmed.
- An escaped file returns to the PR tier for 30 days.

## Verification

- The PR author reports 96 passing tests on Node 22 and 24, using `node
--test scripts/test-tiers/*.test.mjs`. This was not verified in this
review.
- The PR author reports replays against two real versioning runs that
found the breaking merges. This was not verified in this review.
- The PR author reports a flaky failure that was not reproduced and did
not alert. This was not verified in this review.
- The PR author reports a real `main` runner check that passed unit and
package E2E files and errored on a missing file. This was not verified
in this review.

## Scope

- Total: 5 files changed, +1089 / -3 lines.
- No exported or public declarations were removed.
- New exported functions and constants are listed in the `attribute.mjs`
source.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: remy-builder[bot] <324360501+remy-builder[bot]@users.noreply.github.com>
History 2026-11-16 18:27:39 -07:00
..
gh-dane-command refactor(mastracode): split into core, TUI, and standalone web packages (#18986) 2026-07-08 14:59:43 +02:00
gh-issue-triage chore(deps): use the latest @ai-sdk/provider-utils and @ai-sdk/provider patch in each major (#30518) 2026-11-16 15:17:56 +01:00
release-issue-notification chore(deps): update build tools (#27997) 2026-10-22 14:33:21 +02:00
test-tiers ci(test-tiers): name the merge that broke a skipped test and alert its author (#30606) 2026-11-16 18:27:39 -07:00
affected-tests-cache.mjs ci: cache what affected-test detection parses, so PR test selection takes ~4s instead of ~40s (#29133) 2026-11-05 20:02:44 +00:00
affected-tests.mjs ci: cache what affected-test detection parses, so PR test selection takes ~4s instead of ~40s (#29133) 2026-11-05 20:02:44 +00:00
check-core-imports.ts test(libsql): port T81 recovered-history cross-process coverage (#27945) 2026-10-21 10:20:41 -05:00
commonjs-tsc-fixer.js fix: resolve CodeQL sanitization and error-exposure alerts (#18805) 2026-07-02 15:26:54 +02:00
EMBEDDED_DOCS.md docs: make packages frontmatter more precise (#29499) 2026-11-04 17:49:54 -08:00
generate-package-docs.ts fix(docs): generate SOURCE_MAP.json from public export names (#29328) 2026-11-04 09:58:46 -08:00
ignore-example.js chore: migrate linting to oxlint (#20173) 2026-07-25 01:04:02 +02:00
install-example.js chore: migrate linting to oxlint (#20173) 2026-07-25 01:04:02 +02:00
madge.webpack.config.cjs Make the affected-test detector see packages that import with .js specifiers (#24488) 2026-09-21 14:58:17 -07:00
npm-placeholder-publish.mjs feat(ci): fail PRs and publishes on packages missing from npm (#26721) 2026-10-14 16:28:27 -07:00
npm-placeholder-publish.test.mjs feat(ci): fail PRs and publishes on packages missing from npm (#26721) 2026-10-14 16:28:27 -07:00
package-docs-source-map.test.ts fix(docs): generate SOURCE_MAP.json from public export names (#29328) 2026-11-04 09:58:46 -08:00
package-docs-source-map.ts fix(docs): generate SOURCE_MAP.json from public export names (#29328) 2026-11-04 09:58:46 -08:00
README-template.md chore(*): Update/unify/add READMEs (#22858) 2026-09-02 14:36:59 +02:00
test-migration-e2e.sh fix(storage): handle duplicate spans during migration (#12073) 2026-01-20 13:32:49 +01:00
test-observability.cjs chore: migrate linting to oxlint (#20173) 2026-07-25 01:04:02 +02:00
validate-peerdeps.mjs chore: remove stale scripts, workspace entries and package metadata (#29891) 2026-11-10 14:09:52 +01:00
workspace-source-aliases.cjs fix(ci): include exported server subpaths in affected tests (#21140) 2026-08-10 10:53:46 +02:00