* CI: one entry workflow with run admission and a fast pull request tier Pull request checks took a median of ~3 hours from push to done, and ~110 minutes of that was jobs waiting for a runner. Each push asked for ~207 runner-minutes on 15-20 jobs, almost all of it compiling, so bursts of pushes queued hundreds of jobs and every run held a few runners while the rest of its jobs waited behind other runs. - ci.yml is the only pull request / main / merge queue entry point. `plan` picks the suites a change needs; `admit` waits until every runner the run needs is free, then all its jobs start together and each runner is released when its job ends (scripts/ci/admit.py, first come first served with conservative backfilling, state read from the Actions API so fork pull requests work). `CI result` is the one check to require. - Pull requests get a fast tier: Windows, macOS, voice and Android run only when the change looks platform specific (their files, platform conditionals or APIs in changed code); `ci:full` or CI_PR_TIER=full runs everything. Main and the merge queue run the full tier. - Voice: 14 `cargo test` builds capped at 2 jobs (~40 min) become core-tests additions plus one nextest build for audio and the mobile voice core; the workflow no longer runs uncancelled on every push or saves caches from pull requests. - Windows: three test jobs become one nextest build; one shared cache instead of four (~4.5 GB of the 10 GB repository cache). - Linux UI and browser share one job and cache; edge coordinator and vault share one job; the macOS app is built with the test dependencies instead of a `cargo check` that recompiled everything; iOS no longer waits for the frame-source job, which only runs when the GPUI pin can change. - Only main saves Rust caches. * CI admission: cap reservations read from other runs, expire unseen ones A pull request can edit its own ci.yml, so the reservation in its admit job name is untrusted. Cap it at the largest plan.py makes, and stop counting an admitted run's never-created jobs 90 minutes after admission, so one run cannot hold more than a full run's runners or hold unused ones for long. * CI: reuse passes on unchanged inputs; ci:priority, ci:lock and an emergency workflow Reuse. plan fingerprints every suite's inputs in the tree under test (the merge with main for pull requests): workspace manifest, lockfile, toolchain, CI files, the suite's workflow and non-Rust files, and for Rust suites the packages it tests plus every workspace package they depend on, plus any file outside those that their sources name with a ../ path (scanned every run). Every job records its pass in the Actions cache under that fingerprint as its last step; a suite whose jobs all passed is skipped and reserves nothing, and single Windows and voice jobs are skipped on their own. Pull requests trust their own passes and main's; main and the merge queue trust only main's; ci:full and manual runs reuse nothing. Callers are split per suite so each records under its own fingerprint. Priority and locks (labels, so triage or above). ci:priority puts a pull request's runs at the front of the admission queue; ci:lock keeps a full run's runners out of everyone else's budget while the pull request is open (until unlabelled, merged, closed or CI_LOCK_HOURS), and admits its runs first. Runs name their pull request in the admit job; the claim counts only when that pull request's head is the run's commit. ci-emergency.yml (write access) locks a pull request, cancels the other pull requests' CI runs and re-runs its own if needed. * CI reuse: fail closed The merge queue, the last gate before main, reuses nothing. A pull request that changes a Rust build file no selected suite's fingerprint covers reuses nothing. Sources that build a path from CARGO_MANIFEST_DIR's parent in code make their suite read the whole workspace. concat!(.., "/../x") paths and .cargo/ count as inputs. * CI result: fix the result listing (shell quoting) * CI: run less on pull requests, more on schedule, and merge tiny test binaries Measured in CI, test execution is a small part of every job; compiling and starting jobs that run little is the cost. So: - Voice: only the audio stack (zeron-audio, which nothing else builds yet) and the packaged Windows media runtime remain, for their own files. The mobile voice/orb tests were a subset of ios-core, the macOS/Windows `cargo check` of harness+engine duplicated the platform builds, and the voice-media tests join the Windows nextest build. - Fast tier gating of jobs and steps to their own files: Windows packaging (`app`), the iOS simulator tests and their bindings, the Linux and macOS browser/preview fixtures. Android builds the APK on pull requests for the app, the FFI crate and dependency changes; the iOS mobile-core paths come from zeron-mobile's Cargo closure. Gated jobs and optional steps are part of the reuse fingerprint. - GPUI frame-source runs only when the pinned zui revision moves. - Version bumps (only `version = "..."` lines in Cargo.toml/Cargo.lock) select nothing and are left out of fingerprints. - nextest `ci` profile for pull requests and main leaves out the slow/soak tests (48-chat and real-quantum doc_host sync, the codex dying-server backoff, preview leak, two sleep-bound discovery tests); the cursor_shim stress test runs 14 rounds (ZERON_STRESS_ROUNDS, 100 by default). Preview discovery tests run one at a time (fixed port). - Nightly ci.yml schedule runs everything in full; weekly-tests.yml runs the engine integration binaries and crate tests no workflow ran. - Linux core no longer builds the 8 test binaries that run nothing there (Windows-only or all-#[ignore]d). The small harness, engine and preview integration suites move into one `tests/misc` binary per crate. The orb geometry tests move to zeron-orb (they tested its re-export); the assertion-free microbenchmark is dropped. The macOS app builds with the tests instead of rebuilding zeron-ui without the fixture feature. * CI: pass the nextest config to the build-only steps too NEXTEST_PROFILE=ci names a profile from scripts/ci/nextest.toml; the macOS and Windows `--no-run` builds did not load that file, so nextest refused the unknown profile. * Engine: keep the git identity setup inside the home it is given configure_git_identity pinned HOME but inherited XDG_CONFIG_HOME, and `git config --global` writes $XDG_CONFIG_HOME/git/config when that file exists. With a sandbox or test home, the identity landed in the developer's real git config: running the engine tests on a machine with XDG_CONFIG_HOME set replaced the user's name and email with the test's. An XDG_CONFIG_HOME outside the given home now gives way to the home's own, and the test pins it for its own git calls too. * CI: iOS simulator tests in parallel with ios-core; Windows UI tests side by side - The C header and module map the simulator tests link are committed in apps/ios/CoreFFI (outside Xcode's synchronized folders) and refreshed by scripts/ios/build-core.sh like the Swift bindings; ios-core checks all three are current. ios-tests no longer waits for ios-core's artifact, so the iOS path is one job long instead of two. - The simulator core for the unit tests builds in the dev profile, not the optimized `mobile` profile the app ships. - Windows runs zeron-ui's tests side by side: nextest gives each its own process, so the in-process `--test-threads=1` restriction no longer applies. * CI admission: budget defaults at the organization's runner limits 20 jobs in total and 5 on macOS (the organization's concurrency limits), and at most 10 on Windows so Linux keeps half the pool. The repository variables still override them. * CI: skip the browser steps when the Linux setup failed They run after a failed UI test on purpose (a separate signal), but without the system packages (an apt mirror stall) they can only fail too. * CI: retry the WebRTC terminal path test (a known race) remote_terminals' network-change test asserts no keystroke goes through the relay once the path reports Direct; in a loaded run one did. Retried like the other timing-sensitive tests in nextest.toml until the race is fixed. * Fix two races that failed CI on loaded runners - Muse: a crash usually shows first as "Muse stdout closed", racing the process exit; `try_wait` lost that race under load and users got the bare protocol error instead of "Muse exited unexpectedly" with Muse's last stderr lines. Wait up to 250 ms for the exit, as the Pi driver already does (a_crash_mid_turn_reports_muses_last_words failed this way). - Remote terminals: a subscription reported TerminalPath::Direct before attaching the direct writer, so a keystroke typed by whoever saw Direct could still go through the relay (a_terminal_rides_a_real_webrtc_pair_ through_a_network_change failed this way). Report Direct once writes ride the link. The CI retry for that test goes. Both tests pass 25 runs in a row under CPU load. * CI reuse: count only the CI files a suite runs Every suite's fingerprint included all of scripts/ci and the composite actions, so a change to the planner or the scheduler (which decide what runs, not whether it passes) re-ran every suite. Each suite now counts the CI files it executes: nextest.toml and the Linux or Windows setup for the Rust suites, test-core.sh for core, the macOS fixture runner for macOS, apt-install.sh for the installer. ci.yml still counts for all. * docs: changing the fingerprint inputs reruns everything once * CI admission: no circular waits Two ways admission could deadlock (until the fail-open timeout): - Each waiting run counted every other waiting run's admit job as a busy runner, including runs queued behind it. With a full run (16 of 20) at the head and five or more waiting behind it, the head waited for their runners and they waited for it. A run now counts only the admit runners ahead of it. - Two ci:lock labels each reserved 16 of 20 runners and counted the other's reservation, so neither locked pull request (nor anyone else) could start. Locks are now ordered by when their label was added: an older lock ignores a newer one's standing reservation. Every wait now points forward in one order, so no cycle can form. A test simulates 3000 random queues (sizes, priority, overlapping locks, polling order) and checks they all drain; the previous scheduler deadlocked in 309. The fail-open timeout drops from 5 h to 2 h as the backstop for the one case outside the model (waiting admit jobs taking every runner in the pool). * CI admission: waiting admit jobs yield when they starve the pool; ignore runs that never take their turn Waiting costs a hosted runner from the same pool as the suites. With about 20 runs waiting, their admit jobs held the whole pool, admitted runs' jobs got no runner, and nothing moved until the 2-hour fail-open. When an admitted run's Linux or Windows job has queued 10 minutes, every running admit job but the first quarter of the budget now starts its run without a reservation, giving its runner back. A new test models GitHub's runner pool as well as admission: without yielding most 30-run queues never drain, with it all 200 do. A reservation read from another run is not trusted, but a run whose edited admit step never finished still held everyone behind it for 2 hours (and a fake admitted run held a full run for 90 minutes). A waiting run that could have been admitted for 3 minutes but still waits is now ignored, and an admitted run's jobs that have not appeared 10 minutes after admission stop counting, except the late result and publish jobs. Polling backs off further from the head of the queue (5 min, 10 min under 400 requests left): the GITHUB_TOKEN quota is shared with releases, the Cursor SDK update and CI emergency. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com> * CI: voice and the Linux browser fixture nightly only; forks reuse only main's passes Over two weeks the voice jobs left after the move into core-tests (AEC3 and the packaged Windows runtime) never failed, and the Linux browser fixture's failures were window-timing flakes and package downloads. Both now run nightly, on manual runs and with ci:full. The zeron-ui tests stay on every change: 2,046 tests in 29 s, and the only check that caught one real regression in that window. A fork's earlier runs could save a pass entry for the fingerprint of a later, clean-looking commit, so its suites would be skipped and the pull request would still show a green CI result. Until the merge queue (which reuses nothing) is the required gate, forks reuse only main's passes; CI_FORK_REUSE=own turns their own back on (ci.yml passes it to plan in the workflow commit that follows). Docs: admission's new yield and unresponsive-run rules and polling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com> * Engine: keep the user's own XDG_CONFIG_HOME when the git identity home is theirs The XDG_CONFIG_HOME guard replaced any XDG_CONFIG_HOME outside `home` with `home`'s. When `home` is this process's own, that hid an identity the user keeps in an XDG_CONFIG_HOME elsewhere, and the machine identity went into ~/.gitconfig, which wins over it. It now only applies to a sandbox or test home. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com> * CI workflows: Android gets only its own secrets, on pushes to main; CI_FORK_REUSE secrets: inherit gave android.yml every repository secret, and the merge queue (code not on main yet) would have signed with the release keystore. ci.yml now passes the four signing secrets and the R2 token by name, on pushes to main only; pull requests and the merge queue use the debug key, as forks always did. Also: ci.yml passes the CI_FORK_REUSE variable to plan, and the comments in ci.yml, voice-tests.yml and ui-tests.yml say voice and the Linux browser fixture run nightly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * CI: test-core also runs main's crates/doc/tests (attachments_roundtrip is among them) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Wing <44917547+wingleeio@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
14 KiB
CI
Pull requests, main and the merge queue all go through one workflow,
ci.yml. It runs these jobs in order:
- plan (
scripts/ci/plan.py) lists the changed files and picks the suites the change needs. It drops the jobs that already passed on the same inputs (see Reuse), and counts the runners the rest will use. - admit (
scripts/ci/admit.py) waits until all of those runners are free. - The suites all start together. Each is a reusable workflow:
ui-tests.yml(Linux),windows.yml,macos.yml(macOS and iOS),voice-tests.yml,android.yml,preview-tests.yml,linux-installer.ymlandcursor-compatibility.yml. - CI result passes when no suite failed or was cancelled. This is the check to require in the ruleset and the merge queue.
Which suites run
| Pull request (fast tier) | main, merge queue, manual run (full tier) |
|
|---|---|---|
| Linux core and UI, Cursor, edge, installer | when their files change | when their files change |
| iOS core (Linux) | app or mobile core changed | app or mobile core changed |
| iOS simulator | the app, crates/mobile or crates/client changed |
with iOS core |
| Windows tests, macOS | only when the change looks platform specific | when their files change |
Windows packaging (app) |
apps/zeron, dist/windows, packaging scripts, crates/update |
with Windows tests |
| macOS browser / preview fixtures | browser, preview or edge code changed | always |
| Linux browser fixture | nightly or ci:full only |
nightly or manual run only |
| Android APK | the Android app or crates/mobile changed |
when its files change |
| Voice (audio stack, Windows media runtime) | nightly or ci:full only |
nightly or manual run only |
| GPUI frame-source | the pinned GPUI revision moved | the same |
The iOS core paths come from the Cargo graph: every crate zeron-mobile
depends on.
On the fast tier, a change counts as platform specific when it touches:
- the platform's own files: packaging, scripts,
apps/zeron,crates/update, and the voice crates or the mobile core for voice and Android; - a file under
crates/whose name says so, such asbrowser/macos.rs; - changed code (not comments) with a platform conditional or platform API:
target_os,cfg(unix),cfg(windows),objc, WebKit,HWND, ...
Cargo.toml, Cargo.lock and rust-toolchain.toml changes run everything
except a version bump: one that only changes version = "..." lines (the
workspace version and the workspace packages' versions in the lockfile) selects
nothing, and is left out of the reuse fingerprints. Changes to CI itself run
everything.
Scheduled runs
- Nightly (
ci.yml, schedule): every suite, job and fixture, nothing reused, slow tests included. Voice and the Linux browser fixture run only here, on manual runs and withci:full: in the two weeks measured, the voice jobs left after the move tocore-testsnever failed, and the Linux browser fixture's failures were window-timing flakes and package downloads. The rest of voice (protocol, transport, MCP, voice-media) runs incore-tests,ios-coreand the Windows tests. - Nightly (
release-profile-tests.yml): core and UI in the release profile. - Weekly (
weekly-tests.yml): the tests no other workflow runs. That is everyzeron-engineintegration binary, plus the crates whose own tests have no other runner (term, syntax, theme, veil, visual, visual-gpu, zatex, notify).
Slow tests
Pull requests and main use nextest's ci profile (scripts/ci/nextest.toml).
It leaves out these slow and soak tests, which the nightly runs keep:
- the doc_host sync tests at 48-chat scale or against the real 30 s quantum;
- the Codex dying-server backoff test;
- the preview leak test;
- two preview discovery tests that assert nothing happens during a sleep.
The Cursor shim stress test runs 14 rounds instead of 100
(ZERON_STRESS_ROUNDS).
Test binaries that run nothing on Linux are not built there:
- Windows-only files;
- files whose every test is
#[ignore]d.
Small integration suites share one binary per crate (tests/misc/), since
every tests/*.rs file is linked against the whole crate.
To run every suite on a pull request, add the ci:full label, then push or
re-run the CI workflow. To make every pull request run the full tier, set the
repository variable CI_PR_TIER=full.
Reuse
After a new push, a suite runs again only if something it reads changed. That
is checked by content, not by what the push touched: a suite reruns when main
changed something it reads, and is skipped when the push only touched code it
doesn't read.
For each selected suite, plan hashes the inputs in the tree being tested
(scripts/ci/reuse.py). For a pull request that
tree is the merge with the current main. The inputs are:
Cargo.toml,Cargo.lock, the toolchain andci.yml;- the suite's workflow, the CI scripts and setup actions it runs
(
nextest.toml,test-core.sh, the apt and nextest installers…), and the non-Rust files it uses. The planner and the scheduler (plan.py,admit.py,reuse.py,emergency.py) decide what runs, not whether a test passes, so changing them reuses everything; - for Rust suites:
- the packages they test and every workspace package those depend on, of any dependency kind;
- every file outside those packages that their Rust sources name with a
../path, for exampleinclude_str!("../../../ui/src/file-icons.json")orCARGO_MANIFEST_DIRjoined with../harness/tests/fixtures. The sources are scanned on every run, so new references are picked up without a list to maintain. The plan summary lists the files found.
When a job passes, its last step saves a tiny Actions cache entry named after
the suite's hash and the job. plan skips a suite when all its jobs have such
an entry. A single Windows or voice job can be skipped while the suite's other
jobs run. Skipped suites reserve no runners and count as passed in CI result.
The plan summary shows what was reused.
Which recorded passes count:
| Run | Reuses passes from |
|---|---|
| Pull request from a branch of this repository | its own earlier runs, and main |
| Pull request from a fork | main only (its own too with CI_FORK_REUSE=own) |
main |
main only |
Merge queue (the last gate before main), ci:full label, manual run |
nothing |
Reuse fails closed:
- If a pull request changes a Rust build file (
crates/,apps/zeron,Cargo.*,.cargo/) that no selected suite's fingerprint covers, nothing is reused. - If a suite's sources build a path from the package directory's parent in
code (
.parent(),.ancestors(),join("..")), the scan can't follow it. That suite then counts the whole workspace as its input.
A pull request could fake a pass in its own runs: an earlier push whose edited
workflow saves the pass entry for the fingerprint of a later, clean-looking
commit would let that commit's suites be skipped. No other ref ever sees those
entries, but until the merge queue (which reuses nothing) is the required gate,
a green CI result on the pull request is what a reviewer goes by. So forks
reuse only main's passes; set CI_FORK_REUSE=own once the merge queue is
required.
Changing what goes into a fingerprint (SUITE_INPUTS, COMMON,
SUITE_PACKAGES in reuse.py) changes every fingerprint once, so the next run
reruns everything; recorded passes count again after that.
Limits:
- A path assembled from variables is not found by the scan. Write such
references as one
../literal, or add the path to the suite'sSUITE_INPUTS. - A test that depends on the network or on timing and passed by luck stays
passed until its inputs change. Re-run with
ci:fullto check everything again.
Admission
GitHub starts a job as soon as a runner frees up. When several runs compete, each one gets some of its jobs going and holds those runners while its other jobs queue behind other runs, so every run takes hours.
Admission prevents this. A run's admit job waits until the runners for the
whole run are free; only then do the run's jobs start. Each runner is released
when the job using it finishes. Admission is first come, first served. A run
that only needs Linux runners may still start ahead of a run waiting for macOS
runners, but never by taking runners that run is waiting for.
All the state lives in the GitHub API, so admission works with the read-only token that pull requests from forks get:
- The admit job's name carries the run's reservation, for example
admit [linux=4 windows=2 macos=1]. - Other runs read that reservation from their admit jobs.
- How much of it a run still holds comes from how many of its jobs have finished.
Priority and locks
Adding a label needs the triage role or above, so GitHub decides who can use these. A fork's author can't label their own pull request.
-
ci:priority: the pull request's runs go to the front of the queue, behind any locked pull request. Nobody behind them may take runners they wait for, so the next runners to free up go to them. Running jobs are never stopped. -
ci:lock: while the pull request is open, admission treats a full run's worth of runners (10 Linux, 3 Windows, 3 macOS) as taken, whether or not the pull request is running anything. Its own runs skip the queue and are admitted as soon as their runners are free. The lock ends when:- the label is removed;
- the pull request is merged or closed;
CI_LOCK_HOURS(6) have passed since the label was added. Add the label again to extend.
-
CI emergency (
ci-emergency.yml): a workflow you start by hand from the Actions tab with a pull request number. It needs write access. It:- adds
ci:lockto the pull request; - cancels every other pull request's CI run (with
cancel-main, alsomainand merge queue runs); - re-runs the pull request's own CI if none is running and the last run didn't pass.
The cancelled pull requests need a push or a re-run afterwards.
- adds
A run states its pull request in its admit job's name, for example
admit [linux=4 windows=0 macos=1] #1077. The claim only counts when that
pull request's head is the run's commit, so a run can't borrow another pull
request's priority or lock.
Repository variables:
| Variable | Default | |
|---|---|---|
CI_BUDGET_TOTAL |
20 | Hosted runners all CI runs may hold at once: the organization's concurrency limit |
CI_BUDGET_MACOS |
5 | macOS runners: the organization's macOS limit |
CI_BUDGET_WINDOWS |
10 | Windows runners: half the pool, so Linux keeps the other half |
CI_DEBOUNCE_SECONDS |
60 | How long a pull request run waits before queueing. A push within that time cancels it. |
CI_ADMISSION |
on | off starts every run at once |
CI_PR_TIER |
fast | full runs everything on pull requests |
CI_LOCK_HOURS |
6 | How long a ci:lock label holds runners |
CI_FORK_REUSE |
(unset) | own: fork pull requests also reuse their own earlier passes. Set it once the merge queue is required |
A pull request can edit its own copy of ci.yml, so the reservation other runs
read from its admit job is not trusted:
- it is capped at a full run (
MAX_RESERVATION); - jobs an admitted run reserved but never created stop counting 10 minutes
after admission, except the
CI resultjob and Android publishing, which start late (90 minutes); - a waiting run that could have been admitted for 3 minutes but still waits (an edited admit step that never finishes) is ignored: the runs behind it stop waiting for it.
Admission ensures that all of a run's runners were free when it started. It can't stop jobs it doesn't see from taking runners in the meantime: other repositories of the organization, release and deploy workflows. Those can still take a free runner first.
No deadlocks
Every wait points the same way: locked runs (oldest lock first), then
ci:priority, then first come first served. So no two runs can wait for each
other.
- A waiting run's
admitjob holds a runner until the run is admitted. A run counts those runners only for the runs ahead of it, not behind it. - A lock's reservation counts against everyone except older locks. A second lock waits until the first is released; the first never waits for the second.
scripts/tests/test_ci_admit.py simulates 3,000 random queues (any sizes,
priority, overlapping locks, any polling order) and checks that every run gets
admitted. The scheduler before this rule deadlocked in about 1 in 10 of them.
Waiting costs a runner: an admit job runs on the same hosted pool as the
suites. With about 20 runs waiting at once, their admit jobs took the whole
pool, admitted runs' jobs got no runner, and nothing moved until the 2-hour
timeout. So when an admitted run's Linux or Windows job has been queued for 10
minutes, every waiting admit job but the first 5 (a quarter of
CI_BUDGET_TOTAL) in the queue gives its runner back and starts its run
without a reservation. Others count that run as holding its runners, so
admission catches up. The test models GitHub's runner pool as well as
admission: without this rule most of its 30-run queues never drain, with it
all of them do.
Admission never blocks CI. It lets the run start anyway if:
- it waited 2 hours;
- the API keeps failing;
- the API rate limit runs out for more than 10 minutes.
The GITHUB_TOKEN gets 1000 to 5000 API requests an hour per repository,
depending on the plan; zeron has seen 5000. To stay well under that,
admission:
- uses conditional requests, which cost nothing when nothing changed;
- refetches a run's jobs only when the run changed;
- polls every 30 s at the head of the queue, 1 and 2 min at the next two places, 5 min further back, and 10 min when fewer than 400 requests are left. The quota is shared with every workflow (releases, the Cursor SDK update, CI emergency).
Caches
The repository has 10 GB of Actions cache. When it is full, GitHub evicts the
least recently used entries. Only main saves caches, so pull requests can't
push out the caches that main seeds. There is one cache per build: Linux
core, Linux UI, Linux voice, one shared by the Windows jobs, macOS native, iOS
core and simulator, and Android.
Locally
python3 -m unittest discover -s scripts/tests -p 'test_ci_*.py'