* CI: one entry workflow with run admission and a fast pull request tier Pull request checks took a median of ~3 hours from push to done, and ~110 minutes of that was jobs waiting for a runner. Each push asked for ~207 runner-minutes on 15-20 jobs, almost all of it compiling, so bursts of pushes queued hundreds of jobs and every run held a few runners while the rest of its jobs waited behind other runs. - ci.yml is the only pull request / main / merge queue entry point. `plan` picks the suites a change needs; `admit` waits until every runner the run needs is free, then all its jobs start together and each runner is released when its job ends (scripts/ci/admit.py, first come first served with conservative backfilling, state read from the Actions API so fork pull requests work). `CI result` is the one check to require. - Pull requests get a fast tier: Windows, macOS, voice and Android run only when the change looks platform specific (their files, platform conditionals or APIs in changed code); `ci:full` or CI_PR_TIER=full runs everything. Main and the merge queue run the full tier. - Voice: 14 `cargo test` builds capped at 2 jobs (~40 min) become core-tests additions plus one nextest build for audio and the mobile voice core; the workflow no longer runs uncancelled on every push or saves caches from pull requests. - Windows: three test jobs become one nextest build; one shared cache instead of four (~4.5 GB of the 10 GB repository cache). - Linux UI and browser share one job and cache; edge coordinator and vault share one job; the macOS app is built with the test dependencies instead of a `cargo check` that recompiled everything; iOS no longer waits for the frame-source job, which only runs when the GPUI pin can change. - Only main saves Rust caches. * CI admission: cap reservations read from other runs, expire unseen ones A pull request can edit its own ci.yml, so the reservation in its admit job name is untrusted. Cap it at the largest plan.py makes, and stop counting an admitted run's never-created jobs 90 minutes after admission, so one run cannot hold more than a full run's runners or hold unused ones for long. * CI: reuse passes on unchanged inputs; ci:priority, ci:lock and an emergency workflow Reuse. plan fingerprints every suite's inputs in the tree under test (the merge with main for pull requests): workspace manifest, lockfile, toolchain, CI files, the suite's workflow and non-Rust files, and for Rust suites the packages it tests plus every workspace package they depend on, plus any file outside those that their sources name with a ../ path (scanned every run). Every job records its pass in the Actions cache under that fingerprint as its last step; a suite whose jobs all passed is skipped and reserves nothing, and single Windows and voice jobs are skipped on their own. Pull requests trust their own passes and main's; main and the merge queue trust only main's; ci:full and manual runs reuse nothing. Callers are split per suite so each records under its own fingerprint. Priority and locks (labels, so triage or above). ci:priority puts a pull request's runs at the front of the admission queue; ci:lock keeps a full run's runners out of everyone else's budget while the pull request is open (until unlabelled, merged, closed or CI_LOCK_HOURS), and admits its runs first. Runs name their pull request in the admit job; the claim counts only when that pull request's head is the run's commit. ci-emergency.yml (write access) locks a pull request, cancels the other pull requests' CI runs and re-runs its own if needed. * CI reuse: fail closed The merge queue, the last gate before main, reuses nothing. A pull request that changes a Rust build file no selected suite's fingerprint covers reuses nothing. Sources that build a path from CARGO_MANIFEST_DIR's parent in code make their suite read the whole workspace. concat!(.., "/../x") paths and .cargo/ count as inputs. * CI result: fix the result listing (shell quoting) * CI: run less on pull requests, more on schedule, and merge tiny test binaries Measured in CI, test execution is a small part of every job; compiling and starting jobs that run little is the cost. So: - Voice: only the audio stack (zeron-audio, which nothing else builds yet) and the packaged Windows media runtime remain, for their own files. The mobile voice/orb tests were a subset of ios-core, the macOS/Windows `cargo check` of harness+engine duplicated the platform builds, and the voice-media tests join the Windows nextest build. - Fast tier gating of jobs and steps to their own files: Windows packaging (`app`), the iOS simulator tests and their bindings, the Linux and macOS browser/preview fixtures. Android builds the APK on pull requests for the app, the FFI crate and dependency changes; the iOS mobile-core paths come from zeron-mobile's Cargo closure. Gated jobs and optional steps are part of the reuse fingerprint. - GPUI frame-source runs only when the pinned zui revision moves. - Version bumps (only `version = "..."` lines in Cargo.toml/Cargo.lock) select nothing and are left out of fingerprints. - nextest `ci` profile for pull requests and main leaves out the slow/soak tests (48-chat and real-quantum doc_host sync, the codex dying-server backoff, preview leak, two sleep-bound discovery tests); the cursor_shim stress test runs 14 rounds (ZERON_STRESS_ROUNDS, 100 by default). Preview discovery tests run one at a time (fixed port). - Nightly ci.yml schedule runs everything in full; weekly-tests.yml runs the engine integration binaries and crate tests no workflow ran. - Linux core no longer builds the 8 test binaries that run nothing there (Windows-only or all-#[ignore]d). The small harness, engine and preview integration suites move into one `tests/misc` binary per crate. The orb geometry tests move to zeron-orb (they tested its re-export); the assertion-free microbenchmark is dropped. The macOS app builds with the tests instead of rebuilding zeron-ui without the fixture feature. * CI: pass the nextest config to the build-only steps too NEXTEST_PROFILE=ci names a profile from scripts/ci/nextest.toml; the macOS and Windows `--no-run` builds did not load that file, so nextest refused the unknown profile. * Engine: keep the git identity setup inside the home it is given configure_git_identity pinned HOME but inherited XDG_CONFIG_HOME, and `git config --global` writes $XDG_CONFIG_HOME/git/config when that file exists. With a sandbox or test home, the identity landed in the developer's real git config: running the engine tests on a machine with XDG_CONFIG_HOME set replaced the user's name and email with the test's. An XDG_CONFIG_HOME outside the given home now gives way to the home's own, and the test pins it for its own git calls too. * CI: iOS simulator tests in parallel with ios-core; Windows UI tests side by side - The C header and module map the simulator tests link are committed in apps/ios/CoreFFI (outside Xcode's synchronized folders) and refreshed by scripts/ios/build-core.sh like the Swift bindings; ios-core checks all three are current. ios-tests no longer waits for ios-core's artifact, so the iOS path is one job long instead of two. - The simulator core for the unit tests builds in the dev profile, not the optimized `mobile` profile the app ships. - Windows runs zeron-ui's tests side by side: nextest gives each its own process, so the in-process `--test-threads=1` restriction no longer applies. * CI admission: budget defaults at the organization's runner limits 20 jobs in total and 5 on macOS (the organization's concurrency limits), and at most 10 on Windows so Linux keeps half the pool. The repository variables still override them. * CI: skip the browser steps when the Linux setup failed They run after a failed UI test on purpose (a separate signal), but without the system packages (an apt mirror stall) they can only fail too. * CI: retry the WebRTC terminal path test (a known race) remote_terminals' network-change test asserts no keystroke goes through the relay once the path reports Direct; in a loaded run one did. Retried like the other timing-sensitive tests in nextest.toml until the race is fixed. * Fix two races that failed CI on loaded runners - Muse: a crash usually shows first as "Muse stdout closed", racing the process exit; `try_wait` lost that race under load and users got the bare protocol error instead of "Muse exited unexpectedly" with Muse's last stderr lines. Wait up to 250 ms for the exit, as the Pi driver already does (a_crash_mid_turn_reports_muses_last_words failed this way). - Remote terminals: a subscription reported TerminalPath::Direct before attaching the direct writer, so a keystroke typed by whoever saw Direct could still go through the relay (a_terminal_rides_a_real_webrtc_pair_ through_a_network_change failed this way). Report Direct once writes ride the link. The CI retry for that test goes. Both tests pass 25 runs in a row under CPU load. * CI reuse: count only the CI files a suite runs Every suite's fingerprint included all of scripts/ci and the composite actions, so a change to the planner or the scheduler (which decide what runs, not whether it passes) re-ran every suite. Each suite now counts the CI files it executes: nextest.toml and the Linux or Windows setup for the Rust suites, test-core.sh for core, the macOS fixture runner for macOS, apt-install.sh for the installer. ci.yml still counts for all. * docs: changing the fingerprint inputs reruns everything once * CI admission: no circular waits Two ways admission could deadlock (until the fail-open timeout): - Each waiting run counted every other waiting run's admit job as a busy runner, including runs queued behind it. With a full run (16 of 20) at the head and five or more waiting behind it, the head waited for their runners and they waited for it. A run now counts only the admit runners ahead of it. - Two ci:lock labels each reserved 16 of 20 runners and counted the other's reservation, so neither locked pull request (nor anyone else) could start. Locks are now ordered by when their label was added: an older lock ignores a newer one's standing reservation. Every wait now points forward in one order, so no cycle can form. A test simulates 3000 random queues (sizes, priority, overlapping locks, polling order) and checks they all drain; the previous scheduler deadlocked in 309. The fail-open timeout drops from 5 h to 2 h as the backstop for the one case outside the model (waiting admit jobs taking every runner in the pool). * CI admission: waiting admit jobs yield when they starve the pool; ignore runs that never take their turn Waiting costs a hosted runner from the same pool as the suites. With about 20 runs waiting, their admit jobs held the whole pool, admitted runs' jobs got no runner, and nothing moved until the 2-hour fail-open. When an admitted run's Linux or Windows job has queued 10 minutes, every running admit job but the first quarter of the budget now starts its run without a reservation, giving its runner back. A new test models GitHub's runner pool as well as admission: without yielding most 30-run queues never drain, with it all 200 do. A reservation read from another run is not trusted, but a run whose edited admit step never finished still held everyone behind it for 2 hours (and a fake admitted run held a full run for 90 minutes). A waiting run that could have been admitted for 3 minutes but still waits is now ignored, and an admitted run's jobs that have not appeared 10 minutes after admission stop counting, except the late result and publish jobs. Polling backs off further from the head of the queue (5 min, 10 min under 400 requests left): the GITHUB_TOKEN quota is shared with releases, the Cursor SDK update and CI emergency. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com> * CI: voice and the Linux browser fixture nightly only; forks reuse only main's passes Over two weeks the voice jobs left after the move into core-tests (AEC3 and the packaged Windows runtime) never failed, and the Linux browser fixture's failures were window-timing flakes and package downloads. Both now run nightly, on manual runs and with ci:full. The zeron-ui tests stay on every change: 2,046 tests in 29 s, and the only check that caught one real regression in that window. A fork's earlier runs could save a pass entry for the fingerprint of a later, clean-looking commit, so its suites would be skipped and the pull request would still show a green CI result. Until the merge queue (which reuses nothing) is the required gate, forks reuse only main's passes; CI_FORK_REUSE=own turns their own back on (ci.yml passes it to plan in the workflow commit that follows). Docs: admission's new yield and unresponsive-run rules and polling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com> * Engine: keep the user's own XDG_CONFIG_HOME when the git identity home is theirs The XDG_CONFIG_HOME guard replaced any XDG_CONFIG_HOME outside `home` with `home`'s. When `home` is this process's own, that hid an identity the user keeps in an XDG_CONFIG_HOME elsewhere, and the machine identity went into ~/.gitconfig, which wins over it. It now only applies to a sandbox or test home. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com> * CI workflows: Android gets only its own secrets, on pushes to main; CI_FORK_REUSE secrets: inherit gave android.yml every repository secret, and the merge queue (code not on main yet) would have signed with the release keystore. ci.yml now passes the four signing secrets and the R2 token by name, on pushes to main only; pull requests and the merge queue use the debug key, as forks always did. Also: ci.yml passes the CI_FORK_REUSE variable to plan, and the comments in ci.yml, voice-tests.yml and ui-tests.yml say voice and the Linux browser fixture run nightly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * CI: test-core also runs main's crates/doc/tests (attachments_roundtrip is among them) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Wing <44917547+wingleeio@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
13 KiB
Bounded chat synchronization
Historical chat count must not determine concurrent network resource use. Local inspection, a document's lifetime, and its sync client's lifetime are separate. The changes require no manual data cleanup. They do not change the OS file limit.
Admission and lifetime
DocHost::open_local returns the one authoritative writable document and installs
its durable publication subscription. open additionally requests synchronization;
it does not spawn a connection immediately. Boot salvage first inspects snapshots
and rollback sources without registering handles. Legacy epochs are treated as
recovery sources, not evidence the adopted chat2 transcript is healthy. A real
candidate waits for catch-up and rechecks under the command/import locks. Failed
recoveries retry from durable metadata.
Before reserving a slot, the dispatcher selects an operation using snapshot epoch metadata and current registry ownership. Cold snapshots are loaded only after a slot is available, then eligibility is checked again against the opened document. A saturated host can scan durable jobs without deserializing their histories. Modern chats join normally; only the owning host seeds a legacy chat. Wakes for an explicitly foreign legacy chat retire by captured version without touching outgoing batches or consuming a slot. A missing registry row still represents a possible new chat, and a local chat2 epoch still overrides stale legacy registry metadata. A modern registry row overtaking an old handle defers admission until cutover replaces it, without retiring its wake.
A single dispatcher per host admits at most 28 clients, up to four per 100 ms tick. It gives a background selection one in every four admissions, with oldest-service ordering within each class. The disk backlog is paged by chat ID, rather than materialized as thousands of documents or semaphore-waiting tasks. A served quiet background client releases its slot; a warm interactive client has a 10 s reuse grace after catch-up completes. Incomplete catch-up cannot be retired by that idle grace. Closing clients retain their slots until teardown completes, while the dispatcher continues admitting other work and handling durable jobs. The bounded set of close tasks is tracked and joined before the host's final snapshot; actor ownership is retained throughout shutdown so a cancelled close cannot detach the actor. All session waits, including handshake, backfill and backpressured sends, are interruptible by shutdown. Local rotation requires all 28 slots to be occupied and an eligible chat without a client to be waiting. Pending batches on an existing client do not count as contention. Each rotation retires one client and waits for its teardown before another handoff. Transcript/queue views and agent writer leases protect resident documents, but transport ownership is now scheduled separately.
Focus and service turns
FocusChat {chatId} records explicit viewport navigation, including returning to
an already open thread or live subagent panel. It uses a monotonic sequence,
separate from the cache's access timestamp. Automatic opens, MCP reads, agent
output, watch retries and socket reconnects do not refresh this priority. Frozen
subagent transcripts still use static blobs and do not request a socket on focus.
With spare slots, a new document connects without displacing another. At local capacity, a new focus first reclaims an eligible unprotected connection, then a view-only connection, then an active writer's connection. Within those classes, the least recently focused document yields first. One handoff reserves one released slot until teardown finishes; a displaced focus cannot immediately win it back merely by reconnecting. User navigation can request it again explicitly. Stopping sync never stops the agent or removes its writer lease, local document, durable outbox or publication subscription. Offline edits upload on its next turn.
Documents waiting at least 30 seconds become eligible for service in oldest-service order. Three focus admissions are followed by one overdue service admission when both compete. An admitted service turn resists focus preemption for 30 seconds, so repeated navigation cannot continually cancel it. The latest focused resident stays connected while other protected connections rotate. Protected residents yield to fairness after 30 seconds of service and catch-up, or after five minutes if catch-up remains incomplete. Eligible unprotected, caught-up clients without outgoing updates can yield immediately. Pending outboxes can yield after 30 seconds; their durable batches remain queued. Cold-job age survives disk-page boundaries and teardown gaps. A full set of writers therefore delays background work instead of starving it indefinitely. These deadlines bound scheduling opportunities on a healthy transport, not completion time under an unavailable server or arbitrarily large backlog.
The process socket budget separately reports socketWaiting. If another profile
is waiting at the global cap, a host can yield an aged connection even below its
own local cap. Socket admission remains FIFO across profiles; focus ordering is
per profile. The same teardown accounting and service safeguards apply. No
rotation runs solely because an already-connected document has pending updates.
A live subagent's independent transcript consumes one of the same slots as a parent chat. Thus 20 parent chats plus 28 live subagent transcripts request 48 clients and share 28 slots in one profile. Twenty agents is a measured workload recommendation, not an agent-launch limit or a reservation of 20 sync slots.
Desktop transcript, queue and subagent watches, MCP reads and remote watch
proxies own explicit RPC cancellation handles. Closing a quiet view or finishing
a one-shot MCP read releases its lease without waiting for another document
event. subscribe_scoped owns cancellation without waiting for the first item;
subscribe_checked additionally waits for server acknowledgement.
Process-wide chat data budgets allow 32 socket lifetimes, four simultaneous dials and eight HTTP requests. At most six background HTTP requests hold slots, leaving two for interactive requests. Socket permits live in the pump until teardown; HTTP permits cover the response body. Dial starts are spaced by 50 ms. HTTP idle pool retention is bounded separately. Registry/device control links are outside the chat budget; Git, terminal processes, DNS, Happy Eyeballs connection attempts and other features still consume descriptors outside it. These caps leave room for them but cannot guarantee headroom against arbitrary unrelated resource use.
A typed EMFILE/ENFILE cause pauses chat admission for two seconds. Reconnects use jitter and the same budgets, including the HTTP fallback. Admission has at most 128 queued permit requests. Cancelling client construction aborts its actor and fallback; it cannot detach a socket-owning task.
Caller-owned handles protect the open-to-subscribe handoff without the former 30 s document eviction exemption. Watches, explicit agent writer leases, active bounded clients, pending host commands and failed publication writes protect resident state. Legacy document Arc ownership remains a conservative guard. Periodic cache eviction runs even when no document changes. Network teardown does not detach the durable local-update subscription. Failed outbox writes retain a memory copy, pause new client admissions, and retry; already-running agents and persistence continue. Memory containing unpersisted edits is never intentionally reclaimed to satisfy the cache target.
Durable progress and wake protocol
SQLite migrations add a durable work table, a monotonic version clock, and a reconciliation cursor. A wake receipt is retired only after catch-up, command acceptance and successful snapshot persistence, closing the crash-before-debounce window. The dispatcher discovers unacknowledged/rejected outbox batches after restart without a UI open. Checkpoint obligations use the existing idempotent publication path. Snapshot and cursor persistence remain atomic.
A renewed wake on an existing client requests a fresh row read on that transport instead of closing the socket. Monotonic catch-up tickets fence each receipt: neither an older in-flight backfill nor an older HTTP pull can acknowledge a newer wake. Wake reads and gap repairs are serialized on the socket; HTTP fallback can also satisfy a ticket with a complete subsequent pull. Request serialization is separate from replay classification: fresh wake rows remain live. A complete HTTP pull clears an existing row gap only if no newer gap was observed during that pull. Receipt retirement still requires command acceptance and snapshot persistence. Failed reads retain the durable job and use the existing transport recovery deadlines.
New hosts advertise nudgeAck=1. The edge persists a wake before sending it and
keeps it until the host confirms local durable admission. An opaque token fences
old ACKs against newer wakes for the same chat. Delivery uses a 64-receipt window
and a five-second alarm for retry. A reconnect starts a fresh window.
The edge keeps at most 4096 distinct queued chat wakes. At capacity it returns 503 without discarding accepted work, and persists an overflow reconciliation marker. The host accepts that marker durably, waits for registry synchronization, and scans owned chat IDs in resumable pages of 32. The sender's command/outbox remains the source of truth; a wake is only a notification to look for work.
Deploying the edge before or after the engine is supported: old edges omit tokens, and old hosts receive legacy delivery without ACK requirements (in pages rather than a full replay burst). Legacy hosts retain their former best-effort guarantee; they cannot acknowledge or process the new reconciliation marker. No deployment is performed by this change. Database changes are additive; downgrading loses the new delivery/resource guarantees.
Diagnostics and validation
SyncStatus.resources exposes process budget use/limits, pending admissions, open
documents, cumulative document loads, retention reasons, durable batch/job counts, Linux descriptor count
and the Unix soft descriptor limit. A per-chat state and additive
ChatConnectivity.syncState distinguish local, queued, connecting, synchronized,
offline and storage-error states. The selected chat's connection caption displays
queued sync or persistence trouble without exposing internal limits.
Relevant checks:
cargo test --locked -p zeron-engine --test sync_resources -- --nocapture
cargo test --locked -p zeron-engine --lib --test session_publication --test restart_resume --test misc --test born_chat2_race --test message_queue --test transcript_salvage
cargo test --locked -p zeron-sync --lib
cargo test --locked -p zeron-rpc --test device_room
cargo check --locked -p zeron-ui
cd edge
npm run typecheck
npm test
The resource regression lowers RLIMIT_NOFILE to 256 only in an isolated child, queues 1000 chats, serves real loopback WebSockets, drops connections together, checks bounded documents/transports, repeatedly launches Git, waits for durable work to drain, and checks socket release on shutdown. It measures a two-worker headless test process, not the complete desktop application's platform baseline. The workerd tests use real DO SQLite and sockets for old ACKs, retry, overflow and more-than-256-wake delivery. Other coverage checks restart publication, lost ACKs, checkpoint rejection, constructor/shutdown cancellation and the view-attachment race. Loopback lifecycle tests hold 28 valid checkpoints beyond the idle grace, verify durable snapshots after release, and admit an unrelated chat while a renewed wake waits on a peer stalled before ROWS_DONE without replacing it. The 48-chat regression runs three waves of wake receipts and local edits with eight viewed/writer clients, checking zero reconnects for those clients, the 28-client cap, and complete durable backlog drainage. Additional tests cover spare capacity, active outboxes at capacity, retiring only one client for one waiter, and stale socket/HTTP replies arriving after a newer wake. Another 48-chat case fills all 28 slots with fresh views and drains the waiting jobs after one view closes. Lifecycle tests serialize their process-wide budget use while the normal test runner stays parallel. Regressions cover metadata-only cold discovery before a service turn, focus handoffs without immediate reconnect ping-pong, accidental reads that do not count as focus, and writes made while an active writer's sync is queued. Three delivery waves across 48 parent/subagent writer documents assert remote receipt of every marker and both connection caps. A real-time test lets cold work pass after the 30-second deadline without ending any of the 28 resident writers. Forty writers across two profiles exercise global contention below each profile's local cap. Scoped MCP reads and remote watch cancellation still release quiet views; wake/HTTP tests retain replay and gap fencing coverage.
Admission regressions also queue twelve foreign legacy wakes (including missing room-generation metadata), assert prompt interactive admission and receipt retirement, then repeat the wake burst. Controls cover a wake preceding its registry row, modern non-host readers, a local epoch overriding stale registry metadata, host reassignment while waiting, version-fenced retirement, and preservation of pending outbox bytes.