CHAT_RELAY_LAYER="1" (every room offers the relay layer) has been set
on the live production Worker since 2026-11-12 19:57 UTC, but this
branch's wrangler.jsonc, which production is deployed from, had no
CHAT_RELAY_LAYER at all. The next deploy from here would have dropped
it, and unset means nobody gets the relay layer. The config now says
what production runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The engine fetches <edge>/models/v1.json (HTTPS on zeron.sh only, no
redirects, 64 KiB cap, ETag, 6 h TTL, 30 min back-off), keeps the last good
list on disk for offline use, and rejects a bad list whole. The list can
only add a row a known, new-enough CLI runs, reorder, relabel or hide (never
the preselected row, never every row); the CLI's own default wins. The edge
serves it from a checked-in file chosen by MODEL_LIST: staging only, so
production answers 404 and engines keep their built-in catalogs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
One old app anywhere on an account kept every chat on stored rows, and a rollout
had to wait out a 14-day window to see them all. A chat now goes to generation 3
whatever else is on the account. Old clients read that as a chat2 room as they
always have, and when one opens a chat the room tells its host, which stores for
as long as it's there. So storage is paid per chat, only while an old client has
that chat open.
With nothing left to gate, the rollout has one control: CHAT_RELAY_LAYER on the
edge, for everyone or a list of user ids. Engines run the layer by default and
follow the room's answer, with ZERON_RELAY_LAYER=0 as a kill switch. Gone: the
registry's list of old clients, the account-wide rule, and CLOUD_RELAY_LAYER.
The smoke test proves it against shipped v0.2.138, v0.2.137 and v0.2.116.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nothing turned ZERON_RELAY_LAYER on in a Cloud machine's engine, so the
rollout's engine step couldn't reach Cloud. With CLOUD_RELAY_LAYER=1 on the
edge, machines provisioned from then on get it in their unit's environment;
the value is validated like the rest (empty or 1).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The room's checkpoint upload now takes parts (main) and applies the relay
layer's rules (stale refusal, floor coverage, heal) to both the whole and
the staged commit, before reading the body.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An engine from before checkpoint parts posts a long chat's whole snapshot
(35 MB and more) as one checkpoint, gets 413, and posts it again a few
seconds later: the edge read every byte of each before refusing it. Now
/checkpoint and /checkpoint-part read their bodies through `readCapped`:
a declared length past the cap is answered 413 before a byte is read,
and a body of no declared length is read only until it passes the cap.
What's within the cap is read as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
The edge takes at most 32 MiB of checkpoint in one request; a long chat's
snapshot is past that (35 and 39 MB), so its host's every post was refused
413 and the room kept no fresh checkpoint.
`POST /chat2/{id}/checkpoint-part?offset=` appends a piece (at most 8 MiB)
to the room's staged checkpoint, kept as the blob store's chunk rows;
`POST /checkpoint?seqCovered=&staged=<size>` commits it by renaming those
rows, with the same guards (and `incomplete` when it isn't all there).
GET /checkpoint streams it a row at a time (Range as before), and the
nightly backup writes a large one as an object of its own, streamed. A
single POST, its 32 MiB cap and its 413 are as they were; engines send
parts once they run the change (zeron/zgx 3182a9702).
The edge half of zeron/zgx 3182a9702, alone, with its outcome type made
to name `incomplete` (tsc). Checked on staging: a 36 MiB checkpoint in 5
parts committed and read back byte-exact, whole and from a range; a small
one as one POST; one past 32 MiB still 413.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
A reader that came back to a relay chat asked the host for what it missed, and
the room forwarded that to nobody if the host had already left. Asking for the
chat's host now wakes it, within the existing once-a-minute limit; its announce
on joining restarts the reader's request.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Twice on staging a resume sat at the provider for minutes (4 min 46 s here;
7 to 11 min in provisioning earlier) while the user saw only the usual
"waking". Machine-ready times otherwise: p50 about 10 s, slowest of 179
staging wakes 27 s.
- Past 45 s without reaching ready (CLOUD_WAKE_SLOW_SECONDS), the wake reads
stage "slow", and a resume stuck in provisioning is stopped and resumed
again, up to twice, before the slow poll that was there already.
- A machine that is up but whose engine hasn't been heard from past the same
mark also reads "slow" until it is; the restart at a minute and a half is
unchanged.
- The stage clears as soon as the engine talks. The ledger records wake_slow
with what it was stuck on, and the wake's timeline adds slow, kicked,
slowReason and kicks.
- The apps say it plainly: desktop's waiting line and the shared core's setup
step (iOS, Android) read "Waking the (Cloud) machine is taking longer than
usual", and the send isn't called late meanwhile.
Tested with the fake Boat's new stallResumes (a resume stuck until a stop).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On staging a Cloud machine's docs.sqlite3 came back from a sleep with pages
from two moments. Every read of the chat failed, and the message sent next
waited for good while the machine read ready. Three parts, on every engine:
- Prevent: once the quiesce for a sleep is accepted, the engine writes its open
docs into the store and folds the WAL into the database file
(wal_checkpoint(TRUNCATE)); the shutdown does it again last. With the stop
command's sync, the file the provider snapshots is the whole store, at rest.
- Detect: every open runs PRAGMA quick_check first (integrity_check when a
running engine asked for it). A store error that says the file is corrupt is
noticed while running: a Cloud engine restarts at once to rebuild (nobody is
there to do it, and a message would wait), others on their next start.
- Recover: the damaged file is renamed aside (newest three kept, for
diagnosis), a fresh store is made, and every readable row of every table is
copied over, each table read forwards then backwards past the damage. The
ledger comes across, so a handled command never runs twice; docs not
readable come back from their rooms. The heartbeat says when it happened and
the edge records it (store_recovered) and shows storeRecoveredAt.
WAL with synchronous=NORMAL stays: it is crash-consistent as long as writes
reach the disk in order, and after the checkpoint, a clean stop and the sync
nothing is in flight when the snapshot is taken.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The soak found a Cloud machine whose chat database was corrupt on a wake (pages
from two moments: "2nd reference to page"), after an ordinary sleep: the
engine had stopped cleanly and nothing else had run. The message sent next sat
waiting with the session reading ready, since the host couldn't admit the chat.
The provider snapshots the disk from outside the machine right after the
sleep's stop command, and nothing flushed the page cache first. The stop
command now runs sync after stopping the engine, still answering with the
stop's status. Whether the snapshot alone explains the corruption (rather than
the lazy restore on the wake) isn't proven: 1 occurrence in about 200 wakes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only a relay-era checkpoint had to move forward, so a viewer's ordinary checkpoint
(a rejected push, a row threshold) or an older build's could replace a Cloud host's
with less, and a device opening the sleeping chat loaded a truncated one. While the
layer is on, every checkpoint has to include the room's.
A host back on a build without the layer can't heal, so what was queued for it
waited on that heal for good. When a socket without the layer is the chat's host
by the registry, the room now moves the queue into rows for it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The edge counted a machine as gone 3 minutes after its last heartbeat, so a
turn killed by a hard stop sat silent that long before boot recovery resumed it.
The engine now beats every 15 s while a turn or background work runs (every
60 s otherwise), at once when work starts or stops, and says its cadence
(everySecs). The edge counts a machine that says it beats fast as gone after
three of those beats, never under 45 s; an older engine, or one at the idle
pace, keeps 3 minutes. A machine already stopped is put to sleep without the
quiesce and stop commands it can't answer, so its wake follows at once.
A live engine judged dark by mistake still declines the sleep (quiesce answers
busy), so a false alarm costs one command, not the turn.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The desktop transcript now shows Remove beside Retry on a failed send, and
beside "Sending..." while a send's files upload. It calls RemoveHeldSend and
takes the message out of view at once.
The lifecycle rule for stored files goes to 30 days, so a queued message on a
machine asleep for weeks still finds its files. If one has aged out anyway, a
Cloud machine past the 15-minute cap now takes the row out of its queue and
leaves it as its sender's failed message, with a retry that uploads the files
again. Before, the row waited for them forever.
A phone keeps each file it sent through the store until that message is in the
transcript, for queued rows as well as runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Cloud machine's disk is copied back in on every wake while the engine
starts. Two more things came back every time without being needed:
- the release's zeron-engine (78 MB), the desktop app's background engine.
The unit runs `zeron headless`, which links the engine, so the installer
and the updater now leave it out on Cloud, and the tidy and the pre-sleep
cleanup remove it from versions installed before;
- Codex's own downloads in the managed CODEX_HOME: its plugin catalog
(about 60 MB) and the account's plugin bundles (about 30 MB). The
pre-sleep cleanup clears them every sleep; Codex fetches them again in
the background once it runs. Its login, sessions, databases and settings
are never touched.
On staging, a machine on one version went from a median of 17.5 s to
14.2 s send to first output (n=8 each), and its wake downloaded 225 MB,
205 of them the engine it runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A one-row push wrote 5 billed rows: the row, its rows_seq index entry,
seq, backupDirty and pushOutcomes. Measured against the real runtime it
now writes 2, the row and its index entry.
- The head comes from MAX(seq) and the GC floor, so it can't disagree
with the rows. The stored copy flushes with the alarm, at most every
5 minutes, and only when it moved.
- Accepted-push counts are kept in memory and flushed the same way; a
rejection is stored at once. The push log is written once per
notifying push instead of twice.
- The backup mark is written once per backup cycle, with a deadline the
alarm now shares with the flush.
The wire protocol is unchanged. Rolling back past this needs
REGISTRY_ROOM_WRITE_THROUGH=1 and one flush first: older code reads the
stored seq and would stamp new rows below cursors clients hold.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/stats gains writesSinceStart: the billed rows (index entries and alarms
included) the object wrote since it started, counted from each statement's
rowsWritten. The live test runs a streaming turn and a parked session
between two engines on wrangler dev and reports pushes, rows and fields
per phase.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The edge now limits what one user has stored and not yet collected by a Cloud
machine: 1 GiB and 500 files by default, set with ATTACHMENT_QUOTA_BYTES and
ATTACHMENT_QUOTA_OBJECTS. An upload past either gets 507, and the message shows
as failed in its chat with a retry. Collected, deleted and expired files free
their space.
A message to a Cloud-hosted chat now expires after 7 days instead of 1, so it
still runs once its machine comes back from days asleep. Other chats keep their
day. The outbox retries for the same week, and the lifecycle rule for stored
files becomes 14 days.
WatchWaitingMessages lists a chat's messages not yet taken by its agent, with
when each was sent and when it expires, so the composer can say an old one is
about to send.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The room forwarded a queued send to the host only after writing it, and anything
sent once a write has begun waits for the write to be durable. Live peers now get
their copy first. The sender's ack and the copy a host that isn't here reads still
wait for the write. A rejected send never goes early, and a replay goes once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A token already on a machine no longer lives out its hour. Disconnecting a login
revokes its user token and every installation token minted for its sessions with
GitHub, at once. Installation tokens are now kept with the login they were minted
for and never shared between logins, so one login's revocation never reaches
another's sessions.
The disconnect succeeds whether or not GitHub answers: what it didn't revoke is
retried on the object's alarm until GitHub answers or the token would have expired
anyway. A token minted while its login was being disconnected is revoked and never
handed out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Chat links over WebRTC move to the branch zeron/relay-direct-link, whole, so they
can come back. Here a chat's frames take the room and nothing else: no path waits
on a link or prefers one, the room no longer takes a skip list, and a device no
longer serves chatlink/ streams.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A socket can survive the deploy that turns the layer off. Its host went on relaying
into a room that refused the frames, until that socket happened to end. Now it's
closed at its next frame, and the host hears the room's real answer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A credit is meant to be a minute of a small machine, which the provider
bills at 0.5×: 30 billable seconds. The code charged 60.
Balances are worked out from raw usage seconds at the current rate, so
just changing it would re-price all past usage too. The rate now goes by
the UTC day the usage fell on: days before CLOUD_CREDIT_RATE_FROM stay at
60 seconds a credit, and from that day on it's CLOUD_CREDIT_SECONDS. That
covers the balance, plan periods, usage per day and a running machine's
live estimate, which is split at midnight. Provider seconds and dollars
don't change.
Without a real day in CLOUD_CREDIT_RATE_FROM every day stays at 60, even
with CLOUD_CREDIT_SECONDS set. wrangler.jsonc carries the new rate and a
placeholder day, so merging changes nothing until a deployer picks one.
The apps say "a small machine" now, and the "about N hours" line, which
was about the default machine, says small-machine hours.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A machine that sleeps most of the time would otherwise answer on whatever it last
had. Its engine now holds work from assembly, turns a dying machine left to resume
included, and looks for a release. Newer, it installs it, restarts into it and only
answers once the update trial commits. With nothing newer the check costs only what
it outlasts the rest of the boot by.
The message always gets answered. A check that hangs or fails, a failed or slow
download, a refused restart and a restart that never comes all lift the hold on the
version already running, and the background updater tries again later. It can't
loop: a release it went back from or couldn't install isn't tried from a boot again,
and one that failed to download waits 6 hours. A bad new version gives way to the
old one, which finds the rollback and answers; one that hangs gives way after 5
minutes.
This replaces the 2-minute hold on engine updates. What it protected stays: an
update can't restart the engine under a live run or open terminal and, on Cloud,
while a message is waiting. Off Cloud nothing changes. The agent CLI check keeps its
own delay.
A longer wake reads as updating: CloudSession.stage says so, the desktop's waiting
word follows it, the answer's setup steps name the version, and the timeline records
the check, the install, the restart and which version answered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A cached answer kept a device that had just stopped hosting able to confirm rows for
the length of the cache. A real two-device run lost a message to the new host that way.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A machine reports a turn only with its next heartbeat. Stopped within that minute,
it went to sleep as idle, and its chat said the reply was still streaming until
someone sent again. A send reaching a machine that's up now counts as work if no
heartbeat follows it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A chat this device has never loaded reads as empty once it has caught up with a room
that stores nothing. That looks like a chat with no messages. Now, while it has
messages and an empty copy here, it waits on its host and says so if the host is away.
A client that sends no host hint gets the host from the registry, so an old reader
arriving at a relayed chat still brings the host back to store for it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It came up with no engine and billed until its TTL, for a chat that reads error.
A send resumes it just the same from stopped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A resume that doesn't fit leaves the sandbox in Boat's error state, and the wake
gave up on that state, so waking on a bigger size never reached Boat and said
the bigger size was too small. It now resumes from there.
Boat's limit counts what the machine added, not its base image, so judging df
against it warned a fresh machine at 41% and called one that fits critical.
Boot recovery starts a killed turn ahead of the credential broker: the agent came
up with no login and couldn't find its conversation, so no killed turn resumed.
It now waits for the broker.
A machine that goes dark during a turn is looked at once its heartbeats are late,
not a whole idle window on, and Boat's other refusal spelling is read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Old clients and rollback:
- The registry room remembers clients that join without the layer. Phones write no
device row, so this is how a chat's host learns of them.
- The room remembers relaying, and holds the queue back from readers without the
layer until the host's heal, so no stored row can depend on ops only peers hold.
- A build with the flag off owes one heal for a generation 3 chat it hosts.
- A parked row that reading the checkpoint doesn't fill stops redialling.
- Row writers for host and activity write only their own fields.
Checkpoints:
- A relay-era checkpoint waits until this process has caught up, and the room
refuses one that would move backwards.
- Chats over 24 MiB don't flip, and go back to stored rows.
- A chat too big to checkpoint no longer blocks a Cloud sleep.
Queue and links:
- Only the chat's host, as the registry says, can retire a queue row. A new host
re-reads the queue.
- A connected socket without the layer gets queued rows as stored rows.
- The edge flag alone wakes nobody.
- A link ends when its chat's handle goes away, and a handle with a link isn't
evicted. A failed export reads as gone, not empty.
- A round over a dead link ends when the link does.
Local-first: what arrives by relay is journaled locally until a save covers it. The
guarantee and the rollback order are in the design doc.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>