The engine fetches <edge>/models/v1.json (HTTPS on zeron.sh only, no
redirects, 64 KiB cap, ETag, 6 h TTL, 30 min back-off), keeps the last good
list on disk for offline use, and rejects a bad list whole. The list can
only add a row a known, new-enough CLI runs, reorder, relabel or hide (never
the preselected row, never every row); the CLI's own default wins. The edge
serves it from a checked-in file chosen by MODEL_LIST: staging only, so
production answers 404 and engines keep their built-in catalogs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
One old app anywhere on an account kept every chat on stored rows, and a rollout
had to wait out a 14-day window to see them all. A chat now goes to generation 3
whatever else is on the account. Old clients read that as a chat2 room as they
always have, and when one opens a chat the room tells its host, which stores for
as long as it's there. So storage is paid per chat, only while an old client has
that chat open.
With nothing left to gate, the rollout has one control: CHAT_RELAY_LAYER on the
edge, for everyone or a list of user ids. Engines run the layer by default and
follow the room's answer, with ZERON_RELAY_LAYER=0 as a kill switch. Gone: the
registry's list of old clients, the account-wide rule, and CLOUD_RELAY_LAYER.
The smoke test proves it against shipped v0.2.138, v0.2.137 and v0.2.116.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nothing turned ZERON_RELAY_LAYER on in a Cloud machine's engine, so the
rollout's engine step couldn't reach Cloud. With CLOUD_RELAY_LAYER=1 on the
edge, machines provisioned from then on get it in their unit's environment;
the value is validated like the rest (empty or 1).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The room's checkpoint upload now takes parts (main) and applies the relay
layer's rules (stale refusal, floor coverage, heal) to both the whole and
the staged commit, before reading the body.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The edge takes at most 32 MiB of checkpoint in one request; a long chat's
snapshot is past that (35 and 39 MB), so its host's every post was refused
413 and the room kept no fresh checkpoint.
`POST /chat2/{id}/checkpoint-part?offset=` appends a piece (at most 8 MiB)
to the room's staged checkpoint, kept as the blob store's chunk rows;
`POST /checkpoint?seqCovered=&staged=<size>` commits it by renaming those
rows, with the same guards (and `incomplete` when it isn't all there).
GET /checkpoint streams it a row at a time (Range as before), and the
nightly backup writes a large one as an object of its own, streamed. A
single POST, its 32 MiB cap and its 413 are as they were; engines send
parts once they run the change (zeron/zgx 3182a9702).
The edge half of zeron/zgx 3182a9702, alone, with its outcome type made
to name `incomplete` (tsc). Checked on staging: a 36 MiB checkpoint in 5
parts committed and read back byte-exact, whole and from a range; a small
one as one POST; one past 32 MiB still 413.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
A reader that came back to a relay chat asked the host for what it missed, and
the room forwarded that to nobody if the host had already left. Asking for the
chat's host now wakes it, within the existing once-a-minute limit; its announce
on joining restarts the reader's request.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Twice on staging a resume sat at the provider for minutes (4 min 46 s here;
7 to 11 min in provisioning earlier) while the user saw only the usual
"waking". Machine-ready times otherwise: p50 about 10 s, slowest of 179
staging wakes 27 s.
- Past 45 s without reaching ready (CLOUD_WAKE_SLOW_SECONDS), the wake reads
stage "slow", and a resume stuck in provisioning is stopped and resumed
again, up to twice, before the slow poll that was there already.
- A machine that is up but whose engine hasn't been heard from past the same
mark also reads "slow" until it is; the restart at a minute and a half is
unchanged.
- The stage clears as soon as the engine talks. The ledger records wake_slow
with what it was stuck on, and the wake's timeline adds slow, kicked,
slowReason and kicks.
- The apps say it plainly: desktop's waiting line and the shared core's setup
step (iOS, Android) read "Waking the (Cloud) machine is taking longer than
usual", and the send isn't called late meanwhile.
Tested with the fake Boat's new stallResumes (a resume stuck until a stop).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On staging a Cloud machine's docs.sqlite3 came back from a sleep with pages
from two moments. Every read of the chat failed, and the message sent next
waited for good while the machine read ready. Three parts, on every engine:
- Prevent: once the quiesce for a sleep is accepted, the engine writes its open
docs into the store and folds the WAL into the database file
(wal_checkpoint(TRUNCATE)); the shutdown does it again last. With the stop
command's sync, the file the provider snapshots is the whole store, at rest.
- Detect: every open runs PRAGMA quick_check first (integrity_check when a
running engine asked for it). A store error that says the file is corrupt is
noticed while running: a Cloud engine restarts at once to rebuild (nobody is
there to do it, and a message would wait), others on their next start.
- Recover: the damaged file is renamed aside (newest three kept, for
diagnosis), a fresh store is made, and every readable row of every table is
copied over, each table read forwards then backwards past the damage. The
ledger comes across, so a handled command never runs twice; docs not
readable come back from their rooms. The heartbeat says when it happened and
the edge records it (store_recovered) and shows storeRecoveredAt.
WAL with synchronous=NORMAL stays: it is crash-consistent as long as writes
reach the disk in order, and after the checkpoint, a clean stop and the sync
nothing is in flight when the snapshot is taken.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The soak found a Cloud machine whose chat database was corrupt on a wake (pages
from two moments: "2nd reference to page"), after an ordinary sleep: the
engine had stopped cleanly and nothing else had run. The message sent next sat
waiting with the session reading ready, since the host couldn't admit the chat.
The provider snapshots the disk from outside the machine right after the
sleep's stop command, and nothing flushed the page cache first. The stop
command now runs sync after stopping the engine, still answering with the
stop's status. Whether the snapshot alone explains the corruption (rather than
the lazy restore on the wake) isn't proven: 1 occurrence in about 200 wakes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only a relay-era checkpoint had to move forward, so a viewer's ordinary checkpoint
(a rejected push, a row threshold) or an older build's could replace a Cloud host's
with less, and a device opening the sleeping chat loaded a truncated one. While the
layer is on, every checkpoint has to include the room's.
A host back on a build without the layer can't heal, so what was queued for it
waited on that heal for good. When a socket without the layer is the chat's host
by the registry, the room now moves the queue into rows for it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The edge counted a machine as gone 3 minutes after its last heartbeat, so a
turn killed by a hard stop sat silent that long before boot recovery resumed it.
The engine now beats every 15 s while a turn or background work runs (every
60 s otherwise), at once when work starts or stops, and says its cadence
(everySecs). The edge counts a machine that says it beats fast as gone after
three of those beats, never under 45 s; an older engine, or one at the idle
pace, keeps 3 minutes. A machine already stopped is put to sleep without the
quiesce and stop commands it can't answer, so its wake follows at once.
A live engine judged dark by mistake still declines the sleep (quiesce answers
busy), so a false alarm costs one command, not the turn.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Cloud machine's disk is copied back in on every wake while the engine
starts. Two more things came back every time without being needed:
- the release's zeron-engine (78 MB), the desktop app's background engine.
The unit runs `zeron headless`, which links the engine, so the installer
and the updater now leave it out on Cloud, and the tidy and the pre-sleep
cleanup remove it from versions installed before;
- Codex's own downloads in the managed CODEX_HOME: its plugin catalog
(about 60 MB) and the account's plugin bundles (about 30 MB). The
pre-sleep cleanup clears them every sleep; Codex fetches them again in
the background once it runs. Its login, sessions, databases and settings
are never touched.
On staging, a machine on one version went from a median of 17.5 s to
14.2 s send to first output (n=8 each), and its wake downloaded 225 MB,
205 of them the engine it runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A one-row push wrote 5 billed rows: the row, its rows_seq index entry,
seq, backupDirty and pushOutcomes. Measured against the real runtime it
now writes 2, the row and its index entry.
- The head comes from MAX(seq) and the GC floor, so it can't disagree
with the rows. The stored copy flushes with the alarm, at most every
5 minutes, and only when it moved.
- Accepted-push counts are kept in memory and flushed the same way; a
rejection is stored at once. The push log is written once per
notifying push instead of twice.
- The backup mark is written once per backup cycle, with a deadline the
alarm now shares with the flush.
The wire protocol is unchanged. Rolling back past this needs
REGISTRY_ROOM_WRITE_THROUGH=1 and one flush first: older code reads the
stored seq and would stamp new rows below cursors clients hold.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The edge now limits what one user has stored and not yet collected by a Cloud
machine: 1 GiB and 500 files by default, set with ATTACHMENT_QUOTA_BYTES and
ATTACHMENT_QUOTA_OBJECTS. An upload past either gets 507, and the message shows
as failed in its chat with a retry. Collected, deleted and expired files free
their space.
A message to a Cloud-hosted chat now expires after 7 days instead of 1, so it
still runs once its machine comes back from days asleep. Other chats keep their
day. The outbox retries for the same week, and the lifecycle rule for stored
files becomes 14 days.
WatchWaitingMessages lists a chat's messages not yet taken by its agent, with
when each was sent and when it expires, so the composer can say an old one is
about to send.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The room forwarded a queued send to the host only after writing it, and anything
sent once a write has begun waits for the write to be durable. Live peers now get
their copy first. The sender's ack and the copy a host that isn't here reads still
wait for the write. A rejected send never goes early, and a replay goes once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Chat links over WebRTC move to the branch zeron/relay-direct-link, whole, so they
can come back. Here a chat's frames take the room and nothing else: no path waits
on a link or prefers one, the room no longer takes a skip list, and a device no
longer serves chatlink/ streams.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A socket can survive the deploy that turns the layer off. Its host went on relaying
into a room that refused the frames, until that socket happened to end. Now it's
closed at its next frame, and the host hears the room's real answer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A credit is meant to be a minute of a small machine, which the provider
bills at 0.5×: 30 billable seconds. The code charged 60.
Balances are worked out from raw usage seconds at the current rate, so
just changing it would re-price all past usage too. The rate now goes by
the UTC day the usage fell on: days before CLOUD_CREDIT_RATE_FROM stay at
60 seconds a credit, and from that day on it's CLOUD_CREDIT_SECONDS. That
covers the balance, plan periods, usage per day and a running machine's
live estimate, which is split at midnight. Provider seconds and dollars
don't change.
Without a real day in CLOUD_CREDIT_RATE_FROM every day stays at 60, even
with CLOUD_CREDIT_SECONDS set. wrangler.jsonc carries the new rate and a
placeholder day, so merging changes nothing until a deployer picks one.
The apps say "a small machine" now, and the "about N hours" line, which
was about the default machine, says small-machine hours.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A machine that sleeps most of the time would otherwise answer on whatever it last
had. Its engine now holds work from assembly, turns a dying machine left to resume
included, and looks for a release. Newer, it installs it, restarts into it and only
answers once the update trial commits. With nothing newer the check costs only what
it outlasts the rest of the boot by.
The message always gets answered. A check that hangs or fails, a failed or slow
download, a refused restart and a restart that never comes all lift the hold on the
version already running, and the background updater tries again later. It can't
loop: a release it went back from or couldn't install isn't tried from a boot again,
and one that failed to download waits 6 hours. A bad new version gives way to the
old one, which finds the rollback and answers; one that hangs gives way after 5
minutes.
This replaces the 2-minute hold on engine updates. What it protected stays: an
update can't restart the engine under a live run or open terminal and, on Cloud,
while a message is waiting. Off Cloud nothing changes. The agent CLI check keeps its
own delay.
A longer wake reads as updating: CloudSession.stage says so, the desktop's waiting
word follows it, the answer's setup steps name the version, and the timeline records
the check, the install, the restart and which version answered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A cached answer kept a device that had just stopped hosting able to confirm rows for
the length of the cache. A real two-device run lost a message to the new host that way.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A machine reports a turn only with its next heartbeat. Stopped within that minute,
it went to sleep as idle, and its chat said the reply was still streaming until
someone sent again. A send reaching a machine that's up now counts as work if no
heartbeat follows it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A chat this device has never loaded reads as empty once it has caught up with a room
that stores nothing. That looks like a chat with no messages. Now, while it has
messages and an empty copy here, it waits on its host and says so if the host is away.
A client that sends no host hint gets the host from the registry, so an old reader
arriving at a relayed chat still brings the host back to store for it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It came up with no engine and billed until its TTL, for a chat that reads error.
A send resumes it just the same from stopped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A resume that doesn't fit leaves the sandbox in Boat's error state, and the wake
gave up on that state, so waking on a bigger size never reached Boat and said
the bigger size was too small. It now resumes from there.
Boat's limit counts what the machine added, not its base image, so judging df
against it warned a fresh machine at 41% and called one that fits critical.
Boot recovery starts a killed turn ahead of the credential broker: the agent came
up with no login and couldn't find its conversation, so no killed turn resumed.
It now waits for the broker.
A machine that goes dark during a turn is looked at once its heartbeats are late,
not a whole idle window on, and Boat's other refusal spelling is read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Old clients and rollback:
- The registry room remembers clients that join without the layer. Phones write no
device row, so this is how a chat's host learns of them.
- The room remembers relaying, and holds the queue back from readers without the
layer until the host's heal, so no stored row can depend on ops only peers hold.
- A build with the flag off owes one heal for a generation 3 chat it hosts.
- A parked row that reading the checkpoint doesn't fill stops redialling.
- Row writers for host and activity write only their own fields.
Checkpoints:
- A relay-era checkpoint waits until this process has caught up, and the room
refuses one that would move backwards.
- Chats over 24 MiB don't flip, and go back to stored rows.
- A chat too big to checkpoint no longer blocks a Cloud sleep.
Queue and links:
- Only the chat's host, as the registry says, can retire a queue row. A new host
re-reads the queue.
- A connected socket without the layer gets queued rows as stored rows.
- The edge flag alone wakes nobody.
- A link ends when its chat's handle goes away, and a handle with a link isn't
evicted. A failed export reads as gone, not empty.
- A round over a dead link ends when the link does.
Local-first: what arrives by relay is journaled locally until a save covers it. The
guarantee and the rollback order are in the design doc.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rooms from before the window keep a UNIQUE index on batch_id. A replay the window
doesn't reach hit that index on insert and threw, so the batch never acked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gen-3 chats relay their host's ops instead of storing them. Delivery over the
room or a direct WebRTC link is a hint, and version exchange finds and fetches
whatever was missed. A separate offline queue holds messages for a host that
isn't there until it has saved them. Cloud hosts also post checkpoints.
With CHAT_RELAY_LAYER and ZERON_RELAY_LAYER unset, hello is byte-identical, the
room answers new frames as unknown types, no queue table exists and no engine
state or stream service is created.
Nothing deletes existing rows: the queue is its own table, and relay-era
checkpoints prune nothing. Old clients get a current checkpoint and stored rows
while they're around, and anything queued moves into rows they can read.
Mobile isn't part of this, so a phone seen in the last 14 days keeps its
account's chats on stored rows.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Now that a Cloud engine knows it's the installed service, restart_when_superseded
pokes the updater during every boot. A poke ended the initial delay, so each wake
checked for a release, and could start a download, as the waiting message started
its run. Only a forced check cuts that delay short now; a poke asks whether one is
due. Every installed engine gets this: its first check waits the 20 s the delay
always said it would.
The test fake now treats a command that ran as proof the machine is up, as the
wake does, and the durable suite covers a refresh (no timeline) and a machine
brought back from dark (engine started, no send behind its timeline).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A login connected through another device flow used to take over: repositories
and grants came only from the most recent one. Now they are all in use at once.
Repositories merge across logins, each naming the login a new session would
use. A session keeps the login and App installation of its first grant, whatever
is connected later; a delegate takes its parent's. Logins and installations are
separate records, and a session records both.
One login failing leaves the others working and says which, and disconnecting
one refuses its sessions by name until it comes back. Stored logins and old
clients carry on unchanged. docs/design/cloud-device.md states the rules.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The wake polled through Workflow sleeps (2 s, then 10), so a machine already
up waited whole seconds to be noticed, and the engine was started only after
that. fastWake does it in one step: resume, send the start at once (the
provider holds a command until the machine can run it), poll every 250 ms.
A sleep stops the engine's unit, and a machine restored from that snapshot can
come back with it stopped. zeron-cloud-boot.service stays active through the
snapshot and starts it on resume with no exec round trip. The unit no longer
waits on network-online.target, and the sleep's stop command upgrades machines
already provisioned, since nothing re-runs the installer.
Every wake now writes a wake_timing ledger entry and a cloud.wake_timing log
line, joined with the engine's report (POST /runner/wake-timing).
GET /admin/cloud/wakes lists one user's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>