relay_layer_on no longer looks at the edge: on unless ZERON_RELAY_LAYER is
set to anything but 1. Whether a chat relays is still the room's call, so
an edge that doesn't offer the layer (production's today, an old
self-hosted one, a local dev edge) keeps every chat on stored rows.
A new test proves that against a room shaped like production's: the host
stays on generation 2, its commits go up as stored rows, no relay frame
reaches the room, and no heal is owed. The relay task no longer encodes a
version to announce when no room has offered the layer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
Production's edge may not serve the relay layer yet (#1299 deploys it),
so an engine runs the layer only against the staging edge
(ZERON_EDGE_URL=https://staging.zeron.sh) or when ZERON_RELAY_LAYER=1.
Against production's edge, the default, or any other, it stays on stored
rows exactly as before; ZERON_RELAY_LAYER set to anything but 1 is still
the kill switch, staging included. One place: relay_layer_on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
One old app anywhere on an account kept every chat on stored rows, and a rollout
had to wait out a 14-day window to see them all. A chat now goes to generation 3
whatever else is on the account. Old clients read that as a chat2 room as they
always have, and when one opens a chat the room tells its host, which stores for
as long as it's there. So storage is paid per chat, only while an old client has
that chat open.
With nothing left to gate, the rollout has one control: CHAT_RELAY_LAYER on the
edge, for everyone or a list of user ids. Engines run the layer by default and
follow the room's answer, with ZERON_RELAY_LAYER=0 as a kill switch. Gone: the
registry's list of old clients, the account-wide rule, and CLOUD_RELAY_LAYER.
The smoke test proves it against shipped v0.2.138, v0.2.137 and v0.2.116.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nothing turned ZERON_RELAY_LAYER on in a Cloud machine's engine, so the
rollout's engine step couldn't reach Cloud. With CLOUD_RELAY_LAYER=1 on the
edge, machines provisioned from then on get it in their unit's environment;
the value is validated like the rest (empty or 1).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With the layer on, a chat over the relay limit (64 KiB in this debug run) stays on
stored rows after its first save and after its host restarts. With it off, a
message too big to store reaches the other device by checkpoint, and so do the
turns after it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The desktop transcript now shows Remove beside Retry on a failed send, and
beside "Sending..." while a send's files upload. It calls RemoveHeldSend and
takes the message out of view at once.
The lifecycle rule for stored files goes to 30 days, so a queued message on a
machine asleep for weeks still finds its files. If one has aged out anyway, a
Cloud machine past the 15-minute cap now takes the row out of its queue and
leaves it as its sender's failed message, with a retry that uploads the files
again. Before, the row waited for them forever.
A phone keeps each file it sent through the store until that message is in the
transcript, for queued rows as well as runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Cloud machine's disk is copied back in on every wake while the engine
starts. Two more things came back every time without being needed:
- the release's zeron-engine (78 MB), the desktop app's background engine.
The unit runs `zeron headless`, which links the engine, so the installer
and the updater now leave it out on Cloud, and the tidy and the pre-sleep
cleanup remove it from versions installed before;
- Codex's own downloads in the managed CODEX_HOME: its plugin catalog
(about 60 MB) and the account's plugin bundles (about 30 MB). The
pre-sleep cleanup clears them every sleep; Codex fetches them again in
the background once it runs. Its login, sessions, databases and settings
are never touched.
On staging, a machine on one version went from a median of 17.5 s to
14.2 s send to first output (n=8 each), and its wake downloaded 225 MB,
205 of them the engine it runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A failed send nobody retried was deleted after a week, and a send that couldn't
upload for a week stopped trying. Now the outbox never gives up on its own: it
backs off to every 15 minutes after a week, and a failed send stays until it's
retried or removed.
RemoveHeldSend {chatId, id} takes a send out of the outbox, failed or still
uploading. It never goes out after that, even mid-upload, and its stored copies
are deleted. On mobile, PendingSend.removable marks those rows and
SessionHandle::remove_send does the same.
Every try uploads all of a send's files again, so none ages out at the edge
while it's alive. A phone now keeps its copy until the message has run, so a
retry after expiry uploads it again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every release a machine took stayed in ~/.zeron/app (about 270 MB each),
and a resumed Cloud disk is copied back in whole while the engine starts,
so each one kept was read in beside the version starting. On staging, two
machines three updates in answered in about 39 s instead of 18.
Once an update settles, and on every start after, the versions older than
current go, with the one an update gave up on. A Cloud machine keeps none
of them; other Linux installs keep the newest older one to go back to by
hand. Never current, the running executable's folder, or a newer version
(an update being staged). Downloads a stop cut off are cleared too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A phone back from a long background asked a host that wasn't in the room for
the frames it missed, and the room forwards only to who's there. The round
then waited out its 60 s stall: on a real iPhone the rest of a reply landed
61.5 s after foreground. Now a sync round asked of a peer that joins with
nothing back restarts at once. No conflicts. zeron-sync lib tests pass (108).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The edge now limits what one user has stored and not yet collected by a Cloud
machine: 1 GiB and 500 files by default, set with ATTACHMENT_QUOTA_BYTES and
ATTACHMENT_QUOTA_OBJECTS. An upload past either gets 507, and the message shows
as failed in its chat with a retry. Collected, deleted and expired files free
their space.
A message to a Cloud-hosted chat now expires after 7 days instead of 1, so it
still runs once its machine comes back from days asleep. Other chats keep their
day. The outbox retries for the same week, and the lifecycle rule for stored
files becomes 14 days.
WatchWaitingMessages lists a chat's messages not yet taken by its agent, with
when each was sent and when it expires, so the composer can say an old one is
about to send.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A token already on a machine no longer lives out its hour. Disconnecting a login
revokes its user token and every installation token minted for its sessions with
GitHub, at once. Installation tokens are now kept with the login they were minted
for and never shared between logins, so one login's revocation never reaches
another's sessions.
The disconnect succeeds whether or not GitHub answers: what it didn't revoke is
retried on the object's alarm until GitHub answers or the token would have expired
anyway. A token minted while its login was being disconnected is revoked and never
handed out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings the split: run_link, the link inputs and events, CHAT_LINK_PREFIX,
the link stats and the room's skip are gone, and the host no longer serves
chatlink/. The direct-link code is parked on zeron/relay-direct-link.
Conflicts: the engine's ChatLinkService and relay-layer.md's direct-link
design section, both taken from relay-layer as the split has them. The phone
side already dropped its link in 93da4b726.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
Chat links over WebRTC move to the branch zeron/relay-direct-link, whole, so they
can come back. Here a chat's frames take the room and nothing else: no path waits
on a link or prefers one, the room no longer takes a skip list, and a device no
longer serves chatlink/ streams.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The owner parked peer-to-peer. A phone no longer opens a direct link to a
chat's host: the room carries every frame, catch-up and send. The link task,
LinkWanted and its idle clock are gone from crates/client, so nothing on a
phone pairs, holds a WebRTC connection, or sends STUN keepalives for a chat.
Terminals keep their own direct path, as on main.
The parked link and its pacing are on zeron/relay-mobile-direct-link.
relay-mobile.md says so up top and its behaviour tables describe the room
path. The room test keeps its route checks, minus the link.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
Brings the direct-link fixes in crates/preview to the phone: signaling
reconnects keep open connections, `gone` keeps the pairing pace, and private
LAN candidates are skipped when the public addresses differ.
relay.rs: both sides changed the PR-status watch. Kept the presence gate from
this branch and the 2-minute cap and next_watch_retry from relay-layer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
A release ships ahead of any edge flag. This runs the real client against a real edge
with the flag off, or from before the layer: a send is a stored row, as ever.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first versions of these passed with the behaviour broken: the phone-level test didn't
force the bootstrap pull and the socket to overlap, and a second path hid the parked row.
Each now fails when its behaviour is taken out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An idle host may still be opening the chat its wake asked for. A few quick retries get
the link up in seconds, where waiting out the pairing curve took 15 s or more.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The app runs the relay layer, and an empty transcript says why it's empty, with the
desktop's strings. A send waiting on a path or a host no longer ages into "Not
delivered". Debug arguments let a rig play an old phone, a phone with no network, and
a send from outside.
relay_rig drives a headless engine while a phone is tested against it, and
docs/design/relay-mobile.md is the handoff for Android.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A credit is meant to be a minute of a small machine, which the provider
bills at 0.5×: 30 billable seconds. The code charged 60.
Balances are worked out from raw usage seconds at the current rate, so
just changing it would re-price all past usage too. The rate now goes by
the UTC day the usage fell on: days before CLOUD_CREDIT_RATE_FROM stay at
60 seconds a credit, and from that day on it's CLOUD_CREDIT_SECONDS. That
covers the balance, plan periods, usage per day and a running machine's
live estimate, which is split at midnight. Provider seconds and dollars
don't change.
Without a real day in CLOUD_CREDIT_RATE_FROM every day stays at 60, even
with CLOUD_CREDIT_SECONDS set. wrangler.jsonc carries the new rate and a
placeholder day, so merging changes nothing until a deployer picks one.
The apps say "a small machine" now, and the "about N hours" line, which
was about the default machine, says small-machine hours.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A machine that sleeps most of the time would otherwise answer on whatever it last
had. Its engine now holds work from assembly, turns a dying machine left to resume
included, and looks for a release. Newer, it installs it, restarts into it and only
answers once the update trial commits. With nothing newer the check costs only what
it outlasts the rest of the boot by.
The message always gets answered. A check that hangs or fails, a failed or slow
download, a refused restart and a restart that never comes all lift the hold on the
version already running, and the background updater tries again later. It can't
loop: a release it went back from or couldn't install isn't tried from a boot again,
and one that failed to download waits 6 hours. A bad new version gives way to the
old one, which finds the rollback and answers; one that hangs gives way after 5
minutes.
This replaces the 2-minute hold on engine updates. What it protected stays: an
update can't restart the engine under a live run or open terminal and, on Cloud,
while a message is waiting. Off Cloud nothing changes. The agent CLI check keeps its
own delay.
A longer wake reads as updating: CloudSession.stage says so, the desktop's waiting
word follows it, the answer's setup steps name the version, and the timeline records
the check, the install, the restart and which version answered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A cached answer kept a device that had just stopped hosting able to confirm rows for
the length of the cache. A real two-device run lost a message to the new host that way.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A row goes in one write at a time, and a drain on another thread could take it
between them: it left with its id and none of the rest, and the next write failed
on a deleted container. The send failed, or the row vanished. It showed as a queue
test failing about one run in eight.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A chat this device has never loaded reads as empty once it has caught up with a room
that stores nothing. That looks like a chat with no messages. Now, while it has
messages and an empty copy here, it waits on its host and says so if the host is away.
A client that sends no host hint gets the host from the registry, so an old reader
arriving at a relayed chat still brings the host back to store for it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A resume that doesn't fit leaves the sandbox in Boat's error state, and the wake
gave up on that state, so waking on a bigger size never reached Boat and said
the bigger size was too small. It now resumes from there.
Boat's limit counts what the machine added, not its base image, so judging df
against it warned a fresh machine at 41% and called one that fits critical.
Boot recovery starts a killed turn ahead of the credential broker: the agent came
up with no login and couldn't find its conversation, so no killed turn resumed.
It now waits for the broker.
A machine that goes dark during a turn is looked at once its heartbeats are late,
not a whole idle window on, and Boat's other refusal spelling is read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Old clients and rollback:
- The registry room remembers clients that join without the layer. Phones write no
device row, so this is how a chat's host learns of them.
- The room remembers relaying, and holds the queue back from readers without the
layer until the host's heal, so no stored row can depend on ops only peers hold.
- A build with the flag off owes one heal for a generation 3 chat it hosts.
- A parked row that reading the checkpoint doesn't fill stops redialling.
- Row writers for host and activity write only their own fields.
Checkpoints:
- A relay-era checkpoint waits until this process has caught up, and the room
refuses one that would move backwards.
- Chats over 24 MiB don't flip, and go back to stored rows.
- A chat too big to checkpoint no longer blocks a Cloud sleep.
Queue and links:
- Only the chat's host, as the registry says, can retire a queue row. A new host
re-reads the queue.
- A connected socket without the layer gets queued rows as stored rows.
- The edge flag alone wakes nobody.
- A link ends when its chat's handle goes away, and a handle with a link isn't
evicted. A failed export reads as gone, not empty.
- A round over a dead link ends when the link does.
Local-first: what arrives by relay is journaled locally until a save covers it. The
guarantee and the rollback order are in the design doc.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gen-3 chats relay their host's ops instead of storing them. Delivery over the
room or a direct WebRTC link is a hint, and version exchange finds and fetches
whatever was missed. A separate offline queue holds messages for a host that
isn't there until it has saved them. Cloud hosts also post checkpoints.
With CHAT_RELAY_LAYER and ZERON_RELAY_LAYER unset, hello is byte-identical, the
room answers new frames as unknown types, no queue table exists and no engine
state or stream service is created.
Nothing deletes existing rows: the queue is its own table, and relay-era
checkpoints prune nothing. Old clients get a current checkpoint and stored rows
while they're around, and anything queued moves into rows they can read.
Mobile isn't part of this, so a phone seen in the last 14 days keeps its
account's chats on stored rows.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The vault keeps each Cloud session on its own GitHub login. When that login is
disconnected, needs reconnecting or lost the repository, a run now fails naming
it and what to do, instead of working on with no way to push. Asked again before
every run, so a disconnect is noticed at the next one.
New RPCs list the logins, disconnect one and pick the one new sessions try first.
Repositories name the login a new session would use and the App installation
that gives the access; older replies still parse.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Now that a Cloud engine knows it's the installed service, restart_when_superseded
pokes the updater during every boot. A poke ended the initial delay, so each wake
checked for a release, and could start a download, as the waiting message started
its run. Only a forced check cuts that delay short now; a poke asks whether one is
due. Every installed engine gets this: its first check waits the 20 s the delay
always said it would.
The test fake now treats a command that ran as proof the machine is up, as the
wake does, and the durable suite covers a refresh (no timeline) and a machine
brought back from dark (engine started, no send behind its timeline).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Desktop sends with an app capture, and sends queued during a live turn, still
uploaded straight to the host first, so a sleeping machine failed them at send
time. For a Cloud host they now stage locally like any other send.
A capture names its image in an XML attribute, where the ref and path are both
escaped, so the host rewrites a ref in both spellings. QueueMessage takes
transfers: a row waits in the outbox like a command, and its message shows in
the transcript until the queue panel has it. Over a peer link both keep
uploading first, as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A login connected through another device flow used to take over: repositories
and grants came only from the most recent one. Now they are all in use at once.
Repositories merge across logins, each naming the login a new session would
use. A session keeps the login and App installation of its first grant, whatever
is connected later; a delegate takes its parent's. Logins and installations are
separate records, and a session records both.
One login failing leaves the others working and says which, and disconnecting
one refuses its sessions by name until it comes back. Stored logins and old
clients carry on unchanged. docs/design/cloud-device.md states the rules.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The wake polled through Workflow sleeps (2 s, then 10), so a machine already
up waited whole seconds to be noticed, and the engine was started only after
that. fastWake does it in one step: resume, send the start at once (the
provider holds a command until the machine can run it), poll every 250 ms.
A sleep stops the engine's unit, and a machine restored from that snapshot can
come back with it stopped. zeron-cloud-boot.service stays active through the
snapshot and starts it on resume with no exec round trip. The unit no longer
waits on network-online.target, and the sleep's stop command upgrades machines
already provisioned, since nothing re-runs the installer.
Every wake now writes a wake_timing ledger entry and a cloud.wake_timing log
line, joined with the engine's report (POST /runner/wake-timing).
GET /admin/cloud/wakes lists one user's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A shipped engine's CloudState is a closed enum, so one state it doesn't know
fails its whole session list. The edge now sends `data_expired` only to a client
naming it in `x-zeron-cloud-states`. Any other client reads such a session as
`error`, with `failedAction: "wake"` and `dataExpired: true`, on every session
route, and its Retry gets a typed refusal saying why.
This build's engine and client name it, and the state enum gains a catch-all so
a later state can't do this again. A size too small now reaches clients as
`disk_too_small: …` across the RPC's plain-text errors, and the proto types take
what the app reads with defaults.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Attachment bytes only travelled over a direct peer link to the chat's host.
A Cloud machine may be asleep, so a slow wake or a sender going offline lost
the message after 15 minutes with "attachments never arrived".
A send to a Cloud-hosted chat now waits in a durable outbox while its bytes go
up to /blob/{chatId}/attachments/{uploadId}. Only then does the message enter
the chat. The machine fetches the bytes when it wakes, re-drains as an
UploadCommit does, and deletes the object. The 15-minute cap now applies only
when the store has nothing either.
Chats on a user's own computers, hosts without attachment-store-v1, and edges
without the route keep the peer path. The desktop engine and the mobile core
share the outbox. A permanent failure shows in the chat with a retry. Objects
are deleted on delivery, with a 7-day lifecycle rule as the fallback; see
docs/design/cloud-attachments.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Machine sizes. The catalog lives in policy.ts and reaches clients on the status.
A user sets the default for new sessions, and a wake can move a sleeping machine
to another size. The provider's meter stays what's billed; the live estimate
follows the resize.
Wake failures. A send recovers a session in `error` that still has a sandbox.
A resume refused for want of disk is parsed in the Boat provider only, into a
typed failure with the figure. It's never retried and never upsized on its own,
since credits change: the session reports `wakeBlocked`, and a size smaller than
needed gets a typed 409.
Disk. The heartbeat reports the disk, judged against what the size holds: warn
at 80%, critical at 90%. A resumed disk hydrates lazily and under-reads, so the
edge keeps a high-water mark until the provider says every file is there. Before
a sleep the engine clears Cargo's incremental caches, then compiled deps and
package-manager downloads while still over the warning level. It's bounded, and
it never holds up the sleep.
Data expiry. Boat deletes a stopped machine's disk after 14 days, and a resume
restarts that clock, so a machine 12 days asleep comes up and goes straight back
for a minute of machine time. `expiresAt` says when otherwise. A machine whose
disk is gone reads `data_expired`, and `retry` starts a fresh one on the same
repository and branch.
A machine that goes dark during a turn is stopped and woken straight back, so
its boot recovery runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>