local_first's online_runtime_shutdown_stops_edge_workers_and_retires_the_graph
failed every time ("engine graph still reachable after shutdown + drop").
CI never ran it (scripts/ci/test-core.sh lists the engine tests it runs).
After shutdown + drop the doc host stayed at 2 strong references for good;
tracking every DocHost clone named both:
- The version-change hook (zeron_update::set_before_restart, c004d547c)
is process-wide and captured the doc host and workspace strongly, so
every engine ever assembled in the process stayed alive. It now holds
them weakly and upgrades when it runs: only this process's live engine
restarts it (the CLI never sets a hook), and with its graph gone there
is nothing left to prepare.
- A cycle: the engine holds its plugins (sessions.set_plugins), the
plugins' agents hold their host (EngineAgentHost, 564da4414), and the
host holds the doc host, sessions and workspace. Plugins::shutdown now
lets go of that host, as shutdown already breaks the sessions <-> doc
host cycle. (The agents' sink holds the plugins weakly; left alone.)
Each alone still leaks (the test fails with either fix undone); with
both it passes, 5/5. A new restart_hook test (its own binary: the hook
is process-wide) shows the hook still prepares the running engine's
store and holds nothing after shutdown; with the old strong hook it fails.
The test's next check then failed on this Mac only: one `HEAD /` after
shutdown. lsof named the sender: the Zeron Engine already running on
the Mac, whose preview discovery probes this process's listeners (a
listener bound after shutdown got probed too). Not the engine under
test, so the counting edge leaves `HEAD /` out, as the client tests'
mock edge already does; the engine sends the edge no `HEAD /`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
On beta.6 the registry was ~half of the engine's remaining CPU and almost
all of its SQLite writes. Each change (about twice a second on a 5-device
account, mostly session-status rows) re-parsed every chat and session for
every list. Each local write saved the whole 2.56 MB JSON replica. And
the preview service re-read every chat on each scan.
1. Preview: the scan's chat list comes from the published watch channel
(lib.rs). No registry parse per scan; rescans stay event-driven.
2. Reads (crates/doc/src/registry/row_cache.rs): every mutation records
the (kind, id) whose overlay it may change:
- put_authoritative;
- the full replace in apply_state (every row);
- pending ops in enqueue_ops, coalesce_unsent, ack_batch and
set_local_only.
Typed reads (read_chats / read_sessions / read_devices) re-parse only
those rows. Tests check every cached read against a full read, so a
mutation path that forgets to record panics. Per-kind versions let
publish_lists rebuild and send only the lists whose kind changed.
Devices still go out on the presence tick and on relay-verified
presence.
3. Saves: a new row_deltas table (new migration). Each save appends what
changed since the last one (the changed authoritative rows, the
header, the small pending queue), chained by seq on the base whose
saved_at it names. A whole save comes only:
- when the deltas pass max(1 MiB, the base) or 512 rows;
- once the registry has been quiet for 30 s, and at the latest 5 min
after the oldest delta (a build without row deltas reads the base
alone, so what it can miss after a crash is bounded in time);
- at flush and shutdown;
- for the sidebar-import acknowledgement;
- before a restart into another version (stop deltas, save the
registry whole, then the chats and the WAL).
Loads (load_registry, also used by local_import) apply the clean
chain, stop at a gap or an undecodable delta, keep the pending writes
found past a break, and save whole next. append_row_delta refuses
when the base on disk isn't the one its token names.
Equal-terms A/B (fresh scratch store per binary, 300 chats, real mock
turns round-robin on 5 chats, list watches as the app, 5 min, providers
off, env -i):
- base 365383537: 249 whole registry saves (26.3 MB/min), engine CPU
4.5% / 3.6% / 4.4%;
- this change: 7 whole saves (0.74 MB/min) before the final budget; with
it, 2 whole saves + 236 row deltas = ~0.64 MB/min. Engine CPU 2.7% /
2.8%.
Tests (each fails on the old behaviour):
- a_session_status_stream_parses_only_the_rows_that_changed: 200 one-row
changes parse <= 200 rows, against 60,000 with full reads.
- cached_reads_match_full_reads_under_every_mutation: 3,000 random steps
of every mutation kind.
- a_session_status_stream_saves_and_publishes_only_what_changed (engine):
< 4 KB per change, no whole save, the chat list doesn't republish,
reload equals memory, flush compacts. With whole saves it fails at
573 KB.
- row_deltas_fold_into_a_whole_save_once_the_registry_is_quiet.
- a_registry_saved_as_deltas_loads_back_the_same,
a_session_status_change_saves_one_row_not_the_registry,
a_full_replace_needs_a_whole_save,
a_broken_delta_chain_keeps_its_pending_writes.
An independent review found no HIGH issues. Its two MEDIUM findings are
fixed: the time-bounded compaction plus the whole import
acknowledgement, and the relay presence republish. So are LOW 3, 4 and
6; the cache's cross-check now runs in tests only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
From the independent re-review of the previous commit:
- The waiter that kicks the sync dispatcher when a chat's last viewer
leaves held clones of the watch senders, so a viewer's stream outlived
a dropped handle (a cutover or delete) and never resubscribed. It now
ends, releasing them, as the handle drops. It also spawns on the
handle's own runtime when a view attaches from a plain thread.
- A socket request now signals contention only when it actually has to
wait (a failed try-acquire), closing the check-then-wait race. Contention
behind a descriptor-exhaustion pause keeps the dispatcher's timer armed,
since that pause ends on a clock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
The sync dispatcher and remote-wake delivery polled SQLite every 100 ms,
and dev-server discovery walked every user process every 2 s. Idle and
with nothing open, that was 0.3–1.2% CPU and ~85 context switches/s.
Sync dispatcher (doc_host.rs): a pass runs when something it serves
changes: a durable sync job or outbox row (new store change signals), a
sync request, a failed publication, a new handle, a chat's last viewer or
writer leaving, a socket waiter anywhere in the process, a teardown
ending, the registry's chats or its first sync. A 100 ms timer is armed
only while a pass leaves work in flight that moves with time or client
state: a connection catching up, owing a wake receipt or counting down
its idle grace, a client closing, a chat waiting for a slot,
reconciliation or a disk page still to go, a failed write to retry, a
pinned warm set over budget. Passes stay 100 ms apart, so admissions keep
their 4-per-100 ms bound. The fairness handoff, teardown joins and
cursor paging are unchanged.
Remote-wake delivery: wakes on sync-job changes, a delivery ending, or
the workspace being wired. The eight-slot yield keeps its cursor, so
eight dead hosts still can't starve a ninth chat.
Dev-server discovery: scans only the process trees Zeron started (each
process whose own origin marker names a chat or terminal hosted here,
plus their descendants), reading each process's environment once per
lifetime. With none alive, it parks until a run changes state, a terminal
opens, this device's chat folders change, or an open preview loses its
process. Otherwise it keeps its 2 s cadence.
Space git checks: git runs only when an input of its answer changed (the
.git entry, HEAD and the branch it names, config, packed-refs, shallow,
global/system config). Watcher rescans and repository churn cost a few
stats. A failed git call is never cached as the answer.
Measured on a scratch engine (release, symbols), 60 s idle: 0.45% → 0.03%
CPU, 68.8 → 7.3 context switches/s; the 30 s sample has no busy frame
left.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
reconcile_page runs a read-then-write transaction on the main
connection. Deferred, its read could predate a commit on the snapshot
connection (every chat save since B2), and SQLite then fails its first
write at once with SQLITE_BUSY instead of waiting out the busy timeout.
The scheduler (doc_host, every 100 ms) ignores the error and retries the
page next tick, so nothing was lost, but pages failed under load. It now
takes an IMMEDIATE transaction: it waits for the write lock, then reads
fresh.
The other main-connection transactions start with a write (their read
snapshot begins at that write, so they wait normally). append on the
snapshot connection is already IMMEDIATE, and the load is read-only.
Test: a_reconcile_page_waits_out_another_connections_commit. Another
connection holds a write transaction while a page starts, then commits;
the page lands. With a deferred transaction it fails ("database is
locked").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
Coordinator's decision (no chunked format), plus the third review of B2.
Whole saves at quiet moments. Past the delta budget (a quarter of the
snapshot, at least 1 MiB, or 512 rows), a persister with a worker keeps
writing deltas and marks a compaction due. The worker folds them in once
the chat has had no change for 2 s (QUIET_FOR_COMPACTION), or at
eviction or shutdown as before. A hard cap (4x the budget) forces a
whole save even mid-stream, so deltas can't grow without bound. Offline
use (no worker: tools, tests) can't wait for quiet, so it folds at the
budget as before.
Test: a_streaming_chat_is_saved_whole_only_when_quiet_or_at_the_hard_cap.
~32 KB every 100 ms: past the budget, no whole save while commits
arrive; one within the quiet window; and the hard cap forces one
mid-stream. It fails if the budget forces whole saves with a worker.
Review item 3 (data loss). The M3 discard-and-adopt path loads a
pre-chat2 row, then starts a FRESH doc, and the persister adopted the
loaded version anyway. The fresh lineage was written as deltas on the
old row, whose epoch stayed < 2 (only whole saves stamp it), so after a
crash before compaction the next open discarded everything again.
LoadedBase now carries the row's epoch, and a persister adopts only an
epoch-2 row whose version the doc already contains.
Test: a_fresh_doc_never_continues_a_row_it_did_not_grow_from (the M3
shape, then a crash: the reopen has epoch 2 and the new content; plus an
unrelated doc on an epoch-2 row). It fails without the guard.
Review item 4. append_snapshot_delta takes an IMMEDIATE transaction (a
busy store waits out its busy timeout, not fails at the INSERT).
load_merged reads the row and its deltas in one read transaction.
flush_for_sleep holds the snapshot connection, then the main one, so
the WAL is empty at sleep.
Note (review item 5): with two connections, an outbox write during a
whole save longer than the 5 s busy timeout falls into the
failed-publication path, where it used to wait on the mutex. Nothing is
lost (the replay obligation holds), and quiet-time compaction makes such
saves rare.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
Review of df4eecfe0: the phone shares RegistryClient and stops beating in
the background on purpose (each beat wakes the radio). With a 'beating'
flag that never cleared, every background reconnect after a network change
sent a fresh beat and showed the phone online. The client now keeps when
the host last asked for presence; a (re)join beats at once only within
RegistryTuning::rejoin_beat_window (35 s, about two 15 s heartbeats).
Only that rejoin beat emits a Presence event; a steady own beat shows at
the host's next publish instead of pushing device rows every 15 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
The store was one Mutex<Connection>, and every commit writes its outbox
row inside the commit callback (under the chat2 client lock). So each big
snapshot upsert (40-330 ms for a 50-60 MB chat) stalled every commit in
every chat: Wing's freeze-then-burst (found by 13eb2276).
1. Snapshot reads and writes get their own connection (snap_conn). An
outbox write no longer waits for the Rust mutex behind a whole save,
only for SQLite's single writer, and loads never block commits.
Measured (outbox_writes_stay_fast_while_a_whole_snapshot_saves, a
write every 5 ms during 50 MB whole saves): max wait 120-143 ms
before, about 64 ms now. Getting under a ~20 ms budget needs whole
saves written in small transactions; that is a separate change.
2. No whole save after reopening. A load that finds a clean base (no
deltas, or every delta applied on this base) offers it, and the next
ChatPersistence adopts it: deltas continue from the version on disk.
Before, every engine start made the first save of every hot chat
whole (47 MB for the largest). The offer never displaces a base this
process already tracks; any full write invalidates it as before; a
delta append also checks that the base row still exists (primary key
only, no blob read).
Tests:
- a_reopened_chat_saves_a_delta_not_the_whole_doc: with 0 and 3 earlier
deltas, the save after a restart is a delta, the reload equals memory
and has the newest cursor, and compaction still folds everything. It
fails without adoption ("a whole save after reopening").
- a_reopened_chat_saves_whole_when_its_disk_state_is_not_a_clean_base:
no adoption past a bad delta, or after a full write in between.
- a_store_from_before_deltas_migrates_and_reads_unchanged: now both
saves after loading the migrated snapshot are deltas.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
Third review of B:
1. The open path retired parked rows right after importing them into
memory, before the persister or change subscription existed, so
nothing marked the doc dirty. If the chat was evicted or the process
stopped before the next change, those ops were on disk nowhere. Now
the open replays them, marks the doc dirty, saves, and retires the
rows only if that save left the persister clean. Otherwise they stay
and replay at the next open. With no chat2 persister, they stay.
Test: opening_a_chat_after_a_lost_delta_row now reads the store
through a second connection with no change since the open: Q's ops
are there. It fails if the rows retire at once ("the parked ops were
retired before any save held them").
2. base_saved_at and seq were added by editing the snapshot_deltas
migration, so a dev DB from an earlier build failed every delta
insert and load. The table is back to its first shape, and each
column is a new "ADD COLUMN IF MISSING" migration, which migrate()
checks with PRAGMA table_info (pre-release DBs may already have the
column). Old rows (base_saved_at 0, seq 0) claim no cursor and go to
the parked inbox. Test: stores_from_earlier_delta_schemas_open_and_work
covers all three pre-release shapes. It fails with the columns never
added ("no such column: base_saved_at").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
Presence: the registry room never echoes a device's beat back to its
sender, so the engine's own device row only took a lastSeenAt from a
rejoin's hello. After the 03:55 reset on work-laptop it froze there while
the heartbeat kept going. RegistryClient now keeps the beat it last sent on
a live session (own_presence) and the engine folds it into its own row.
Every (re)join also beats at once, so peers see the device live again
right after a reset instead of up to 15 s later.
Queue edit: begin_queued_message_edit refused any unexpired lease, even one
held by the same device and window asking again (its Acquired reply lost
to a dropped app socket, or the chat reopened). The app said "being edited
on another device" for up to a minute. The same editor now takes its own
lease over; any other window or device is still locked out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
The mock registry room now matches the DO: a presence beat goes to every
socket except its sender, and drop_all_connections() cuts every socket at
once (the 2026-11-11 03:55 reset). Both tests fail on this commit; the fix
follows.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
Second review of the delta store (a18277f50):
(a) HIGH: a lost row is not always a causal gap. Row 12 holds peer P's
ops and is lost; row 13 holds peer Q's concurrent ops, which don't build
on P. Row 13 imports with nothing pending, so the prefix merge moved the
cursor to 13 and P's ops were skipped for good. Each delta now carries
its place in its base's chain (seq 1, 2, ... per base saved_at, written
with the row). The merge stops at the first missing number, as at an
undecodable row or a pending import.
(b) MEDIUM: the rows after a stop held ops that a clean save had already
retired from the outbox or relay journal, and the next whole save
(which clears the chain) deleted them. The load now moves them, in one
transaction, into the doc's parked inbox (<chat>#parked). An open
replays that inbox until nothing waits. Only an undecodable row is
dropped. The chain then ends at its clean run.
(c) PERF: the merge ran twice per open: the loader, then
EngineChatSink::new -> snapshot_cursor. EngineChatSink::for_handle
shares the open handle's persister, which already holds the loader's
cursor, and starts no throwaway persister; the sync join uses it.
DocsStore::delta_merges() counts merges (diagnostics).
Tests (each fails on a mutation of the behavior it guards):
- a_lost_row_before_a_concurrent_row_keeps_the_cursor_before_it: cursor
11, not 13 ("the cursor passed a lost row" without the chain check).
- rows_after_a_pending_stop_survive_the_next_whole_save: the tail is in
the parked inbox and survives a whole save (empty without parking).
- a_lost_middle_delta_ends_the_load_at_the_last_whole_one: now also
checks the parked tail.
- opening_a_chat_after_a_lost_delta_row (end to end through DocHost):
cursor 11, Q's ops back from the parked inbox, P's not, the parked
inbox retired, and the sync sink adds no merge ("the sink merged the
doc again" without for_handle).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
Review finding (HIGH) on the delta store. merge_deltas imported every
delta with import_batch and ignored its ImportStatus. A delta whose ops
build on ops the store doesn't hold (a lost middle row: salvage after
damage, a second writer) parks as pending and returns Ok. The export
then dropped those ops, while the load paired it with the NEWEST delta's
cursor, so the room never resent them: a permanent hole. And one
undecodable delta threw the whole batch back to the base.
- R1: the base, then each delta one at a time, in ordinal order. The run
stops at the first error, or at a status with pending set. A stop
rebuilds from the base and the clean run only, so no parked op rides
along. The cursor is the LAST CLEANLY APPLIED delta's (still only if
it was saved on this base), and the room resends the rest.
- R5: snapshot_cursor returns what the loader returns. With deltas on
disk it merges (its one caller, EngineChatSink::new, runs at chat
open); without, it reads the column as before.
- R4: ChatPersistence::new notes deltas already on disk (left by a
crash). Its next compaction (eviction, shutdown) folds them even if
the chat never changes.
Tests, each failing on 6f4e0e511:
- a_lost_middle_delta_ends_the_load_at_the_last_whole_one: was
("base one", 13), a hole marked received; now ("base one", 11).
- good_then_garbage_then_good_stops_at_the_garbage: was ("base", 10);
now ("base one", 11).
- deltas_left_by_a_crash_fold_at_the_next_compaction: was 3 deltas left;
now 0, and the reload equals memory.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
An older engine ignores snapshot_deltas. Rejoin backfill recovers chats
whose rows the room stores, but not relay (generation 3) chats or
local-only data. So nothing may be left in that table when the version
can change.
- Before any restart into another version (restart_service: updates, a
newer install, the update a channel switch brings), the engine runs
DocHost::prepare_for_version_change through a new
zeron_update::set_before_restart hook: the store refuses deltas from
then on (every save is whole), every open chat is saved whole, and the
WAL is checkpointed. A stop cut short after that (launchd's SIGKILL
after its timeout) still leaves a store every version reads.
- Clean stops already compact (flush_all on shutdown and Cloud sleep).
Now a test proves it.
Tests:
- a_clean_stop_leaves_zero_snapshot_deltas: flush_all leaves zero
deltas, reloads equal memory, and after prepare_for_version_change
further edits save whole.
- dying_between_export_and_commit_keeps_the_last_commit: a child aborts
after export and before the write, in the first whole save, in deltas,
and in the compacting save (ops 1, 2, 37, 513, 514, 515). The reopened
store holds exactly ops 1..at-1 with cursor at-1.
- deltas_stay_whole_while_the_cursor_advances_concurrently: a streaming
thread, the persister's worker and a forcing thread race (281k ops,
3,287 deltas). Every reload holds ops 1..k with no gap or repeat and
k >= cursor; the final reload equals memory.
- a_delta_save_retires_a_failed_publication_only_when_it_commits (relay
chats): a failed delta write keeps the replay obligation and the old
cursor; a committed one clears both in its transaction.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
The downgrade story for snapshot deltas. Say a user crashes with deltas
outstanding, moves to a build without snapshot_deltas, and that build
rewrites the snapshots row, perhaps after a room reset gave it a lower
cursor. On upgrading back, the stale deltas still sat in the table. Their
ops are this doc's, so merging them is harmless, but their cursor from
the old base could claim rows the new base doesn't hold.
Each delta now records the saved_at of the base it extends. Loads and
snapshot_cursor take a delta's cursor only while that base is still the
row on disk; otherwise the row's own cursor stands and the room backfills.
The base's saved_at is kept in memory with its token: reading it from the
row walked the whole snapshot's overflow pages (saved_at is stored after
the blob), which made 300 delta saves of the 49 MB chat take 1.9 s
instead of 7.6 ms.
Test: a_downgrade_rewrite_strips_stale_deltas_of_their_cursor. The real
49 MB reload test still passes: 300 delta saves took 7.6 ms and 36 KB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
ChatPersistence exported and wrote the WHOLE Loro snapshot on every save:
up to once a second per changing chat (immediately when urgent). On the
live beta.5 engine that was about 354 MB/min (about 21 GB/hour) into
SQLite: 60 rewrites of the 47 MB coordinator chat in 10 minutes, each
30-125 ms of export. The WAL also sat at 64 MB.
Now each save after the first in a process appends the Loro updates
since the last saved version (export(updates(&saved_vv))) to a new
snapshot_deltas table. The delta and the cursor its state covers commit
in one transaction. When the deltas pass max(1 MiB, snapshot/4) or 512
rows, and on eviction and shutdown (flush_all), the next save writes the
whole snapshot again and clears them.
Why it's safe:
- Reads are unchanged for every caller. load_snapshot and
load_snapshot_with_cursor return base + deltas merged into one
snapshot, with the newest delta's cursor. snapshot_cursor reads it too.
A merge that fails returns the base with ITS cursor (a torn delta
can't claim a cursor its bytes don't have).
- The snapshots row keeps the base's own bytes and cursor. A build that
predates the table (a beta -> stable downgrade) reads an older but
honest pair and backfills from the room.
- A delta is only accepted on the base this process last wrote in full
(an in-memory token per doc; one engine per store via the instance
lock). Any other full write (a rebuild, a heal, a delete) clears that
doc's deltas in its transaction, and the next save is whole.
- The version is read BEFORE each export, like the cursor. An op that
races an export is at worst written twice. Loro imports are idempotent,
so nothing is lost or doubled.
- Outbox and inbox retirement still key on is_clean(): a delta is a
durable save of every change counted.
Tests (chat_persistence, store):
- a_kill_mid_save_loses_and_doubles_nothing: a child process streams
numbered ops, saving after each; SIGKILL at 5 points (cursors 1,282 to
15,334, up to 461 deltas outstanding, through several whole saves).
Every reopen is exactly ops 1..=cursor.
- a_torn_delta_falls_back_to_the_base_pair,
unappliable_deltas_fall_back_to_the_base_pair
- deltas_leave_the_snapshot_row_an_honest_older_pair (old readers)
- a_store_from_before_deltas_migrates_and_reads_unchanged
- compaction_folds_deltas_into_one_snapshot,
deltas_past_their_budget_save_the_whole_doc_again,
another_full_write_makes_the_next_save_whole,
a_lowered_cursor_is_the_one_read_back,
a_save_after_the_first_writes_what_changed_not_the_whole_doc
- real_whale_reloads_equal_to_before_through_deltas (ignored; needs
ZERON_DELTA_WHALE). On a local copy of the 49 MB chat: whole save
229 ms. 300 delta saves took 7.7 ms total and 36 KB on disk (whole
saves would have written 14.7 GB). Reload equals the doc in memory
(op version vector and state) after 99/199/299 deltas and after
compaction.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
The relay task sent to the room with try_send and dropped the frame when
the 256-frame path was full. A dropped update parked every later one on
the peers until a sync round; a dropped SYNC_REQ or SYNC_DATA left the
round waiting out SYNC_STALL (60 s), then everything landed at once.
Frames that find the queue full now wait in order (up to 1024) and are
offered again every 100 ms. A frame is only given up when the room has
gone (a new session starts its round over anyway) or too many wait; each
is counted (RelayStats::room_frames_dropped) and logged at INFO, and a
lost SYNC_REQ of this device's restarts its round after GAP_GRACE, not
after SYNC_STALL.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
relay_layer_on no longer looks at the edge: on unless ZERON_RELAY_LAYER is
set to anything but 1. Whether a chat relays is still the room's call, so
an edge that doesn't offer the layer (production's today, an old
self-hosted one, a local dev edge) keeps every chat on stored rows.
A new test proves that against a room shaped like production's: the host
stays on generation 2, its commits go up as stored rows, no relay frame
reaches the room, and no heal is owed. The relay task no longer encodes a
version to announce when no room has offered the layer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
Main's #1307 reads chats in windows and stops reading before a send, which
replaces most of this branch's handling of sleeping hosts. Kept: a read of a chat
falls back to this engine's replica when its host can't answer, now as window_of.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A long transcript was one RPC message across the device relay. Past 16 MiB
the Durable Object's single frame hit tungstenite's default frame cap in the
reader, ending the whole peer link; past 32 MiB workerd closed the host's
socket, ending every client's. Each read of such a chat from another device
failed with "WatchDocMessages: stream ended before the transcript arrived",
and send_message read the transcript before delivering, so sends failed too.
- Relay: clients that reassemble say so in the RPC stream id; hosts send them
large replies in 512 KiB pieces. Older clients get an error for that request
instead of a frame that ends their link. Relay dials take frames up to the
local socket's cap, so older hosts' big frames arrive.
- WatchDocMessages takes a window (limit, offset, after): frames carry only
the slice and its facts. Older hosts ignore it and the reader slices.
- send_message delivers first. The turn's marker is the sent message's own id,
so nothing about the transcript can hold up or undo a send. It no longer
needs the host's harness catalog when the chat has a harness.
- read_chat, get_chat, respond_to_input and waits ask for windows, and waits
follow deltas rather than re-reading.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
One old app anywhere on an account kept every chat on stored rows, and a rollout
had to wait out a 14-day window to see them all. A chat now goes to generation 3
whatever else is on the account. Old clients read that as a chat2 room as they
always have, and when one opens a chat the room tells its host, which stores for
as long as it's there. So storage is paid per chat, only while an old client has
that chat open.
With nothing left to gate, the rollout has one control: CHAT_RELAY_LAYER on the
edge, for everyone or a list of user ids. Engines run the layer by default and
follow the room's answer, with ZERON_RELAY_LAYER=0 as a kill switch. Gone: the
registry's list of old clients, the account-wide rule, and CLOUD_RELAY_LAYER.
The smoke test proves it against shipped v0.2.138, v0.2.137 and v0.2.116.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A host woken for a reader joined after that reader's announce had gone by, and
the reader only asks once it hears the host's. The host read the quiet as idle
and left within a tenth of a second of joining, before the question came, and
nothing woke it again. Joining now counts as recent activity, so it lingers like
it does after a peer's frame.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Frames in and out of a chat session (relay frames at trace), each sync round
with its peer, each direct frame and whether it left, and the room coming up.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A session only sent the relay layer's frames once its row backfill and any
checkpoint download were done. On a relay chat those frames are the catch-up:
a host that had just rejoined waited on this device's announce and sync
request for as long as the stored-row catch-up took.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A phone back in the foreground asks the chat on screen for a fresh socket. When
the socket had already dropped, that request waited in the channel and ended
the session the next dial joined, so the chat dialed twice and caught up about
a second later. The next dial answers every request made before it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A phone back from a suspension sometimes never rejoins its chat room, and
nothing in the reconnect path said where it stopped. Dial start, join, session
end and stop log at info; the dial's slot, token and connect steps and each
backoff at debug.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On staging a Cloud machine's docs.sqlite3 came back from a sleep with pages
from two moments. Every read of the chat failed, and the message sent next
waited for good while the machine read ready. Three parts, on every engine:
- Prevent: once the quiesce for a sleep is accepted, the engine writes its open
docs into the store and folds the WAL into the database file
(wal_checkpoint(TRUNCATE)); the shutdown does it again last. With the stop
command's sync, the file the provider snapshots is the whole store, at rest.
- Detect: every open runs PRAGMA quick_check first (integrity_check when a
running engine asked for it). A store error that says the file is corrupt is
noticed while running: a Cloud engine restarts at once to rebuild (nobody is
there to do it, and a message would wait), others on their next start.
- Recover: the damaged file is renamed aside (newest three kept, for
diagnosis), a fresh store is made, and every readable row of every table is
copied over, each table read forwards then backwards past the damage. The
ledger comes across, so a handled command never runs twice; docs not
readable come back from their rooms. The heartbeat says when it happened and
the edge records it (store_recovered) and shows storeRecoveredAt.
WAL with synchronous=NORMAL stays: it is crash-consistent as long as writes
reach the disk in order, and after the checkpoint, a clean stop and the sync
nothing is in flight when the snapshot is taken.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rejoin that tells a parked row about a newer checkpoint waited five minutes,
too long for a reader whose host just made up for a rejected push: the
two-engine smoke's flags-off run timed out on it. It now rejoins once a row has
waited 20 s, and the wait keeps doubling across sessions, so a row nothing can
fill still costs a handful of dials an hour. The smoke's big messages are
incompressible, so their size is what it says.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A queue read the room's full channel refused was counted as asked but never
sent, so catch-up waited on it all session. It's asked again.
- A host reconnecting mid-transfer left the requester waiting out the 60 s
stall. Its join now asks for the rest from where the transfer got to.
- The list of clients without the layer is process-wide: a new registry
connection, perhaps another account's, starts without one.
- Deleting a chat's snapshot also clears its parked relay updates.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A row that parked after reading the room's checkpoint stayed parked until the
socket dropped, even with the layer off: the recheck only asked for rows again,
and only a hello says the checkpoint changed. A row parked that long now joins
again, and the checkpoint is read only if its frontier changed. This also covers
a viewer whose host made up for a rejected push with a new checkpoint.
A send meant for the queue waited for a socket until the network was called
socketless, three failed dials and 30 s, where it used to go over HTTPS at once.
After 5 s of failed dials it goes as a stored row.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A failed send nobody retried was deleted after a week, and a send that couldn't
upload for a week stopped trying. Now the outbox never gives up on its own: it
backs off to every 15 minutes after a week, and a failed send stays until it's
retried or removed.
RemoveHeldSend {chatId, id} takes a send out of the outbox, failed or still
uploading. It never goes out after that, even mid-upload, and its stored copies
are deleted. On mobile, PendingSend.removable marks those rows and
SessionHandle::remove_send does the same.
Every try uploads all of a send's files again, so none ages out at the edge
while it's alive. A phone now keeps its copy until the message has run, so a
retry after expiry uploads it again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A phone back from a long background asked a host that wasn't in the room for
the frames it missed, and the room forwards only to who's there. The round
then waited out its 60 s stall: on a real iPhone the rest of a reply landed
61.5 s after foreground. Now a sync round asked of a peer that joins with
nothing back restarts at once. No conflicts. zeron-sync lib tests pass (108).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A phone back from a long background asked the chat's host for what it missed
while the host's socket was closed. The room only forwards to who's there, so the
ask went nowhere, and the phone waited out the 60 s stall although the host
joined within a second. A peer's join now restarts a round asked of it that
nothing has come back for. On a real phone, catch-up after 34 minutes away took
61.5 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Whether to send into the queue was set on the chat's client by the maintenance
tick, a second after the client existed. A client that dialed and pushed before
then stored the send as a row: on a real Cloud machine, a message sent to a
sleeping chat from a device that had just opened it went into stored rows. It's
set on the relay handle now, before the first dial. The room's answer for each
session still decides.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The edge now limits what one user has stored and not yet collected by a Cloud
machine: 1 GiB and 500 files by default, set with ATTACHMENT_QUOTA_BYTES and
ATTACHMENT_QUOTA_OBJECTS. An upload past either gets 507, and the message shows
as failed in its chat with a retry. Collected, deleted and expired files free
their space.
A message to a Cloud-hosted chat now expires after 7 days instead of 1, so it
still runs once its machine comes back from days asleep. Other chats keep their
day. The outbox retries for the same week, and the lifecycle rule for stored
files becomes 14 days.
WatchWaitingMessages lists a chat's messages not yet taken by its agent, with
when each was sent and when it expires, so the composer can say an old one is
about to send.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings the split: run_link, the link inputs and events, CHAT_LINK_PREFIX,
the link stats and the room's skip are gone, and the host no longer serves
chatlink/. The direct-link code is parked on zeron/relay-direct-link.
Conflicts: the engine's ChatLinkService and relay-layer.md's direct-link
design section, both taken from relay-layer as the split has them. The phone
side already dropped its link in 93da4b726.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
Android can leave a backgrounded process running for hours where iOS suspends it, and
each open socket's keepalive wakes the radio. Once delivery has had its time, the app
tells the core, and every socket closes and stays down until the next foreground, with
no polling or timed retries. Outboxes hold their rows and the version exchange catches
up on return. iOS doesn't call this.
A socket on an old path can die without saying so, so a path change now asks the rooms
and the registry instead of waiting for their next quiet probe.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
Chat links over WebRTC move to the branch zeron/relay-direct-link, whole, so they
can come back. Here a chat's frames take the room and nothing else: no path waits
on a link or prefers one, the room no longer takes a skip list, and a device no
longer serves chatlink/ streams.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first versions of these passed with the behaviour broken: the phone-level test didn't
force the bootstrap pull and the socket to overlap, and a second path hid the parked row.
Each now fails when its behaviour is taken out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
crates/client joins relay rooms when asked to: it says so to the registry and each
room, reads live frames, catches up from the host by version exchange, sends through
the outbox into the room's queue, and reads a Cloud machine's checkpoint. What came
by relay is journaled until a save holds it, so a kill loses nothing, and any chat
loaded before opens with no network and no host.
A chat on screen and doing something gets a direct link over the WebRTC stack
terminals use, and gives it and its connection up when it stops. A checkpoint a chat
off screen lacks waits until someone looks. The bootstrap pull and the socket share
one download, and a parked row doesn't read a checkpoint this doc already holds.
An empty transcript says why: syncing, offline, or that the chat lives on another
device. Off by default in zeron-client, and off, every byte is what shipped phones send.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>