0
Fork 0
mirror of https://github.com/zeronsh/comet.git synced 2026-11-16 18:57:19 +00:00
Commit graph

126 commits

Author SHA1 Message Date
Wing
1a7dc2e49c engine: a shut-down engine is freed (two holds kept its whole graph alive)
local_first's online_runtime_shutdown_stops_edge_workers_and_retires_the_graph
failed every time ("engine graph still reachable after shutdown + drop").
CI never ran it (scripts/ci/test-core.sh lists the engine tests it runs).
After shutdown + drop the doc host stayed at 2 strong references for good;
tracking every DocHost clone named both:

- The version-change hook (zeron_update::set_before_restart, c004d547c)
  is process-wide and captured the doc host and workspace strongly, so
  every engine ever assembled in the process stayed alive. It now holds
  them weakly and upgrades when it runs: only this process's live engine
  restarts it (the CLI never sets a hook), and with its graph gone there
  is nothing left to prepare.
- A cycle: the engine holds its plugins (sessions.set_plugins), the
  plugins' agents hold their host (EngineAgentHost, 564da4414), and the
  host holds the doc host, sessions and workspace. Plugins::shutdown now
  lets go of that host, as shutdown already breaks the sessions <-> doc
  host cycle. (The agents' sink holds the plugins weakly; left alone.)

Each alone still leaks (the test fails with either fix undone); with
both it passes, 5/5. A new restart_hook test (its own binary: the hook
is process-wide) shows the hook still prepares the running engine's
store and holds nothing after shutdown; with the old strong hook it fails.

The test's next check then failed on this Mac only: one `HEAD /` after
shutdown. lsof named the sender: the Zeron Engine already running on
the Mac, whose preview discovery probes this process's listeners (a
listener bound after shutdown got probed too). Not the engine under
test, so the counting edge leaves `HEAD /` out, as the client tests'
mock edge already does; the engine sends the edge no `HEAD /`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
2026-11-11 13:38:55 -07:00
Wing
98199a5b66 engine: the registry reads, publishes and saves only what changed
On beta.6 the registry was ~half of the engine's remaining CPU and almost
all of its SQLite writes. Each change (about twice a second on a 5-device
account, mostly session-status rows) re-parsed every chat and session for
every list. Each local write saved the whole 2.56 MB JSON replica. And
the preview service re-read every chat on each scan.

1. Preview: the scan's chat list comes from the published watch channel
   (lib.rs). No registry parse per scan; rescans stay event-driven.

2. Reads (crates/doc/src/registry/row_cache.rs): every mutation records
   the (kind, id) whose overlay it may change:
   - put_authoritative;
   - the full replace in apply_state (every row);
   - pending ops in enqueue_ops, coalesce_unsent, ack_batch and
     set_local_only.
   Typed reads (read_chats / read_sessions / read_devices) re-parse only
   those rows. Tests check every cached read against a full read, so a
   mutation path that forgets to record panics. Per-kind versions let
   publish_lists rebuild and send only the lists whose kind changed.
   Devices still go out on the presence tick and on relay-verified
   presence.

3. Saves: a new row_deltas table (new migration). Each save appends what
   changed since the last one (the changed authoritative rows, the
   header, the small pending queue), chained by seq on the base whose
   saved_at it names. A whole save comes only:
   - when the deltas pass max(1 MiB, the base) or 512 rows;
   - once the registry has been quiet for 30 s, and at the latest 5 min
     after the oldest delta (a build without row deltas reads the base
     alone, so what it can miss after a crash is bounded in time);
   - at flush and shutdown;
   - for the sidebar-import acknowledgement;
   - before a restart into another version (stop deltas, save the
     registry whole, then the chats and the WAL).
   Loads (load_registry, also used by local_import) apply the clean
   chain, stop at a gap or an undecodable delta, keep the pending writes
   found past a break, and save whole next. append_row_delta refuses
   when the base on disk isn't the one its token names.

Equal-terms A/B (fresh scratch store per binary, 300 chats, real mock
turns round-robin on 5 chats, list watches as the app, 5 min, providers
off, env -i):
- base 365383537: 249 whole registry saves (26.3 MB/min), engine CPU
  4.5% / 3.6% / 4.4%;
- this change: 7 whole saves (0.74 MB/min) before the final budget; with
  it, 2 whole saves + 236 row deltas = ~0.64 MB/min. Engine CPU 2.7% /
  2.8%.

Tests (each fails on the old behaviour):
- a_session_status_stream_parses_only_the_rows_that_changed: 200 one-row
  changes parse <= 200 rows, against 60,000 with full reads.
- cached_reads_match_full_reads_under_every_mutation: 3,000 random steps
  of every mutation kind.
- a_session_status_stream_saves_and_publishes_only_what_changed (engine):
  < 4 KB per change, no whole save, the chat list doesn't republish,
  reload equals memory, flush compacts. With whole saves it fails at
  573 KB.
- row_deltas_fold_into_a_whole_save_once_the_registry_is_quiet.
- a_registry_saved_as_deltas_loads_back_the_same,
  a_session_status_change_saves_one_row_not_the_registry,
  a_full_replace_needs_a_whole_save,
  a_broken_delta_chain_keeps_its_pending_writes.

An independent review found no HIGH issues. Its two MEDIUM findings are
fixed: the time-bounded compaction plus the whole import
acknowledgement, and the relay presence republish. So are LOW 3, 4 and
6; the cache's cross-check now runs in tests only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
2026-11-11 11:19:43 -07:00
Wing
3ea858b9b6 Engine idle: views' streams close with their handle; no lost socket waiter
From the independent re-review of the previous commit:

- The waiter that kicks the sync dispatcher when a chat's last viewer
  leaves held clones of the watch senders, so a viewer's stream outlived
  a dropped handle (a cutover or delete) and never resubscribed. It now
  ends, releasing them, as the handle drops. It also spawns on the
  handle's own runtime when a view attaches from a plain thread.
- A socket request now signals contention only when it actually has to
  wait (a failed try-acquire), closing the check-then-wait race. Contention
  behind a descriptor-exhaustion pause keeps the dispatcher's timer armed,
  since that pause ends on a clock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
2026-11-11 05:01:11 -07:00
Wing
bd3796114e Engine: an idle engine wakes for nothing
The sync dispatcher and remote-wake delivery polled SQLite every 100 ms,
and dev-server discovery walked every user process every 2 s. Idle and
with nothing open, that was 0.3–1.2% CPU and ~85 context switches/s.

Sync dispatcher (doc_host.rs): a pass runs when something it serves
changes: a durable sync job or outbox row (new store change signals), a
sync request, a failed publication, a new handle, a chat's last viewer or
writer leaving, a socket waiter anywhere in the process, a teardown
ending, the registry's chats or its first sync. A 100 ms timer is armed
only while a pass leaves work in flight that moves with time or client
state: a connection catching up, owing a wake receipt or counting down
its idle grace, a client closing, a chat waiting for a slot,
reconciliation or a disk page still to go, a failed write to retry, a
pinned warm set over budget. Passes stay 100 ms apart, so admissions keep
their 4-per-100 ms bound. The fairness handoff, teardown joins and
cursor paging are unchanged.

Remote-wake delivery: wakes on sync-job changes, a delivery ending, or
the workspace being wired. The eight-slot yield keeps its cursor, so
eight dead hosts still can't starve a ninth chat.

Dev-server discovery: scans only the process trees Zeron started (each
process whose own origin marker names a chat or terminal hosted here,
plus their descendants), reading each process's environment once per
lifetime. With none alive, it parks until a run changes state, a terminal
opens, this device's chat folders change, or an open preview loses its
process. Otherwise it keeps its 2 s cadence.

Space git checks: git runs only when an input of its answer changed (the
.git entry, HEAD and the branch it names, config, packed-refs, shallow,
global/system config). Watcher rescans and repository churn cost a few
stats. A failed git call is never cached as the answer.

Measured on a scratch engine (release, symbols), 60 s idle: 0.45% → 0.03%
CPU, 68.8 → 7.3 context switches/s; the 30 s sample has no busy frame
left.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
2026-11-11 04:50:38 -07:00
Wing
b9cf705c36 sync: reconcile_page takes the write lock before it reads
reconcile_page runs a read-then-write transaction on the main
connection. Deferred, its read could predate a commit on the snapshot
connection (every chat save since B2), and SQLite then fails its first
write at once with SQLITE_BUSY instead of waiting out the busy timeout.
The scheduler (doc_host, every 100 ms) ignores the error and retries the
page next tick, so nothing was lost, but pages failed under load. It now
takes an IMMEDIATE transaction: it waits for the write lock, then reads
fresh.

The other main-connection transactions start with a write (their read
snapshot begins at that write, so they wait normally). append on the
snapshot connection is already IMMEDIATE, and the load is read-only.

Test: a_reconcile_page_waits_out_another_connections_commit. Another
connection holds a write transaction while a page starts, then commits;
the page lands. With a deferred transaction it fails ("database is
locked").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
2026-11-11 01:56:25 -07:00
Wing
a943eba34a Merge zeron/engine-snapshot-writes: snapshot writes on their own connection, no whole save after open, whole saves at quiet moments
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 01:51:04 -07:00
Wing
95d7bfc029 engine: whole saves wait for a quiet moment, and adoption needs a chat2 row
Coordinator's decision (no chunked format), plus the third review of B2.

Whole saves at quiet moments. Past the delta budget (a quarter of the
snapshot, at least 1 MiB, or 512 rows), a persister with a worker keeps
writing deltas and marks a compaction due. The worker folds them in once
the chat has had no change for 2 s (QUIET_FOR_COMPACTION), or at
eviction or shutdown as before. A hard cap (4x the budget) forces a
whole save even mid-stream, so deltas can't grow without bound. Offline
use (no worker: tools, tests) can't wait for quiet, so it folds at the
budget as before.
Test: a_streaming_chat_is_saved_whole_only_when_quiet_or_at_the_hard_cap.
~32 KB every 100 ms: past the budget, no whole save while commits
arrive; one within the quiet window; and the hard cap forces one
mid-stream. It fails if the budget forces whole saves with a worker.

Review item 3 (data loss). The M3 discard-and-adopt path loads a
pre-chat2 row, then starts a FRESH doc, and the persister adopted the
loaded version anyway. The fresh lineage was written as deltas on the
old row, whose epoch stayed < 2 (only whole saves stamp it), so after a
crash before compaction the next open discarded everything again.
LoadedBase now carries the row's epoch, and a persister adopts only an
epoch-2 row whose version the doc already contains.
Test: a_fresh_doc_never_continues_a_row_it_did_not_grow_from (the M3
shape, then a crash: the reopen has epoch 2 and the new content; plus an
unrelated doc on an epoch-2 row). It fails without the guard.

Review item 4. append_snapshot_delta takes an IMMEDIATE transaction (a
busy store waits out its busy timeout, not fails at the INSERT).
load_merged reads the row and its deltas in one read transaction.
flush_for_sleep holds the snapshot connection, then the main one, so
the WAL is empty at sleep.

Note (review item 5): with two connections, an outbox write during a
whole save longer than the 5 s busy timeout falls into the
failed-publication path, where it used to wait on the mutex. Nothing is
lost (the replay obligation holds), and quiet-time compaction makes such
saves rare.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
2026-11-11 01:47:34 -07:00
Wing
9ab031c58e Merge zeron/self-presence-on-zgx: own presence moves after a reconnect; the same editor takes its own queue-edit lease back
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 01:44:16 -07:00
Wing
7a062eebd2 Registry: a rejoin beats only for a host still beating; steady beats don't republish
Review of df4eecfe0: the phone shares RegistryClient and stops beating in
the background on purpose (each beat wakes the radio). With a 'beating'
flag that never cleared, every background reconnect after a network change
sent a fresh beat and showed the phone online. The client now keeps when
the host last asked for presence; a (re)join beats at once only within
RegistryTuning::rejoin_beat_window (35 s, about two 15 s heartbeats).

Only that rejoin beat emits a Presence event; a steady own beat shows at
the host's next publish instead of pushing device rows every 15 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
2026-11-11 01:43:35 -07:00
Wing
88d5e3e76f Merge zeron/engine-delta-snapshots: a chat's save writes what changed, not the whole doc
Three independent reviews; every data-loss finding fixed with a test that fails first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 01:35:57 -07:00
Wing
8ada39b77f engine: snapshot writes hold up commits less, and reopening saves no whole doc
The store was one Mutex<Connection>, and every commit writes its outbox
row inside the commit callback (under the chat2 client lock). So each big
snapshot upsert (40-330 ms for a 50-60 MB chat) stalled every commit in
every chat: Wing's freeze-then-burst (found by 13eb2276).

1. Snapshot reads and writes get their own connection (snap_conn). An
   outbox write no longer waits for the Rust mutex behind a whole save,
   only for SQLite's single writer, and loads never block commits.
   Measured (outbox_writes_stay_fast_while_a_whole_snapshot_saves, a
   write every 5 ms during 50 MB whole saves): max wait 120-143 ms
   before, about 64 ms now. Getting under a ~20 ms budget needs whole
   saves written in small transactions; that is a separate change.
2. No whole save after reopening. A load that finds a clean base (no
   deltas, or every delta applied on this base) offers it, and the next
   ChatPersistence adopts it: deltas continue from the version on disk.
   Before, every engine start made the first save of every hot chat
   whole (47 MB for the largest). The offer never displaces a base this
   process already tracks; any full write invalidates it as before; a
   delta append also checks that the base row still exists (primary key
   only, no blob read).

Tests:
- a_reopened_chat_saves_a_delta_not_the_whole_doc: with 0 and 3 earlier
  deltas, the save after a restart is a delta, the reload equals memory
  and has the newest cursor, and compaction still folds everything. It
  fails without adoption ("a whole save after reopening").
- a_reopened_chat_saves_whole_when_its_disk_state_is_not_a_clean_base:
  no adoption past a bad delta, or after a full write in between.
- a_store_from_before_deltas_migrates_and_reads_unchanged: now both
  saves after loading the migrated snapshot are deltas.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
2026-11-11 01:35:53 -07:00
Wing
9103383591 engine: parked rows retire only after a save, and delta columns migrate
Third review of B:

1. The open path retired parked rows right after importing them into
   memory, before the persister or change subscription existed, so
   nothing marked the doc dirty. If the chat was evicted or the process
   stopped before the next change, those ops were on disk nowhere. Now
   the open replays them, marks the doc dirty, saves, and retires the
   rows only if that save left the persister clean. Otherwise they stay
   and replay at the next open. With no chat2 persister, they stay.
   Test: opening_a_chat_after_a_lost_delta_row now reads the store
   through a second connection with no change since the open: Q's ops
   are there. It fails if the rows retire at once ("the parked ops were
   retired before any save held them").

2. base_saved_at and seq were added by editing the snapshot_deltas
   migration, so a dev DB from an earlier build failed every delta
   insert and load. The table is back to its first shape, and each
   column is a new "ADD COLUMN IF MISSING" migration, which migrate()
   checks with PRAGMA table_info (pre-release DBs may already have the
   column). Old rows (base_saved_at 0, seq 0) claim no cursor and go to
   the parked inbox. Test: stores_from_earlier_delta_schemas_open_and_work
   covers all three pre-release shapes. It fails with the columns never
   added ("no such column: base_saved_at").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
2026-11-11 01:35:37 -07:00
Wing
df4eecfe04 Engine: this device's own lastSeenAt keeps moving; the same editor gets its queue lease back
Presence: the registry room never echoes a device's beat back to its
sender, so the engine's own device row only took a lastSeenAt from a
rejoin's hello. After the 03:55 reset on work-laptop it froze there while
the heartbeat kept going. RegistryClient now keeps the beat it last sent on
a live session (own_presence) and the engine folds it into its own row.
Every (re)join also beats at once, so peers see the device live again
right after a reset instead of up to 15 s later.

Queue edit: begin_queued_message_edit refused any unexpired lease, even one
held by the same device and window asking again (its Acquired reply lost
to a dropped app socket, or the chat reopened). The app said "being edited
on another device" for up to a minute. The same editor now takes its own
lease over; any other window or device is still locked out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
2026-11-11 01:18:13 -07:00
Wing
6248f09823 Tests: own lastSeenAt after every socket drops; same editor re-asking for its queue lease
The mock registry room now matches the DO: a presence beat goes to every
socket except its sender, and drop_all_connections() cuts every socket at
once (the 2026-11-11 03:55 reset). Both tests fail on this commit; the fix
follows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
2026-11-11 01:18:13 -07:00
Wing
705b308499 engine: a lost delta row stops the load, and the rows after it are kept
Second review of the delta store (a18277f50):

(a) HIGH: a lost row is not always a causal gap. Row 12 holds peer P's
ops and is lost; row 13 holds peer Q's concurrent ops, which don't build
on P. Row 13 imports with nothing pending, so the prefix merge moved the
cursor to 13 and P's ops were skipped for good. Each delta now carries
its place in its base's chain (seq 1, 2, ... per base saved_at, written
with the row). The merge stops at the first missing number, as at an
undecodable row or a pending import.

(b) MEDIUM: the rows after a stop held ops that a clean save had already
retired from the outbox or relay journal, and the next whole save
(which clears the chain) deleted them. The load now moves them, in one
transaction, into the doc's parked inbox (<chat>#parked). An open
replays that inbox until nothing waits. Only an undecodable row is
dropped. The chain then ends at its clean run.

(c) PERF: the merge ran twice per open: the loader, then
EngineChatSink::new -> snapshot_cursor. EngineChatSink::for_handle
shares the open handle's persister, which already holds the loader's
cursor, and starts no throwaway persister; the sync join uses it.
DocsStore::delta_merges() counts merges (diagnostics).

Tests (each fails on a mutation of the behavior it guards):
- a_lost_row_before_a_concurrent_row_keeps_the_cursor_before_it: cursor
  11, not 13 ("the cursor passed a lost row" without the chain check).
- rows_after_a_pending_stop_survive_the_next_whole_save: the tail is in
  the parked inbox and survives a whole save (empty without parking).
- a_lost_middle_delta_ends_the_load_at_the_last_whole_one: now also
  checks the parked tail.
- opening_a_chat_after_a_lost_delta_row (end to end through DocHost):
  cursor 11, Q's ops back from the parked inbox, P's not, the parked
  inbox retired, and the sync sink adds no merge ("the sink merged the
  doc again" without for_handle).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
2026-11-11 01:07:47 -07:00
Wing
a18277f509 engine: a snapshot load stops at the last delta that applies
Review finding (HIGH) on the delta store. merge_deltas imported every
delta with import_batch and ignored its ImportStatus. A delta whose ops
build on ops the store doesn't hold (a lost middle row: salvage after
damage, a second writer) parks as pending and returns Ok. The export
then dropped those ops, while the load paired it with the NEWEST delta's
cursor, so the room never resent them: a permanent hole. And one
undecodable delta threw the whole batch back to the base.

- R1: the base, then each delta one at a time, in ordinal order. The run
  stops at the first error, or at a status with pending set. A stop
  rebuilds from the base and the clean run only, so no parked op rides
  along. The cursor is the LAST CLEANLY APPLIED delta's (still only if
  it was saved on this base), and the room resends the rest.
- R5: snapshot_cursor returns what the loader returns. With deltas on
  disk it merges (its one caller, EngineChatSink::new, runs at chat
  open); without, it reads the column as before.
- R4: ChatPersistence::new notes deltas already on disk (left by a
  crash). Its next compaction (eviction, shutdown) folds them even if
  the chat never changes.

Tests, each failing on 6f4e0e511:
- a_lost_middle_delta_ends_the_load_at_the_last_whole_one: was
  ("base one", 13), a hole marked received; now ("base one", 11).
- good_then_garbage_then_good_stops_at_the_garbage: was ("base", 10);
  now ("base one", 11).
- deltas_left_by_a_crash_fold_at_the_next_compaction: was 3 deltas left;
  now 0, and the reload equals memory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
2026-11-11 00:23:48 -07:00
Wing
c004d547c7 engine: no snapshot deltas survive a clean stop or a version change
An older engine ignores snapshot_deltas. Rejoin backfill recovers chats
whose rows the room stores, but not relay (generation 3) chats or
local-only data. So nothing may be left in that table when the version
can change.

- Before any restart into another version (restart_service: updates, a
  newer install, the update a channel switch brings), the engine runs
  DocHost::prepare_for_version_change through a new
  zeron_update::set_before_restart hook: the store refuses deltas from
  then on (every save is whole), every open chat is saved whole, and the
  WAL is checkpointed. A stop cut short after that (launchd's SIGKILL
  after its timeout) still leaves a store every version reads.
- Clean stops already compact (flush_all on shutdown and Cloud sleep).
  Now a test proves it.

Tests:
- a_clean_stop_leaves_zero_snapshot_deltas: flush_all leaves zero
  deltas, reloads equal memory, and after prepare_for_version_change
  further edits save whole.
- dying_between_export_and_commit_keeps_the_last_commit: a child aborts
  after export and before the write, in the first whole save, in deltas,
  and in the compacting save (ops 1, 2, 37, 513, 514, 515). The reopened
  store holds exactly ops 1..at-1 with cursor at-1.
- deltas_stay_whole_while_the_cursor_advances_concurrently: a streaming
  thread, the persister's worker and a forcing thread race (281k ops,
  3,287 deltas). Every reload holds ops 1..k with no gap or repeat and
  k >= cursor; the final reload equals memory.
- a_delta_save_retires_a_failed_publication_only_when_it_commits (relay
  chats): a failed delta write keeps the replay obligation and the old
  cursor; a committed one clears both in its transaction.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
2026-11-11 00:05:34 -07:00
Wing
29ad352469 engine: a delta's cursor counts only on the base it was saved on
The downgrade story for snapshot deltas. Say a user crashes with deltas
outstanding, moves to a build without snapshot_deltas, and that build
rewrites the snapshots row, perhaps after a room reset gave it a lower
cursor. On upgrading back, the stale deltas still sat in the table. Their
ops are this doc's, so merging them is harmless, but their cursor from
the old base could claim rows the new base doesn't hold.

Each delta now records the saved_at of the base it extends. Loads and
snapshot_cursor take a delta's cursor only while that base is still the
row on disk; otherwise the row's own cursor stands and the room backfills.
The base's saved_at is kept in memory with its token: reading it from the
row walked the whole snapshot's overflow pages (saved_at is stored after
the blob), which made 300 delta saves of the 49 MB chat take 1.9 s
instead of 7.6 ms.

Test: a_downgrade_rewrite_strips_stale_deltas_of_their_cursor. The real
49 MB reload test still passes: 300 delta saves took 7.6 ms and 36 KB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
2026-11-10 23:58:56 -07:00
Wing
cd3d002046 engine: chat snapshots save what changed, not the whole doc
ChatPersistence exported and wrote the WHOLE Loro snapshot on every save:
up to once a second per changing chat (immediately when urgent). On the
live beta.5 engine that was about 354 MB/min (about 21 GB/hour) into
SQLite: 60 rewrites of the 47 MB coordinator chat in 10 minutes, each
30-125 ms of export. The WAL also sat at 64 MB.

Now each save after the first in a process appends the Loro updates
since the last saved version (export(updates(&saved_vv))) to a new
snapshot_deltas table. The delta and the cursor its state covers commit
in one transaction. When the deltas pass max(1 MiB, snapshot/4) or 512
rows, and on eviction and shutdown (flush_all), the next save writes the
whole snapshot again and clears them.

Why it's safe:
- Reads are unchanged for every caller. load_snapshot and
  load_snapshot_with_cursor return base + deltas merged into one
  snapshot, with the newest delta's cursor. snapshot_cursor reads it too.
  A merge that fails returns the base with ITS cursor (a torn delta
  can't claim a cursor its bytes don't have).
- The snapshots row keeps the base's own bytes and cursor. A build that
  predates the table (a beta -> stable downgrade) reads an older but
  honest pair and backfills from the room.
- A delta is only accepted on the base this process last wrote in full
  (an in-memory token per doc; one engine per store via the instance
  lock). Any other full write (a rebuild, a heal, a delete) clears that
  doc's deltas in its transaction, and the next save is whole.
- The version is read BEFORE each export, like the cursor. An op that
  races an export is at worst written twice. Loro imports are idempotent,
  so nothing is lost or doubled.
- Outbox and inbox retirement still key on is_clean(): a delta is a
  durable save of every change counted.

Tests (chat_persistence, store):
- a_kill_mid_save_loses_and_doubles_nothing: a child process streams
  numbered ops, saving after each; SIGKILL at 5 points (cursors 1,282 to
  15,334, up to 461 deltas outstanding, through several whole saves).
  Every reopen is exactly ops 1..=cursor.
- a_torn_delta_falls_back_to_the_base_pair,
  unappliable_deltas_fall_back_to_the_base_pair
- deltas_leave_the_snapshot_row_an_honest_older_pair (old readers)
- a_store_from_before_deltas_migrates_and_reads_unchanged
- compaction_folds_deltas_into_one_snapshot,
  deltas_past_their_budget_save_the_whole_doc_again,
  another_full_write_makes_the_next_save_whole,
  a_lowered_cursor_is_the_one_read_back,
  a_save_after_the_first_writes_what_changed_not_the_whole_doc
- real_whale_reloads_equal_to_before_through_deltas (ignored; needs
  ZERON_DELTA_WHALE). On a local copy of the 49 MB chat: whole save
  229 ms. 300 delta saves took 7.7 ms total and 36 KB on disk (whole
  saves would have written 14.7 GB). Reload equals the doc in memory
  (op version vector and state) after 99/199/299 deltas and after
  compaction.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
2026-11-10 23:45:06 -07:00
Wing
7c553f437d Relay: a frame the room's queue can't take yet waits, in order
The relay task sent to the room with try_send and dropped the frame when
the 256-frame path was full. A dropped update parked every later one on
the peers until a sync round; a dropped SYNC_REQ or SYNC_DATA left the
round waiting out SYNC_STALL (60 s), then everything landed at once.

Frames that find the queue full now wait in order (up to 1024) and are
offered again every 100 ms. A frame is only given up when the room has
gone (a new session starts its round over anyway) or too many wait; each
is counted (RelayStats::room_frames_dropped) and logged at INFO, and a
lost SYNC_REQ of this device's restarts its round after GAP_GRACE, not
after SYNC_STALL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
2026-11-10 22:33:27 -07:00
Wing
5ac4775867 Relay layer: engines run it against every edge, ZERON_RELAY_LAYER=0 turns it off
relay_layer_on no longer looks at the edge: on unless ZERON_RELAY_LAYER is
set to anything but 1. Whether a chat relays is still the room's call, so
an edge that doesn't offer the layer (production's today, an old
self-hosted one, a local dev edge) keeps every chat on stored rows.

A new test proves that against a room shaped like production's: the host
stays on generation 2, its commits go up as stored rows, no relay frame
reaches the room, and no heal is owed. The relay task no longer encodes a
version to announce when no room has offered the layer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
2026-11-10 12:57:04 -07:00
Wing
04e09f8938 Merge main into zeron/cloud-relay-v3
Main's #1307 reads chats in windows and stops reading before a send, which
replaces most of this branch's handling of sleeping hosts. Kept: a read of a chat
falls back to this engine's replica when its host can't answer, now as window_of.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 06:50:01 -07:00
Wing
68c7ea286c
Chats reach long chats on other devices (#1307)
A long transcript was one RPC message across the device relay. Past 16 MiB
the Durable Object's single frame hit tungstenite's default frame cap in the
reader, ending the whole peer link; past 32 MiB workerd closed the host's
socket, ending every client's. Each read of such a chat from another device
failed with "WatchDocMessages: stream ended before the transcript arrived",
and send_message read the transcript before delivering, so sends failed too.

- Relay: clients that reassemble say so in the RPC stream id; hosts send them
  large replies in 512 KiB pieces. Older clients get an error for that request
  instead of a frame that ends their link. Relay dials take frames up to the
  local socket's cap, so older hosts' big frames arrive.
- WatchDocMessages takes a window (limit, offset, after): frames carry only
  the slice and its facts. Older hosts ignore it and the reader slices.
- send_message delivers first. The turn's marker is the sent message's own id,
  so nothing about the transcript can hold up or undo a send. It no longer
  needs the host's harness catalog when the chat has a harness.
- read_chat, get_chat, respond_to_input and waits ask for windows, and waits
  follow deltas rather than re-reading.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 06:28:32 -07:00
Wing
aec924c131 Relay: old clients no longer hold a chat or an account back
One old app anywhere on an account kept every chat on stored rows, and a rollout
had to wait out a 14-day window to see them all. A chat now goes to generation 3
whatever else is on the account. Old clients read that as a chat2 room as they
always have, and when one opens a chat the room tells its host, which stores for
as long as it's there. So storage is paid per chat, only while an old client has
that chat open.

With nothing left to gate, the rollout has one control: CHAT_RELAY_LAYER on the
edge, for everyone or a list of user ids. Engines run the layer by default and
follow the room's answer, with ZERON_RELAY_LAYER=0 as a kill switch. Gone: the
registry's list of old clients, the account-wide rule, and CLOUD_RELAY_LAYER.

The smoke test proves it against shipped v0.2.138, v0.2.137 and v0.2.116.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 00:46:21 -07:00
Wing
960a21734b Merge zeron/cloud-attachments-r2
# Conflicts:
#	crates/client/src/live/mod.rs
#	crates/client/src/session/mod.rs
#	crates/client/tests/support/mock_edge.rs
#	crates/sync/src/store.rs
#	crates/ui/src/state.rs
2026-11-07 19:37:42 -07:00
Wing
acfdd9a440 Merge zeron/cloud-durable 2026-11-07 19:36:35 -07:00
Wing
6095e58cb6 Merge main into zeron/cloud-attachments-r2
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 18:14:01 -07:00
Wing
9ff5deb707 Merge zeron/relay-mobile-ios: main
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 17:46:06 -07:00
Wing
accc4f044e Merge main into zeron/cloud-durable
Both sides added MCP tests at the same spot; both kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 17:40:58 -07:00
Wing
346044f908 Merge zeron/relay-layer (279a9858): main
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 17:38:18 -07:00
Wing
279a985854 Merge main into zeron/relay-layer
The room's checkpoint upload now takes parts (main) and applies the relay
layer's rules (stale refusal, floor coverage, heal) to both the whole and
the staged commit, before reading the body.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 17:38:02 -07:00
Wing
81bd5feeee Merge zeron/relay-mobile-ios (518ba017): reconnect and woken-host fixes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 13:27:47 -07:00
Wing
518ba0171b Merge zeron/relay-layer (110842d8): a woken host stays for the reader's request
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 13:00:49 -07:00
Wing
a0eca081dc Chat relay: a host stays a while after joining
A host woken for a reader joined after that reader's announce had gone by, and
the reader only asks once it hears the host's. The host read the quiet as idle
and left within a tenth of a second of joining, before the question came, and
nothing woke it again. Joining now counts as recent activity, so it lingers like
it does after a peer's frame.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 13:00:41 -07:00
Wing
d3391b2c4e Chat relay: debug logs for frames, rounds and room changes
Frames in and out of a chat session (relay frames at trace), each sync round
with its peer, each direct frame and whether it left, and the room coming up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 11:05:43 -07:00
Wing
2f2be5388d Chat client: relay frames go out during the backfill and checkpoint fetch
A session only sent the relay layer's frames once its row backfill and any
checkpoint download were done. On a relay chat those frames are the catch-up:
a host that had just rejoined waited on this device's announce and sync
request for as long as the stored-row catch-up took.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 11:05:32 -07:00
Wing
b987c50ece Chat client: a redial asked while down doesn't end the next session
A phone back in the foreground asks the chat on screen for a fresh socket. When
the socket had already dropped, that request waited in the channel and ended
the session the next dial joined, so the chat dialed twice and caught up about
a second later. The next dial answers every request made before it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 10:39:20 -07:00
Wing
5f0f79d181 Chat client: log each reconnect step
A phone back from a suspension sometimes never rejoins its chat room, and
nothing in the reconnect path said where it stopped. Dial start, join, session
end and stop log at info; the dial's slot, token and connect steps and each
backoff at debug.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 10:25:51 -07:00
Wing
3324f850f7 A damaged chat store is found, kept aside and rebuilt
On staging a Cloud machine's docs.sqlite3 came back from a sleep with pages
from two moments. Every read of the chat failed, and the message sent next
waited for good while the machine read ready. Three parts, on every engine:

- Prevent: once the quiesce for a sleep is accepted, the engine writes its open
  docs into the store and folds the WAL into the database file
  (wal_checkpoint(TRUNCATE)); the shutdown does it again last. With the stop
  command's sync, the file the provider snapshots is the whole store, at rest.
- Detect: every open runs PRAGMA quick_check first (integrity_check when a
  running engine asked for it). A store error that says the file is corrupt is
  noticed while running: a Cloud engine restarts at once to rebuild (nobody is
  there to do it, and a message would wait), others on their next start.
- Recover: the damaged file is renamed aside (newest three kept, for
  diagnosis), a fresh store is made, and every readable row of every table is
  copied over, each table read forwards then backwards past the damage. The
  ledger comes across, so a handled command never runs twice; docs not
  readable come back from their rooms. The heartbeat says when it happened and
  the edge records it (store_recovered) and shows storeRecoveredAt.

WAL with synchronous=NORMAL stays: it is crash-consistent as long as writes
reach the disk in order, and after the checkpoint, a clean stop and the sync
nothing is in flight when the snapshot is taken.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 04:28:44 -07:00
Wing
ecb53e2e99 Merge relay mobile iOS updates
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
2026-11-07 08:00:16 +00:00
Wing
9402f3cba5 Merge zeron/relay-layer (e9e78b38): parked rows rejoin sooner 2026-11-07 00:53:01 -07:00
Wing
e9e78b3880 Chat client: a long-parked row rejoins sooner, backing off across sessions
The rejoin that tells a parked row about a newer checkpoint waited five minutes,
too long for a reader whose host just made up for a rejected push: the
two-engine smoke's flags-off run timed out on it. It now rejoins once a row has
waited 20 s, and the wait keeps doubling across sessions, so a row nothing can
fill still costs a handful of dials an hour. The smoke's big messages are
incompressible, so their size is what it says.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 00:52:44 -07:00
Wing
5be004e327 Merge zeron/relay-layer: the second review's fixes 2026-11-07 00:41:56 -07:00
Wing
fe5f24a822 Relay layer: queue reads that didn't go out, transfers cut off by a reconnect, another account's old clients
- A queue read the room's full channel refused was counted as asked but never
  sent, so catch-up waited on it all session. It's asked again.
- A host reconnecting mid-transfer left the requester waiting out the 60 s
  stall. Its join now asks for the rest from where the transfer got to.
- The list of clients without the layer is process-wide: a new registry
  connection, perhaps another account's, starts without one.
- Deleting a chat's snapshot also clears its parked relay updates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 00:41:27 -07:00
Wing
d1d862f3da Chat client: a parked row hears about a newer checkpoint, and a queued send doesn't wait half a minute
A row that parked after reading the room's checkpoint stayed parked until the
socket dropped, even with the layer off: the recheck only asked for rows again,
and only a hello says the checkpoint changed. A row parked that long now joins
again, and the checkpoint is read only if its frontier changed. This also covers
a viewer whose host made up for a rejected push with a new checkpoint.

A send meant for the queue waited for a socket until the network was called
socketless, three failed dials and 30 s, where it used to go over HTTPS at once.
After 5 s of failed dials it goes as a stored row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 00:41:27 -07:00
Wing
6e8b4c8a99 Keep every send until its person removes it
A failed send nobody retried was deleted after a week, and a send that couldn't
upload for a week stopped trying. Now the outbox never gives up on its own: it
backs off to every 15 minutes after a week, and a failed send stays until it's
retried or removed.

RemoveHeldSend {chatId, id} takes a send out of the outbox, failed or still
uploading. It never goes out after that, even mid-upload, and its stored copies
are deleted. On mobile, PendingSend.removable marks those rows and
SessionHandle::remove_send does the same.

Every try uploads all of a send's files again, so none ages out at the edge
while it's alive. A phone now keeps its copy until the message has run, so a
retry after expiry uploads it again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 23:36:10 -07:00
Wing
022da49f95 Merge zeron/relay-layer (bee81e89): a host joining restarts a stalled sync
A phone back from a long background asked a host that wasn't in the room for
the frames it missed, and the room forwards only to who's there. The round
then waited out its 60 s stall: on a real iPhone the rest of a reply landed
61.5 s after foreground. Now a sync round asked of a peer that joins with
nothing back restarts at once. No conflicts. zeron-sync lib tests pass (108).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 22:50:19 -07:00
Wing
bee81e898d Relay layer: a host joining restarts a sync round asked of it
A phone back from a long background asked the chat's host for what it missed
while the host's socket was closed. The room only forwards to who's there, so the
ask went nowhere, and the phone waited out the 60 s stall although the host
joined within a second. A peer's join now restarts a round asked of it that
nothing has come back for. On a real phone, catch-up after 34 minutes away took
61.5 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 22:48:36 -07:00
Wing
12c03242fe Relay layer: a viewer's first send goes to the queue
Whether to send into the queue was set on the chat's client by the maintenance
tick, a second after the client existed. A client that dialed and pushed before
then stored the send as a row: on a real Cloud machine, a message sent to a
sleeping chat from a device that had just opened it went into stored rows. It's
set on the relay handle now, before the first dial. The room's answer for each
session still decides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 21:32:26 -07:00
Wing
6bd35c92b1 Cap a user's waiting attachments, and give Cloud messages a week
The edge now limits what one user has stored and not yet collected by a Cloud
machine: 1 GiB and 500 files by default, set with ATTACHMENT_QUOTA_BYTES and
ATTACHMENT_QUOTA_OBJECTS. An upload past either gets 507, and the message shows
as failed in its chat with a retry. Collected, deleted and expired files free
their space.

A message to a Cloud-hosted chat now expires after 7 days instead of 1, so it
still runs once its machine comes back from days asleep. Other chats keep their
day. The outbox retries for the same week, and the lifecycle rule for stored
files becomes 14 days.

WatchWaitingMessages lists a chat's messages not yet taken by its agent, with
when each was sent and when it expires, so the composer can say an old one is
about to send.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 21:17:50 -07:00