* Sync: update loro to 1.17 so streaming chats stop leaking memory
loro before 1.17 left another copy of a doc's text in its string arena on
every snapshot export (fixed in loro-dev/loro 089a4e6). chat_persistence
exports a dirty chat each second, so a chat with a streaming agent grew
quadratically until the arena passed 4 GiB and export panicked (#905).
* Sync: upgrade edge Loro and guard snapshot memory growth
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
---------
Co-authored-by: gratenes <claude@storage.gay>
Co-authored-by: Wing <44917547+wingleeio@users.noreply.github.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
Root cause of the 'Add Tweets' invisible-transcript incident (2026-08-18),
diagnosed from room stats + phone log archives + the absent on-device doc:
the room's checkpoint (seq 5, rows 1-5 trimmed) carries an empty frontier
label, and both clients treated empty-frontier as 'already contained' — so
every FRESH reader skipped the checkpoint, imported rows 6..75 raw, and
Loro parked everything depending on the chat's founding ops: transcript
invisible past the reader's own-lineage rows, no import errors (parking
is not an error), and snapshot export fails with pending ops so the doc
never persists. Converged devices (synced from row 1 pre-trim) unaffected,
which is why the bug only shows on new devices/fresh installs.
- Both clients: drop the empty-frontier shortcut; empty fails the decode
path → not contained → fetch. Fetching is always safe (full-state merge;
an empty-doc seed applies as a no-op) — the shortcut contradicted the
adjacent comment's own principle ('never silently skips history').
Applying the poisoned room's real checkpoint bytes heals it client-side
with no server repair.
- Edge: reject checkpoint POSTs with an empty frontier when seqCovered>0
(empty stays legal for the M1 empty-doc seed) so no new rooms are
poisoned; chat2-check codifies the guard (66 checks green).
- Regression tests: EngineChatSink::contains_frontier(&[]) must be false;
plan-level decision-table entry for the fresh-reader fetch.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The airplane-wifi fix. The WS needs 4+ serial round trips (DNS, TCP, TLS,
upgrade, hello) before byte one, and hostile middleboxes strip the upgrade
entirely — the user-reported 'never connects on the plane'. A plain HTTPS
GET is 1 RTT on a warm connection and indistinguishable from a webpage to
any proxy. So: push/pull/poke.
Edge (worker + DOs):
- GET /registry/{org}/rows?since= — the WS hello's exact delta answer
(full/gcFloor rules shared with handleHello); ?beat=1 doubles as a
presence beat. POST /registry/{org}/push — the WS push twin via a shared
applyPushBatch core (validate, atomic apply, rows broadcast to live
sockets); LWW clocks make replays no-ops.
- GET /chat2/{id}/rows?after= — one request collapsing connect→hello→
state→rowsReq→backfill: u32-LE length-prefixed frames (state with
frontier payload, rows, rowsDone), byte-identical encoding to the WS.
POST /chat2/{id}/rows?batchId= — push twin, batchId-deduped.
Desktop (Rust): RegistryTransport/ChatTransport seams; with a transport,
construction resolves immediately (local-first), an HTTP pull bootstraps
in ~1 RTT while the socket dials, and every backoff cycle pushes pending +
pulls over HTTPS — a WS-hostile network degrades to ≤30s polling instead
of silence. Engine wires both via the existing bearer/reqwest plumbing
(registry: URL derived from the WS provider; chat: EdgeChatTransport).
iOS (Swift): WorkspaceStore pull-first bootstrap + 20s poll-while-down +
HTTP flush of pending writes; the header spinner now means 'server state
reached this session' (flips on the ~1-RTT pull, not the 4-RTT socket).
ChatRoomClient.pullSync: HTTP push of pending sends (permanent rejections
retire, mirroring the error-frame path), framed rows pull applied through
the exact socket path, checkpoint fetch shared via fetchInFlight.
Validation: new edge/scripts/pullpush-check.mjs green against wrangler dev
(push/ack/no-op replay, delta-at-cursor, presence beat, framed pull,
excludeOwn, batchId dedupe); existing smoke + chat2-check (65) + reset
suites green; zeron-sync 34, engine+rpc 180; iOS sim build clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Attachments already travel device-to-device (UploadChunk/UploadCommit to
the chat's host, ReadAttachmentChunk for read-back); the R2 copy was
never read — the sha was discarded after the PUT, so the objects were
unaddressable. Drop the durable mirror outbox + drain worker from the
engine and the /attachments/:sha256 routes from the edge.
The edge keeps a 200 stub for PUT /attachments/* so deployed clients
(≤0.1.62) drain their outboxes once instead of retrying a 404 forever.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- binary/CLI `comet` -> `zeron`; crates comet-* -> zeron-*; COMET_* -> ZERON_*
- data dir ~/.comet-native -> ~/.zeron, with a one-shot rename migration
- services: comet-native.service -> zeron.service; launchd sh.zeron.comet ->
sh.zeron.app; macOS bundle id dev.comet.native -> sh.zeron.app
- release artifacts comet-<ver>-* -> zeron-<ver>-*; drop the Comet.app
tarball-path compat hack in package-macos.sh
- iOS: Comet -> Zeron targets/dirs, bundle id sh.zeron.Zeron, zeron:// scheme
- clients default to https://edge.zeron.sh; installer + landing on zeron.sh
- kept on purpose: repo name, worker names (comet-native-edge/comet-landing;
DO storage is bound to the script name), R2 bucket names, and legacy
comet.zeron.sh routes so 0.1.x clients keep working
zeron-<ver>-* artifacts don't exist until the next release, so curl-install
and landing downloads 404 between this deploy and the 0.2.0 tag.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Artifacts from the verification pass, for the C3 session and CI-someday:
- edge/scripts/chat2-check.mjs: 65-check wire-level E2E against a deployed
worker (protocol, guards, checkpoint lifecycle, quota, sidecars, blob-route
isolation + '#' encoding, reset).
- edge/scripts/chat2-crosscheck.mjs + crates/sync/examples/chat2_live.rs:
real ChatClient (tungstenite + actor) vs a JS Loro peer through a live
ChatRoom — checkpoint-then-rows leg, live push/ack, byte-level convergence.
- edge/scripts/chat2-persist-check.mjs: seed/verify pair proving rows,
checkpoint, frontier meta and sidecars survive DO eviction (redeploy).
- crates/doc/examples/rebuild_whale.rs: M1 rebuild against a real snapshot
file; the 1b65e93d whale: 1,079,986 -> 152,934 bytes (7.1x), 384 sidecar
payloads, idempotent, 8ms.
- edge/wrangler.chat2test.jsonc: throwaway staging config (dev auth, isolated
bucket, workers.dev) used to run all of the above; deploy, test, delete.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two tiers: pure-logic unit tests stay on Node+FakeSql; a new workerd tier
runs against a real SQLite-backed DO so platform limits (the ~2MB
SQLITE_TOOBIG row cap behind the 2026-08-05 whale sync freeze) are the
runtime's own. Extends whale-check with a fold-backpressure phase and the
unit tier with a real-LoroDoc fold-cycle test over chunked rows.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two DO-side bugs that only extremely long sessions trigger, compounding
into "newest messages no longer sync":
- A single update above the ~2MB SQL row cap (bulk import, full re-upload
span) imported, acked Ok, and relayed — then every flush() threw
SQLITE_TOOBIG forever: nothing persisted, and each hibernation reverted
the room to stale state. Update-log rows now chunk across continuation
rows (update-log.ts, same CHUNK_BYTES discipline as blobs.ts), with a
guarded ALTER TABLE migration; durability no longer depends on row caps
or a whale snapshot export succeeding.
- handleJoin's backfill export({mode:"update", from}) on a SHALLOW doc
does not throw for a client behind the trimmed root — it silently
exports ~90 bytes of post-root ops that import as forever-pending:
fresh readers of any force-trimmed whale room got an empty transcript
with zero errors. Detect via isShallow() + from.compare(shallowSinceVV)
and serve the §3.1 full-snapshot resync instead.
Silence-killers so this class can't hide again: the debounced flush now
catches + escalates (was an unhandled rejection — the only flush call
site with no handler above it), /stats wraps failures as JSON 500
stats_failed, and inbound fragment batches are capped at the header
(32MB / 4096 fragments → PayloadTooLarge, before any reassembly buffer).
Tests: update-log.test.ts (FakeSql enforcing the row cap; migration +
byte-identical reassembly), scripts/whale-check.mjs E2E regression (3MB
single-update push → durable, fresh-reader full transcript, /tail 200;
verified across a wrangler dev restart). smoke.mjs + fold-check.mjs pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Diagnosing why every device still showed offline after the fold fix
surfaced two more bugs that together wedged the per-user workspace room
permanently:
1. Engine never retried the FIRST join. crates/sync RoomClient only
self-reconnects AFTER a successful join; an initial failure (a 500 from
an overloaded workspace DO, a token racing a refresh, an edge deploy)
ended workspace_host's join task with "staying offline" and the device
stayed offline until an app restart — presence dead while the relay and
per-chat rooms worked. Three engines restarting together reliably hit
this: they hammered the cold DO at once, all got 500s, all gave up.
Now join_room retries on a capped, jittered backoff (jitter so a
simultaneous fleet restart doesn't resynchronize into a herd).
2. The DO was already wedged and no client-side retry could heal it. Its
update log had grown huge (the pre-fix byte-only cap never tripped on
tiny writes), so every cold `ensureDoc` replay exceeded the DO CPU
limit and reset before any join finished — which also blocked the
compaction that would have shrunk it. A solo join hung 50s+ and never
landed. Add POST /workspace/:orgId/reset-log: it drops the persisted
update log + snapshot WITHOUT calling ensureDoc (so it stays cheap
enough to land on a wedged DO) and boots attached sockets. State is not
lost — every engine re-uploads its full local workspace doc on the next
join ("export everything the server lacks", room.rs:19), the same
recovery as an ws-namespace bump but without client version skew.
Verified against wrangler dev: 51 rows cleared to 0, empty room then
accepts a fresh join + write.
The fold fix (row-count trigger) keeps a healthy room from re-bloating;
this pair recovers one that already did and stops a transient blip from
stranding a device.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-user workspace SessionRoom DO was hitting "Durable Object
exceeded its CPU time limit and was reset" on production. A cold
`ensureDoc` (after any hibernation or reset) replays the snapshot plus
every logged update, and each `doc.import()` crosses the WASM boundary —
so cold-start cost scales with the NUMBER of update rows, not just their
bytes. The workspace doc is high-churn (three devices writing device and
session rows continuously) but each write is tiny, so the 8 MiB byte cap
never tripped while the row count climbed into the thousands. Cold replay
eventually blew the CPU budget; the runtime reset the DO; every client
reconnected into another cold start; and the room's memory-only presence
store was wiped each time — which is why every device showed permanently
offline even though the device-room relay (a different DO) was healthy.
Fold on either budget: bytes bounds one huge update, a new
COMPACT_LOG_ROWS (1500) bounds many small ones. Also drop COMPACT_LOG_BYTES
8 MiB -> 2 MiB so the byte path keeps cold starts cheap too. The fold is
lossless (snapshot re-export + clear log); COUNT(*) is one indexed read
per 5s flush. Verified against wrangler dev: 1700 tiny commits peaked at
1401 rows, crossed 1500, folded back to 99 rows behind a 7 KB snapshot.
Also route GET /workspace/:orgId/stats through the Worker (org-membership
already checked) so the log/snapshot sizes are observable — how I'll
confirm the budget holds in production after this ships.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Teammates in one org must not see each other's spaces/sessions. Session
rooms (per-chat claim-on-first-join owner), device rooms (owner user),
and R2 attachments (att/{userId}/ prefix) were already user-scoped; the
org-wide workspace doc was the gap — it broadcast every member's spaces,
chat titles/previews/paths/branches, devices, and presence.
- Edge: workspace room is now ws3/{orgId}/{userId}, the user segment
derived from the caller's OWN verified claim — another user's room
cannot even be addressed. Org-claim 403 unchanged. Legacy org-wide
ws2 rooms are orphaned (unreachable via routes) on deploy.
- Engine: Auth::user_id() (WorkOS sub; dev bearer `user@org` prefix,
mirroring the edge's parsing), WorkspaceHost joins the per-user room,
EngineCore::assemble_with_org -> assemble_with_identity(org, user).
- Local storage: orgs/{org}/{user}/ for snapshots, command ledger, and
run journals — two users on one machine can't read each other's cache.
- Embedded (headed dev) engine derives identity from its dev token so
the room join matches what the edge derives.
- Live smoke rewritten to assert the new semantics: one user's devices
converge; a same-org teammate lands in an isolated empty doc. The
live Rust convergence test now runs two devices of ONE user (two
users converging is exactly what must no longer happen).
Destructive break (pre-production, per ws->ws2 precedent): fresh
workspace room + fresh local path on next boot.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Streaming: FollowMode::Tail + per-splice scroll_to_end hard-snapped growth
(one 56px jump/commit). Replaced with an owned pin + per-frame StickSpring
(mugen/use-stick-to-bottom constants: damping .7 stiffness .05 mass 1.25,
growth-EMA feed-forward, catch-up, teleport guard); splice-glue anchors the
offset so layout holds and the spring glides. Frame evidence: 2,13,13,10,6,
4,2,2,1,0 px/frame ramp vs a single 56px step. Doc->render path verified
O(changed) already.
Traffic lights: intended macOS inset was dead (.pl overwritten by .px).
TitlebarOptions (transparent, lights at 14,15, title None); animated 200ms
spacer 88px<->12px on fullscreen (per window-controls.tsx); all titlebar
strips are drag regions (start_window_move) w/ double-click zoom.
Pickers: double-highlight fixed (selected wash vs keyboard cursor wash),
search-input inset, w-72 widths, section separators, traits max-height,
skeleton sizes — all per harness-model-picker.tsx. Model description line
deferred (proto lacks the field).
Ports: comet-native defaults moved to 27654 (IPC) / 27640 (edge dev) so it
never conflicts with the original comet (26634/26638/26640/26654) running
on the same machine. 192 UI tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
gen_fixture example (streamed assistant entry + tool + command outcome) +
edge/scripts/compat-check.mjs — 17 shape assertions pass. Tool calls are
canonically kind-tagged (serde) in comet-native; edge runtime verified not to
depend on the old _tag shape.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>