0
Fork 0
mirror of https://github.com/zeronsh/comet.git synced 2026-11-16 18:57:19 +00:00
Commit graph

15 commits

Author SHA1 Message Date
Chance
6973500bc0
Sync: update loro to 1.17 so streaming chats stop leaking memory (#1116)
* Sync: update loro to 1.17 so streaming chats stop leaking memory

loro before 1.17 left another copy of a doc's text in its string arena on
every snapshot export (fixed in loro-dev/loro 089a4e6). chat_persistence
exports a dirty chat each second, so a chat with a streaming agent grew
quadratically until the arena passed 4 GiB and export panicked (#905).

* Sync: upgrade edge Loro and guard snapshot memory growth

Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>

---------

Co-authored-by: gratenes <claude@storage.gay>
Co-authored-by: Wing <44917547+wingleeio@users.noreply.github.com>
Co-authored-by: zeronsh[bot] <346184242+zeronsh[bot]@users.noreply.github.com>
2026-10-31 11:55:18 -07:00
Wing
1b3e335c36
Edge: profile routes (#1115) 2026-10-30 16:42:51 -07:00
wing-anara
2c9e306edf sync: empty checkpoint frontier is NOT contained — fetch, never skip history
Root cause of the 'Add Tweets' invisible-transcript incident (2026-08-18),
diagnosed from room stats + phone log archives + the absent on-device doc:
the room's checkpoint (seq 5, rows 1-5 trimmed) carries an empty frontier
label, and both clients treated empty-frontier as 'already contained' — so
every FRESH reader skipped the checkpoint, imported rows 6..75 raw, and
Loro parked everything depending on the chat's founding ops: transcript
invisible past the reader's own-lineage rows, no import errors (parking
is not an error), and snapshot export fails with pending ops so the doc
never persists. Converged devices (synced from row 1 pre-trim) unaffected,
which is why the bug only shows on new devices/fresh installs.

- Both clients: drop the empty-frontier shortcut; empty fails the decode
  path → not contained → fetch. Fetching is always safe (full-state merge;
  an empty-doc seed applies as a no-op) — the shortcut contradicted the
  adjacent comment's own principle ('never silently skips history').
  Applying the poisoned room's real checkpoint bytes heals it client-side
  with no server repair.
- Edge: reject checkpoint POSTs with an empty frontier when seqCovered>0
  (empty stays legal for the M1 empty-doc seed) so no new rooms are
  poisoned; chat2-check codifies the guard (66 checks green).
- Regression tests: EngineChatSink::contains_frontier(&[]) must be false;
  plan-level decision-table entry for the fresh-reader fetch.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-17 17:40:35 +03:00
wing-anara
22b5a67dc6 sync: pull-first HTTPS transport — 1-RTT bootstrap, WS demoted to enhancement
The airplane-wifi fix. The WS needs 4+ serial round trips (DNS, TCP, TLS,
upgrade, hello) before byte one, and hostile middleboxes strip the upgrade
entirely — the user-reported 'never connects on the plane'. A plain HTTPS
GET is 1 RTT on a warm connection and indistinguishable from a webpage to
any proxy. So: push/pull/poke.

Edge (worker + DOs):
- GET /registry/{org}/rows?since= — the WS hello's exact delta answer
  (full/gcFloor rules shared with handleHello); ?beat=1 doubles as a
  presence beat. POST /registry/{org}/push — the WS push twin via a shared
  applyPushBatch core (validate, atomic apply, rows broadcast to live
  sockets); LWW clocks make replays no-ops.
- GET /chat2/{id}/rows?after= — one request collapsing connect→hello→
  state→rowsReq→backfill: u32-LE length-prefixed frames (state with
  frontier payload, rows, rowsDone), byte-identical encoding to the WS.
  POST /chat2/{id}/rows?batchId= — push twin, batchId-deduped.

Desktop (Rust): RegistryTransport/ChatTransport seams; with a transport,
construction resolves immediately (local-first), an HTTP pull bootstraps
in ~1 RTT while the socket dials, and every backoff cycle pushes pending +
pulls over HTTPS — a WS-hostile network degrades to ≤30s polling instead
of silence. Engine wires both via the existing bearer/reqwest plumbing
(registry: URL derived from the WS provider; chat: EdgeChatTransport).

iOS (Swift): WorkspaceStore pull-first bootstrap + 20s poll-while-down +
HTTP flush of pending writes; the header spinner now means 'server state
reached this session' (flips on the ~1-RTT pull, not the 4-RTT socket).
ChatRoomClient.pullSync: HTTP push of pending sends (permanent rejections
retire, mirroring the error-frame path), framed rows pull applied through
the exact socket path, checkpoint fetch shared via fetchInFlight.

Validation: new edge/scripts/pullpush-check.mjs green against wrangler dev
(push/ack/no-op replay, delta-at-cursor, presence beat, framed pull,
excludeOwn, batchId dedupe); existing smoke + chat2-check (65) + reset
suites green; zeron-sync 34, engine+rpc 180; iOS sim build clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-17 16:22:26 +03:00
wing-anara
6994ba4fd6 uploads: remove the write-only R2 attachment mirror
Attachments already travel device-to-device (UploadChunk/UploadCommit to
the chat's host, ReadAttachmentChunk for read-back); the R2 copy was
never read — the sha was discarded after the PUT, so the objects were
unaddressable. Drop the durable mirror outbox + drain worker from the
engine and the /attachments/:sha256 routes from the edge.

The edge keeps a 200 stub for PUT /attachments/* so deployed clients
(≤0.1.62) drain their outboxes once instead of retrying a 404 forever.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-14 17:19:08 +00:00
wing-anara
b0be3dea2f rename: comet -> zeron everywhere (binary, crates, env, data dir, iOS)
- binary/CLI `comet` -> `zeron`; crates comet-* -> zeron-*; COMET_* -> ZERON_*
- data dir ~/.comet-native -> ~/.zeron, with a one-shot rename migration
- services: comet-native.service -> zeron.service; launchd sh.zeron.comet ->
  sh.zeron.app; macOS bundle id dev.comet.native -> sh.zeron.app
- release artifacts comet-<ver>-* -> zeron-<ver>-*; drop the Comet.app
  tarball-path compat hack in package-macos.sh
- iOS: Comet -> Zeron targets/dirs, bundle id sh.zeron.Zeron, zeron:// scheme
- clients default to https://edge.zeron.sh; installer + landing on zeron.sh
- kept on purpose: repo name, worker names (comet-native-edge/comet-landing;
  DO storage is bound to the script name), R2 bucket names, and legacy
  comet.zeron.sh routes so 0.1.x clients keep working

zeron-<ver>-* artifacts don't exist until the next release, so curl-install
and landing downloads 404 between this deploy and the 0.2.0 tag.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-14 10:49:19 +00:00
wing-anara
c359862f1e chat2 test harnesses: live E2E, cross-language crosscheck, whale rebuild probe
Artifacts from the verification pass, for the C3 session and CI-someday:

- edge/scripts/chat2-check.mjs: 65-check wire-level E2E against a deployed
  worker (protocol, guards, checkpoint lifecycle, quota, sidecars, blob-route
  isolation + '#' encoding, reset).
- edge/scripts/chat2-crosscheck.mjs + crates/sync/examples/chat2_live.rs:
  real ChatClient (tungstenite + actor) vs a JS Loro peer through a live
  ChatRoom — checkpoint-then-rows leg, live push/ack, byte-level convergence.
- edge/scripts/chat2-persist-check.mjs: seed/verify pair proving rows,
  checkpoint, frontier meta and sidecars survive DO eviction (redeploy).
- crates/doc/examples/rebuild_whale.rs: M1 rebuild against a real snapshot
  file; the 1b65e93d whale: 1,079,986 -> 152,934 bytes (7.1x), 384 sidecar
  payloads, idempotent, 8ms.
- edge/wrangler.chat2test.jsonc: throwaway staging config (dev auth, isolated
  bucket, workers.dev) used to run all of the above; deploy, test, delete.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-10 03:44:26 +00:00
wing-anara
ca8a6ba656 edge tests: split unit/workerd tiers via vitest-pool-workers
Two tiers: pure-logic unit tests stay on Node+FakeSql; a new workerd tier
runs against a real SQLite-backed DO so platform limits (the ~2MB
SQLITE_TOOBIG row cap behind the 2026-08-05 whale sync freeze) are the
runtime's own. Extends whale-check with a fold-backpressure phase and the
unit tier with a real-LoroDoc fold-cycle test over chunked rows.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 22:09:16 +00:00
wing-anara
6cee4af40a Whale-session sync fix: chunked update-log rows + shallow-aware join backfill
Two DO-side bugs that only extremely long sessions trigger, compounding
into "newest messages no longer sync":

- A single update above the ~2MB SQL row cap (bulk import, full re-upload
  span) imported, acked Ok, and relayed — then every flush() threw
  SQLITE_TOOBIG forever: nothing persisted, and each hibernation reverted
  the room to stale state. Update-log rows now chunk across continuation
  rows (update-log.ts, same CHUNK_BYTES discipline as blobs.ts), with a
  guarded ALTER TABLE migration; durability no longer depends on row caps
  or a whale snapshot export succeeding.

- handleJoin's backfill export({mode:"update", from}) on a SHALLOW doc
  does not throw for a client behind the trimmed root — it silently
  exports ~90 bytes of post-root ops that import as forever-pending:
  fresh readers of any force-trimmed whale room got an empty transcript
  with zero errors. Detect via isShallow() + from.compare(shallowSinceVV)
  and serve the §3.1 full-snapshot resync instead.

Silence-killers so this class can't hide again: the debounced flush now
catches + escalates (was an unhandled rejection — the only flush call
site with no handler above it), /stats wraps failures as JSON 500
stats_failed, and inbound fragment batches are capped at the header
(32MB / 4096 fragments → PayloadTooLarge, before any reassembly buffer).

Tests: update-log.test.ts (FakeSql enforcing the row cap; migration +
byte-identical reassembly), scripts/whale-check.mjs E2E regression (3MB
single-update push → durable, fresh-reader full transcript, /tail 200;
verified across a wrangler dev restart). smoke.mjs + fold-check.mjs pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-05 18:04:31 +00:00
wing-anara
88714bd966 Workspace room: retry initial join + an edge wedge-break for a bloated DO
Diagnosing why every device still showed offline after the fold fix
surfaced two more bugs that together wedged the per-user workspace room
permanently:

1. Engine never retried the FIRST join. crates/sync RoomClient only
   self-reconnects AFTER a successful join; an initial failure (a 500 from
   an overloaded workspace DO, a token racing a refresh, an edge deploy)
   ended workspace_host's join task with "staying offline" and the device
   stayed offline until an app restart — presence dead while the relay and
   per-chat rooms worked. Three engines restarting together reliably hit
   this: they hammered the cold DO at once, all got 500s, all gave up.
   Now join_room retries on a capped, jittered backoff (jitter so a
   simultaneous fleet restart doesn't resynchronize into a herd).

2. The DO was already wedged and no client-side retry could heal it. Its
   update log had grown huge (the pre-fix byte-only cap never tripped on
   tiny writes), so every cold `ensureDoc` replay exceeded the DO CPU
   limit and reset before any join finished — which also blocked the
   compaction that would have shrunk it. A solo join hung 50s+ and never
   landed. Add POST /workspace/:orgId/reset-log: it drops the persisted
   update log + snapshot WITHOUT calling ensureDoc (so it stays cheap
   enough to land on a wedged DO) and boots attached sockets. State is not
   lost — every engine re-uploads its full local workspace doc on the next
   join ("export everything the server lacks", room.rs:19), the same
   recovery as an ws-namespace bump but without client version skew.
   Verified against wrangler dev: 51 rows cleared to 0, empty room then
   accepts a fresh join + write.

The fold fix (row-count trigger) keeps a healthy room from re-bloating;
this pair recovers one that already did and stops a transient blip from
stranding a device.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-25 04:49:51 +03:00
wing-anara
05fdc9369b Workspace room: fold the update log on row count, not just bytes
The per-user workspace SessionRoom DO was hitting "Durable Object
exceeded its CPU time limit and was reset" on production. A cold
`ensureDoc` (after any hibernation or reset) replays the snapshot plus
every logged update, and each `doc.import()` crosses the WASM boundary —
so cold-start cost scales with the NUMBER of update rows, not just their
bytes. The workspace doc is high-churn (three devices writing device and
session rows continuously) but each write is tiny, so the 8 MiB byte cap
never tripped while the row count climbed into the thousands. Cold replay
eventually blew the CPU budget; the runtime reset the DO; every client
reconnected into another cold start; and the room's memory-only presence
store was wiped each time — which is why every device showed permanently
offline even though the device-room relay (a different DO) was healthy.

Fold on either budget: bytes bounds one huge update, a new
COMPACT_LOG_ROWS (1500) bounds many small ones. Also drop COMPACT_LOG_BYTES
8 MiB -> 2 MiB so the byte path keeps cold starts cheap too. The fold is
lossless (snapshot re-export + clear log); COUNT(*) is one indexed read
per 5s flush. Verified against wrangler dev: 1700 tiny commits peaked at
1401 rows, crossed 1500, folded back to 99 rows behind a 7 KB snapshot.

Also route GET /workspace/:orgId/stats through the Worker (org-membership
already checked) so the log/snapshot sizes are observable — how I'll
confirm the budget holds in production after this ships.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-25 04:29:41 +03:00
Wing
12357b5643 Per-user privacy: workspace docs, local storage scoped to user id
Teammates in one org must not see each other's spaces/sessions. Session
rooms (per-chat claim-on-first-join owner), device rooms (owner user),
and R2 attachments (att/{userId}/ prefix) were already user-scoped; the
org-wide workspace doc was the gap — it broadcast every member's spaces,
chat titles/previews/paths/branches, devices, and presence.

- Edge: workspace room is now ws3/{orgId}/{userId}, the user segment
  derived from the caller's OWN verified claim — another user's room
  cannot even be addressed. Org-claim 403 unchanged. Legacy org-wide
  ws2 rooms are orphaned (unreachable via routes) on deploy.
- Engine: Auth::user_id() (WorkOS sub; dev bearer `user@org` prefix,
  mirroring the edge's parsing), WorkspaceHost joins the per-user room,
  EngineCore::assemble_with_org -> assemble_with_identity(org, user).
- Local storage: orgs/{org}/{user}/ for snapshots, command ledger, and
  run journals — two users on one machine can't read each other's cache.
- Embedded (headed dev) engine derives identity from its dev token so
  the room join matches what the edge derives.
- Live smoke rewritten to assert the new semantics: one user's devices
  converge; a same-org teammate lands in an isolated empty doc. The
  live Rust convergence test now runs two devices of ONE user (two
  users converging is exactly what must no longer happen).

Destructive break (pre-production, per ws->ws2 precedent): fresh
workspace room + fresh local path on next boot.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 21:38:38 +00:00
Wing
56e19819e8 Mac feedback: scroll spring, traffic-light chrome, picker polish, own ports
Streaming: FollowMode::Tail + per-splice scroll_to_end hard-snapped growth
(one 56px jump/commit). Replaced with an owned pin + per-frame StickSpring
(mugen/use-stick-to-bottom constants: damping .7 stiffness .05 mass 1.25,
growth-EMA feed-forward, catch-up, teleport guard); splice-glue anchors the
offset so layout holds and the spring glides. Frame evidence: 2,13,13,10,6,
4,2,2,1,0 px/frame ramp vs a single 56px step. Doc->render path verified
O(changed) already.

Traffic lights: intended macOS inset was dead (.pl overwritten by .px).
TitlebarOptions (transparent, lights at 14,15, title None); animated 200ms
spacer 88px<->12px on fullscreen (per window-controls.tsx); all titlebar
strips are drag regions (start_window_move) w/ double-click zoom.

Pickers: double-highlight fixed (selected wash vs keyboard cursor wash),
search-input inset, w-72 widths, section separators, traits max-height,
skeleton sizes — all per harness-model-picker.tsx. Model description line
deferred (proto lacks the field).

Ports: comet-native defaults moved to 27654 (IPC) / 27640 (edge dev) so it
never conflicts with the original comet (26634/26638/26640/26654) running
on the same machine. 192 UI tests green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 16:08:52 +00:00
Wing
65618eb9d3 Cross-language compat gate: Rust-written doc reads in loro-crdt JS
gen_fixture example (streamed assistant entry + tool + command outcome) +
edge/scripts/compat-check.mjs — 17 shape assertions pass. Tool calls are
canonically kind-tagged (serde) in comet-native; edge runtime verified not to
depend on the old _tag shape.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 18:09:34 +00:00
Wing
240b008bd3 Foundation: architecture, research, Rust workspace scaffold, TS edge port
- ARCHITECTURE.md: full design — Loro docs (session + per-org workspace) through
  Durable Objects, TS edge / Rust everything-else split, gpui UI plan with
  mugen/pretext techniques, headed+headless single binary, milestones M0-M6
- docs/research/: feature-parity inventory, gpui guide, mugen/pretext techniques,
  loro Rust ecosystem, DO-language decision (TS, evidence-backed), harness
  integration (claude stream-json / codex app-server)
- Cargo workspace (proto, doc, sync, harness, engine, rpc, ui + comet binary):
  builds green incl. gpui pinned to zed@f14fea9b; comet-proto wire types and
  comet-doc command-ledger rules 1-3, event->parts fold, render-parts privacy
  policy, continuation splitting all ported with tests (15 passing)
- edge/: comet's Worker + DOs ported standalone (session room w/ hibernation +
  two-level compaction + alarms + R2 backups, device room relay, attachments),
  vendored session-doc modules, NEW workspace rooms (ws/{orgId}, org-claim authz),
  absorbed WorkOS auth routes; tsc clean, wrangler dry-run bundles

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 18:04:51 +00:00