0
Fork 0
mirror of https://github.com/zeronsh/comet.git synced 2026-11-16 18:57:19 +00:00
comet/docs/transport-reliability.md

9.3 KiB

Transport reliability: measured Cloudflare results

What changed

High round-trip latency exposed unnecessary retransmission: every new edit sent all previously unacknowledged chat batches again. The actor now tracks batches in flight per WebSocket session. New edits send only new batches; reconnects still replay the durable SQLite outbox with the same deduplication IDs. Explicit quota retries retain their existing head-of-queue policy.

The shared Rust socket pump drives reads and writes independently. Its watchdogs measure byte inactivity, including incomplete incoming WebSocket frames, instead of imposing an absolute duration on an otherwise progressing transfer. A full consumer queue remains bounded. Closing the consumer cancels both halves.

Messages larger than 16 KiB use 4 KiB WebSocket fragments with empty control pings between fragments. Cloudflare answers those pings before the application message is complete. This matters when the whole message fits in the local TCP buffer: local writes can finish immediately while the remote uplink takes over a minute. The application still receives one ordered message, with identical bytes. Small messages and idle heartbeat cadence are unchanged. Large-message fragmentation adds about 0.4% framing/control traffic; it is not a compression feature.

The document heartbeat remains 15 seconds with a 45-second inactivity lease; relay transport retains its 10-second heartbeat and 25-second lease. The dialer bounds DNS, TCP racing, TLS and upgrade together to 20 seconds. An interrupted send closes the session because resuming a partial frame is unsafe. Documents recover with their batch IDs and cursors; uncertain RPC mutations are not replayed.

Live comparison against main

On 2026-09-15, the production Rust ChatClient, Loro document, SQLite outbox and actual Cloudflare Worker/SQLite Durable Object handled synthetic test rooms. The worker used unchanged edge source from main 67c960f4, isolated bindings, dev authentication and no production routes. Worker version: c2c7587e-b5f7-49ae-aefe-d9b54fc95039.

A local proxy established verified TLS to Cloudflare and limited aggregate traffic across both clients to 2,048 bytes/second per direction, with 600 ms added one-way delay. Each run queued 20 edits at 100 ms intervals: 4,900 application text bytes / 6,756 Loro update bytes. Every run used a fresh room and proxy. Main and fixed versions ran alternately, three times each.

Metric Main, median (range) Fixed, median (range)
Successful push attempts for 20 unique batches 30 (30–61) 20 (20–20)
Outbound HTTP/WebSocket bytes 12,648 (12,648–24,338) 8,912 (8,912–8,912)
Inbound HTTP/WebSocket bytes 16,373 (15,542–16,397) 11,922 (11,920–13,526)
Completion including final stats verification 9.84 s (9.29–13.87) 7.28 s (7.25–7.62)
Delivered rows / pending batches 20 / 0, every run 20 / 0, every run
Sender and receiver disconnects 0, every run 0, every run

Median outbound bytes fell 29.5%, inbound bytes 27.2%, and completion time 26.0%. An earlier single comparison showed larger gains; the table deliberately uses the repeated experiment. Three runs establish this workload's behavior, not a fleet-wide performance distribution. These are decrypted HTTP/WebSocket counts, not physical network bytes; TLS/TCP/IP overhead is excluded.

Profile Actual result
1 KiB/s each direction, 1.2 s added one-way delay 20/20 rows, 20 pushes, pending=0, no disconnects; completion 19.14 s
30-second blackout, active connections reset, then 2 KiB/s + 600 ms restored 20/20 rows, pending=0, 20 pushes; completion 41.19 s including outage
Every WebSocket upgrade rejected; first successful HTTP POST response discarded 20/20 unique rows, 21 pushes, pending=0; completion 48.66 s; 18 upgrades blocked, 1 ACK lost
128 KiB fresh-reader catchup, 2 KiB/s + 600 ms, WebSocket only Main fails with chat2 handshake failed; fixed completes in 70.22 s, cursor=1, zero disconnects
128 KiB upload and delivery to a second client, 2 KiB/s + 600 ms, WebSocket only Exact document equality, pending=0, both clients stay connected; 139.80 s including setup

The catchup update is 131,167 bytes, seeded directly before starting the impaired reader. The upload sends that update through the impaired uplink. Neither large transfer uses HTTP fallback. The HTTP fault is injected after Cloudflare returns success, proving accepted-but-unacknowledged replay against actual server dedupe. The live test uses persistent SQLite storage but does not restart the process; a separate regression reopens the SQLite outbox before replay.

Raw run outputs and metadata are committed beside this report. The proxy forces HTTP connection close, so its connection counter is not a production connection-pooling benchmark. Arrival-percentile recording was refined during testing; p95 fields in raw output are not used for comparisons.

Regression evidence

85 Rust tests pass; 3 pre-existing live-edge tests remain ignored. The live matrix above is a separate executable. Clippy completes with existing warnings in document/sync code. The isolated worker configuration also passes Wrangler's deployment dry run.

  • A locally buffered 132,000-byte UTF-8 message traverses a remote 2 KiB/s bottleneck; control pongs arrive before message completion, and reassembly preserves characters crossing fragment boundaries.
  • A 128 KiB frame downloading at 2 KiB/s fails the original pump at 45 seconds; the new pump completes at 65 seconds under the same virtual byte transport.
  • Healthy idle traffic over 601 virtual seconds remains 400 outbound / 240 inbound WebSocket bytes, one connection, in both versions. No extra idle traffic was introduced.
  • Twenty edits with every acknowledgment withheld produce exactly 20 pushes.
  • Blocked writes do not suppress inbound ACKs or cancellation. Inbound queue stalls close the session. Incoming pongs cannot keep a blocked writer alive; locally buffered unanswered echo/ping traffic cannot keep a dead peer alive.
  • An accepted TCP socket that never answers its upgrade times out within 20 s.
  • Existing tests cover lost-ACK replay, cursor repair, quota retry, persistence, relay supersede, credential rotation and zombie-host recovery.

The original negative controls are reproducible at 94cc6753. Comparative byte and idle tests also carry the original pump as a control. The initial PR's absolute write timeout was insufficient: the slow-transfer tests exposed that regression, and the final implementation uses byte progress and fragmentation.

Reproduce

cargo test -p zeron-sync -p zeron-rpc --features zeron-sync/mock-server
cargo test -p zeron-sync --lib socket:: -- --nocapture
cargo clippy -p zeron-sync -p zeron-rpc --all-targets --features zeron-sync/mock-server
cargo build -p zeron-sync --example transport_live

# Requires authenticated Wrangler and an account with Workers/R2 enabled.
cd edge
npm ci
node_modules/.bin/wrangler r2 bucket create zeron-transport-385-20260915
node_modules/.bin/wrangler deploy --config wrangler.transport-test.jsonc
cd ..

# Use the isolated workers.dev URL printed by deployment.
python3 scripts/run-transport-matrix.py \
  --binary target/debug/examples/transport_live \
  --origin https://zeron-transport-385-20260915.YOUR-SUBDOMAIN.workers.dev \
  --profile stream --output /tmp/transport-stream.json
# Repeat with: very-slow, outage, http, catchup, upload.

For the baseline, create a detached worktree at 67c960f4, copy only crates/sync/examples/transport_live.rs into its examples directory, and build that same example there. Run the same proxy and streaming/catchup profiles with its binary. The matrix driver owns and stops each proxy. Large transfers allow up to 360 seconds for completion; the driver has a 420-second process limit.

After testing, delete only the isolated resources:

cd edge
node_modules/.bin/wrangler delete --config wrangler.transport-test.jsonc
node_modules/.bin/wrangler r2 bucket delete zeron-transport-385-20260915

Scope and remaining limits

These measurements demonstrate lighter, successful document synchronization on the stated impaired links. They do not prove a globally most reliable transport or that every feature works on every aircraft network.

  • Generic remote RPC/uploads still require WebSocket; preview uses WebRTC with STUN and lacks an HTTPS byte relay. Document HTTP fallback already exists.
  • Relay end-to-end echo and room-probe deadlines are separate from transport liveness. Large RPC transfers and congested host legs need their own live measurements; document results cannot establish those guarantees.
  • Swift/iOS lifecycle, sleep/wake, IP changes, authentication expiry, captive portals and actual packet loss/retransmission were not exercised here.
  • Checkpoint Range resume still needs a strong content validator when content changes at the same sequence. HTTP retry ownership and stalled RPC consumer isolation remain separate issues.
  • A proxy shaping decrypted bytes is not a physical airline network. Live requests here use dev auth and a test HTTP adapter with a 180-second timeout; production authentication and engine HTTP timeouts need separate coverage.