0
Fork 0
mirror of https://github.com/zeronsh/comet.git synced 2026-11-16 18:57:19 +00:00
comet/docs/performance-workspace-search.md
Wing 1bc74d5dad
Projectless chats get their own folder; home is never indexed by fff (#876)
* Projectless chats get their own folder; home is never indexed by fff

Workspace search shipped in 0.2.111 indexed the home folder for
projectless chats with fff, whose post-scan opens every file up to 10 MB
to sniff binaries. On Windows that downloaded every OneDrive cloud-only
file (iCloud Drive / File Provider folders on macOS likewise).

- New projectless chats run in their own folder,
  `<data dir>/scratch/<date>-<id>` (`~/.zeron/scratch/…` by default), as
  t3code does under `~/.t3/scratch`. The cwd is stored home-relative
  (`zeron_proto::projectless`) and the host creates the folder on first
  run, terminal or file browse. The engine, desktop composer, client
  library (iOS) and MCP all mint it; runs from older clients that send
  `~` follow the chat's row. Hosts advertise `projectless-scratch-v1`;
  older hosts keep getting `~`.
- Chats from before keep `~`, so their harness sessions still resume.
  Home gets a names-only walk instead of fff (`home_index.rs`): hidden,
  dependency and machine folders, cloud roots (OneDrive*, iCloud Drive,
  Dropbox, Google Drive, Box, ~/Library, AppData) and cloud placeholder
  folders are skipped; no content search there. The new-project folder
  search uses the same index.
- Projectless chats no longer show `~` or a project name: sidebar and
  palette rows name the device, groups and the icon tooltip read
  "No project", the files header reads "Files · device".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Composer stays centered on the transcript with the terminal open

With the session card open, the composer is shifted left to stay
centered on the transcript. Reserving the bottom terminal re-roots the
composer's layout behind a centered slot, which dropped that style
inset, so the composer jumped back to the window's center. The shift
now lives on `DockedComposer` and is applied where it positions the
child, holding on both paths.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 13:21:03 -07:00

8.9 KiB

Workspace search on fff

One engine-side index, fff-search =0.11.0, now serves every workspace search surface:

Surface RPC Before Now
Composer @ SearchFiles nucleo-matcher over a walk cached for 10 s fff names (files and folders)
File tree search SearchWorkspaceFiles a full disk walk per query fff names; "show ignored" still walks
cmd+K Files SearchWorkspaceFiles — fff names, focused chat's folder
cmd+K In Files SearchWorkspaceContent (new) — fff plain-text grep, smart case

WarmWorkspaceSearch (new) lets the UI start an index before the first query. Both new methods are forwardable, so remote chats search on their host. Older hosts answer UnknownMethod: warm-ups are skipped silently and the palette's In Files section explains that the device needs an update.

The code lives in crates/engine/src/workspace_search.rs (WorkspaceSearch, owned by Repos). SearchFiles and SearchWorkspaceFiles keep their wire shapes; iOS's @ uses SearchFiles and only sees the new ranking.

Index lifecycle

One fff file picker per working folder (repository, worktree or plain folder), keyed by the canonical root that WorkspaceFiles::resolve_target returns:

  • Created by the first warm-up or search of that root. It scans in the background and keeps itself current with its own file watcher.
  • A search waits up to 300 ms (SCAN_WAIT) for the initial scan, then answers from whatever is indexed; content answers carry indexing: true then.
  • Dropped after 15 minutes without a search or warm-up (IDLE_EVICT; a reaper thread checks every 60 s, so up to 16 minutes in practice).
  • At most 7 live indexes (MAX_LIVE); creating an eighth evicts the least recently used. Each index runs its own file watcher, so the cap also bounds inotify instances.
  • The UI warms the focused chat's index once, when the chat gains focus. There is no heartbeat or pinning: a focused chat left without searches for 15 minutes loses its index and the next search (or refocus) rebuilds it.
  • An index a chat warmed survives release_unless_warmed, which the new-project folder picker calls on the home index when it closes.
  • Each index caps its cached file contents at 16 MB (CACHE_BUDGET_BYTES); fff would otherwise size it up to 512 MB.
  • Eviction stops the scan and watcher, drops the index, and asks glibc to return the freed memory (malloc_trim) on Linux.

Fixed options: no content (bigram) index — see "Paths-only index" below —, mmap warm-up off, no frecency database (SharedDb::noop), git recency on, symlinks not followed. fff's tracing and panic hooks are not installed.

Home folder

The host's home is never given to fff. After a scan, fff opens every file up to 10 MB to tell text from binary, so indexing home downloaded every cloud-only file under it (OneDrive on Windows, iCloud Drive and File Provider folders on macOS; reported against 0.2.111). Home, which chats from before scratch folders and the new-project folder search still use, gets a names-only walk instead (crates/engine/src/home_index.rs):

  • It reads directory listings only and skips hidden folders, .git, dependency folders (node_modules, venv, …), cloud folders directly under home (OneDrive*, iCloudDrive, Dropbox, Google Drive, Box, …), Library, AppData and Applications, and any folder a cloud client still has to fetch (Windows recall/offline attributes, macOS SF_DATALESS).
  • At most 250,000 paths, ranked by case-insensitive substring: exact name, name prefix, name, then path; shallower paths first.
  • No watcher: an index older than 60 s is rewalked in the background while searches keep answering from the old one. Idle eviction and release_unless_warmed work as for fff indexes.
  • No content search: SearchWorkspaceContent on home answers no matches.

New projectless chats don't run in home at all: each gets its own folder, ~/.zeron/scratch/<date>-<id> (zeron_proto::projectless), created by the host when it first runs or browses the chat.

Measurements

crates/engine/examples/workspace_search_bench.rs, release build, Linux (Fedora 44, x86_64), 10 timed runs per query after a warm-up; medians. "Before" is the same benchmark at commit 0c6b94ed, before the port. comet is this repository (≈2.7k tracked files, 3.3k entries walked, 139k with ignored ones); t3code is a ≈27.7k-file TypeScript monorepo.

Median per query comet before comet after t3code before t3code after
SearchFiles (@, 8 results) 0.13–0.58 ms¹ 0.70–1.57 ms 1.0–31 ms¹ 1.0–3.6 ms
SearchWorkspaceFiles (tree, ≤200) 27.7–30.1 ms 0.70–2.6 ms 209–409 ms 1.8–4.4 ms
…including ignored (walk) 310–781 ms 237–248 ms 105–320 ms 91–109 ms
SearchWorkspaceContent (100, 3 per file) — 0.17–0.43 ms — 0.72–1.99 ms
Index build (scan + content index) 8–11 ms walk² 51–61 ms 29 ms walk² 625–685 ms

¹ The old @ index was rebuilt by a full walk whenever it was older than 10 s; the benchmark's runs all hit the cache. ² The old walk had no content index.

Queries: composer, repos, cmdpal, Cargo.toml, workspace_files.

Process RSS comet t3code
Baseline (no index) 6.1 MB 6.0 MB
Index built 25.5 MB (+19.4) 90.5 MB (+84.5)
After every query above 39.3 MB 123.0 MB
2 s after evicting the index 16.6 MB (+10.5) 56.7 MB (+50.7)

The residual after eviction is allocator state, not a live index: repeated create/evict cycles of comet's index plateau at about +20 MB over the first baseline after four cycles and stop growing (glibc keeps per-thread arenas for fff's search and scan threads). Before, the old @ cache cost 3.6 MB for comet and 12 MB for t3code, but every tree query re-walked the disk.

Against the plan's targets: tree and @ p50 ≤ 2 ms in comet — met (0.70–2.6 ms; the tree's Cargo.toml query, 200 results, is the slowest). Content p50 ≤ 50 ms — met by two orders of magnitude. ≤ 20 MB per index in comet — met for the index itself (+19.4 MB); query-time allocations add more until eviction. Back within ±10 MB of baseline after eviction — not met on Linux (+10.5 MB after one cycle, plateauing near +20 MB).

Paths-only index

The tables above were taken with fff's content index on. It is now off: indexes hold paths only and SearchWorkspaceContent greps the indexed files without a bigram prefilter. Re-measured the same way:

comet, content index comet, paths only t3code, content index t3code, paths only
Index ready 51–61 ms 41 ms 625–685 ms 303 ms
RSS with the index 25.5 MB 19.5 MB 90.5 MB 76.7 MB
RSS after every query 39.3 MB 43.8 MB 123.0 MB 91.7 MB
SearchWorkspaceContent median 0.17–0.43 ms 0.48–2.3 ms 0.72–1.99 ms 1.1–21 ms

Name searches are unchanged. Content queries with many hits stay near 1 ms; rare or absent strings scan every candidate (~20 ms on t3code's 27.7k files, page cache warm). A cold disk makes the first such query slower; that was not measured. A full rescan of t3code drops from ~1.7 s to ~1.05 s of CPU, and build CPU from ~2.0 s to ~0.9 s.

Behavior changes

  • Ranking follows fff: typo-tolerant fuzzy matching with filename, git-recency and modification-time boosts. The top results are better; a long tail of weak matches fills the tree's 200 slots where the old substring ranking returned fewer rows.
  • An empty @ query lists the chat's featured paths first, then recently changed files (it used to list the shallowest paths).
  • Dotfiles are indexed inside git repositories only; in a plain folder fff skips them. .git is never indexed.
  • Symbolic links are not indexed (fff does not follow them), so name and content searches skip them; the "show ignored" walk still lists them.
  • The tree's "show ignored" mode keeps a direct walk (fff has no include-ignored mode), now ranked by plain case-insensitive substring: name prefix, then name, then path matches, shorter paths first.
  • Binary files and files over 10 MB are never content-searched.

Cost

  • Dependencies: 53 new lockfile entries (most are Windows target shims), including C builds of libgit2-sys, libz-sys and lmdb-master-sys (a C compiler is required on every platform) and notify 9.0.0-rc alongside our notify 7. nucleo-matcher is gone.
  • Threads: fff's global search and background pools, plus one watcher per live index (at most seven).

Reproduce

BENCH_ROOT=/path/to/checkout cargo run --release -p zeron-engine \
    --example workspace_search_bench

BENCH_QUERIES=a,b,c changes the queries and BENCH_RUNS the timed runs. Index creation and eviction are logged at info (workspace search index created / scanned / evicted, with root, file count, scan time and arena bytes).