0
Fork 0
mirror of https://github.com/zeronsh/comet.git synced 2026-11-16 18:57:19 +00:00
comet/docs/reference/desktop-dictation.md
Wing 63774eef24 zgx: dictation as a feature of the app, with Settings → Voice
Dictation was a built-in plugin in name only: its model, microphone and
protected insert all lived in the app, and the plugin needed a new engine
permission. Make it what it is:

- src/dictation.rs talks to zeron_voice directly. No plugin lease, backend
  trait or catalog, and no engine, MCP or plugin-system changes.
- ds.Composer takes one `dictation` prop instead of twelve, and sends on its
  own, so lane_tab's send no longer has to move.
- Settings → Voice: turning dictation on downloads the model, then the
  microphone, shortcut and Remove, as the shipping app's page has them.
- The shortcut is in Settings → Shortcuts' table.

Fixes along the way: Send while transcribing never sent (retiring the words
ran the effect again, which stopped its timer); the waveform stayed flat (its
level was never passed), so it breathes from time again and nothing publishes
a level every 40 ms; and an unknown data folder panicked the window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 03:36:57 -07:00

7.7 KiB

Desktop dictation

Settings → Voice downloads NVIDIA Parakeet TDT 0.6B v3 for this device and enables the composer microphone. The optional INT8 ONNX download is 670,479,942 bytes, derived from the pinned manifest, and is independent of the chat's engine host. Turn off hides dictation without deleting the model; Remove model disables it and removes the local cache.

Dictation is hold to talk: hold the microphone (or Enter/Space while it is focused) or the Hold to dictate shortcut, rebindable in Settings → Voice (default Cmd/Ctrl+D), and release to transcribe. Releasing outside the microphone still counts; for the shortcut, letting go of its key or any of its modifiers ends the hold, since macOS does not deliver key-up for Command chords. A release within 300 ms is treated as a tap and explains hold to talk instead of transcribing. Assistive-technology Click cannot hold, so it starts and then finishes. Recording is limited to one minute. Release transcribes locally into the selected draft range. It never invokes an agent or submits on its own. Explicit Send waits for finalization, then follows the normal Send/Queue path once. If no speech is recognized, the composer explains this and preserves the draft without sending. Esc cancels; cancelling or a failed/timed-out finalization never sends. Microphone permission is requested on first use; if the permission dialog takes focus, retry after granting permission.

The model detects the spoken language automatically and does not translate. Supported languages: Bulgarian, Croatian, Czech, Danish, Dutch, English, Estonian, Finnish, French, German, Greek, Hungarian, Italian, Latvian, Lithuanian, Maltese, Polish, Portuguese, Romanian, Russian, Slovak, Slovenian, Spanish, Swedish, Ukrainian. Technical names may need correction, especially in mixed-language speech.

No audio files are created by production dictation. Audio stays in a bounded, transient buffer on the desktop running the UI; it is not attached, logged, synchronized or sent to the coding-agent host. Dictated text is an ordinary editable draft and is handled like typed text. Model downloads contact Hugging Face; transcription does not use the network.

Implementation and limits

zeron-voice owns downloads, model integrity, CPAL capture, anti-aliased conversion of device-rate audio to 16 kHz, a capture thread, and a single background Parakeet worker. zeron-ui::dictation owns permissions, local settings and the small transcription event interface. Composer inputs own draft-range/generation protection and grouped undo. A manual edit, selection move, IME update, focus leaving the composer, chat switch or queue-draft replacement invalidates the session. Attachments retain their existing behavior.

Capture starts as soon as permission and the audio device allow, independently of background model verification/loading. “Listening” is emitted only after the microphone stream starts; there is no hover prewarming. Stop closes capture and retains the bounded audio buffer until the model is ready, within the existing finalization deadline. A load failure closes capture before reporting failure. The production adapter performs offline final transcription on Stop. No live partials or native streaming are advertised. Editor partial-result behavior is tested through a deterministic adapter. Audio is capped at 60 seconds; UI finalization has a 30-second deadline. An in-flight native ONNX call cannot be forcibly interrupted safely: cancellation signals capture to stop independently of model loading (the capture loop checks every 10 ms) and invalidates results; one worker keeps native work bounded and rejects overlapping sessions until it returns. Cached weights unload after 30 idle seconds or model removal. Settings refuses removal while capture, loading, or inference is still active; after cancelling uninterruptible native work, retry removal once it returns.

Pinned runtime: parakeet-rs = 0.3.8, ort/ort-sys = 2.0.0-rc.13, ONNX Runtime 1.28.0, CPU execution; cpal = 0.17.3, rubato = 0.16.2. The download manifest and conversion attribution are in crates/voice/model.json and crates/voice/NOTICE.md. The conversion pins immutable bytes but its publisher does not provide the source-weight revision or exact INT8 conversion tool version; this provenance limitation is explicit.

Release targets are macOS arm64, Linux x86_64/aarch64 and Windows x86_64. Linux CI installs ALSA headers. macOS uses AVFoundation only for permission, and its bundle includes microphone usage text and the hardened-runtime audio-input entitlement. The selected ONNX binary distribution has no Intel macOS build; Intel development requires an independently built matching ONNX Runtime. Windows' prebuilt ONNX archive links the system DirectML/DX12 libraries even though the chosen execution provider is CPU. Real-model inference, desktop UI, packaging and physical-microphone validation are separate checks; the PR records the platform evidence. iOS does not depend on this crate.

On Windows, microphone access for desktop apps must be enabled in the system privacy settings. On Linux, the selected ALSA input must be available to the user and the audio server. Settings → Voice lists available inputs and refreshes while visible; a disconnected selection falls back to the system default. Test built-in, USB and Bluetooth microphones on the target OS: a synthetic WAV test cannot establish permission, device routing or capture behavior.

Linux releases dynamically link the system ALSA runtime (libasound.so.2). It is required even with dictation disabled or in headless mode; no audio device or microphone permission is needed to run the engine. Both Linux installers check executable startup before activating an installation. The updater also checks a staged binary before replacing the active version.

Verification

Deterministic editor fixtures cover partial replacement, Unicode, undo/redo, failure, stale events, cancellation, Send finalization, queue editing, navigation, focus and input-request takeover. cargo test -p zeron-voice covers pinned manifest shape/size, cancellation, corrupt cache rejection, and deterministic capture/load barriers for Stop, cancellation, failure, device startup, recording limits, and busy admission. The verify example runs explicitly supplied synthetic/public PCM16 WAV files and prints transcripts/timings for development only; production never prints transcripts.

For native startup timing, launch a development bundle with RUST_LOG=info,zeron_ui::dictation=debug. The activation_to_listening_ms field measures from composer dictation activation to the UI entering Listening, including permission waiting and the 40 ms event polling cadence. It contains no audio or transcript; display presentation may follow on the next frame. Measure cold and cached model runs separately.

Real-model verification is an explicit development check using the verify example and external synthetic/public WAV files. Routine CI does not download the model or retain speech fixtures in the repository.

Future mobile implementation is scoped in the PR #591 handoff, including native capture, lifecycle cancellation, mobile inference benchmarks, UTF-16/UTF-8 selection handling and preservation of iOS delivery modes.

See the task evidence under /Users/gaelcado/zeron/evidence/voice/ for actual runtime transcripts, hardware measurements and native build captures. Standalone synthetic model accuracy is distinct from a live microphone test.

ZGX

ZGX has the same dictation as a feature of the app (apps/zgx/src/dictation.rs, ui/screens/chat/dictation.luau, Settings → Voice), with the same zeron-voice model cache. It runs in the window: the engine has no part in it.