0
Fork 0
mirror of https://github.com/zeronsh/comet.git synced 2026-11-16 18:57:19 +00:00
comet/crates/voice/examples
Gaël Cado 80b946b153
Add opt-in on-device dictation to the desktop composer (#591)
* feat(voice): add local Parakeet v3 capture and transcription backend

* feat(ui): add opt-in dictation settings and composer controls

* build(macos): package microphone permissions and voice attribution

* docs: describe desktop dictation behavior and runtime limits

* fix(ui): clarify dictation states and cancellation controls

* fix(ui): keep composer model controls right aligned

* feat(ui): give Voice a dedicated settings page

* fix(voice): support all native microphone sample formats

* fix(voice): retain model cleanup after interrupted downloads

* fix(ui): preserve dictation while pasted references resolve

* fix(ui): cancel dictation with Escape across composer controls

* fix(ui): refresh dictation state after empty undo and redo

* fix(ui): preserve dictated words on blank recognition results

* fix(voice): clarify retries while native inference finishes

* test(ui): verify repeated Send preserves unfinished dictation

* style(ui): format settings modules

* feat(voice): add microphone selection and streamline Voice settings

* feat(ui): visualize live dictation with a waveform and recording clock

* fix(ui): preserve waveform paint opacity

* feat(ui): morph composer controls into dictation track

* test(voice): exercise real inference on every desktop release target

* fix(voice): keep device choices fresh and cancellation accessible

* ci(voice): verify Linux release packages on x86 and ARM

* ci(voice): keep model and package probes out of permanent CI

* fix(voice): detect missing ALSA before activating Linux installs

* fix(voice): capture audio while Parakeet loads

Run capture independently of serialized model loading and inference. Retain stopped audio until the model is ready, close capture before reporting load failure, and hold admission through native work and stream teardown.

Cover loading, cancellation, device startup, rapid Stop, admission, and recording limits with deterministic coordinator tests.

* test(ui): verify and measure nonblocking dictation startup

Cover selected-draft preservation, delayed final results, single Send delivery, and late results after loading failures or finalization timeouts.

Add transcript-free debug timing from dictation activation to the Listening state and clarify asynchronous capture cancellation.

* docs(voice): explain concurrent capture and startup timing

Document audio retention during loading, cancellation and model-removal limits, race coverage, and the native startup timing procedure.

* fix(ui): stop Shell saves from reverting settings chosen elsewhere

The Shell republished its whole cached UiSettings on every save
(navigation, pane resize) and only refreshed a hand-kept list of fields
first. Anything missing from that list was silently reverted: leaving
Settings → Voice turned dictation off, and the microphone choice,
compact transcript, fit-content code fences, and diff split/wrap toggles
had the same bug.

Replace the list with a three-way merge against the store snapshot the
Shell last synced: fields the Shell changed win, every other field keeps
the store's value. The merge destructures UiSettings exhaustively, so a
new field fails to compile until it is merged too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(voice): hold to talk, default Cmd+D, shortcut on the Voice page

Dictation is now hold to talk. Holding the microphone (pointer, or
Enter/Space while it is focused) or the shortcut records; releasing
transcribes. A pointer release outside the button still counts. For the
shortcut, letting go of its key or any of its modifiers ends the hold,
because macOS does not deliver key-up for Command chords. A release
within 300 ms explains hold to talk instead of transcribing a blip, and a
release that only answered the permission prompt records nothing.
Assistive-technology Click still starts and then finishes, since it
cannot hold.

The default shortcut is Cmd/Ctrl+D (unbound elsewhere). Its rebinding row
moves from Settings → Shortcuts to Settings → Voice through a reusable
ShortcutField that shares the Shortcuts page's chip and conflict rules.
Settings → Shortcuts re-reads the stored keymap while not recording, so
its own commits cannot revert a binding changed on the Voice page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(voice): harden capture, worker and download against edge cases

- Keep the capture flags outside the sample lock. The capture loop polled
  them under the mutex every 10 ms, so the real-time callback's try_lock
  could fail and drop a buffer of speech.
- Catch a native panic on the worker. A panicking job killed the only
  worker thread, leaving every later session "unavailable" until restart.
- Keep loaded weights when a session fails for a microphone or recording
  reason; dropping them made the next attempt reload 670 MB.
- Time out a download request whose response headers never arrive.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(ui): dictation review fixes for focus, shortcuts and Windows

- Enter/Space hold on the focused microphone no longer moves focus to the
  editor, which sent its key-up (and auto-repeat spaces, or Enter-to-send)
  to the draft and left the microphone live.
- A key release caused by focus leaving no longer pulls focus back, and an
  unparseable stored shortcut falls back to the default when matching its
  release, as apply_keymap already does.
- Another source can take over a hold whose release went missing, so the
  microphone or shortcut can always end it.
- Dictation cannot start while a queued row is acquiring its edit lease,
  which would replace the draft and silently drop the transcript.
- With dictation off (the default) ⌘D propagates to any other binding of
  the chord instead of being swallowed in the composer.
- The Voice page's shortcut row is always shown, conflicts with a feature
  page's shortcut name that page, and recording stops if the window closes
  so Appshots' capture shortcut is never left suspended.
- Show Settings → Voice on Windows; the PR ships the Windows backend and
  only Appshots is macOS/Linux-only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(ui): give the composer microphone its siblings' hover

At rest the microphone snapped to a different surface colour on hover;
it now uses the paperclip's eased ink wash. While live it dims like Send,
and its focus ring and tooltip match the neighbouring composer buttons.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Wing <contact@winglee.io>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
History 2026-10-01 15:16:51 -07:00
..
verify.rs Add opt-in on-device dictation to the desktop composer (#591) 2026-10-01 15:16:51 -07:00