0
Fork 0
mirror of https://github.com/zeronsh/comet.git synced 2026-11-16 18:57:19 +00:00
comet/crates/audio
José Gurruchaga 4a5fb8b443
Internal session plumbing and packaging cleanup (#819)
* test(codex): document and probe native voice capabilities

* feat(proto): define local voice session contracts

* fix(harness): bound realtime media and prioritize control messages

* feat(codex): attach realtime voice to an idle thread runtime

* feat(engine): own voice leases and enforce subscription eligibility

* feat(mcp): create standalone sessions on selected hosts

Add chat/side kinds with compatible defaults, resolve projects within the selected device, and validate that host's harness and model catalogs before writing. Route transcript reads to the chat host and preserve send baselines across waits so missing session rows and delayed replies cannot return an old response.

Cover invalid requests, remote catalogs, mixed batches, wait races, two-engine execution and interrupt delivery. Add an isolated MCP stdio smoke fixture.

* feat(doc): persist canonical voice transcripts once

* docs(mcp): document standalone and remote session workflows

* feat(audio): add native capture and streaming playback

* feat(audio): handle echo cancellation and voice interruptions

* feat(ui): coordinate voice ownership over local engine RPC

* feat(ui): extract the Bezel orb into a native Zeron component

* feat(ui): replace the composer with a central orb during voice sessions

* feat(mcp): constrain voice orchestration to subscription policy

* build(voice): add microphone permissions and offline audio CI

* test(voice): verify lifecycle, billing boundaries and rollout gates

* fix(voice): preserve text transport and idle billing boundaries

* feat(codex): connect subscription voice through native WebRTC

* feat(engine): activate native voice ownership and preserve delegated work

* feat(ui): enable native voice with central orb and retained drafts

* fix(mcp): keep voice delegation on subscription-backed Codex

* test(voice): document native transport and add opt-in connectivity smoke

* fix(voice): close audit races and validate project subscription context

* Fix Codex voice initialization and macOS microphone permissions

* Allow Codex voice agents to delegate through upstream MCP

* Make voice a persistent orchestrator with a dedicated stage

* Add call controls and elapsed time to the voice stage

* Remove decorative rings around the voice orb

* Add voice selection menu and simplify stage artwork

* Remove voice status labels beneath the orb

* Fix dictation and realtime voice integration after rebase

* Move Codex voice style selection into Voice settings

* Restore window dragging from the voice stage titlebar

* Open the voice stage immediately when starting a call

* Smooth voice orb motion and speaking transitions

* Remove the green audio pulse from the voice microphone control

* Show the live voice orb above the orchestrator chat composer

* Use full chat IDs in MCP reply instructions

* Correlate MCP replies by message IDs instead of remote timestamps

* Keep the voice stage open when launched from Settings

* test(voice): add a split-host Codex voice compatibility probe

* feat(voice): package a standalone desktop media runtime

* feat(proto): define client-media voice control contracts

* refactor(codex): separate voice signaling from local audio

* feat(engine): own remote voice sessions with ephemeral leases

* feat(rpc): route versioned voice controls to execution hosts

* feat(voice): coordinate local media with remote Codex control

* feat(ui): start voice on a selected remote Codex host

* test(voice): cover cancelled preparations and stale session controls

* test(voice): prepare desktop rollout validation and offline CI

* feat(ios): add a pinned native WebRTC compatibility endpoint

* feat(mobile): expose remote voice sessions through UniFFI

* feat(ios): bridge native audio lifecycle to remote voice calls

* feat(ios): add remote voice controls and registered host selection

* test(voice): cover relay and iOS lifecycle and prepare live acceptance

Add half-open and native callback regressions, stage-only diagnostics, complete runtime notices, and development test instructions. Keep real Mac/Fedora/iPhone acceptance opt-in for user validation.

* refactor(orb): share the thinking-orb engine through zeron-orb

Move the renderer-agnostic engine, presets, motion and a new OrbAnimator
(clock, audio response, state crossfades) into crates/orb. The GPUI widget
now delegates to the animator, so desktop and mobile paint identical frames.

* feat(voice): rebuild iOS remote voice around the desktop orb and phone-call audio

- Share the call view (phase, caption, levels, speaking, orb state) in
  voice-session; the client derives speaking because client media has no
  host meter. Keep owner-event headroom so level meters never overflow it.
- Title remote orchestrator chats "Voice session" like local ones.
- UniFFI: typed call state and end reasons instead of JSON, a distinct
  denied-microphone failure, and OrbRenderer frames from zeron-orb.
- iOS: OrbView paints the desktop orb; start voice from the New session bar
  (live strip while a call runs), the session nav bar and the iPad sidebar;
  full-screen call stage with caption, transcript (flagged when Codex waits
  for an answer), mute, route picker and hang-up.
- Phone-call audio: proximity sensor moves output to the earpiece, headsets
  win, calls survive screen lock, headset loss re-routes instead of ending.
  Install the WebRTC audio configuration globally so it is not reverted when
  the audio unit starts.

* fix(voice): install the macOS voice runtime under codex-resources/voice

Codex's helper only initializes from a codex-resources/voice directory and
exits with code 23 on initializeRuntime anywhere else, which the desktop
reported as "Voice could not connect". Bundle the runtime at
Contents/Resources/codex-resources/voice, reject other locations up front,
and report a helper that cannot start as an unavailable runtime.

Ship the runtime byte-for-byte with OpenAI's Developer ID signatures instead
of re-signing it ad hoc.

* fix(ios): reopen the call stage from the live strip and redo voice settings

The live strip in the bottom bar is a plain UIControl, which never sends
.primaryActionTriggered, so tapping it did nothing; listen for touchUpInside.
UI tests cover reopening the stage after the transcript and after minimizing.

Voice settings now follow the Settings list style: the resting orb, the
devices Codex can run on with their status, the voice as a pop-up menu, and
what to expect during a call.

* fix(ios): resume the orb animation when the app becomes active again

At willEnterForeground the app still reports .background, so the display
link never restarted and the orb stayed frozen after returning from Home.

* fix(ios): guard voice preview state in Release builds

* feat(voice): veil streamed caption words in on desktop and iPhone

Move the transcript's renderer-agnostic streaming veil into zeron-veil so
the voice call caption can use the same chunk fades on both platforms. A
new CaptionVeil keeps the veil aligned with the ellipsized tail and fades
the previous utterance out when the speaker turn changes; reduced motion
shows text directly. Desktop recolors GPUI runs; iOS gets UTF-16 spans
through CaptionFader and paints an attributed caption on a display link.
The desktop transcript keeps its behavior; the iOS transcript is untouched.

* fix(ui): step the voice stage aside when opening any session

Opening a session only hid the stage indirectly, when the selection changed,
so clicking the thread already under the stage (or the one you came from
Settings) did nothing. open_chat now hides the stage explicitly; the call
keeps running in the background.

* feat(voice): reuse host orchestrator chats across calls

* fix(voice): reserve orchestrator atomically and preserve rotation retries

* feat(voice): rotate the orchestrator transcript after 1000 entries

* feat(voice): package remote media runtime for Linux and Windows

* feat(voice): use the installed Codex helper everywhere and stop bundling it

Remote voice now plays audio through the client's own standalone Codex
installation, like local voice, instead of a runtime packaged inside
Zeron. The resolver moves to zeron-voice-media and is shared by the
harness and the desktop media endpoint; ZERON_VOICE_MEDIA_DIR remains a
development override. macOS, Linux and Windows packages no longer
contain Codex binaries, GStreamer/GLib or the Visual C++ runtime, so
Zeron takes on none of their redistribution or update obligations.
package-voice-runtime.py stays as a development projection tool.

Missing-runtime messages point to Codex's official installer, since npm
installs lack the voice runtime.

Notices and disclosures:
- THIRD_PARTY_NOTICES: full MIT chain for the orb (Bezel,
  gpui-thinking-orbs, thinking-orbs), Codex NOTICE, OpenAI trademark and
  non-affiliation statement, and WebRTC for iOS.
- The iOS app carries the orb's MIT notice; the Linux tarball now ships
  LICENSE and THIRD_PARTY_NOTICES.md.
- Microphone purpose strings on macOS and iOS state that voice sessions
  stream audio to OpenAI; dictation stays on device.
- Docs describe the voice data flow, and the acceptance checklist lists
  the legal and privacy decisions that block a public release.

* feat(ios): show the voice call in the Dynamic Island and on the Lock Screen

* fix(voice): hide the voice trigger when this device has no Codex

The microphone in the sidebar footer only shows when Codex is installed and
enabled on this device, following the same catalog the composer offers. It
re-reads after Settings → Agents changes and when returning to a window that
hides it. A start without Codex now asks for an install instead of a retry.

* test(voice): seed the real rotation threshold in the remote rotation test

* feat(voice): sound the start and end of a call on desktop and iPhone

A call sounds as it is placed, before the microphone opens, and again when
the user hangs up or a connected call ends. A failure to connect stays quiet
and explains itself. Nothing sounds in between. The iPhone plays them as
system sounds, so they follow the ringer switch.

* perf(voice): skip the rate limits round trip before each call

The pre-call probe read account/rateLimits/read, a ~1 s backend round
trip, and ran twice per call (prepare and negotiate). Its result was never
shown: the native backend makes the final usage decision. Dropping it
saves about 1.7 s per call on desktop and iPhone.

* perf(voice): release the orchestrator runtime 5 minutes after hang-up

The voice orchestrator's Codex runtime and MCP servers hold ~200 MB. Its
idle window was the generic 30 minutes, counted from the runtime's start,
so a call ending late in the window left a runtime reaped right after the
hang-up. It now idles 5 minutes, restarted at each hang-up.

Hang-ups wake the reaper directly, so even a short call restarts the
window, and a reap armed before a call began rechecks for it instead of
cutting the call.

* fix(voice): enable client-owned remote voice by default

* fix(harness): read whole resumed histories and report why stdout failed

509818ed capped every JSON-RPC line at 8 MiB on the transport shared by
text chats. A resumed long chat returns its history in one response
(54-71 MiB measured), so resume failed and was reported as an exit even
though Codex was alive. Lift the cap (voice bounds its own messages) and
keep the read error's cause for every waiting and later request.

* fix(harness): name the rejected request in JSON-RPC errors again

dca9a05f made peer rejections structured but dropped the method from
their message, so chats showed 'JSON-RPC rejection -32600: …' without
saying which request failed. Prefix the method again and render it as
upstream did, keeping the code available for auth detection.

* test: skip POSIX mode check on Windows hosts

* fix: launch Codex batch shims from plain paths and answer account/read in fake

* refactor(engine): remove the superseded local session RPC surface

The client-media protocol replaced these methods; nothing calls them anymore.

* refactor(harness): restore the shared JSON-RPC transport

Return the stdio transport every agent uses to its upstream behavior
(unbounded writes, no timeouts or polling timers) and keep only an
additive notification route that is registered while a subscriber
exists.

* fix(ui): let the full-window stage show the window glass

On translucent appearances the stage painted an opaque fill unlike the
new-thread page. Drop the fill there and fade out the page it covers.

* fix(ui): show the full-window stage over Settings

Settings replaces the workspace layout and never mounted the stage, so
opening it from there did nothing. Mount it beside the settings sidebar
and fade the page it covers.

* fix(voice-session): keep the live caption on the newest turn

Turns arrive interleaved: a user final often lands after the answer has
started streaming. Keep each turn's text separately and caption the
newest one, so a late final never takes the caption back and deltas
never restart it. Desktop and mobile now share this state.

* ci: build the desktop UI only in the UI workflow

The offline-linux job compiled zeron-ui without the WebKitGTK and
JSON-GLib packages its build script requires, so it failed every run.
Drop its UI step; the UI workflow already runs those lib tests and now
also runs the orb geometry test.
History 2026-10-06 23:47:32 -07:00
..
src Internal session plumbing and packaging cleanup (#819) 2026-10-06 23:47:32 -07:00
Cargo.toml Internal session plumbing and packaging cleanup (#819) 2026-10-06 23:47:32 -07:00