0
Fork 0
mirror of https://github.com/zeronsh/comet.git synced 2026-11-16 18:57:19 +00:00
comet/docs/research/codex-voice.md
José Gurruchaga 4a5fb8b443
Internal session plumbing and packaging cleanup (#819)
* test(codex): document and probe native voice capabilities

* feat(proto): define local voice session contracts

* fix(harness): bound realtime media and prioritize control messages

* feat(codex): attach realtime voice to an idle thread runtime

* feat(engine): own voice leases and enforce subscription eligibility

* feat(mcp): create standalone sessions on selected hosts

Add chat/side kinds with compatible defaults, resolve projects within the selected device, and validate that host's harness and model catalogs before writing. Route transcript reads to the chat host and preserve send baselines across waits so missing session rows and delayed replies cannot return an old response.

Cover invalid requests, remote catalogs, mixed batches, wait races, two-engine execution and interrupt delivery. Add an isolated MCP stdio smoke fixture.

* feat(doc): persist canonical voice transcripts once

* docs(mcp): document standalone and remote session workflows

* feat(audio): add native capture and streaming playback

* feat(audio): handle echo cancellation and voice interruptions

* feat(ui): coordinate voice ownership over local engine RPC

* feat(ui): extract the Bezel orb into a native Zeron component

* feat(ui): replace the composer with a central orb during voice sessions

* feat(mcp): constrain voice orchestration to subscription policy

* build(voice): add microphone permissions and offline audio CI

* test(voice): verify lifecycle, billing boundaries and rollout gates

* fix(voice): preserve text transport and idle billing boundaries

* feat(codex): connect subscription voice through native WebRTC

* feat(engine): activate native voice ownership and preserve delegated work

* feat(ui): enable native voice with central orb and retained drafts

* fix(mcp): keep voice delegation on subscription-backed Codex

* test(voice): document native transport and add opt-in connectivity smoke

* fix(voice): close audit races and validate project subscription context

* Fix Codex voice initialization and macOS microphone permissions

* Allow Codex voice agents to delegate through upstream MCP

* Make voice a persistent orchestrator with a dedicated stage

* Add call controls and elapsed time to the voice stage

* Remove decorative rings around the voice orb

* Add voice selection menu and simplify stage artwork

* Remove voice status labels beneath the orb

* Fix dictation and realtime voice integration after rebase

* Move Codex voice style selection into Voice settings

* Restore window dragging from the voice stage titlebar

* Open the voice stage immediately when starting a call

* Smooth voice orb motion and speaking transitions

* Remove the green audio pulse from the voice microphone control

* Show the live voice orb above the orchestrator chat composer

* Use full chat IDs in MCP reply instructions

* Correlate MCP replies by message IDs instead of remote timestamps

* Keep the voice stage open when launched from Settings

* test(voice): add a split-host Codex voice compatibility probe

* feat(voice): package a standalone desktop media runtime

* feat(proto): define client-media voice control contracts

* refactor(codex): separate voice signaling from local audio

* feat(engine): own remote voice sessions with ephemeral leases

* feat(rpc): route versioned voice controls to execution hosts

* feat(voice): coordinate local media with remote Codex control

* feat(ui): start voice on a selected remote Codex host

* test(voice): cover cancelled preparations and stale session controls

* test(voice): prepare desktop rollout validation and offline CI

* feat(ios): add a pinned native WebRTC compatibility endpoint

* feat(mobile): expose remote voice sessions through UniFFI

* feat(ios): bridge native audio lifecycle to remote voice calls

* feat(ios): add remote voice controls and registered host selection

* test(voice): cover relay and iOS lifecycle and prepare live acceptance

Add half-open and native callback regressions, stage-only diagnostics, complete runtime notices, and development test instructions. Keep real Mac/Fedora/iPhone acceptance opt-in for user validation.

* refactor(orb): share the thinking-orb engine through zeron-orb

Move the renderer-agnostic engine, presets, motion and a new OrbAnimator
(clock, audio response, state crossfades) into crates/orb. The GPUI widget
now delegates to the animator, so desktop and mobile paint identical frames.

* feat(voice): rebuild iOS remote voice around the desktop orb and phone-call audio

- Share the call view (phase, caption, levels, speaking, orb state) in
  voice-session; the client derives speaking because client media has no
  host meter. Keep owner-event headroom so level meters never overflow it.
- Title remote orchestrator chats "Voice session" like local ones.
- UniFFI: typed call state and end reasons instead of JSON, a distinct
  denied-microphone failure, and OrbRenderer frames from zeron-orb.
- iOS: OrbView paints the desktop orb; start voice from the New session bar
  (live strip while a call runs), the session nav bar and the iPad sidebar;
  full-screen call stage with caption, transcript (flagged when Codex waits
  for an answer), mute, route picker and hang-up.
- Phone-call audio: proximity sensor moves output to the earpiece, headsets
  win, calls survive screen lock, headset loss re-routes instead of ending.
  Install the WebRTC audio configuration globally so it is not reverted when
  the audio unit starts.

* fix(voice): install the macOS voice runtime under codex-resources/voice

Codex's helper only initializes from a codex-resources/voice directory and
exits with code 23 on initializeRuntime anywhere else, which the desktop
reported as "Voice could not connect". Bundle the runtime at
Contents/Resources/codex-resources/voice, reject other locations up front,
and report a helper that cannot start as an unavailable runtime.

Ship the runtime byte-for-byte with OpenAI's Developer ID signatures instead
of re-signing it ad hoc.

* fix(ios): reopen the call stage from the live strip and redo voice settings

The live strip in the bottom bar is a plain UIControl, which never sends
.primaryActionTriggered, so tapping it did nothing; listen for touchUpInside.
UI tests cover reopening the stage after the transcript and after minimizing.

Voice settings now follow the Settings list style: the resting orb, the
devices Codex can run on with their status, the voice as a pop-up menu, and
what to expect during a call.

* fix(ios): resume the orb animation when the app becomes active again

At willEnterForeground the app still reports .background, so the display
link never restarted and the orb stayed frozen after returning from Home.

* fix(ios): guard voice preview state in Release builds

* feat(voice): veil streamed caption words in on desktop and iPhone

Move the transcript's renderer-agnostic streaming veil into zeron-veil so
the voice call caption can use the same chunk fades on both platforms. A
new CaptionVeil keeps the veil aligned with the ellipsized tail and fades
the previous utterance out when the speaker turn changes; reduced motion
shows text directly. Desktop recolors GPUI runs; iOS gets UTF-16 spans
through CaptionFader and paints an attributed caption on a display link.
The desktop transcript keeps its behavior; the iOS transcript is untouched.

* fix(ui): step the voice stage aside when opening any session

Opening a session only hid the stage indirectly, when the selection changed,
so clicking the thread already under the stage (or the one you came from
Settings) did nothing. open_chat now hides the stage explicitly; the call
keeps running in the background.

* feat(voice): reuse host orchestrator chats across calls

* fix(voice): reserve orchestrator atomically and preserve rotation retries

* feat(voice): rotate the orchestrator transcript after 1000 entries

* feat(voice): package remote media runtime for Linux and Windows

* feat(voice): use the installed Codex helper everywhere and stop bundling it

Remote voice now plays audio through the client's own standalone Codex
installation, like local voice, instead of a runtime packaged inside
Zeron. The resolver moves to zeron-voice-media and is shared by the
harness and the desktop media endpoint; ZERON_VOICE_MEDIA_DIR remains a
development override. macOS, Linux and Windows packages no longer
contain Codex binaries, GStreamer/GLib or the Visual C++ runtime, so
Zeron takes on none of their redistribution or update obligations.
package-voice-runtime.py stays as a development projection tool.

Missing-runtime messages point to Codex's official installer, since npm
installs lack the voice runtime.

Notices and disclosures:
- THIRD_PARTY_NOTICES: full MIT chain for the orb (Bezel,
  gpui-thinking-orbs, thinking-orbs), Codex NOTICE, OpenAI trademark and
  non-affiliation statement, and WebRTC for iOS.
- The iOS app carries the orb's MIT notice; the Linux tarball now ships
  LICENSE and THIRD_PARTY_NOTICES.md.
- Microphone purpose strings on macOS and iOS state that voice sessions
  stream audio to OpenAI; dictation stays on device.
- Docs describe the voice data flow, and the acceptance checklist lists
  the legal and privacy decisions that block a public release.

* feat(ios): show the voice call in the Dynamic Island and on the Lock Screen

* fix(voice): hide the voice trigger when this device has no Codex

The microphone in the sidebar footer only shows when Codex is installed and
enabled on this device, following the same catalog the composer offers. It
re-reads after Settings → Agents changes and when returning to a window that
hides it. A start without Codex now asks for an install instead of a retry.

* test(voice): seed the real rotation threshold in the remote rotation test

* feat(voice): sound the start and end of a call on desktop and iPhone

A call sounds as it is placed, before the microphone opens, and again when
the user hangs up or a connected call ends. A failure to connect stays quiet
and explains itself. Nothing sounds in between. The iPhone plays them as
system sounds, so they follow the ringer switch.

* perf(voice): skip the rate limits round trip before each call

The pre-call probe read account/rateLimits/read, a ~1 s backend round
trip, and ran twice per call (prepare and negotiate). Its result was never
shown: the native backend makes the final usage decision. Dropping it
saves about 1.7 s per call on desktop and iPhone.

* perf(voice): release the orchestrator runtime 5 minutes after hang-up

The voice orchestrator's Codex runtime and MCP servers hold ~200 MB. Its
idle window was the generic 30 minutes, counted from the runtime's start,
so a call ending late in the window left a runtime reaped right after the
hang-up. It now idles 5 minutes, restarted at each hang-up.

Hang-ups wake the reaper directly, so even a short call restarts the
window, and a reap armed before a call began rechecks for it instead of
cutting the call.

* fix(voice): enable client-owned remote voice by default

* fix(harness): read whole resumed histories and report why stdout failed

509818ed capped every JSON-RPC line at 8 MiB on the transport shared by
text chats. A resumed long chat returns its history in one response
(54-71 MiB measured), so resume failed and was reported as an exit even
though Codex was alive. Lift the cap (voice bounds its own messages) and
keep the read error's cause for every waiting and later request.

* fix(harness): name the rejected request in JSON-RPC errors again

dca9a05f made peer rejections structured but dropped the method from
their message, so chats showed 'JSON-RPC rejection -32600: …' without
saying which request failed. Prefix the method again and render it as
upstream did, keeping the code available for auth detection.

* test: skip POSIX mode check on Windows hosts

* fix: launch Codex batch shims from plain paths and answer account/read in fake

* refactor(engine): remove the superseded local session RPC surface

The client-media protocol replaced these methods; nothing calls them anymore.

* refactor(harness): restore the shared JSON-RPC transport

Return the stdio transport every agent uses to its upstream behavior
(unbounded writes, no timeouts or polling timers) and keep only an
additive notification route that is registered while a subscriber
exists.

* fix(ui): let the full-window stage show the window glass

On translucent appearances the stage painted an opaque fill unlike the
new-thread page. Drop the fill there and fade out the page it covers.

* fix(ui): show the full-window stage over Settings

Settings replaces the workspace layout and never mounted the stage, so
opening it from there did nothing. Mount it beside the settings sidebar
and fade the page it covers.

* fix(voice-session): keep the live caption on the newest turn

Turns arrive interleaved: a user final often lands after the answer has
started streaming. Keep each turn's text separately and caption the
newest one, so a late final never takes the caption back and deltas
never restart it. Desktop and mobile now share this state.

* ci: build the desktop UI only in the UI workflow

The offline-linux job compiled zeron-ui without the WebKitGTK and
JSON-GLib packages its build script requires, so it failed every run.
Drop its UI step; the UI workflow already runs those lib tests and now
also runs the orb geometry test.
2026-10-06 23:47:32 -07:00

13 KiB

Native Codex subscription voice

Zeron connects a local Codex chat to the installed Codex voice runtime. Voice uses ChatGPT authentication and the normal Codex plan budget. Codex may use additional credits according to the user's account settings and usage limits. There is no OpenAI API-key fallback, separate public Realtime API client, or promise of an included-quota-only spending cap.

Sources checked 2026-10-01:

Transport and authentication

Codex 0.159's WebSocket audio path calls realtime_api_key and requires API-key credentials, including an environment-key fallback. That path is unsuitable for this integration. Native subscription voice uses WebRTC V3 instead:

  1. Prepare or reuse an idle native thread with the normal Zeron MCP server. No initial turn/start, empty user message, title request, or submitted draft. Load the initial account/read snapshot before attaching the voice router: Codex announces initial authentication with account/updated, which must not be mistaken for a later identity change. Subsequent updates still retire voice.
  2. Check account/read on that same process for chatgpt authentication, reject custom providers and realtime backend overrides in the effective project configuration (config/read with the resolved thread cwd), validate the provider returned by thread/start or thread/resume, inspect ordinary usage permission, and obtain thread/realtime/listVoices. Exhausted ordinary quota alone does not reject permitted credits; the native backend makes the final usage and spend-control decision.
  3. Resolve the helper inside the physical standalone Codex package (layout 1, version 0.159 or later). Its protocol/build handshake must succeed.
  4. Initialize the helper and gather its SDP offer. Call thread/realtime/start with WebRTC, V3, audio output, the lease session id, startup context, and clientManagedHandoffs: false. Codex owns background task delegation.
  5. The request response acknowledges submission. Independently wait for both thread/realtime/started and thread/realtime/sdp; validate session/version.
  6. Apply the answer and wait for transport readiness. Open default local audio devices while muted/suppressed. Only enable them for an attached owner.

Audio, Opus, resampling, echo cancellation, reference audio, native interruption, and bounded playout remain in Codex's helper. No PCM or SDP reaches the UI, document, sync journal, or owner RPC stream. Helper stderr is discarded; errors and diagnostics use typed rejection reasons. Child processes are killed and reaped on cancellation, including when a helper control reply is stalled.

The earlier zeron-audio PCM/AEC prototype remains isolated and tested offline; it is not a dependency of the released UI or the subscription voice path. voice-experimental is retained as an empty compatibility build feature. The Codex voice control is available in ordinary desktop builds.

Lifecycle, transcript and MCP

One engine lease owns local voice. Tokens are redacted from Debug. Ownership is exclusive, generation checked, ephemeral, and cancelled when its scoped stream is dropped. A five-second unattached-owner watchdog prevents abandoned starts from creating a provider call. A successor waits for native stop and its terminal requested notification from that single stop; a spontaneous terminal cannot satisfy the barrier and leave our terminal queued for a successor. Late events cannot acquire the successor's generation. Native capture termination is independent of the actor's pending control I/O. The stdout reader also cancels capture on identity changes, overflow and EOF; it does not wait for a blocked audio-control reply. Identity retirement invalidates pending start reservations as well as existing leases. The physical Codex release is pinned before spawning app-server so an installer symlink change cannot select a different helper for a warm runtime. A reader identity change also permanently invalidates the voice bridge before any new token can be installed. A fresh explicit request can renew that idle runtime after cleanup; continuing delegated work is preserved and must finish before renewal.

Voice protects the warm runtime from idle reaping and updates. Voice stop leaves delegated Codex work running. Account/profile retirement, engine/window owner loss, audio failure and provider termination close voice; navigating between threads does not. An agent question keeps voice open: the stage reports it and links to the session transcript, where the usual input UI answers it. Reconnection and microphone resumption require a fresh user action. Changes to the chat host, configuration or checkout invalidate pending starts and active ownership; workspace monitoring also covers synced host/configuration changes.

Canonical thread/realtime/item/completed transcript segments are committed by the serialized document owner, deduplicated by session/item identity. Legacy transcript events and partial text never become durable messages. Only a newly inserted canonical final updates sidebar activity and preview; replay does not bump the timestamp. Native BEM promotion uses the existing Codex task transcript; it does not resubmit a spoken request or create another task response. Native turn/started publishes the engine turn boundary before its text/tools, including on an idle bootstrap.

Every session is an orchestrator. thread/realtime/start carries one short English instruction, as realtimeStartInstructions for the backing Codex model and as a developer initialItems entry for the voice model: use the Zeron MCP to create, message and monitor chats, and delegate coding work to them.

The voice agent uses the unchanged upstream Zeron MCP. Creating chats, sending messages and delegating tasks use the normal provider selection, authentication and device routing. A Codex voice session can create a Grok, Claude Code or other available provider session; those children use their own provider credentials. Codex/ChatGPT authentication and local-host checks apply to starting voice itself, not to the sessions it controls through MCP. Ending voice does not restrict continuing tasks or their children.

Interface and packaging

Voice starts from the microphone in the sidebar footer, next to Settings. Each session creates a fresh projectless Codex chat (cwd ~) on this device whose id starts with voice-orchestrator-. That prefix keeps it out of the sidebar, jump slots, command palette, archive and completion notifications, and the engine drops it as a parentChatId: chats the orchestrator creates are ordinary top-level sessions. The chat is deleted again if startup fails before a lease exists. It uses the composer's Codex model when Codex is selected, otherwise Codex's default. The composer is never replaced or suspended: the user keeps typing in and switching between threads while voice runs.

While a session is live the microphone becomes a small orb. Pressing it toggles a stage over the conversation area (the whole window when the sidebar is collapsed); the titlebar cluster stays above it, and picking another thread, a new session or Settings steps it aside without ending voice. The stage shows the orb extracted from Bezel at 6141af9c16f7353cdf36003f7404e0a94566a163 (magnified), over the new-thread hero artwork with its soft bottom fade and a live caption. A video-call style bar at the bottom carries the session clock, microphone mute (with a live input ring), a next-session voice menu, transcript, a red End button and the way back to the chats (also Escape). Bezel is not a dependency. MIT notices accompany the extracted component. The orb tracks listening, speaking, task work and pending questions, uses the theme and reduced-motion setting, and stops hidden/background timers.

Native voice ids from the provider catalog can be selected for the next session; only that preference is persisted. Devices use the operating system defaults. Codex's current helper protocol does not expose device selection.

The native helper/runtime is supplied by the standalone Codex installation, not redistributed by Zeron, on macOS, Linux and Windows; the desktop client of a remote call uses its own installation the same way. npm/CLI-only installations without those resources report nativeRuntimeUnavailable. macOS production/dev bundles declare microphone usage and audio-input entitlements. The viewport verifies microphone authorization and requests it when voice is explicitly started, before creating a provider session or opening devices. Denied access shows a settings hint. Cargo builds embed the microphone purpose string in the executable's __TEXT,__info_plist section, so direct development launches can also request access. Missing metadata shows a rebuild hint instead of requesting permission and being terminated by macOS. scripts/run-macos-dev.sh builds the signed development bundle; ZERON_DEV_BUILD_ONLY=1 prepares it without launching. No new DSP DLLs or C++ build dependency are added to the production UI. Apache-2.0 attribution for the adapted native helper protocol is included in THIRD_PARTY_NOTICES.md.

Data flow:

  • Audio goes from the device with the microphone (Codex's helper on desktop, WebRTC on iOS) directly to OpenAI, under the user's own ChatGPT-authenticated Codex account and OpenAI's terms. It never transits Zeron servers.
  • Remote calls exchange SDP signaling and controls over the authenticated owner-only device relay; they are ephemeral and never persisted.
  • Canonical final transcript text is stored in the orchestrator chat's session document and syncs like any chat, through Zeron's relay. Orchestrator chats are hidden from the sidebar but are not deleted when a call ends.
  • Usage is billed to the user's Codex plan or credits by OpenAI.

The microphone purpose strings (dist/macos/Info*.plist, apps/ios/Zeron/Info.plist) state that voice sessions stream audio to OpenAI; dictation stays on device.

Notices: THIRD_PARTY_NOTICES.md covers the Apache-2.0 Codex protocol adaptation (with its upstream NOTICE and an OpenAI trademark/non-affiliation statement), the MIT orb chain (Bezel, gpui-thinking-orbs, thinking-orbs) and WebRTC for iOS. macOS, Windows and Linux packages ship LICENSE and THIRD_PARTY_NOTICES.md; the iOS app ships Voice/WebRTC-LICENSE.txt and Voice/ThinkingOrbs-LICENSE.txt. No Codex binaries are redistributed.

Pending before a public release (owner decisions, not code):

  • iOS: App Review guideline 5.1.2(i) requires disclosing that personal data is shared with a third-party AI and obtaining explicit permission before doing so. The permission string discloses it; an explicit in-app consent step before the first call is not implemented.
  • iOS privacy manifest (PrivacyInfo.xcprivacy) and App Store privacy labels declare no collected data. Decide whether audio sent to OpenAI and synced voice transcripts must be declared.
  • Zeron's public privacy policy and terms (outside this repository) should describe the OpenAI audio flow and transcript sync.
  • WebRTC's embedded third-party licenses (BoringSSL, libsrtp, Opus, libyuv, Abseil and others) are referenced, not reproduced; generate the full list from the pinned XCFramework before distribution.

Validation and practical limits

Offline CI never starts a provider session or microphone. Fake app-server and helper peers exercise actual engine RPC, independent start acknowledgements, SDP signaling, owner attachment, native controls, transcript replay, task continuation, stop/restart and authentication rejection. Synthetic audio tests remain offline. Cross-platform checks compile the native bridge without loading hardware; signed package/device trials require their actual operating systems.

Read-only discovery:

python3 scripts/codex-voice-probe.py

Explicit opt-in connectivity smoke (consumes normal Codex voice usage, does not open microphone devices):

python3 scripts/codex-voice-smoke.py --live

Optional device/playout check opens default devices with the microphone muted and requests a short greeting:

python3 scripts/codex-voice-smoke.py --live --devices

Evidence on 2026-10-01, Linux, installed Codex 0.159.0: ChatGPT authentication, packaged helper initialization, real V3 WebRTC negotiation, transport readiness and native stop passed. No API fallback was used. The optional device trial failed at opening audio devices: this environment exposes a network output monitor, without a usable default microphone. No real speech or acoustic echo trial is claimed. macOS/Windows signed package and physical-device trials also remain unverified; protocol/fixture success does not establish those results.

Implementation review and local test results are recorded in the ignored .personal/codex-voice/ working notes. No push, deployment or remote PR is part of this task.