mirror of
https://github.com/mastra-ai/mastra.git
synced 2026-11-17 08:44:46 +00:00
2 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
7e90b39302 |
docs(voice): rename LiveKit page to Realtime Voice (#20274)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
5e7f09c4b3 |
feat(livekit): add @mastra/livekit voice integration with Studio voice mode (#17896)
LiveKit runs the audio loop (WebRTC, streaming STT, semantic turn detection, barge-in, TTS); the Mastra agent generates every reply with its own model, tools, and memory via an llmNode bridge. Includes a Studio voice mode (phone button, live captions, per-call memory threads), a call-center example app, docs, and changesets. **@mikhael28 -** Building on Shane's foundation, the integration picked up several capabilities during development to bring it closer to production-ready: - **Workflow-driven replies.** Beyond the agent path, a turn's reply can now be generated by a Mastra workflow that runs once to completion per detected turn (`createWorkflowReplyGenerator`), with a `pipeAgentReplyToWriter` helper that streams the agent's text *and* forwards its tool calls into a workflow step. The workflow path is at full feature parity with the agent path. - **Per-turn and end-of-call lifecycle hooks.** `onTurnComplete` runs after each reply finishes streaming (off the audio path, fire-and-forget) and `onCallEnd` runs once when the call ends (awaited within LiveKit's shutdown grace window). Together they give a clean seam for memory maintenance, CRM writes, analytics, and flushing observational memory once per call — none of it adding to the caller's latency. - **Voice-call observability.** Each session opens a root "voice call" trace: LiveKit's STT/TTS/turn-detection/VAD metrics attach as child spans, per-turn agent runs nest underneath, and the call closes with a token/audio usage roll-up. - **Working + observational memory showcase.** The example app demonstrates layering per-caller working memory and observational memory into a live voice agent, with a deterministic end-of-call reconciliation pass. - **Production hardening from review.** Config-validation guards (mutually-exclusive reply sources, dispatch credential checks), structured logging through the Mastra logger, backpressure and cancellation safety on barge-in, and two full rounds of CodeRabbit feedback addressed. ## Description <!-- Required: Provide a brief description of the changes in this PR. --> ## Related issue(s) <!-- Required: Link to the issue(s) this PR addresses, e.g. with: Fixes #123. PRs without linked issues may be closed. --> ## Type of change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Code refactoring - [ ] Performance improvement - [ ] Test update ## Checklist - [ ] I have linked the related issue(s) in the description above - [ ] I have made corresponding changes to the documentation (if applicable) - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have addressed all Coderabbit comments on this PR <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## ELI5 This PR adds a “talk by phone” voice mode to Mastra Studio. When you click the phone button, LiveKit handles the audio + live captions, and Mastra decides what to say back—without waiting for the caller to finish (you can interrupt/“barge in” mid-sentence). ## Overview Introduces a new `@mastra/livekit` integration that turns Mastra agents into realtime, interruptible voice agents by wiring LiveKit’s audio loop (WebRTC, STT, VAD/turn detection, barge-in, TTS) to Mastra’s normal reply generation. Studio/Playground adds a voice call UI with a phone button, live captions, agent state indicators, and per-call “voice thread” continuity alongside text chat. ## Core integration: `@mastra/livekit` - **LiveKit ↔ Mastra bridge (`bridge.ts`)**: `MastraVoiceAgent` / `createMastraVoiceAgent` streams Mastra outputs into LiveKit voice responses, supports tool-call “spoken feedback”, and implements barge-in by canceling/aborting the active Mastra generation. - **Worker support (`worker.ts`, `worker-setup.ts`, `run.ts`)**: `createLiveKitWorker` + `runLiveKitWorker` configure STT/TTS/VAD/turn detection, end-of-utterance (turn) handling, optional greetings, and per-call lifecycle hooks. When memory is enabled, the worker ensures a dedicated voice thread and can persist spoken greetings. - **Reply generation options (agent or workflow)**: per-turn replies can be produced via a Mastra **agent stream** or a **workflow** (run-to-completion per detected turn), with a shared generation seam for streaming text + tool calls. - **Server routing + dispatch** - `liveKitConnectionRoute()` mints LiveKit connection-details tokens and embeds typed session metadata (e.g., `agentId`, `threadId`, `resourceId`). - `dispatchVoiceSession()` programmatically creates LiveKit dispatches (server-side), reading `LIVEKIT_*` env vars when not provided. - **Session/message helpers** - `metadata.ts` provides typed parse/serialize helpers for session metadata. - `messages.ts` converts LiveKit chat context into Mastra “turn” messages and can extract only “new turns” to avoid duplicating persisted history. - `voice-thread.ts` creates/ensures the per-call “Voice call” memory thread and persists the spoken greeting (skipping empty greetings). - **Observability (`observability.ts`)**: optional voice-call tracing with a root “voice call” span, nested turn metrics, and per-model usage rollups. ## Studio/Playground voice mode (UI) - Adds a voice domain with `useVoiceCall()` to fetch connection details, connect to a LiveKit room, enable the microphone, and stream/aggregate captions into UI state (with stale-async protection via epochs/abort). - Adds: - `VoiceCallButton` (idle/connecting/active + start/stop) - `VoiceCallPanel` (agent state + latest user/agent caption) - Wires the phone button into the chat composer so starting a call refreshes the correct thread/messages and keeps voice + text in the same memory continuity. ## Example app: Meridian Trades call center Adds `examples/voice-agent`, a working call-center voice demo using `@mastra/livekit`, including: - A caller-facing `callCenterAgent` plus a fast `triageAgent` intent classifier. - Call-center CRM-style tools (customer lookup, availability, booking/rescheduling/cancellation) and intake reconciliation. - Two interchangeable voice worker entrypoints (agent-driven vs workflow-driven) with feature parity. - Seeded in-memory scheduling data, deterministic end-of-call reconciliation, per-caller memory layers, and a single-file SQLite trace output. ## Tests, docs, and release notes - Adds unit tests for the LiveKit↔Mastra bridge (streaming, tool feedback, barge-in/abort behavior), message extraction, session metadata parsing/round-trips, route token minting, worker/session setup, workflow reply generation, observability, and the Playground voice call lifecycle. - Adds LiveKit documentation (guide + API reference) and sidebar entries. - Adds Changesets for the new `@mastra/livekit` package and for Studio voice mode changes (including new per-turn/per-call lifecycle hook coverage). <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: mikhael28 <mikhael@hey.com> |