0
Fork 0
mirror of https://github.com/mastra-ai/mastra.git synced 2026-11-17 08:44:46 +00:00
Commit graph

2 commits

Author SHA1 Message Date
Michael 'Misha' Litchev
7e90b39302
docs(voice): rename LiveKit page to Realtime Voice (#20274)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 11:15:06 +02:00
Shane Thomas
5e7f09c4b3
feat(livekit): add @mastra/livekit voice integration with Studio voice mode (#17896)
LiveKit runs the audio loop (WebRTC, streaming STT, semantic turn
detection, barge-in, TTS); the Mastra agent generates every reply with
its own model, tools, and memory via an llmNode bridge. Includes a
Studio voice mode (phone button, live captions, per-call memory
threads), a call-center example app, docs, and changesets.

**@mikhael28 -** Building on Shane's foundation, the integration picked
up several capabilities during development to bring it closer to
production-ready:

- **Workflow-driven replies.** Beyond the agent path, a turn's reply can
now be generated by a Mastra workflow that runs once to completion per
detected turn (`createWorkflowReplyGenerator`), with a
`pipeAgentReplyToWriter` helper that streams the agent's text *and*
forwards its tool calls into a workflow step. The workflow path is at
full feature parity with the agent path.
- **Per-turn and end-of-call lifecycle hooks.** `onTurnComplete` runs
after each reply finishes streaming (off the audio path,
fire-and-forget) and `onCallEnd` runs once when the call ends (awaited
within LiveKit's shutdown grace window). Together they give a clean seam
for memory maintenance, CRM writes, analytics, and flushing
observational memory once per call — none of it adding to the caller's
latency.
- **Voice-call observability.** Each session opens a root "voice call"
trace: LiveKit's STT/TTS/turn-detection/VAD metrics attach as child
spans, per-turn agent runs nest underneath, and the call closes with a
token/audio usage roll-up.
- **Working + observational memory showcase.** The example app
demonstrates layering per-caller working memory and observational memory
into a live voice agent, with a deterministic end-of-call reconciliation
pass.
- **Production hardening from review.** Config-validation guards
(mutually-exclusive reply sources, dispatch credential checks),
structured logging through the Mastra logger, backpressure and
cancellation safety on barge-in, and two full rounds of CodeRabbit
feedback addressed.

## Description

<!-- Required: Provide a brief description of the changes in this PR.
-->

## Related issue(s)

<!-- Required: Link to the issue(s) this PR addresses, e.g. with: Fixes
#123. PRs without linked issues may be closed. -->

## Type of change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [ ] Test update

## Checklist

- [ ] I have linked the related issue(s) in the description above
- [ ] I have made corresponding changes to the documentation (if
applicable)
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## ELI5
This PR adds a “talk by phone” voice mode to Mastra Studio. When you
click the phone button, LiveKit handles the audio + live captions, and
Mastra decides what to say back—without waiting for the caller to finish
(you can interrupt/“barge in” mid-sentence).

## Overview
Introduces a new `@mastra/livekit` integration that turns Mastra agents
into realtime, interruptible voice agents by wiring LiveKit’s audio loop
(WebRTC, STT, VAD/turn detection, barge-in, TTS) to Mastra’s normal
reply generation. Studio/Playground adds a voice call UI with a phone
button, live captions, agent state indicators, and per-call “voice
thread” continuity alongside text chat.

## Core integration: `@mastra/livekit`
- **LiveKit ↔ Mastra bridge (`bridge.ts`)**: `MastraVoiceAgent` /
`createMastraVoiceAgent` streams Mastra outputs into LiveKit voice
responses, supports tool-call “spoken feedback”, and implements barge-in
by canceling/aborting the active Mastra generation.
- **Worker support (`worker.ts`, `worker-setup.ts`, `run.ts`)**:
`createLiveKitWorker` + `runLiveKitWorker` configure STT/TTS/VAD/turn
detection, end-of-utterance (turn) handling, optional greetings, and
per-call lifecycle hooks. When memory is enabled, the worker ensures a
dedicated voice thread and can persist spoken greetings.
- **Reply generation options (agent or workflow)**: per-turn replies can
be produced via a Mastra **agent stream** or a **workflow**
(run-to-completion per detected turn), with a shared generation seam for
streaming text + tool calls.
- **Server routing + dispatch**
- `liveKitConnectionRoute()` mints LiveKit connection-details tokens and
embeds typed session metadata (e.g., `agentId`, `threadId`,
`resourceId`).
- `dispatchVoiceSession()` programmatically creates LiveKit dispatches
(server-side), reading `LIVEKIT_*` env vars when not provided.
- **Session/message helpers**
- `metadata.ts` provides typed parse/serialize helpers for session
metadata.
- `messages.ts` converts LiveKit chat context into Mastra “turn”
messages and can extract only “new turns” to avoid duplicating persisted
history.
- `voice-thread.ts` creates/ensures the per-call “Voice call” memory
thread and persists the spoken greeting (skipping empty greetings).
- **Observability (`observability.ts`)**: optional voice-call tracing
with a root “voice call” span, nested turn metrics, and per-model usage
rollups.

## Studio/Playground voice mode (UI)
- Adds a voice domain with `useVoiceCall()` to fetch connection details,
connect to a LiveKit room, enable the microphone, and stream/aggregate
captions into UI state (with stale-async protection via epochs/abort).
- Adds:
  - `VoiceCallButton` (idle/connecting/active + start/stop)
  - `VoiceCallPanel` (agent state + latest user/agent caption)
- Wires the phone button into the chat composer so starting a call
refreshes the correct thread/messages and keeps voice + text in the same
memory continuity.

## Example app: Meridian Trades call center
Adds `examples/voice-agent`, a working call-center voice demo using
`@mastra/livekit`, including:
- A caller-facing `callCenterAgent` plus a fast `triageAgent` intent
classifier.
- Call-center CRM-style tools (customer lookup, availability,
booking/rescheduling/cancellation) and intake reconciliation.
- Two interchangeable voice worker entrypoints (agent-driven vs
workflow-driven) with feature parity.
- Seeded in-memory scheduling data, deterministic end-of-call
reconciliation, per-caller memory layers, and a single-file SQLite trace
output.

## Tests, docs, and release notes
- Adds unit tests for the LiveKit↔Mastra bridge (streaming, tool
feedback, barge-in/abort behavior), message extraction, session metadata
parsing/round-trips, route token minting, worker/session setup, workflow
reply generation, observability, and the Playground voice call
lifecycle.
- Adds LiveKit documentation (guide + API reference) and sidebar
entries.
- Adds Changesets for the new `@mastra/livekit` package and for Studio
voice mode changes (including new per-turn/per-call lifecycle hook
coverage).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: mikhael28 <mikhael@hey.com>
2026-07-06 07:56:07 -07:00