Skip to main content

Kara Voice: A Realtime Voice Agent

A production voice layer for Kara where a realtime model is the receptionist and every data question runs the same agent turn as typed chat.

  • Architecture
  • Voice Agent Engineering
  • Safety & Evaluation

  • 0.77 s

    first audio from Gemini Live

  • 100%

    barge-ins stopped Kara mid-answer on iPhone

  • 40–250 ms

    interrupt to silence

  • 858

    voice tests passing

The problem

Kara is the AI assistant inside One Mount's meeting app. Voice lets people ask it about their own meetings, contacts and to-dos in the moments when typing is not an option: standing outside a client's door, or walking out of a meeting.

The hard requirement is that a spoken answer must equal the typed answer: same data, same numbers, same refusals, same safety rules. The answer also stays on screen, because people forget most of what they only hear.

Live in production for Kara users, running on the Gemini Live service.

  • Two languagesVietnamese with English mixed in. Kara answers in the language the user spoke.
  • Never silentWhile a lookup runs, the user must not hear more than 2 seconds of silence.
  • No invented dataNumbers, names, dates and refusals come from the agent, never from the voice model.
  • Off by defaultEvery voice flag is off by default and fails closed.
Kara Voice: User, Receptionist, Backend and Expert layers, question down and answer upUserReceptionistBackendExpertKara appmic · speaker · screenGemini LiveVAD · acksnever answersdata itselfVoice sessionreceptionist promptanswer shapingVoice relay · shared turn runnerauth ticket · PCM16 relay · same chat turn as typed chatKara's existing agent:same data, numbers,refusals and safetyrules as typed chat.Main agent graphsame agent as typed chatMCP toolsread allowlistconsent for writes1 SPEAK9 HEAR + SEE2 ASK_KARA8 WHEN_IDLE3 TURN REQUEST7 TURN RESULT4 CHAT TURN6 ANSWER5 TOOL CALL
Four layers. The question goes down (steps 1–5): the Receptionist hears it and calls ask_kara, the Backend runs the same chat turn as typed chat, and the Expert answers with its tools. The answer comes back up (steps 6–9) to be spoken and shown on screen.

The Receptionist and the Expert

A realtime voice model is good at conversation and risky as a source of facts. So I split the job. Gemini Live is the Receptionist: it greets, acknowledges and handles small talk. The Expert is Kara's existing agent, reached through one tool.

The app connects to the backend over a public WebSocket with a one-time ticket that expires in 60 seconds, and sends PCM16 audio. The backend hands the audio to the agent service, which holds the Live session. When the user asks a data question, the agent service calls the backend, and the backend runs the chat turn. Auth, data scope, quota and storage are therefore the same as in text chat. The result streams to the screen as a normal chat reply and goes back to Live to be spoken.

Same turn, not a second brain

Before writing any voice code I refactored the 1,160-line chat handler into one shared turn runner. Typed chat and voice now call the same function, so there is no second copy of chat logic to drift. The baseline tests passed unchanged before and after.

The messages that flow between the app, backend, agent service and clients follow one frame contract, byte-identical across the web app, the mobile app and both services. A sha256 check and contract tests fail the build if any copy drifts.

One turn runner, one frame contract. Voice is a new way in, not a new brain.

Never silent, never inventing

Lookups take time, and a quiet line sounds like a dead app. The receptionist has to fill the gap without making anything up.

  • Data-free acknowledgementsThe receptionist opens with a short phrase from a fixed list. An eval fails any filler that contains digits, names or dates, or runs past 15 words.
  • Spoken-answer shapingMarkdown is stripped, URLs become "link on screen", long lists are shortened and refusals are read verbatim. It now uses a natural paraphrase that keeps every number, name and date, and an eval checks it.
  • Sentence-by-sentence streamingThe answer reaches the model as it is written, so the first chunk is spoken about one second after it arrives.
  • Device-side metricsFirst sentence, longest silence and interrupt-to-stop are measured on the device itself.

Interruptions: what the spike taught

Before building, I ran a one-day spike against the Live API. First audio arrived in 0.77 seconds. More importantly, it caught a problem early: when a user interrupts, the cancellation event for an in-flight tool call never arrived, in any of four interruption styles I tried.

The fix was a second tool, cancel_ask_kara, that the model calls itself. A non-blocking version of it worked in all 5 trials, and the cancelled call is answered with silence. On an iPhone loudspeaker with system echo cancellation on, every interruption stopped Kara, with playback ending 40 to 250 milliseconds after Gemini's interrupt signal.

A common assumption is that iOS echo cancellation mutes the microphone and makes barge-in impossible. I measured it instead: barge-in works on the iPhone loudspeaker.

Voice turn lifecycle on the server: Idle, Listening, Acknowledge, Pending, Queued, Speaking, Drain, with Refused for turns without fresh speechIdlewaiting for speechListeningPCM16 to LiveAcknowledgedata-free phrasePendingtoolCall · waitingQueuedanswer heldSpeakingLive reads answerDrainplays bufferSPEECH STARTEND OF SPEECHTOOLCALLTURN_RESULTWHEN_IDLEAUDIO LEFTRefusedSILENTno audio outNO SPEECHBARGE-INDRAINED
Server-side life of one voice turn. A turn with no fresh speech is refused and stays silent, and the answer is spoken only once Live is idle.

Safety in code, not in the prompt

The voice model hears raw speech, and anything a model hears can be an attempt to steer it. So the rules that matter live in the agent service's code, where a prompt cannot talk its way around them.

  • Consent from the user, not the modelA write is allowed only when the user's own speech transcript contains a strict affirmative. A negation counts as no.
  • Default-deny write gateOnly tools on a read allowlist run freely. An unknown tool is blocked.
  • Fresh speech for lookupsA lookup or cancel needs new user speech in the current turn, so nothing but the user's own words can start one.
  • A receptionist that never refusesEverything except small talk and controls goes to the expert. Turns that call no tool are logged.
Six protection layers for a voice turn: guidance prompt at the top, code and server config below01Live promptroutes data questions to ask_karalanguage & address · ack before a toolPROMPT02Gemini server VADend of speech 550 ms · prefix padding 150 msbarge-in: user speech stops playbackCONFIG03Transcript cleanupmarkup dropped from the user transcriptrepeated phrases de-duplicatedCODE04Provenance gatelookups need fresh speech in this turnhold ≤ 1.5 s · question ≤ 1000 charsCODE05Tool permissionsread allowlist · unknown tools blockedwrites need consentCODE06Backend queue1 running + 1 queued turnoverload returns busyCODEPrompt guidanceCodeGemini server configFocal layer
Six layers guard a voice turn. The top layer is a prompt; the other five are code or server settings, so they hold even when the prompt is talked around.

Test the path, not just the function

With unit tests green, I ran the whole stack locally with the real Gemini Live service and fake tools. That run caught five integration issues before any user could meet them, including a consent gate that had to be wired on a second code path. Unit tests check parts; the full path checks how they connect.

The same run drove a protocol upgrade. I replaced timer-based turn tracking in the clients with explicit frames for turn accepted, queued, done and interrupted, so the client reports what happened rather than inferring it.

Green unit tests said the parts worked. Only the full path said the product did.

How it was built

I used the AI-native SDLC I built and my team runs: intent, then a specification that went through many revisions with numbered concerns, then a plan where every step names its proof. The plan review uses an independent verifier. Work then fans out as parallel lanes, one per repository, each in its own worktree, with a deviations log recording every departure from the plan.

Spikes come before code. The one-day Live API spike is why cancellation is a designed feature and not a late patch. Operationally, a deploy drains sessions with a shutdown notice, then the client resumes through a stored handle on another instance. Quota and rate limits are re-checked on every turn.

325 backend tests and 533 agent-service voice tests pass.

Tech stack

  • Gemini Live
  • WebSocket
  • PCM16 audio
  • FastAPI
  • LangGraph
  • Redis
  • React
  • React Native
  • Expo
  • Native iOS audio engine
  • OpenTelemetry