# Realtime Voice Agents with Frontier Intelligence

Bohan Li, EliseAI | AI Engineer World's Fair 2026 | 13:14

Source: https://www.youtube.com/watch?v=MBHOH1NmDqc
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/realtime-voice-agents-with-frontier-intelligence
Published: 2026-09-15
Tags: caching, latency, tool-use, voice

## TL;DR
- Bohan Li maps a cascaded voice agent onto the self-driving stack, with transcription as perception, the language model as planning, and speech synthesis as control.
- EliseAI combines a fast streaming transcriber with a slower corrective transcriber, then uses background agents to prepare tool results while the main model is generating.
- A prefix cache lets the agent begin speaking from previously generated audio while a speech provider produces the rest of the sentence with natural prosody.

## Summary
Bohan Li explains how EliseAI makes voice agents respond quickly while still using slow, intelligent language models. He compares the system with a self-driving car: transcription turns audio into usable data, the language model plans a response, and text-to-speech turns that response into action. The transcription layer combines a fast streaming engine with a slower, more accurate engine that can correct earlier text. Background agents prepare tool results, such as a caller's name and date of birth, while the main model starts generating. For speech, a prefix cache reuses audio for repeated phrases, then hands the full sentence to Cartesia so the remaining audio has natural prosody. Cached audio already played is removed from the provider's output. Li ends with a clinic booking call where the caller hears a normal conversation despite the work happening behind the scenes.

## Key ideas
### A voice agent can be designed like a self-driving system
[00:35](https://www.youtube.com/watch?v=MBHOH1NmDqc&t=35s)
Li breaks the voice stack into perception, planning, and control. In a self-driving car, perception uses cameras, lidar, and bounding boxes to turn real-world signals into data. In a voice agent, transcription does that job. The language model is the planning layer: it takes the transcribed input and decides what to say. The control layer turns that text into audio. Li says EliseAI applies separate speed improvements to each layer so the system can use highly intelligent models without making the caller wait for every step.

### A fast transcriber and a slower transcriber can correct each other
[02:06](https://www.youtube.com/watch?v=MBHOH1NmDqc&t=126s)
EliseAI places a fast streaming transcriber, Flux, alongside Scribe V2, a slower batch transcription system that uses more context. The streaming layer emits early text, while the accurate layer checks and corrects it. If newer audio arrives, an older corrective result is canceled. In Li's example, the question asks for a name and date of birth. Scribe V2 uses that context to identify which words belong to the name and which belong to the birth date. Punctuation does not matter to the agent, so the system releases the useful text after the relevant corrections are complete.

### Background agents prepare tool results before the caller finishes speaking
[03:55](https://www.youtube.com/watch?v=MBHOH1NmDqc&t=235s)
Slow language models can lose time when every tool call requires another round trip. Li's design uses background agents to make the tool call and place its result into the main agent's context. The main model can start generating from early transcriptions, but its response is held until the system knows the user has stopped speaking. In the example, an early generation that asks for spelling is canceled when the corrective transcription finds the caller's name and date of birth. The agent then runs name correction and phonetic matching before generating again with the needed context.

### Streaming speech can hide the time needed to finish a response
[05:59](https://www.youtube.com/watch?v=MBHOH1NmDqc&t=359s)
The agent produces text as a stream, so EliseAI tries to start audio before the full response exists. Li introduces a prefix cache that checks whether a sequence of words already has generated audio from an earlier generation in the call. The cache waits for several words before deciding it has a match. Repeated phrases such as "you said your name is" are likely to appear again, while a caller's particular name causes the cache to miss. When that happens, the cached opening can play while the remaining text continues through the speech provider.

### The speech provider can generate the full sentence while cached audio plays
[08:28](https://www.youtube.com/watch?v=MBHOH1NmDqc&t=508s)
Cartesia receives the full text and does not know that the prefix cache has already played part of it. That lets Cartesia generate the sentence with its normal prosody instead of generating only an isolated continuation. EliseAI suppresses the portion of Cartesia's audio that overlaps with the cached opening, then plays the remaining audio after the cache finishes. Li says there may be a small hiccup, but the goal is for the caller not to notice the handoff. The recorded clinic call demonstrates the result while background streaming and tool work happen out of earshot.

### The recorded clinic call combines all three optimizations
[10:21](https://www.youtube.com/watch?v=MBHOH1NmDqc&t=621s)
Li's example call starts with a pregnancy ultrasound request and moves through identity collection, insurance upload, appointment availability, and booking. The agent asks for a name and date of birth, confirms the caller is new, sends an insurance link, and offers appointment times at North Clinic. The caller pauses to check a calendar and then chooses an appointment. The conversation sounds ordinary, while the system is handling speculative transcription, background information gathering, streamed generation, and cached speech.

## Notable quotes
- "To me, cascaded voice agents makes a lot of sense when you view it in lens of kind of breaking it down into perception, planning, and controls." (00:35)
- "One way to get rid of that is by having background agents do the tool calling for you and kind of push the tools back into the context of the main agent so that it thinks it made the tool call, but it really didn't." (03:55)
- "At this point, the user hears the agent and user doesn't really know what's going on. They just looks like really fast response times to them." (08:08)
- "Cartesia has seen the entire transcript up to this point. It doesn't know about the existence of this prefix cache." (08:56)

## Tools & references mentioned
- EliseAI
- Flux
- Scribe V2
- Cartesia

## Who should watch
- You are building a cascaded voice agent and need to reduce perceived latency without replacing a capable language model.
- Your system receives partial transcriptions that can be corrected later, especially when names and other caller details matter.
- You want to stream speech from repeated response patterns while keeping the final audio prosody natural.

## Related talks

- [Your Voice Agent Doesn't Need a Frontier Model](https://aietalks.com/talks/your-voice-agent-doesnt-need-a-frontier-model) (Joel Allou & Ornella Bahidika, Microsoft, 05:45)
- [Giving a Voice to AI Agents](https://aietalks.com/talks/giving-a-voice-to-ai-agents) (Scott Stephenson, Deepgram, 13:08)
- [Engineering voice agents: Latency, quality, and scale](https://aietalks.com/talks/engineering-voice-agents-latency-quality-and-scale) (Rishabh Bhargava, Together AI, 24:35)
- [Designing Voice Agents for Real Conversations](https://aietalks.com/talks/designing-voice-agents-for-real-conversations) (Chintan Agrawal & Daniel Wirjo, AWS, 32:57)
- [Serving Voice AI at Scale](https://aietalks.com/talks/serving-voice-ai-at-scale) (Arjun Desai, Cartesia & Rohit Talluri, AWS, 17:05)