# Engineering voice agents: Latency, quality, and scale

Rishabh Bhargava, Together AI | AI Engineer Europe 2026 | 24:35

Source: https://www.youtube.com/watch?v=N7b1PJc7SFc
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/engineering-voice-agents-latency-quality-and-scale
Published: 2026-05-31
Tags: deployment, inference, latency, voice

## TL;DR
- Voice agents must keep response latency below the point where users notice delays or hang up, while also handling complex instructions, tool calls, natural speech, and concurrent traffic.
- A production voice agent usually combines streaming speech to text, an LLM, and text to speech, with the LLM taking most of the latency and cost budget.
- Co-locating the orchestrator and models can reduce network latency substantially, while a small thinker model can delegate complex work to a larger model without slowing the fast path.

## Summary
Rishabh Bhargava describes voice agents as a real-time systems problem with four simultaneous requirements: low latency, enough intelligence for complex workflows, natural speech, and reliability at scale. He explains the common cascading pipeline, where streamed audio passes through speech to text, an LLM, and text to speech. Each component has different measures, from word error rate and transcript completion time to time to first token, time to first audio, and real-time factor. The LLM usually takes most of the budget, which limits model size. Network placement also matters. In his example, reducing network latency from 75 milliseconds to 5 milliseconds produces a large improvement in an already optimized system. Bhargava then compares this approach with pure speech-to-speech models, which can preserve tone and support interruptions more naturally but still struggle with instruction following and tool calls. For complex workflows, he describes a thinker-talker pattern that keeps a small model on the conversational path and delegates difficult work to a larger one.

## Key ideas
### Voice agents have to satisfy several constraints at once
[02:28](https://www.youtube.com/watch?v=N7b1PJc7SFc&t=148s)
Bhargava says a useful voice agent must be real time, smart enough to complete real work, natural enough to sound pleasant, and reliable under concurrent load. People notice responses that take more than 500 milliseconds, and they may hang up after one or two seconds. Complex workflows also require good instruction following and tool calling. Naturalness includes language coverage, accents, name pronunciation, and emotional delivery. A system that works for one demo call still has to work for 100, 1,000, or 10,000 concurrent calls. These requirements are coupled, so improving one part does not remove the need to meet the others.

### The production pipeline streams audio through three model stages
[04:20](https://www.youtube.com/watch?v=N7b1PJc7SFc&t=260s)
The common architecture streams audio from the user to an agent orchestrator such as Pipecat, LiveKit, or a homegrown system. Speech to text converts the audio into text. An LLM decides whether to make a tool call and produces a response. Text to speech converts that response into audio chunks that stream back to the user. Bhargava presents this cascading design as the dominant production pattern because each stage can be measured and controlled separately. The trade-off is that the system has to coordinate several models and account for the latency added between them.

### Speech to text needs both accurate words and fast turn completion
[05:14](https://www.youtube.com/watch?v=N7b1PJc7SFc&t=314s)
Bhargava gives word error rate as the main speech-to-text quality measure and says state-of-the-art models are typically around 6% on open benchmarks. Errors in names, drug names, or other important terms propagate through the LLM and text-to-speech stages. He also tracks the time from the end of an utterance until a complete transcript is ready for the LLM, with a P90 target around 100 milliseconds for some Together models. Turn detection remains difficult because a pause may mean either that the speaker has finished or that they are continuing. Starting audio too early creates an interruption that feels unnatural.

### Streaming-native speech models avoid the workarounds required by Whisper
[07:12](https://www.youtube.com/watch?v=N7b1PJc7SFc&t=432s)
Bhargava contrasts batch speech models with newer streaming-native architectures. Whisper was trained on 30-second audio clips, so real-time systems have had to add chunking, silence padding, repeated calls, and transcript stitching. He describes a newer Nvidia model whose encoder uses different amounts of look-ahead, from roughly 80 milliseconds to up to a second, instead of waiting for 30 seconds. The encoder also caches activations as audio advances in small frames, so it avoids repeating the heaviest computation. This design is intended for streaming conversations rather than offline transcription.

### The LLM dominates the latency budget and constrains model size
[08:26](https://www.youtube.com/watch?v=N7b1PJc7SFc&t=506s)
For the LLM stage, time to first token is the main performance measure because the first tokens can immediately feed the text-to-speech model. Bhargava says 200 to 300 milliseconds is a useful target range. That pushes teams toward models in the 8 to 30 billion parameter range. Larger models can consume the latency budget, while smaller models may lose the intelligence and tool-calling ability needed for meaningful tasks. He places the LLM ahead of text to speech and speech to text in both the rough latency and cost budgets.

### Text to speech must start quickly and keep pace with playback
[09:27](https://www.youtube.com/watch?v=N7b1PJc7SFc&t=567s)
Text to speech is measured by time to first audio and real-time factor. A real-time factor below one means the system can produce audio faster than that audio is played, which helps avoid buffering. Bhargava says listening to samples remains the best way to judge whether a voice has the right quality for a particular customer experience. Other requirements include naturalness across voices and languages, accurate pronunciation of names and products, and some control over emotion. Some models accept tags such as happy, angry, or sad, although this capability is still developing.

### Model placement can matter as much as model speed
[13:18](https://www.youtube.com/watch?v=N7b1PJc7SFc&t=798s)
Bhargava describes a system with engine latency in the intended 100 to 200 millisecond range that still suffers from 75 milliseconds of network latency between its orchestrator and models. Moving the components into the same data center, or even the same building, can reduce that network delay to about 5 milliseconds. In his example, this changes the overhead from roughly 30% to around 5% in an already optimized setup. He recommends detailed observability because small delays accumulate across a real-time pipeline. Global deployment also helps place models near users and supports regional data residency requirements.

### Speech-to-speech models preserve conversational signals but still need better tool calling
[14:20](https://www.youtube.com/watch?v=N7b1PJc7SFc&t=860s)
A pure speech-to-speech model can replace the separate speech-to-text, LLM, and text-to-speech stages. Bhargava mentions OpenAI's real-time API and Nvidia's VoiceChat as examples. These systems can retain tone, emotion, and hesitation instead of reducing speech to text first. They may also support full-duplex interaction, back-channel responses such as 'I see' or 'aha', and more natural handling of interruptions. Their current weakness is instruction following and tool calling. Teams often spend time prompt engineering before returning to the more controllable pipeline architecture.

### A small thinker model can keep complex work off the fast path
[21:33](https://www.youtube.com/watch?v=N7b1PJc7SFc&t=1293s)
For workflows that need stronger reasoning or more guardrails, Bhargava describes a thinker-talker pattern. A small LLM handles the conversation and can answer with something like 'let me think about it' while issuing one tool call to a larger model. That larger model has the complex instructions, tools, and possibly additional guardrails, then returns a cleaner response for text to speech. This pattern keeps the conversational path responsive while reserving the larger model for difficult requests. Bhargava says the added components increase the need for independent scaling, clear latency budgets, and observability for every stage.

## Notable quotes
- "If it takes a second, if it takes 2 seconds, people will just hang up." (03:05)
- "The LLM is going to take up a majority of it followed by TTS followed by speech to text both from a latency and a cost perspective." (11:17)
- "That drop from 75 milliseconds to five basically gets you a 30% reduction in already a fairly optimized voice agent setup." (14:01)
- "You can't take back things that are spoken." (21:16)
- "The model will natively understand what was the tone, what was the emotion, was the user hesitant?" (15:32)

## Tools & references mentioned
- Together AI
- Refuel
- Cursor
- Deck
- ChatGPT Advanced Voice Mode
- Claude Code
- Pipecat
- LiveKit
- Whisper
- Nvidia
- OpenAI real-time API
- Nvidia VoiceChat

## Who should watch
- You are building a phone or conversational agent and need to set latency targets for speech to text, the LLM, and text to speech.
- Your current system works in a demo but becomes slow or unreliable when calls run concurrently or when models sit in different regions.
- You are evaluating speech-to-speech models and need to weigh their conversational benefits against weaker instruction following and tool calling.

## Related talks

- [Giving a Voice to AI Agents](https://aietalks.com/talks/giving-a-voice-to-ai-agents) (Scott Stephenson, Deepgram, 13:08)
- [Serving Voice AI at Scale](https://aietalks.com/talks/serving-voice-ai-at-scale) (Arjun Desai, Cartesia & Rohit Talluri, AWS, 17:05)
- [Building and Scaling an AI Agent Swarm of Low-Latency Real-Time Voice Bots](https://aietalks.com/talks/building-and-scaling-an-ai-agent-swarm-of-low-latency-real-time-voice-bots) (Damien Murphy, Deepgram, 1:07:23)
- [Designing Voice Agents for Real Conversations](https://aietalks.com/talks/designing-voice-agents-for-real-conversations) (Chintan Agrawal & Daniel Wirjo, AWS, 32:57)
- [Building Effective Voice Agents](https://aietalks.com/talks/building-effective-voice-agents) (Toki Sherbakov & Anoop Kotha, OpenAI, 17:17)