# Designing Voice Agents for Real Conversations

Chintan Agrawal & Daniel Wirjo, AWS | AI Engineer World's Fair 2026 | 32:57

Source: https://www.youtube.com/watch?v=hMlLw1LeIK8
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/designing-voice-agents-for-real-conversations
Published: 2026-07-20
Tags: latency, voice

## TL;DR
- Voice turn-taking is mainly an audio engineering problem, because a strong LLM still produces a poor experience when the pipeline misses interruptions or waits too long.
- Silero VAD, STT-provider turn detection, and local Smart Turn models offer increasing control and context, with different trade-offs in latency, portability, and transparency.
- Cloud voice pipelines commonly take roughly 1,100 to 1,300 milliseconds, so interruption handling and turn detection must make the delay feel acceptable.

## Summary
Chintan Agrawal and Daniel Wirjo explain why voice agents need different engineering from chat agents. Human turn switches happen in about 200 milliseconds, while delays around 800 milliseconds already feel awkward. The talk maps turn-taking onto a pipeline containing VAD, STT, the LLM, TTS, and an interruption handler. It compares three approaches: a locally controlled Silero VAD timer, turn detection provided by an STT service, and local Smart Turn layered on top of VAD. The first is simple but cannot understand whether silence means a completed thought, a pause, or an acknowledgement. The second uses audio and language context but hides its decisions. The third adds portability and a fallback timer. The speakers also break down latency, explain why LLM time-to-first-token and STT consume much of the budget, and show a Pipecat demo using all three configurations. Production concerns include false interruptions, long-turn prompt drift, and operating several independently scaling systems.

## Key ideas
### Voice turn-taking is an audio pipeline problem
[00:03](https://www.youtube.com/watch?v=hMlLw1LeIK8&t=3s)
Chintan Agrawal says a voice agent can use the same model and prompt in two very different interactions. In one example, a user starts to say, "I want to fly," then tries to correct the sentence, but the agent continues for almost two seconds. In the better version, the pipeline detects the interruption in under 200 milliseconds and backs off. The difference comes from the audio pipeline, which must notice that another person has started speaking and stop the agent. The LLM is identical in both cases. This makes turn-taking a separate engineering concern from model quality.

### Voice agents have a much smaller timing budget than chat agents
[00:42](https://www.youtube.com/watch?v=hMlLw1LeIK8&t=42s)
The speakers use 200 milliseconds as the target for human turn switching. At about 800 milliseconds, a response starts to feel wrong, and at 1.5 seconds a user may hang up. Chintan compares this with chat agents, which can often take around five seconds without the same effect. He cites a measured 755-millisecond response time from Salesforce as an example that is still nearly four times slower than natural turn switching. When the full target cannot be reached, accurate turn-taking can still make the delay feel less disruptive.

### VAD controls the basic turn boundary, but it cannot understand intent
[03:49](https://www.youtube.com/watch?v=hMlLw1LeIK8&t=229s)
Voice activity detection sits at the front of the pipeline and answers whether someone is speaking. Silero VAD uses raw audio and spectral features, with convolutional layers, an LSTM, and a small model footprint. Its minimum-silence setting strongly changes the experience. A low value can cut off users who are thinking, while a high value creates dead air. Chintan gives 200 milliseconds as an example for a sales agent and 1,000 to 1,200 milliseconds for a domain that needs more time. VAD cannot tell whether a pause means a completed sentence, a breath, an incomplete thought, or a backchannel acknowledgement.

### STT turn detection adds context while hiding the decision
[08:18](https://www.youtube.com/watch?v=hMlLw1LeIK8&t=498s)
The second approach lets the speech-to-text provider decide when a turn has ended. The service receives streaming audio, performs transcription, and emits a turn-complete event. Chintan mentions Cartesia and Deepgram endpointing, with reported P50 latencies of about 300 and 250 milliseconds. These systems can use the complete audio signal and linguistic context, which gives them more information than VAD alone. The trade-off is limited visibility. When the service cuts a user off at the wrong moment, the application cannot inspect the internal reason because the decision happened on the provider's server.

### Smart Turn combines a local fallback with a model that reads pauses
[09:49](https://www.youtube.com/watch?v=hMlLw1LeIK8&t=589s)
The third approach keeps local VAD for the basic speech signal and adds Smart Turn during silence. Chintan gives Smart Turn V3.2 a recall of 58.9 percent and precision of 68.4 percent. When the model is confident that a sentence has ended, the agent can respond quickly. When it is not confident, the VAD timer remains as a fallback, so the user is not left waiting indefinitely. Chintan also mentions a Meta paper with 87.7 percent recall, but says its code was not released. Smart Turn is described as an 8 MB BSD-2 licensed model that can be installed with pip.

### Interruption handling should classify what the user said
[11:46](https://www.youtube.com/watch?v=hMlLw1LeIK8&t=706s)
Pipecat mechanically handles a barge-in by detecting speech, flushing downstream audio, stopping TTS, and cancelling LLM generation. Chintan says the pipeline can be ready for new input within about 50 milliseconds after the interruption is handled. The harder question is whether every sound should stop the agent. An acknowledgement such as "Yeah" may mean the agent should continue, while "Okay, wait, no. That was wrong" requires an immediate stop. Filler sounds, coughs, and background noise may be ignored or handled differently. Owning the classification makes these policies possible.

### The three implementations share a pipeline and change the turn decision
[13:20](https://www.youtube.com/watch?v=hMlLw1LeIK8&t=800s)
The three Python examples use nearly the same voice pipeline. The first passes a Silero VAD analyzer and ends a turn after a configured silence period. The second replaces that analyzer with a turn-detection service from the STT provider. The third keeps Silero VAD and adds a local Smart Turn analyzer that examines prosody and intonation during silence. The code structure stays similar, but the configuration produces different behavior. This lets teams change the turn-taking method without rebuilding the rest of the STT, LLM, and TTS pipeline.

### Cloud components usually push voice-to-voice latency above one second
[14:54](https://www.youtube.com/watch?v=hMlLw1LeIK8&t=894s)
The latency breakdown includes about 40 milliseconds for audio capture, 52 milliseconds for network and jitter buffering, around 300 milliseconds for transcription and endpointing, 500 to 650 milliseconds for LLM time to first byte, and roughly 90 to 120 milliseconds for outbound network and TTS playback. That produces about 1,100 to 1,300 milliseconds in a standard cloud API setup. A co-located GPU deployment was demonstrated at about 500 milliseconds by removing internal network hops. Chintan says STT and the LLM together consume about two-thirds of the latency budget, so they are the main places to improve response time.

### Voice systems must be judged by tail latency and long conversations
[18:18](https://www.youtube.com/watch?v=hMlLw1LeIK8&t=1098s)
The speakers benchmark LLMs for time to first token, using an under-700-millisecond target. Their June 2026 observations put Nemotron-3 Ultra at 529 milliseconds P50 and GPT-4.1 at 536 milliseconds P50. P95 matters more in voice because one slow turn can damage the interaction. GPT-4.1 reached 1.7 seconds at P95, while Claude 3 went above four seconds in the cited comparison. Long sessions create another problem: after 15 or 20 turns, models may ignore parts of the system prompt or become too verbose. Context pruning or session resets may then be needed.

## Notable quotes
- "They are not LLM problems because you can have the perfect model, perfect track but the experience still might feel broken if the turn taking is off." (00:21)
- "The difference is purely because of the audio pipeline." (02:18)
- "There is no universally right answer. It depends on the domain." (06:48)
- "One slow response and your entire flow is gone." (19:18)
- "The code-wise the same pipeline, but the configuration changes, which leads to completely different behavior." (14:07)

## Tools & references mentioned
- AWS
- Pipecat
- Daily.co
- Silero VAD
- Smart Turn V3.2
- Cartesia
- Deepgram
- Deepgram Nova 3
- Meta
- Nemotron-3 Ultra
- GPT-4.1
- Claude 3
- Picovoice Cloud
- Whisper
- AssemblyAI
- Anthropic Claude

## Who should watch
- You are building a real-time voice agent and need to decide whether silence timers, provider turn detection, or a local model fits your product.
- Your agent cuts users off, leaves dead air, or fails to stop when someone corrects it. The talk gives concrete causes and implementation patterns.
- You need to estimate whether a cloud STT, LLM, and TTS pipeline can meet your latency target and understand what happens when conversations run for many turns.