# How to Build Voice Agents That Don't Fall Apart When You Interrupt

Jesse Hall, LiveKit | AI Engineer NYC 2026 | 20:55

Source: https://www.youtube.com/watch?v=OmyqelTwpOo
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/how-to-build-voice-agents-that-dont-fall-apart-when-you-interrupt
Published: 2026-10-26
Tags: evals, latency, testing, voice

## TL;DR
- Voice agents should be evaluated as complete conversations under realistic conditions, rather than by comparing individual model leaderboard scores.
- Time to first audio and time to first sentence are better measures of perceived responsiveness than time to first byte or time to first token.
- Async tool calls, audio-aware turn detection, and repeated simulations help an agent keep talking naturally while back-end work and interruptions happen.

## Summary
Jesse Hall explains why voice agents need different engineering and evaluation methods from background agents or ordinary chatbots. A voice system has to handle interruptions, know when a caller has finished speaking, respond inside a limited latency window, and continue while back-end tools run. Hall argues that model leaderboards are a poor basis for choosing a stack because they score components separately, while a deployed agent is a complete conversation. He recommends measuring time to first audio and time to first sentence, then testing a complete agent with simulated callers who interrupt, change their minds, mumble, and speak over background noise. The same scenarios can be run with different model combinations and checked against actual outcomes, such as a correct reservation or payment. The talk also covers LiveKit's audio-aware turn detector and async tool calls that let the agent keep the conversation moving.

## Key ideas
### Real-time agents fail when interruptions break the conversation
[00:01](https://www.youtube.com/watch?v=OmyqelTwpOo&t=1s)
Background agents can take extra seconds or hours because the user walks away and returns later. Voice agents have to respond while the caller is still present. Hall describes calling a pharmacy, airline, or car dealership, speaking over the system, and watching the conversation fall apart. In financial services, the failure can be more serious. A caller might change a payment amount or date midway through the call, and the agent has to keep up. The target is a system that feels human when the caller interrupts, changes direction, or causes confusion.

### A voice stack combines several models with different timing behavior
[03:46](https://www.youtube.com/watch?v=OmyqelTwpOo&t=226s)
Hall describes the traditional cascaded pipeline as speech-to-text, an LLM, and text-to-speech working together. Speech-to-speech models can replace that pipeline, but he says they are newer and do not fit every use case yet. The number of available models and providers makes selection difficult. A transcription error can propagate through the whole system. If speech-to-text hears 'the 30th' when the caller said 'the 3rd,' the LLM plans for the wrong date and the error reaches the back end. The stack therefore has to be judged as a working system.

### First audible sound matters more than first byte or first token
[06:46](https://www.youtube.com/watch?v=OmyqelTwpOo&t=406s)
Hall says a text-to-speech provider may report a fast time to first byte while leaving the caller with silence afterward. He gives an example of providers returning up to three quarters of a second of silence after that first byte. The useful measure is time to first audio, which tells you when the caller actually hears something. For language models, time to first token has the same problem. Text-to-speech needs enough text to begin speaking, so a fast token stream does not necessarily produce a fast spoken response. Hall recommends measuring time to first sentence instead.

### Async tool calls let the agent speak while back-end work continues
[08:38](https://www.youtube.com/watch?v=OmyqelTwpOo&t=518s)
Slow tools cannot be hidden by silence. Hall says a human would tell the caller that they are looking something up or ask another question while the system works. An async tool call lets the conversation and the tool stop sharing a thread. In the hotel demonstration, the agent saves booking details in the back end while immediately asking for the next piece of information. If a tool returns an error, the agent can verify or correct the information, and it can hand the caller to a human when necessary.

### Turn detection needs audio signals as well as transcript text
[12:33](https://www.youtube.com/watch?v=OmyqelTwpOo&t=753s)
Hall says LiveKit's turn detector listens to the audio itself, including rhythm, pitch, and intonation, rather than relying only on the transcript. That helps it distinguish a completed thought from a pause inside a longer amount. His payment example is a caller saying, 'I'd like to pay 500 and $55.23.' A detector that sees '500' and then a pause may interrupt too early. An audio-aware detector can hear that the caller is still speaking. Hall reports 9.9% interruptions for a leading alternative and 4.5% for LiveKit in the stated use cases.

### Every voice response fits inside a finite latency window
[14:02](https://www.youtube.com/watch?v=OmyqelTwpOo&t=842s)
The latency window runs from the caller's last syllable to the agent's first audible response. Turn detection decides that the caller is done, the language model prepares a response from a provisional transcript, and text-to-speech starts generating audio. Hall says the conversation still holds at roughly 1.5 seconds, although it begins to feel off there. At 500 or 600 milliseconds, the exchange starts to feel truly human-like. The engineering task is to fit the most natural behavior possible inside that fixed window, rather than optimizing one isolated component.

### Simulated callers reveal which stack fits a specific agent
[15:44](https://www.youtube.com/watch?v=OmyqelTwpOo&t=944s)
Hall recommends taking a complete working agent with a real back end and testing it with simulated callers. The callers can pursue goals, change their minds, interrupt, mumble, and speak with background noise. Text simulations test business logic, while audio simulations test speech recognition, turn-taking, and the full voice pipeline. Scenarios define the caller's goal, the actions the agent should take, and the expected outcome. The team then swaps model stacks while keeping the input identical, checks the actual result, and builds a benchmark for that particular use case.

### Pre-launch checks must include outcomes and regression cases
[18:35](https://www.youtube.com/watch?v=OmyqelTwpOo&t=1115s)
Hall's checklist includes setting the latency budget, measuring time to first audio and time to first sentence, confirming financial dates and amounts, and verifying identity before account access. The agent should keep listening while slow tools run. Tests should check the final state, privacy safeguards, unnecessary tool calls, and turn-taking, rather than completion alone. When a failure appears, the team should turn it into a regression scenario and run it repeatedly. This catches cases where an agent completes a task through the wrong steps or invents a function call.

## Notable quotes
- "We need to be able to score them together." (05:08)
- "A byte does not equal audio." (07:19)
- "This is allowing the conversation and the tool to stop sharing a thread." (09:48)
- "A voice agent isn't just a chatbot with a speaker attached." (19:10)

## Tools & references mentioned
- LiveKit
- OpenAI
- ChatGPT advanced voice mode
- Gemma 4 31B
- LiveKit Inference

## Who should watch
- You are choosing speech, language, and text-to-speech models for a production voice workflow and need to evaluate the whole stack.
- Your agent handles payments, reservations, identity, or other actions where interruptions and wrong details can cause real damage.
- You want a practical test plan for latency, turn-taking, async tools, and simulated callers before launch.

## Editor's note

Jesse Hall says teams should turn each voice-agent failure into a regression scenario and run it repeatedly. Kitaru records a real agent run, including every model call and tool result, then replays the code against the same inputs and responses. That lets a team check a fix against the failed run without touching real systems.

Written by the AIE Talks editors (the Kitaru team), not by the speaker.

## Related talks

- [Designing Voice Agents for Real Conversations](https://aietalks.com/talks/designing-voice-agents-for-real-conversations) (Chintan Agrawal & Daniel Wirjo, AWS, 32:57)
- [Building Effective Voice Agents](https://aietalks.com/talks/building-effective-voice-agents) (Toki Sherbakov & Anoop Kotha, OpenAI, 17:17)
- [Voice Agent Engineering](https://aietalks.com/talks/voice-agent-engineering) (Nik Caryotakis, SuperDial, 19:07)
- [5 Voice Agent Failure Modes You'll Hit in Week One](https://aietalks.com/talks/5-voice-agent-failure-modes-youll-hit-in-week-one) (Venky B, Plivo, 26:46)
- [Voice Agents: the good, the bad, and the ugly](https://aietalks.com/talks/voice-agents-the-good-the-bad-and-the-ugly) (Eddie Seagull, Fractional AI, 18:48)
