# Latency Is a Budget. Humanlike Is the Goal. Jesse Hall, LiveKit | AI Engineer World's Fair 2026 | 17:01 Source: https://www.youtube.com/watch?v=j4AmIK6OvHc Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/latency-is-a-budget-humanlike-is-the-goal Published: 2026-10-11 Tags: benchmarks, evals, latency, voice ## TL;DR - A voice agent is a pipeline, so model leaderboards do not predict how the complete system handles real callers. - Perceived latency is the time until the caller hears audio, and around 600 milliseconds starts to feel human while 1.5 seconds is close to the limit. - Agents should keep the conversation moving during tool calls, use audio-based turn detection, and test complete conversations against human-like outcomes. ## Summary Jesse Hall argues that voice agents should be designed around human-like conversation rather than raw model speed. A voice system combines speech-to-text, an LLM, and text-to-speech, so a strong score for one component says little about the shipped pipeline. A transcription mistake can send every later step in the wrong direction. Hall also separates dashboard latency from what callers hear. Time to first byte may include hundreds of milliseconds of silence, while the useful measure is time to first audible audio. His design uses audio-based turn detection, provisional transcripts, complete first sentences, and asynchronous tools that let the agent keep talking while work happens. He gives a practical latency window, with 1.5 seconds as the edge and about 600 milliseconds as the point where interaction starts to feel human. LiveKit's open-source benchmark evaluates complete agents with simulated interruptions, mistakes, and changing requests. ## Key ideas ### Human-like behavior requires more than faster responses [01:03](https://www.youtube.com/watch?v=j4AmIK6OvHc&t=63s) Hall says speed alone cannot make a voice agent feel natural. A useful agent listens while it talks, distinguishes a completed thought from someone still thinking, and handles a caller who changes their mind. He describes a common failure: people talk over voice agents, and the agent falls apart. The usual advice is to choose faster models and remove another 100 milliseconds, but Hall says the real work is choosing models and configuring the pipeline. The goal is an agent that responds in a way people recognize as conversational, rather than one that merely returns audio quickly. ### A voice agent is a pipeline whose errors compound [02:57](https://www.youtube.com/watch?v=j4AmIK6OvHc&t=177s) LiveKit's voice pipeline has speech-to-text, an LLM, and text-to-speech working together. Real-time models add another choice, but they are not suitable for every use case. Hall's example shows why component leaderboards are misleading: a caller says they want an appointment on Tuesday the 3rd, while speech-to-text hears Tuesday the 30th. From then on, even a capable language model may confidently book the wrong date. The input used for many scores is clean audio in a quiet room, while real callers may use speakerphones in traffic with children nearby. The relevant question is which complete stack to ship. ### Measured latency is different from the delay a caller hears [05:18](https://www.youtube.com/watch?v=j4AmIK6OvHc&t=318s) Hall warns that dashboards can report an attractive time to first byte while the audio still begins with silence. He says LiveKit measured popular providers sending hundreds of milliseconds of silence, with the worst case reaching three quarters of a second before actual audio. Time to first audio is therefore more useful than time to first byte. The same problem appears with language models. Time to first token does not tell you when text-to-speech can speak, because TTS needs a complete sentence. A model that streams tokens fastest may still finish the first sentence later than another model. ### Agents should buy time while tools run [07:12](https://www.youtube.com/watch?v=j4AmIK6OvHc&t=432s) Real agents need to look up orders, check availability, and book rooms. Those tools may take two, ten, or forty seconds, and model selection cannot hide a long tool delay. Hall compares the desired behavior to a receptionist saying, 'Give me a sec. I'm looking up your account.' In the code he shows, the first tool update hands the microphone back to the agent while the tool continues in the background. The conversation and the tool no longer block each other. His hotel demo keeps talking without awkward pauses, handles an interruption, and accepts a caller changing the dates before completing the booking. ### Audio turn detection lets the system respond before the transcript is complete [09:58](https://www.youtube.com/watch?v=j4AmIK6OvHc&t=598s) Humans manage turn taking within a few hundred milliseconds, largely from tone. A pipeline that waits for the full transcript is already late. LiveKit's turn detector listens to rhythm, pitch, and intonation in the audio itself. Hall reports that, at the same latency budget, the best alternative cut users off 9.9% of the time while LiveKit's system cut them off 4.5% of the time. End-of-turn detection can fire before the transcript arrives, allowing the language model to run on a provisional transcript. If the final words match, the system saves roughly half a second. ### The useful latency window ends at 1.5 seconds and feels human near 600 milliseconds [11:30](https://www.youtube.com/watch?v=j4AmIK6OvHc&t=690s) Hall defines the important interval as the gap between the caller's last syllable and the agent's first audible audio. TTS should start on the first complete sentence, rather than waiting for the last token. Each model contributes to this fixed window. Hall says an end-to-end delay within 1.5 seconds can still hold a conversation, although it is close to the edge. At around 600 milliseconds, the interaction starts to feel really human. The design question is therefore how much human-like behavior can fit inside the budget, rather than how fast one component can become in isolation. ### Complete-agent benchmarks should test outcomes under realistic conversation [12:19](https://www.youtube.com/watch?v=j4AmIK6OvHc&t=739s) LiveKit's benchmark starts with a complete working agent, including the hotel receptionist's real backend, and runs simulated callers against it. The callers interrupt, mumble, and change their minds. The benchmark checks whether the correct reservation reached the database, whether the caller was cut off, whether names and dates were correct, and whether the agent invented tool calls or leaked personal data. Completion alone is insufficient because an agent can guess its way to a completed task. The benchmark also penalizes unnecessary questions and taking ten steps for work that should take two. The same scenarios and byte-for-byte inputs can be used while swapping model stacks. ### The checklist puts latency and conversation quality ahead of isolated model scores [15:53](https://www.youtube.com/watch?v=j4AmIK6OvHc&t=953s) Hall's checklist starts with setting a latency budget and measuring perceived latency. Any remaining budget should go toward model quality, rather than speed that does not affect what the caller hears. Work taking more than a second should run asynchronously so the conversation does not stop. Teams should test the whole conversation instead of individual components and continue testing in production. LiveKit does not make the speech-to-text, LLM, or text-to-speech models in its stack, so Hall presents the benchmark as a way to compare providers in the context of an actual agent. ## Notable quotes - "The goal is being the voice AI being as human-like as possible." (01:26) - "Leaderboards score components, but you are shipping a cascade, a pipeline." (04:04) - "Bytes are not sound." (05:38) - "If you can stay within a second and a half end to end, the conversation still holds, but you're right on the edge." (11:34) - "A voice agent isn't a chatbot with a speaker stacked onto it." (15:49) ## Tools & references mentioned - LiveKit - ChatGPT advanced voice mode - OpenAI - Gemma 4 31B - LiveKit Inference ## Who should watch - You are shipping a voice agent and your dashboard latency looks good, but callers still interrupt it or describe it as robotic. - You need to choose speech-to-text, LLM, and text-to-speech providers without trusting isolated leaderboard scores. - You want an evaluation setup that tests interruptions, changing requests, tool calls, dates, names, and database outcomes on your own agent. ## Related talks - [Engineering voice agents: Latency, quality, and scale](https://aietalks.com/talks/engineering-voice-agents-latency-quality-and-scale) (Rishabh Bhargava, Together AI, 24:35) - [Designing Voice Agents for Real Conversations](https://aietalks.com/talks/designing-voice-agents-for-real-conversations) (Chintan Agrawal & Daniel Wirjo, AWS, 32:57) - [Your Voice Agent Doesn't Need a Frontier Model](https://aietalks.com/talks/your-voice-agent-doesnt-need-a-frontier-model) (Joel Allou & Ornella Bahidika, Microsoft, 05:45) - [Giving a Voice to AI Agents](https://aietalks.com/talks/giving-a-voice-to-ai-agents) (Scott Stephenson, Deepgram, 13:08) - [Realtime Voice Agents with Frontier Intelligence](https://aietalks.com/talks/realtime-voice-agents-with-frontier-intelligence) (Bohan Li, EliseAI, 13:14)