Voice agents need low time to first audio, so Venky B recommends disabling long-form model thinking and considering self-hosted open-source models for latency-sensitive calls.
2
Transcription should be treated as unreliable, with dynamic keyword boosting, LLM post-processing, and transliteration used to clean inputs before they reach the agent.
3
Data collection works better when each answer has a typed field, validation rules, confirmation steps, and field-level evaluations instead of one large end-to-end test.
Summary
Venky B describes what breaks when a voice agent moves from a working demo into production. Latency is the first problem: teams often aim for under 550 milliseconds from the end of a user's speech to the first audio, but many land between 750 and 1,200 milliseconds. He argues that voice agents usually need thinking turned off, then recommends choosing models and infrastructure around token speed, language coverage, and tool calling. Transcription is also unreliable with names, phone numbers, domain terms, accents, and code-switched language. His proposed fixes include dynamic keyword boosting, LLM post-processing, and transliteration. For data collection, he recommends typed fields with validators and confirmation flows. He says this change raised accuracy from about 30 percent to the mid-90s without fine-tuning. A normalization layer should also clean LLM output before speech synthesis, while turn detection, barge-in, and backchanneling need separate attention.
Voice agents fail when demo latency meets real callers
Venky B says voice agents often sound fine during development, then fail when moved into production. The first production issue is the delay between a caller finishing and the agent starting to speak. Teams commonly target less than 550 milliseconds, but most end up between 750 milliseconds and 1.2 seconds. Calls with more than 1.2 seconds of delay can lead users to hang up. He frames the design problem as a balance between cost, intelligence, and latency because improving one can affect the others.
Long-form model thinking usually has to be disabled for calls
Venky B argues that many recent language-model gains came from allowing models to think longer, but that behavior does not fit a voice interaction where the response must start quickly. Frontier models can have roughly 450 to 500 milliseconds of time to first token on a good day, with higher-percentile latency rising above 1.2 seconds. Dedicated fast inference capacity can reduce delay, but he says it is expensive and may require booking capacity far in advance.
Self-hosted open-source models can balance speed, cost, and capability
For production agents targeting under 300 milliseconds, Venky B recommends considering open-source models hosted on a team's own GPUs. He names Qwen 3.5 and Gemma 4 as models his team has benchmarked. Gemma 4 works better for multilingual use in his experience, partly because it needs fewer tokens to produce a word in a language. He says three- or four-billion-parameter mixture-of-experts models can work without fine-tuning, while teams that need domain fine-tuning should start with at least an eight- or twelve-billion-parameter model.
Transcription should be treated as brittle even when the vendor looks accurate
Venky B says teams should assume transcription will fail even with a strong engine. He cites four to six percent word error rates on known evaluation sets, with real noisy calls and domain vocabulary often reaching double digits. The failures include proper nouns, jargon, phone numbers, addresses, and accents. Code-switched language creates another problem when the transcription uses a different script or writes one language in another language's alphabet. The resulting errors can spread through the language model and into speech synthesis.
Dynamic keyword boosting and post-processing repair predictable transcription errors
Static keyword lists can pollute a transcription engine's context and cause it to hallucinate terms. Venky B recommends adding relevant keywords dynamically during the part of the call where they are likely to occur. He also recommends sending transcripts through an LLM with domain context. That step can correct errors such as a letter being transcribed inside a phone number when the intended value is a digit. For multilingual calls, a transliteration layer can normalize text before it reaches the language model.
Typed fields make voice data collection easier to validate
Venky B says data collection causes serious problems in roughly half to 60 percent of AI agent deployments. He recommends treating a phone number, date, name, or address as a typed field rather than asking an LLM to interpret an unconstrained transcript. A phone number field has a length and validation rules. If a transcription produces an unexpected letter, the system can correct it with confirmation or ask the caller to repeat it. He compares this approach with Python data classes, Pydantic, Zod, and form fields.
Field-level unit tests are more useful than large end-to-end evals
Relative values such as "next week Wednesday at 8" become manageable when the system knows it is collecting a datetime field and can resolve the value against the current date. Venky B recommends evaluating each field as a unit test instead of running hundreds of full agent conversations to find one broken collection step. He says this structured approach helped his team reach roughly 95 to 97 percent accuracy without fine-tuning, by defining the agent's current state and the rules for that field.
LLM output needs normalization before it reaches speech synthesis
Venky B advises putting a normalization layer between the language model and text-to-speech system. It should remove emojis and markdown, apply pronunciation dictionaries for names, brands, and acronyms, and normalize emails, currencies, and dates. The agent can slow down around a difficult entity, such as a phone number or a name spelled letter by letter. He tests voice systems by checking whether they can pronounce his surname and Plivo correctly, since failures on those names expose weaknesses in pronunciation control.
Turn detection, barge-in, and backchanneling need explicit design
Venky B closes with turn detection, barge-in, and backchanneling as additional production concerns. He does not go into the same detail as the earlier failure modes because he is out of time. He says teams can build these behaviors within speech-to-speech pipelines and do not necessarily need a speech-to-speech model to support them.
"The irony with voice agents is like almost always the LLM or the agent that's talking has to have thinking turned off."06:56
Who should watch
You are moving a voice agent from a proof of concept into production and need to understand why a working pipeline starts failing on real calls.
Your agent collects names, phone numbers, addresses, dates, or other structured values from callers, and end-to-end tests are not showing which field is broken.
Your system handles multiple languages or sends language-model output directly to text-to-speech without a normalization layer.
