Speech-to-speech models now make low-latency voice applications practical at production scale.
2
Voice agents need different choices for architecture, prompts, tools, evaluations, and guardrails than text agents.
3
Teams should begin with observability and labelled examples, then add transcription, audio, and synthetic-conversation evaluations as their systems mature.
Summary
Toki Sherbakov and Anoop Kotha describe how voice agents are changing as speech-to-speech models become faster, more expressive, and better at handling interruptions. They compare a chained design, which connects transcription, an LLM, and text-to-speech, with a speech-to-speech design such as the real-time API. The latter reduces latency and preserves more conversational meaning, though the right choice depends on the application. Consumer applications usually value speed and expressiveness, while customer service systems place more weight on accuracy, integrations, and dependable tool use. Kotha recommends using a frontline voice agent that delegates harder tasks to other agents through tools. He also covers voice-specific prompting, limited tool sets, context summaries during handoffs, observability, labelled data, audio evaluations, synthetic conversations, and asynchronous guardrails. Examples from Lemonade and Tinder show how early evaluation work and brand-specific voice customization affected their launches.
Speech-to-speech models have reached a practical production threshold
Sherbakov contrasts older voice systems with current models. Earlier systems were slow, robotic, and brittle. In the demonstration, the model took a long time to respond and failed to follow an interruption asking for a poem about stars. The newer system responded much faster, sounded expressive, changed direction when interrupted, and followed the requested topic. Sherbakov says this combination of speed, expressiveness, and accuracy makes it possible to build more reliable voice applications at scale. He describes audio as reaching a point where it is becoming usable for production systems rather than remaining mainly an experimental modality.
A speech-to-speech architecture reduces latency and conversational loss
The established chained approach connects incoming audio to transcription, sends the text through an intelligence layer based on an LLM, and then converts the response back to speech. Sherbakov says this design is slower because it stitches together three models, and that meaning can be lost as the conversation passes between them. A speech-to-speech architecture combines those stages in one model. In OpenAI's example, the real-time API handles transcription, reasoning, and speech output. This simplifies the system, shortens the time before audio is returned, and preserves semantic understanding across the conversation.
Architecture choices depend on the application's trade-offs
Sherbakov groups the design questions around latency, cost, accuracy and intelligence, user experience, and integrations and tooling. A consumer application usually needs fast, attractive interaction and expressive answers, while cost and highly exact answers may matter less. A customer-service system has different requirements. It cannot confuse an order number or delete an order when the user asked to update it. Internal-system integrations, SIP, and phone providers also become more important. Sherbakov says the real-time API may fit cases where latency dominates, while a chained architecture can make sense when determinism and accuracy matter more.
Voice agents can delegate difficult work through tools
Kotha defines an agent as a model, its instructions, its tools, and the runtime or guardrails around execution. For voice systems, he describes a frontline real-time agent that handles routine interaction and delegates harder questions through tool calls. Those tools can reach agents powered by stronger models. In the return example, a smaller voice model handles the conversation while another model checks the return policy and supplies the result. This keeps the voice interaction responsive while giving specialized tasks access to a more capable agent. The agents SDK supports this delegation pattern.
Voice prompts control how the agent sounds as well as what it does
Text prompts mainly specify the behavior the model should follow. Voice prompts can also set the expressive qualities of the response. Kotha names demeanor, tone, and enthusiasm as controls that help define a brand's voice. He recommends experimenting with available voices and sample prompts because prompt wording can produce a large change in expressiveness and performance. Few-shot conversation examples can also be adapted to voice. A prompt can describe a greeting, give instructions for the user, and lay out the steps the agent should follow for a more complex interaction.
Small tool sets and summarized handoffs help preserve context
Kotha advises starting with a limited number of tools for each agent instead of attaching ten or twenty tools at once. More capabilities can be added as the system needs them, while separate agents handle separate tasks. When one agent hands work to another, the conversation context needs to remain available. His suggested pattern is to summarize the conversation state and pass that summary to the next agent. This avoids losing important context during the handoff. Delegation to another model can then happen through the agents SDK.
Voice evaluation should start with traces and labelled examples
Kotha says observability is the first step because teams need traces of agent activity, audio, and related events before they can improve the system. The next step is labelling data and using those labels to iterate on prompts. He describes this as the most effective approach his team has seen with customers, even when the process is not yet scalable. Transcription-based evaluations can then test function calling and business criteria with a rubric. Audio evaluations add information about tone, pacing, and intonation that is difficult to capture from text alone.
Asynchronous guardrails and synthetic conversations extend testing
Real-time API responses generate text faster than the corresponding speech, so Kotha recommends running guardrails asynchronously. Teams can choose how often to run them, such as after every 100 characters in the example shown. For broader testing, two real-time clients can simulate a conversation, with one acting as the agent and the other using customer personas. These synthetic conversations can produce many test cases, after which teams can evaluate both transcripts and audio. Kotha says this gives teams a fuller view of how the voice system behaves across simulated interactions.
"The models are getting, I would say, good enough and are kind of at this tipping point where you can start to really meaningfully build high quality production applications at scale."01:53
Who should watch
You are deciding whether a voice application should use a chained pipeline or a speech-to-speech design.
Your team is building a customer-service voice agent and needs to balance accuracy, latency, integrations, and delegation.
You need a practical starting point for voice prompts, evaluation data, synthetic conversations, or asynchronous guardrails.
