Voice agents already handle translation, learning, speech therapy, enterprise software, and phone support at scale.
2
The voice AI stack runs from models through real-time APIs and orchestration frameworks to application code, while difficult capabilities gradually move down the stack.
3
The Gemini Live API and Pipecat support multimodal voice applications with turn detection, tool use, screen updates, and conversational context, but the demo shows that models still behave unpredictably.
Summary
Kwindla Kramer and Shrestha Basu Mallick describe the stack behind real-time voice applications, from Gemini models and the Gemini Live API to Pipecat and application code. They argue that voice agents are already used for translation, tutoring, speech therapy, enterprise software, and phone calls, while many difficult parts of the stack remain early and only partly solved. Turn detection shows how capabilities move downward over time, from application code into frameworks, then APIs and models. Kramer demonstrates a personal voice assistant that manages grocery, reading, and work lists, updates a screen, assigns tasks, and generates a small animated app. The demo succeeds at some tasks and fails at others, including names, book titles, list consolidation, and turn handling. Those failures expose how much the application depends on model interpretation and conversational context. Basu Mallick closes by describing Gemini's multimodal design and her belief that voice will become a major way people interact with language models.
Voice is already used in practical applications, including phone calls
The speakers present voice as a natural interface because people talk before they read, often speak faster than they type, and use vocal expression to convey emotion. They point to deployed examples including language translation between patients and doctors, directed learning for children, speech therapy, and copilots for complex enterprise software. At Daily, Kramer says callers often do not realize they are speaking with a voice agent, even after being told that the call involves AI. Voice agents are therefore presented as an existing product category, not only as a future interface.
Real-time voice applications require work across several layers
The stack begins with large language models, followed by real-time APIs such as Google's Gemini Live API. Orchestration libraries and frameworks such as Pipecat sit above the APIs and manage much of the complexity involved in real-time multimodal applications. Application code sits at the top and implements the product-specific behavior. Kramer and Basu Mallick map difficult problems onto this stack and their level of maturity. They say none of the listed areas feels more than about 50% solved, by Kramer's own deliberately arbitrary estimate.
Capabilities tend to move from applications into frameworks and APIs
The speakers describe a pattern in which developers first solve difficult problems inside individual applications. When enough developers encounter the same problem, the solution moves into an orchestration framework and may later become part of an API. Turn detection is their example. Kramer initially implemented it in application code because suitable tools did not exist. Pipecat later included more general turn detection, and the Gemini Live API then added turn detection inside its own interface. Basu Mallick adds that developers can turn off server-side turn detection and use systems such as Daily or LiveKit instead.
Model-driven applications behave differently from traditional programs
Kramer says he keeps relearning that a model can drive the application cycle in ways his code did not anticipate. An unexpected action can be a bug, but it can also be a useful result. The demo is intentionally close to his personal code, without unit tests or evals, and it uses current model and Pipecat branches. Basu Mallick says the model's behavior varies from session to session. This makes the boundary between code written by the developer and behavior inferred by the model less predictable than in conventional software.
The live assistant can manage lists and update a visual interface through voice
Kramer's assistant creates a grocery list for asparagus pizza, adds ingredients, creates reading and work lists, calculates the date for an end-of-day Thursday deadline, combines and separates lists, fixes a duplicated work entry, and assigns items to people. It can also display the lists on screen. Later, the assistant generates an app with the words 'hello world' jumping in Google colors and two animated neon green ASCII cats. Kramer explains that he gave the model a function for displaying text but provided little instruction about when to clear the screen, so the model inferred some of the behavior in context.
The demo's failures come from names, turn handling, and uncertain context
The assistant repeatedly mishears or mishandles 'Dream Count', adding items such as 'Quick' and 'segmentation fault' instead. It also struggles with the names Quinn and Shrestha, sometimes interprets requests incorrectly, and needs repeated instructions for the Three-Body Problem books. Kramer says the assistant loads many previous conversational sessions as user and assistant messages. Depending on the code version, it may summarize those sessions or leave them intact. The model is therefore responsible for understanding what a list means and which earlier context applies. The speakers describe the results as impressive but uneven.
Gemini is designed to handle multiple modalities from the start
Basu Mallick connects the discussion to a broader view of language-model interfaces. She describes Gemini models as multimodal from the ground up, able to ingest text, voice, images, and video. Her example of grandmothers tying knots in a sari or strings around fingers illustrates a memory aid that preserves the reminder but not the thing being remembered. She says her work at Google aims to build technologies that support creative possibilities, and she expects much interaction with language models to happen through voice.
"We believe that voice is the most natural of interfaces and there will come a most of the interaction with language models will happen via voice."21:04
Who should watch
You are building a voice agent and need to decide what belongs in application code, Pipecat, or a real-time API.
Your assistant needs tool calls, persistent conversational context, screen updates, or multimodal interaction, and you want to see the failure modes in a live example.
You are evaluating voice interfaces for support, tutoring, gaming, translation, or telephony and want a grounded view of what works today.
