Voice agents have moved from Siri's constrained app actions to open-ended conversation and then to systems that can call tools and complete tasks.
2
Speech-to-speech models reduce latency and preserve tone, but current systems still lose intelligence and usually rely on half-duplex turn-taking.
3
A hybrid design could use a small full-duplex voice interface for natural conversation and delegate reasoning, tool calls, and other agent work to a background text model.
Summary
Neil Zeghidour traces voice agents through three stages. Siri mapped a transcript to a narrow app action. OpenAI's original voice mode made conversation open-ended but had little agency. Cascaded voice agents added tools, planning, and reasoning, while passing through text introduced latency and removed vocal detail. Speech-to-speech models improved latency and naturalness, yet current systems remain less intelligent than their text-based counterparts and still stop listening when they speak. Zeghidour demonstrates the problem with backchanneling, where a simple "mm-hmm" interrupts the model. He explains how neural codecs compress audio into tokens and how multi-stream language models let both speakers produce audio at once. The remaining problem is capacity: adding speech input and output takes capacity away from reasoning. He proposes scaling speech-to-speech models, or separating the natural voice interface from a background text model that handles thinking and tools. He favors the second approach because it can reduce cost and let teams change the backend model.
Voice agents began as narrow pipelines that mapped speech to app actions
Neil Zeghidour uses Siri's 2011 launch to describe the first stage of voice agents. Siri could answer questions such as the weather or the NASDAQ, but the system was tightly constrained. It converted speech to a transcript, classified the user's intent, identified the app, and selected an action. There was no language model doing open-ended reasoning. The system used natural language understanding and a complex pipeline for specific use cases. Zeghidour calls this close-ended dialogue. It could trigger real actions, but only inside the set of interactions the designers had prepared.
Open-ended voice chat gained natural conversation while losing agency
The next stage was represented by OpenAI's original voice mode, before advanced voice mode. It could discuss almost anything, which made the interaction much less constrained than Siri. Zeghidour contrasts that openness with its lack of agentic ability. The system could tell a bedtime story about a sunflower hedgehog, but it could not reliably give the NASDAQ or the weather. The logic that had previously been built into a task-specific pipeline was replaced by a language model. Naturalness improved substantially, although latency still took several seconds.
Cascaded voice agents can act, but text in the middle removes vocal information
The third stage adds real agency to voice interaction. In Zeghidour's drive-through demonstration, the model takes a chicken sandwich order, remembers the items, and calculates a total. This kind of system has tools, reasoning, and planning behind the conversational interface. Cascaded systems regained and generalized the action-taking abilities of earlier assistants. Their weakness is the route through speech-to-text, a language model, and text-to-speech. The route adds latency compared with human conversation and loses information about tone and emotion because speech is reduced to text before the response is produced.
Speech-to-speech improves latency but still behaves like a walkie-talkie
Speech-to-speech models combine speech recognition, language processing, and speech generation in one model. Zeghidour says this makes latency very good and allows the model to use non-linguistic information. It does not solve turn-taking. A typical real-time model is half duplex: it listens or speaks. During a live demonstration, Zeghidour says "mm-hmm" and "yeah" while the model is talking. The model treats these backchannels as interruptions and stops. Human conversations allow people to backchannel, interrupt, and speak over one another. Zeghidour says that relatives on the phone can speak at the same time for up to 20 percent of the call.
Neural codecs make audio manageable by turning it into learned tokens
Raw audio is far too long to feed directly into a language model. Zeghidour gives an example of eight spoken words that take about three seconds. At 24 kHz, that becomes 72,000 audio time steps. Since transformer attention has quadratic complexity with sequence length, a sequence 10,000 times longer is 100 million times more expensive to process. Neural codecs, also called audio tokenizers, solve the length problem by encoding audio into a dense representation and decoding it back into high-quality sound. A language model can then predict audio tokens in a way similar to predicting text tokens.
Full-duplex models need two token streams so both sides can speak at once
A single sequence of user and system audio tokens still creates a half-duplex interaction. Zeghidour describes multi-stream language models as the solution used for full duplex. Instead of modeling one sequence, the transformer models two streams. Each participant can be active or inactive independently, so both can speak at the same time. He shows a Moshi demonstration in which the model guesses the end of a question, answers over the speaker, and continues through noise, coughing, and interruptions. The result is more resilient to the messy timing of ordinary conversation, although the early system was less intelligent than other speech-to-speech models.
Naturalness currently takes capacity away from reasoning
Zeghidour describes a repeated tradeoff in the history of voice agents. Cascaded agents can use strong text models and retain their tool-calling reliability. Speech-to-speech models sound more natural, but they are less intelligent. A text model has a fixed number of weights, or capacity. When the same model must understand speech and produce speech as well as reason, part of that capacity is spent on the added modality. That capacity is no longer available for intelligence. Full duplex adds another naturalness requirement, so a system cannot simply gain conversational flow by accepting a large loss in reasoning and agency.
A small voice interface could delegate hard work to a separate text model
Zeghidour gives two possible paths. One is to make speech-to-speech models larger and improve their pretraining and post-training until their intelligence is sufficient for more uses. The other splits the system. A small, possibly on-device, full-duplex model handles natural conversation and knows when it does not know. It asynchronously delegates reasoning, tool calls, and other agent work to a background text model. Zeghidour connects this design to the MoshiRAG paper and says it can lower costs because simple conversation does not need a huge model that can solve difficult problems. It also lets users change the backend model instead of being tied to one speech-to-speech provider.
"If you're having a relative on the phone, there is up to 20% of the time where you are both speaking at the same time."09:39
Who should watch
You are building a voice agent and need both natural conversation and tool use, rather than a demo that only chats.
Your current system uses a speech-to-text, text model, and text-to-speech pipeline, and you want to understand what speech-to-speech changes and what it does not.
You are deciding whether to invest in one large speech-to-speech model or split the voice interface from the reasoning backend.
