Cascaded voice systems remain practical, but their latency and dependence on text make them unlike real conversation.
2
Speech-to-speech models reduce latency, yet full-duplex interaction is needed for overlap, interruptions, and backchanneling.
3
Voice AI still needs better paralinguistic understanding, production reliability, and lower costs before it can reach mass-market use.
Summary
Neil Zeghidour argues that voice AI has improved without reaching the conversational quality shown in Her. Cascaded systems combine speech-to-text, an LLM, and text-to-speech, which makes them useful and easier to control, but their latency becomes painful when tools take 500 milliseconds to 4 seconds. Speech-to-speech systems reduce latency by handling audio directly, yet most are half-duplex. They listen or speak, so ordinary behaviors such as interruptions, overlap, coughing, and backchanneling make them fail. Moshi demonstrated that full-duplex interaction is possible, but it lacked the intelligence, tools, observability, and personalization needed for production. Zeghidour also points to paralinguistic understanding, since tone and hesitation carry information that transcription removes. Cost remains a practical barrier, especially for TTS, and privacy pushes some processing onto user devices. Gradium's Phoneon is an on-device TTS model designed to run on a smartphone CPU.
Gradium is building voice models rather than complete voice agents
Zeghidour describes Gradium AI as a model provider for teams building voice agents and voice products. The company trains speech-to-text, text-to-speech, and speech-to-speech models for transformation, translation, and dialogue. It does not build orchestration systems or focus on specific business verticals. He says the company grew from a nonprofit research lab funded by philanthropists including Eric Schmidt and Xavier Niel. That lab developed Moshi, which he calls the first speech-to-speech model for conversation, along with speech-to-speech translation and Pocket TTS, a CPU model. Gradium is the for-profit structure intended to turn that research into production software beyond open source.
Current voice demos sound better, but they still do not behave like conversations
Using Her as a deliberately overused comparison, Zeghidour contrasts the movie's Samantha with recent voice demos. He shows a government helper from ElevenLabs and a streaming voice model running in a Replika-style demonstration. The voices sound more natural than before, and the underlying agents are becoming useful. The weaknesses remain visible: latency is high, simultaneous speaking is unsupported, and much of the system is still a text model with a voice added around it. Information carried only in the audio cannot be used when the interaction is reduced to text. For Zeghidour, better intelligence has created momentum around voice agents, but smoother speech alone has not produced the experience promised by the movie.
Cascaded systems hit a latency floor before they perform real work
The standard cascade sends audio through speech-to-text, an LLM, and text-to-speech. Gradium uses streaming speech-to-text, streaming speech synthesis with voice cloning, and semantic VAD. Even a fast TTS system can take more than 200 milliseconds, while a human conversation needs understanding, answer generation, and pronunciation to fit around that total. The problem gets worse when the agent uses a tool. A tool call or router can take 500 milliseconds to 4 seconds, making small TTS improvements less meaningful. Zeghidour suggests splitting the LLM's work so it can issue the tool call while producing a filler or a relevant comment, then insert the result when it arrives. His travel-agent demo uses this approach, though he says it still needs polishing.
Speech-to-speech lowers latency but half-duplex systems cannot handle ordinary overlap
Speech-to-speech replaces the separate speech-to-text, LLM, and text-to-speech blocks with one model that accepts speech and produces speech. This reduces latency, but Zeghidour says it does not automatically create human conversation. Except for Moshi, he describes speech-to-speech models as half-duplex. They listen or speak, rather than doing both at once. That prevents them from handling interruptions, coughs, overlapping speech, and small acknowledgements such as "mhm." He notes that backchanneling varies across languages and cultures. In Japanese, frequent acknowledgements can signal politeness and active listening. Human speakers can overlap for up to 20 percent of the conversation, and systems that cannot handle this sound mechanical even when their voices are convincing.
Moshi shows that full-duplex interaction can support interruption and prediction
Zeghidour presents Moshi, which he describes as a full-duplex system. In the demonstration, the model begins answering before the speaker has finished, while still processing speech spoken over its response. It does not simply discard the interruption. This creates a more continuous exchange and remains usable with noise and multiple people speaking. He says full-duplex models are the type of interactive system needed for an experience as natural as human conversation. Moshi's conversational flow is strong, but the original model was not useful enough for production. It lacked tools and agent capabilities, and its behavior was difficult to inspect or control.
Paralinguistic understanding requires training the model to use vocal cues
Voice carries information that transcription does not preserve, including tone, hesitation, discomfort, and other signals in the way someone speaks. Zeghidour gives a demonstration where the AI infers from his tone that a character is uncomfortable. The raw information is present in speech-to-speech models, since it is not discarded during transcription, but the model will not use it unless training and outputs reward that behavior. An audio version of a factual question-answering dataset gives the system little reason to learn these cues. Zeghidour treats this as a research and training problem. Natural-sounding speech alone is insufficient if the model cannot respond to what the speaker's voice is communicating.
Full-duplex models still need the reliability and intelligence of cascaded systems
Zeghidour is candid about Moshi's limits. After a few minutes, the model could become pointless because it was not an agent and could not call tools or perform tasks. Production systems also need observability, including ways to detect unacceptable content or understand what the model is doing. He says Moshi did not provide enough paralinguistic understanding either. His position has changed from opposing cascaded systems to accepting their practicality and control. Full-duplex models may provide the right interaction pattern, but they must gain the reliability, intelligence, and personalization that make cascaded systems useful. Until then, he does not see them replacing cascades.
Voice cost and privacy push inference toward local devices
Even a technically successful speech-to-speech model would face a difficult business problem if people used it for hours each day. Zeghidour says voice is expensive, and that the voice modes of major hyperscalers run at a loss. In consumer applications, LLM costs, speech-to-text, and diarization are relatively small compared with TTS. He has seen teams spend their fundraising on TTS bills before building a user base. Privacy adds another reason to keep data local, since users may be uncomfortable putting private information into a central database. Gradium's Phoneon is designed to run on a smartphone CPU. Zeghidour says it has fewer than 100 million parameters, supports voice cloning, and can avoid API fees.
