Speech-to-speech models can combine translation, task completion, multimodal input, and natural conversation in one promptable system.
2
The model must balance conversational speed, intelligence, and multimodal ability because improving one can damage another.
3
Live translation, roadside assistance, and proactive audio show how the same model can handle different voice-agent settings.
Summary
Valeria Wu Fon and Tom Ouyang describe Google's move from separate speech components toward a natively multimodal speech-to-speech model. Older systems chained feature extraction, acoustic modeling, pronunciation modeling, language modeling, and rescoring before producing text. End-to-end speech recognition simplified that chain, but it still handled transcription as one isolated task. Gemini models are trained on interleaved audio, video, and text, which lets one model translate, answer questions, call tools, and produce speech. The speakers frame the research around three competing goals: low-latency conversation, useful intelligence, and multimodal input and output. They show live multilingual meeting translation, a roadside assistance agent that handles registration plates and postcodes, and proactive audio that ignores background noise when it is not the user's turn. They also discuss visual avatars and argue that spoken interaction will be part of future general-purpose AI.
Older speech systems depended on a long chain of specialised components
Before about 2018, automatic speech recognition commonly used feature extraction, acoustic modeling, pronunciation modeling, language modeling, and a second rescoring pass to turn audio into text. End-to-end neural systems later learned much of the mapping from acoustic input to transcription without as much hand-built domain knowledge. Tom Ouyang says these systems still handled one job. They did not respond or translate, and features such as tone, emotion, speaking speed, or image conditioning had to be added separately. That made the systems harder to scale when a product needed more than transcription.
Multimodal pre-training lets one model connect audio, video, and text
Gemini models are trained with interleaved multimodal examples during pre-training. One example combines a text request with video and audio, then asks the model to produce a summary and timestamps for interesting events. Other examples caption video, generate audio from video or text, or combine speech recognition and speech synthesis with agent tasks. Ouyang says these abilities are learned within one unified token embedding space. The result is a model that already understands relationships between audio, video, and text, rather than requiring a separate system for every transition.
Streaming translation can approach offline quality while speakers are still talking
The live translation application translates conversations across languages as people speak. Ouyang says the streaming result is about as good as offline translation that has access to the full utterance from the start. The system must switch between languages, preserve the source speaker's voice, handle multiple speakers and noise, and respond in real time. He presents this as an ability that would be difficult to engineer across cascaded components. With the Gemini models, much of the behaviour can be requested through prompting because the relevant capabilities come from pre-training.
A useful speech-to-speech model must balance conversation, intelligence, and multimodality
Valeria Wu Fon describes three goals for the model. It should be conversational, with low latency and natural responses. It should be intelligent enough to follow instructions, reason, and complete tasks. It should accept more than audio, including video, screen shares, and PDFs, and produce multimodal output. Wu Fon also says the team focuses on languages beyond US English because most Gemini users are non-English speakers. These goals are connected, so the product cannot be judged only by how pleasant the voice sounds.
Increasing thinking can improve intelligence while damaging response latency
The three goals create direct trade-offs. Wu Fon gives the example of increasing the model's thinking time before it calls a tool or answers. Evaluations show that more thinking can improve intelligence, but the extra work affects latency, time to first audio, and the naturalness of the conversation. The team is researching ways to combine the goals without sacrificing too much of any one of them. The demos that follow are presented as evidence of progress toward that balance, rather than as a claim that the trade-off has disappeared.
Live meeting translation is designed to avoid rigid turn-taking
The live model supports streaming translation in more than 70 languages and can follow several people speaking different languages in a Google Meet conversation. Users choose the target language, then translation begins shortly after someone starts speaking. Wu Fon says this low latency makes the exchange feel less like a sequence of forced turns. The demo includes speakers discussing weather and a birthday across languages. The system can translate multiple speakers in a shared conversation instead of waiting for the entire meeting or requiring everyone to speak one at a time.
A roadside agent needs both precise data handling and awareness of its surroundings
The roadside assistance demo uses the same speech-to-speech model for a more constrained task. The agent asks for a policy number, then switches to a registration plate and postcode when the caller does not have it. Wu Fon points to the need for accurate handling of alphanumeric information such as postcodes and addresses. She also describes proactive audio, where the model decides whether external sound should interrupt it. A person talking nearby or noise on a train should not make the agent stop mid-response. The intended setting is everyday movement, rather than a quiet office.
The final example adds a visual presence to the conversation. Wu Fon describes a pilot with City that supports customised real-time avatars, from hyperrealistic people to cartoons. The system uses the same speech-to-speech model and supports low-latency multilingual lip-syncing. The demo combines spoken conversation, Spanish and English, tool use, and awareness that another person has joined the interaction. Wu Fon presents this as a direction where multimodal input, multimodal output, task actions, and conversational flow can operate together.
"There's like this feature that we have called proactive audio, which is the idea that the LLM knows when or when not to respond to an external input."Valeria Wu Fon13:30
Who should watch
You are building a voice agent and need to decide between a speech-to-speech model and a speech-to-text pipeline.
Your product must handle live translation, multiple speakers, noise, or languages beyond English.
You want to understand the engineering trade-off between response speed, reasoning time, and multimodal capability.
