Gemini 3 Flash Preview can turn one audio API request into speaker labels, timestamps, language detection, translations, emotion tags, and a summary.
2
Gemini's speech generation uses a small set of base voices directed by a written performance note, which can specify accents, scenes, and delivery.
3
Gemini 3.1 Flash Live handles real-time multimodal conversation directly in audio, while Lyria 3 generates short jingles or full songs with lyrics.
Summary
Thor Schaeff presents the audio work around Gemini, starting with audio understanding. Gemini 3 Flash Preview can identify speakers by name, add timestamps, detect languages, translate non-English speech, classify emotions, and summarise a recording in one structured API response. He then shows how speech generation uses a base voice plus a director's note that describes the setting, accent, and performance. The talk moves to Gemini 3.1 Flash Live, a full-duplex model that accepts text, audio, and video over a WebSocket connection and returns audio with a transcript. Schaeff contrasts its built-in reasoning with a pipeline that converts speech to text before sending it to a separate language model. He finishes with Lyria 3 and a Live Jukebox application, where the live model calls Lyria as a tool to create a German techno schlager about the UK startup scene.
Gemini can extract structured meaning from an audio recording in one request
Thor Schaeff describes Echo Script, a Google AI Studio application built with Gemini 3 Flash Preview. A single request can identify distinct speakers, label them by name when context is available, produce timestamps, detect the language, translate non-English speech into English, classify emotion as happy, sad, angry, or neutral, and write a brief summary at the beginning. He uses structured outputs to place this result directly into an application interface. The model is intended to understand more than words, including overlapping speech, accents, pacing, and changes between languages.
Speech generation starts with a base voice and a written performance direction
Gemini speech generation does not require choosing from a large catalogue of fully specified voices. Schaeff says Gemini has roughly 30 base voices, which can be directed to perform in a particular way. The prompt can define an audio profile, a scene, the delivery, and the transcript. His example turns a standard American voice into Finian, a high-pitched Irish male voice speaking in a crowded pub on the coast of County Clare. Another example directs Zephyr to speak with Singaporean phrasing in a hawker centre setting.
Audio understanding gives speech generation control over accents and situations
Schaeff connects speech generation to the same audio understanding research used elsewhere in Gemini. The model can modify a voice according to a director's note because it has learned how different accents, pacing, and situations sound. He plays a standard voice and then an Irish-directed version of the same text, followed by a Singaporean delivery with phrases such as 'lah' and 'faster queue'. The examples show that the prompt describes how the voice should act, rather than selecting a fixed voice that already has every desired characteristic.
Gemini 3.1 Flash Live reasons inside a real-time audio model
Gemini 3.1 Flash Live is presented as a speech-to-speech, full-duplex multimodal model. It can receive text, audio, and video in real time through a WebSocket connection and return an audio response along with a text transcript. Schaeff says its thinking and reasoning are built directly into the audio model. This differs from a cascaded design in which speech becomes text, a separate language model handles the reasoning, and another stage produces the response. Developers can try the model in AI Studio Live.
Live conversation can combine audio, vision, languages, and system instructions
In the live demonstration, Schaeff gives the model a system instruction to use a friendly Irish accent and lets it view his camera. It comments on his Gemini shirt and backwards hat, then responds to a request for a German poem. The model switches languages during the exchange, although it also carries the Irish accent into the German response. Schaeff points out that developers would need to change the system instructions if they did not want the accent applied to every language. The live system can also receive screen video, currently at a maximum of one frame per second.
Lyria 3 can generate either short music clips or full songs with lyrics
Schaeff introduces Lyria 3 as the music generation part of the stack. Lyria 3 Clip generates 30-second jingles, while Lyria 3 Pro generates full-length songs. He combines it with Gemini Live in an application called Live Jukebox. The live model receives a request from the user, asks for details about the desired style and lyrics, and then uses a tool to call Lyria. The stage request is for a German techno schlager about the UK startup scene, with manic energy and lyrics chosen by the model.
"So, this was one API call to Gemini 3 Flash preview. And, you know, we got all this information out."06:55
Who should watch
You are building transcription or media-processing software and need speaker identity, timestamps, translations, emotion labels, and summaries in one response.
You want generated speech with a specific accent or performance style without maintaining a large catalogue of finished voices.
You are exploring real-time voice agents, camera or screen input, or tool calls from a conversational model into music generation.
