# Can Your Agent Hear You Now? Building Live Voice Agents with Gemini

Thor Schaeff, Google DeepMind | AI Engineer World's Fair 2026 | 20:14

Source: https://www.youtube.com/watch?v=xfPVBzDYjH4
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/can-your-agent-hear-you-now-building-live-voice-agents-with-gemini
Published: 2026-10-11
Tags: multimodal, tool-use, voice

## TL;DR
- Gemini's stateful Interactions API lets an image-generation workflow carry visual style and characters from one image to the next.
- Gemini 3.5 Live Translate can listen to a speaker and stream near-real-time translation into a listener's chosen language.
- The Gemini Live API uses a native audio model over a stateful WebSocket, with audio and video input, streamed audio output, transcripts, interruptions, tools, Google Search grounding, and support for 90 languages.

## Summary
Thor Schaeff demonstrates several ways to build multimodal, real-time agents with Gemini. A story canvas combines Gemini Live with Nano Banana Light, while the stateful Interactions API carries earlier image context forward so characters and visual style remain consistent. A translation demo lets audience members listen to the talk in their chosen language. An AI radio DJ uses Gemini Live to gather a music request and call Lyria 3 to generate a song. Schaeff then explains the Live API as a stateful WebSocket that accepts text, audio, and video frames and streams back audio and a transcript. Its native audio model avoids a separate speech-to-text and text-to-speech chain. The API supports interruptions, tool calls, Google Search grounding, and 90 languages. He finishes with a voice and vision demo in Google AI Studio, where the model identifies clothing, switches to Mandarin, and can use search for current information.

## Key ideas
### Gemini TTS can produce expressive, steerable speech
[01:11](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=71s)
Schaeff opens by playing an audio sample and asks the audience to identify the model. It is Gemini text to speech, which he describes as an expressive speech model. Developers can steer it with inline audio text, and speech can be streamed. He says its latency is much lower than it used to be, making Gemini TTS an option for applications that need generated speech.

### Nano Banana Light makes a live illustrated story practical
[03:15](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=195s)
Schaeff builds a story canvas that uses Gemini Live to create a narrative and Nano Banana Light to illustrate it. The audience supplies a setting, 1980s Taipei, and Gemini describes a scene about the emergence of bubble tea. The application then generates an image through a tool call. Schaeff says image generation still takes two to three seconds, which is fast enough for this interactive demonstration.

### The Interactions API carries visual context between generations
[05:42](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=342s)
The first image request has no previous interaction ID. Later requests pass the earlier interaction ID to the stateful Interactions API. That gives the next generation the context of the previous interaction. In the story canvas, this lets the application continue a specific visual feeling and preserve characters across images. Schaeff describes the Interactions API as a unified way to interact with Google DeepMind models and agents.

### Live Translate streams speech into a listener's chosen language
[07:07](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=427s)
Audience members scan a QR code, choose a language, and listen through their phones. The session receives Schaeff's microphone audio and translates it as he speaks. He says Gemini 3.5 Live Translate recognizes the context of an utterance and starts translating in near real time. It can automatically detect the speaker's language, and the audience can listen in languages such as German or Chinese.

### A voice agent can call a music-generation tool
[10:21](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=621s)
The radio DJ demo uses Gemini 3.1 Flash Live as the conversational agent. It asks the caller for a genre and lyrical direction, then receives a request for a high-energy reggaeton song about the AI engineering world's fair. Gemini Live calls the Lyria 3 music-generation model through a function call and returns a generated track. Schaeff says the same pattern could connect an agent to a CRM, retrieve customer information, or create a ticket.

### The Live API connects multimodal input to streamed audio
[14:13](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=853s)
Schaeff describes the Live API as a low-latency voice and vision API. A client creates a stateful WebSocket session and sends text, audio, and video frames over it. The model responds with audio chunks that an application can stream, along with a transcript. This lets an agent handle multimodal, real-time interactions within one ongoing session.

### Native audio removes the usual speech pipeline
[15:07](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=907s)
The Live API uses a native audio model rather than a cascading voice-agent pipeline. The model does not first turn speech into text before reasoning over it. Schaeff says the intelligence is built into the audio model, which understands speech directly and responds in any of its 90 supported languages. The API also supports barge-in, so users can interrupt the model, plus tools and audio transcription.

### Google AI Studio exposes voice, vision, search, and language controls
[16:31](https://www.youtube.com/watch?v=xfPVBzDYjH4&t=991s)
Schaeff presents Google AI Studio as the easiest way to try the Live API. Developers can select Gemini 3.1 Flash Live, choose a voice, enable Google Search grounding, and talk to the model. In the demo, it hears Schaeff, sees his black shirt and hat, reads the word Gemini on his sweatshirt, and switches to Mandarin. Google Search gives the model access to current information beyond its training data.

## Notable quotes
- "So Gemini text to speech. Actually a really incredible model for expressive speech." (01:11)
- "So what that means specifically for our demo here is that you can see is our first interaction, it generated an image with a very specific kind of visual feeling and also with characters, right?" (06:13)
- "And so what this means is you have this stateful websocket API where you create a session and then over this websocket you can send in real time text, audio and video frames to the model and the model will then respond in real time back in audio chunks." (14:08)
- "So this is a little bit different from kind of your cascading voice agent pipeline because this is a native audio model." (15:01)
- "We can barge in, we can interrupt the model, we can give it the tools and we have the audio transcription as well." (15:45)

## Tools & references mentioned
- Gemini TTS
- Gemini Live
- Gemini API
- Google AI Studio
- Google DeepMind
- Omni Flash
- Nano Banana Light
- Interactions API
- Gemini 3.5 Live Translate
- Gemini 3.1 Flash Live
- Lyria 3
- LiveKit
- Pipecat
- Google Search

## Who should watch
- You are building a voice agent and need to understand how a native audio API differs from a speech-to-text and text-to-speech pipeline.
- Your application needs live translation, interruptions, vision input, tool calls, or responses in multiple languages.
- You want concrete examples of connecting Gemini Live to image generation, music generation, or Google Search before building your own prototype.

## Related talks

- [Building Conversational Agents](https://aietalks.com/talks/building-conversational-agents) (Thor Schaeff & Philipp Schmid, Google DeepMind, 1:47:34)
- [From Transcription to Live Music: Gemini's Audio Stack](https://aietalks.com/talks/from-transcription-to-live-music-geminis-audio-stack) (Thor Schaeff, Google DeepMind, 19:34)
- [Any-to-Any: Building Native Multimodal Agents](https://aietalks.com/talks/any-to-any-building-native-multimodal-agents) (Patrick Löber, Google DeepMind, 16:21)
- [Milliseconds to Magic: Real-Time Workflows Using the Gemini Live API and Pipecat](https://aietalks.com/talks/milliseconds-to-magic-real-time-workflows-using-the-gemini-live-api-and-pipecat) (Kwindla Kramer, Daily & Shrestha Basu Mallick, Google DeepMind, 21:43)
- [Let's Go Bananas with GenMedia](https://aietalks.com/talks/lets-go-bananas-with-genmedia) (Guillaume Vernade, Google DeepMind, 1:17:14)
