# Building Conversational Agents

Thor Schaeff & Philipp Schmid, Google DeepMind | AI Engineer Europe 2026 | 1:47:34

Source: https://www.youtube.com/watch?v=cVzf49yg0D8
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/building-conversational-agents
Published: 2026-04-30
Tags: agents, multimodal, tool-use, voice

## TL;DR
- The Interactions API gives models and agents one interface with server-side state, streaming, tool calls, and asynchronous execution.
- A coding agent can combine Gemini with file and shell tools, but it needs explicit instructions and its generated code still needs review.
- The Gemini Live API streams text, audio, and video over WebSockets, although the workshop demos exposed gaps in grounding, latency, and production readiness.

## Summary
Thor Schaeff and Philipp Schmid present a hands-on path from a basic Gemini agent to realtime voice and video applications. Philipp starts with the Interactions API, showing server-side conversation state, branching from earlier turns, content blocks, tool calls, and agent skills that give coding tools current Gemini documentation. The workshop then builds a small coding agent with file and Bash tools. Thor introduces the Gemini Live API through a music-generation jukebox and explains its native audio model, multimodal inputs, voice prompting, search grounding, and WebSocket setup. The examples include both server-to-server connections and ephemeral-token connections from a browser. The demos do not always work: Google Search grounding fails repeatedly, and the server proxy adds latency. The speakers are candid that some business applications may still be better served by cascading speech pipelines, which offer more observability. They also discuss session limits, context compression, transcripts, speaker identification, personalization, privacy, and evaluation.

## Key ideas
### The Interactions API unifies model and agent calls
[08:38](https://www.youtube.com/watch?v=cVzf49yg0D8&t=518s)
Philipp Schmid describes the Interactions API as a unified interface for ordinary models and agents such as Deep Research. A request can name a Gemini model or an agent, and later interactions can continue from the same history. Inputs and outputs use content blocks with types such as function call, thought signature, text, audio, video, and image. The API also supports streaming with server-sent events, built-in tools, remote MCP, and combinations such as Google Search with a custom function. Philipp says the interface is closer to APIs developers already know and less tied to Google's protobuf and gRPC conventions.

### Server-side state reduces client history work and can improve caching
[11:38](https://www.youtube.com/watch?v=cVzf49yg0D8&t=698s)
With server-side state, an application stores the previous interaction ID and sends only the next input. The server keeps the user input and model output, so the client does not need to rebuild the entire conversation on every turn. Applications can still manage their own history when they need to remove or alter context. Philipp explains that preserving the server's exact context also helps implicit caching. Removing whitespace or other small changes can break a cache, while cached input tokens are much cheaper. He says startups using the Interactions API have seen two to three times better cache rates.

### Agent skills give coding tools current API guidance
[17:31](https://www.youtube.com/watch?v=cVzf49yg0D8&t=1051s)
The workshop uses agent skills instead of having attendees write every line manually. Philipp shows a Gemini Interactions API skill that can be installed in a project through an NPX command and used by tools such as Cursor, Antigravity, Gemini CLI, and potentially Claude Code. The skill tells the coding agent which Gemini models and agents are available and points it to current documentation in Markdown. That design avoids copying the entire documentation into the skill and means the documentation can be updated independently. The speakers say a skill is useful when a model cannot reliably perform a workflow or when a team has a required development preference.

### A coding agent needs a tool loop and clear instructions
[25:10](https://www.youtube.com/watch?v=cVzf49yg0D8&t=1510s)
The first agent has a client, a model ID, a run method, and a previous interaction ID. The workshop then adds read-file and write-file functions with JSON schemas, followed by a Bash command tool. The agent checks its outputs for function calls, executes the matching local function, sends back a function result, and repeats until the model returns text. When the agent creates a CSV but does not write it to disk, Philipp adds a system instruction that explicitly defines it as a coding agent with access to the local file system. After that change, it writes an SVG file when asked. The example is deliberately simple, and Philipp warns that generated changes still need checking.

### The Live API streams multimodal input over a stateful WebSocket
[55:29](https://www.youtube.com/watch?v=cVzf49yg0D8&t=3329s)
Thor Schaeff introduces Gemini 3.1 Flash Live as a stateful WebSocket API for streaming text, audio, and video into a model. Audio arrives in buffer chunks, while video can be sent at up to one frame per second from a camera, canvas, or screen share. The service returns audio buffers, transcriptions, and tool calls. Google Search grounding is built in, and the model supports interruption through voice activity detection. Thor explains that the native audio model works directly with sound tokens instead of using a separate speech-to-text, text model, and speech-generation pipeline. He also describes multilingual conversations, including mixed German and English.

### Client connection design changes latency and credential handling
[1:06:10](https://www.youtube.com/watch?v=cVzf49yg0D8&t=3970s)
The first Live API example uses a server-to-server WebSocket connection. A browser sends audio and video to the application's server, and the server proxies them to Gemini. Thor says this adds latency because the media makes an extra hop. The second example uses an ephemeral token. The server keeps the long-lived Gemini API key, creates a short-lived token, and gives that token to the browser or phone. The client then opens its WebSocket directly to the Live API. Thor shows the raw bidirectional WebSocket setup, including the model, real-time input configuration, tools, and access token. He points attendees to Python examples, a FastAPI server, and Live API coding skills.

### Realtime voice still has practical limits in production
[1:26:57](https://www.youtube.com/watch?v=cVzf49yg0D8&t=5217s)
The speakers name Shopify Sidekick, Waymo, Stitch, and Hey Otto as examples of work around Live API applications. Hey Otto uses voice companions for elderly people and connects them with an app for caretakers. Thor says that for many business cases, a cascading speech pipeline may still be a better choice because each stage is observable and easier to modify before the response is spoken. The Live API currently does not provide downloadable session transcripts, so applications need to store them themselves or use an integration partner such as LiveKit or PipeCast. Thor also says the model does not reliably restrict listening to one speaker, and specialised user knowledge must be supplied through context or tools.

### Long sessions depend on context compression and input volume
[1:25:14](https://www.youtube.com/watch?v=cVzf49yg0D8&t=5114s)
Thor explains that audio-only Live API sessions are limited to 15 minutes without compression, while audio-video sessions are limited to 2 minutes. The service sends a go-away signal before the session ends. Context window compression provides a sliding window that keeps a chosen amount of recent context and drops older material as the conversation continues. Video consumes context faster because each frame adds input, so applications should avoid sending images continuously when the scene has not changed. For an editor assistant, the speakers suggest sending an image of the editor and then audio rather than repeatedly streaming an unchanged image.

## Notable quotes
- "We really want to build this unified interface where you can continue your conversations, no matter what model you used." (15:41)
- "What I can guarantee you is that the Gemini 3 flash has never seen any code of the interactions API because the model was trained before we even released the API." (39:19)
- "So, the real world, especially in kind of live audio, does often look a bit different." (54:44)
- "For certain business use cases it might just not be there yet." (1:29:59)
- "You would have to store them on your end." (1:30:21)

## Tools & references mentioned
- Google DeepMind
- Gemini API
- Google AI Studio
- Interactions API
- Generate Content API
- Deep Research
- Gemini 3 Flash
- Gemini 3.1 Flash Live
- Gemini Live API
- Lyria 3
- Google Search grounding
- remote MCP
- Gemini agent skills
- Cursor
- Antigravity
- Gemini CLI
- Claude Code
- Chrome
- Meta Ray-Ban
- Vision Claw
- Shopify Sidekick
- Stitch
- Waymo
- Hey Otto
- LiveKit
- PipeCast
- Software Mansion
- Fishjam
- Vision Agents
- Voximplant
- NVIDIA Parakeet
- Vertex AI

## Who should watch
- You are building an agent that needs tool calls, persistent turns, or a coding workflow and want concrete Interactions API patterns.
- You are prototyping voice, screen-sharing, or camera-based assistants and need to understand the Live API's WebSocket and ephemeral-token options.
- You are deciding whether native realtime audio is ready for a business workflow and want an honest account of latency, observability, transcripts, grounding, and speaker limitations.

## Related talks

- [Any-to-Any: Building Native Multimodal Agents](https://aietalks.com/talks/any-to-any-building-native-multimodal-agents) (Patrick Löber, Google DeepMind, 16:21)
- [Milliseconds to Magic: Real-Time Workflows Using the Gemini Live API and Pipecat](https://aietalks.com/talks/milliseconds-to-magic-real-time-workflows-using-the-gemini-live-api-and-pipecat) (Kwindla Kramer, Daily & Shrestha Basu Mallick, Google DeepMind, 21:43)
- [Building in the Gemini Era](https://aietalks.com/talks/building-in-the-gemini-era) (Kat Kampf & Ammaar Reshi, Google, 17:57)
- [From Transcription to Live Music: Gemini's Audio Stack](https://aietalks.com/talks/from-transcription-to-live-music-geminis-audio-stack) (Thor Schaeff, Google DeepMind, 19:34)
- [Let's Go Bananas with GenMedia](https://aietalks.com/talks/lets-go-bananas-with-genmedia) (Guillaume Vernade, Google DeepMind, 1:17:14)