# Realtime Voice AI

Mark Backman, Daily | AI Engineer World's Fair 2025 | 1:09:41

Source: https://www.youtube.com/watch?v=nxuTVd7v7dg
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/realtime-voice-ai
Published: 2025-08-03
Tags: agents, latency, tool-use, voice

## TL;DR
- Pipecat provides a modular Python pipeline for building realtime voice and multimodal agents.
- Realtime voice agents need fast responses, natural audio, accurate turn detection, and access to application data and tools.
- Developers can use Gemini Live with Pipecat while keeping the transport, processors, tools, and client components replaceable.

## Summary
Mark Backman and the Daily team introduce Pipecat through a hands-on workshop. Pipecat is an open source Python framework for building voice and multimodal agents. The talk explains its pipeline model, where transport, speech recognition, language models, text-to-speech, tools, and utility processors pass frames between one another. Gemini Live can combine several of those functions in one speech-to-speech service, while Pipecat still handles orchestration, interruptions, tools, recording, and other application logic. The workshop walks through a small Python bot using Daily transport, Gemini Live, and Silero VAD. The speakers also discuss WebRTC versus WebSockets, phone integrations, context management, guardrails, noise cancellation, local models, and semantic end-of-turn detection. Their advice is practical: keep task-oriented conversations divided into smaller steps, control the context window, run voice activity detection locally when possible, and choose transports based on whether the application is client-server or server-server.

## Key ideas
### Realtime voice agents have to meet several demanding human expectations
[01:59](https://www.youtube.com/watch?v=nxuTVd7v7dg&t=119s)
Mark Backman says voice applications are difficult because people expect a machine to listen well, respond intelligently, connect to data, sound natural, and answer quickly. He gives roughly 800 milliseconds as a practical end-to-end latency benchmark, with human conversation closer to 500 milliseconds. He points to Gemini Live's native audio dialogue as an example of more natural voice interaction. The network is part of this problem, since a voice agent streams audio continuously rather than sending occasional small requests. The workshop audience is warned that weak conference Wi-Fi can make the experience noticeably worse.

### Pipecat models a voice agent as a stream of processors
[03:33](https://www.youtube.com/watch?v=nxuTVd7v7dg&t=213s)
Alles explains Pipecat's multimedia pipeline as a chain of boxes that receive, modify, and forward audio, video, text, or other data. In a cascaded voice system, transport receives the user's audio, speech-to-text transcribes it, the LLM generates tokens, text-to-speech produces audio, and transport sends that audio back to the user. A speech-to-speech model such as Gemini Live can combine transcription, language generation, and speech synthesis, although applications may still need processors for recording, transcripts, tools, or custom logic. Pipecat's basic abstraction is the frame moving through this pipeline.

### Provider modularity lets applications change services without rewriting the pipeline
[06:51](https://www.youtube.com/watch?v=nxuTVd7v7dg&t=411s)
Pipecat gives developers a common interface around different AI and transport providers. The speakers say a developer can replace one speech-to-text, LLM, or text-to-speech service without changing the underlying application code. Pipecat also has a universal function schema that translates tool definitions into each provider's native format, covering differences among OpenAI, Anthropic, Gemini, Bedrock, and other services. Parallel pipelines allow separate branches for audio and video, vendor failover, or multiple agents in one interaction. The workshop presents this modularity as a way to keep application logic separate from provider-specific APIs.

### The workshop bot connects Daily transport directly to Gemini Live
[10:24](https://www.youtube.com/watch?v=nxuTVd7v7dg&t=624s)
The example bot is written in Python and runs inside an asynchronous session. Its main pieces are Daily transport, context aggregation, Gemini's multimodal live service, tools, and a pipeline task. The sample defines simple weather and restaurant handlers, then passes their schemas to Gemini so the model can call them. A client connection injects an initial context frame to start the conversation. The runner executes the task, while the transport handles incoming and outgoing audio. The code keeps the speech-to-text and text-to-speech stages out of the application because Gemini Live handles those functions.

### WebRTC fits client-server voice apps, while WebSockets fit server-to-server connections
[16:30](https://www.youtube.com/watch?v=nxuTVd7v7dg&t=990s)
Mark recommends WebRTC for client-server applications because it provides properties that matter for production audio, including error correction and better audio quality. WebSockets are a reasonable choice for server-to-server systems, including some phone chatbot setups. Pipecat includes a FastAPI WebSocket transport for exchanging messages, and it also supports phone connections through providers such as Twilio, Telnyx, Pivo, and Exotel. SIP gives more call control but adds complexity. The speakers also warn that phone agents need fast startup, since users should not wait for an unprovisioned agent to come online.

### Local voice activity detection improves interruption handling at low cost
[17:54](https://www.youtube.com/watch?v=nxuTVd7v7dg&t=1074s)
Voice activity detection, or VAD, identifies when the user starts speaking. Pipecat emits a user-started-speaking frame and can interrupt audio that the bot is currently playing. The team recommends Silero as an open source, on-device VAD because it is fast and uses very little CPU. Developers can tune how long the system listens before emitting the event. Mark says the local CPU cost is a fraction of one percent, while text-to-speech characters are much more expensive in their cost analysis. VAD must balance quick responses against giving people enough time to finish a thought.

### Context should be divided, trimmed, or summarized when conversations grow
[26:47](https://www.youtube.com/watch?v=nxuTVd7v7dg&t=1607s)
The speakers advise splitting task-oriented conversations into discrete steps. A restaurant reservation bot can collect a name, obtain a time, and write the result to a database as separate tasks. A patient intake bot might remove a verified date of birth from the context after it has served its initial purpose. Long conversations can also be summarized with an out-of-band text-model call, then returned to the realtime agent in compressed form. Large contexts slow processing and can reduce accuracy. Mark describes a tradeoff in the World's Fair bot: putting speaker data directly into Gemini Live was faster than retrieving it through a tool, but large, repetitive JSON made the model more likely to confuse entries.

### Turn-taking remains an unsolved part of natural voice interaction
[1:00:44](https://www.youtube.com/watch?v=nxuTVd7v7dg&t=3644s)
Fixed VAD timeouts create a difficult tradeoff. A short timeout reduces latency but can make the agent interrupt a person who is still thinking. A longer timeout gives the speaker room but makes the bot feel slower. Pipecat's smart-turn project uses a native audio classifier that returns either complete or incomplete for the user's turn. If the turn is incomplete, Pipecat can extend the VAD timeout dynamically before responding. The classifier uses audio and language cues such as pauses, filler words, intonation, and context. Mark describes semantic end-of-turn detection as an active area that still needs better solutions.

## Notable quotes
- "Pipecat is a Python framework for building voice and AI multimodal agents." (01:35)
- "If you're building client server, you really want WebRTC." (16:49)
- "The VAD serves a really important purpose of detecting when a user starts speaking." (17:47)
- "The issue being when you get a tool, you need the entire JSON response before you can actually do anything with it." (32:50)
- "It is very much an unsolved problem." (1:03:04)

## Tools & references mentioned
- Daily
- Pipecat
- Gemini Live
- Google DeepMind
- Google AI Studio
- OpenAI
- Anthropic
- AWS Bedrock
- AWS Nova Sonic
- NVIDIA
- Silero
- Twilio
- Telnyx
- Pivo
- Exotel
- SIP
- Crisp
- Cartesia
- ElevenLabs
- Rime
- Llama
- Whisper
- Modal
- Word Wrangler

## Who should watch
- You are building a realtime voice agent and want a Python framework that can connect different model, transport, and tool providers.
- Your application needs phone support, interruptions, tool calls, or multiple audio and video processing branches.
- You are deciding how to manage latency, context size, voice activity detection, and turn-taking in a production voice experience.

## Related talks

- [Milliseconds to Magic: Real-Time Workflows Using the Gemini Live API and Pipecat](https://aietalks.com/talks/milliseconds-to-magic-real-time-workflows-using-the-gemini-live-api-and-pipecat) (Kwindla Kramer, Daily & Shrestha Basu Mallick, Google DeepMind, 21:43)
- [Pipecat Cloud: Enterprise Voice Agents Built On Open Source](https://aietalks.com/talks/pipecat-cloud-enterprise-voice-agents-built-on-open-source) (Kwindla Hultman Kramer, Daily, 26:46)
- [Realtime Conversational Video with Pipecat and Tavus](https://aietalks.com/talks/realtime-conversational-video-with-pipecat-and-tavus) (Chad Bailey, Daily & Brian Johnson, Tavus, 18:46)
- [How to build the world's fastest voice bot](https://aietalks.com/talks/how-to-build-the-worlds-fastest-voice-bot) (Kwindla Hultman Kramer, Daily, 20:38)
- [Giving a Voice to AI Agents](https://aietalks.com/talks/giving-a-voice-to-ai-agents) (Scott Stephenson, Deepgram, 13:08)