# How to build the world's fastest voice bot

Kwindla Hultman Kramer, Daily | AI Engineer World's Fair 2024 | 20:38

Source: https://www.youtube.com/watch?v=dRQHikOrH2A
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/how-to-build-the-worlds-fastest-voice-bot
Published: 2025-02-10
Tags: deployment, latency, open-source, voice

## TL;DR
- A voice bot has to keep audio transport, transcription, endpointing, language-model inference, and speech generation within a conversational response window of about 500 milliseconds.
- Self-hosting and colocating the pipeline in one compute container gives engineers more control over latency than calling separate hosted services.
- Pipecat provides an open-source, vendor-neutral framework for building real-time multimodal applications such as voice bots, translation apps, and voice-controlled agents.

## Summary
Kwindla Hultman Kramer explains why voice bots are harder to build than a clean diagram suggests. The system must handle audio capture, echo cancellation, encoding, network transport, jitter buffers, transcription, phrase endpointing, interruptions, language-model inference, and text-to-speech. Latency matters more than throughput because people expect a conversational response in roughly half a second. His approach is to place as much of the pipeline as possible in one compute container, then use WebRTC and edge networking to move audio efficiently. The demo uses Deepgram models for transcription and voice generation, with Llama 3 8B for language-model inference. Kwindla also introduces Pipecat, an open-source framework from Daily for real-time and multimodal AI. He is direct about the trade-off: hosted services are convenient, but teams that need very low latency need to host, tune, and combine components themselves.

## Key ideas
### Voice bots combine many systems that each affect the user experience
[01:16](https://www.youtube.com/watch?v=dRQHikOrH2A&t=76s)
Kwindla starts with a simple picture of a user speaking to a device while cloud GPUs do the computation. The production version quickly becomes much larger. It includes audio processing, CPU management, firewall traversal, transcription, real-time data pipelines, buffer management, model selection, external system calls, phrase endpointing, interruption handling, state management, echo cancellation, and text-to-speech. He says phrase endpointing determines when a person has finished talking and expects the AI to respond. Interruptions create another problem because the bot must decide what to do while maintaining state. Echo cancellation remains necessary because users will not reliably wear headphones.

### Latency matters more than throughput in conversational voice
[07:47](https://www.youtube.com/watch?v=dRQHikOrH2A&t=467s)
Kwindla separates throughput from latency. Current language models can often produce content as fast as people can read or hear it, but the delay before the first response is what makes a voice interaction feel slow. He says human replies commonly arrive within 200 or 300 milliseconds, and that a target around 500 milliseconds matters because voice systems enter an uncanny valley above it. Gemini Pro's time to first token is given as about 900 milliseconds before network calls and other services are added. Model choice and pipeline design therefore constrain the result.

### The useful latency measurement runs from the end of speech to the start of the bot's speech
[12:00](https://www.youtube.com/watch?v=dRQHikOrH2A&t=720s)
The measurement Kwindla cares about begins when the user stops talking and ends when the user first hears the language model speak. It includes audio encoding, network transfer, packet handling, jitter buffering, decoding, transcription, endpointing, language-model inference, text-to-speech, and the return trip through the network. A laptop microphone on macOS can already add 30 to 40 milliseconds because of echo cancellation, noise removal, and gain control. Careful transcription and endpointing may take a couple hundred milliseconds, while the model ideally contributes about 100 milliseconds to time to first token.

### Colocating the pipeline gives engineers control over the latency budget
[10:20](https://www.youtube.com/watch?v=dRQHikOrH2A&t=620s)
Kwindla's main technical recommendation is to put natural-language processing, speech-to-text, phrase endpointing, language-model inference, and voice output together in one compute container whenever possible. Most AI applications call hosted services, and he acknowledges that there are good reasons to do so. That pattern becomes difficult when latency is the priority because each service call adds processing and network delay. Self-hosting lets a team tune the components, combine them, and control how they interact. His open-source demo reaches roughly 500 to 700 milliseconds of voice-to-voice latency.

### The demo measures both the whole interaction and each pipeline stage
[14:35](https://www.youtube.com/watch?v=dRQHikOrH2A&t=875s)
The FastBot demo shows a geography conversation while exposing latency data. The large number in the middle measures voice-to-voice latency on the client. Smaller metrics around the edges come from the server handling the pipeline, and are sent back so developers can compare individual stages and see variation. Kwindla says the goal of the demo was to test and make latency visible rather than to demonstrate sophisticated prompt engineering. The bot uses Deepgram models for transcription and voice generation, running in local Docker containers, and Llama 3 8B for language-model inference.

### Pipecat packages recurring real-time multimodal engineering work
[18:08](https://www.youtube.com/watch?v=dRQHikOrH2A&t=1088s)
Daily created Pipecat as an open-source framework for real-time and multimodal AI. Kwindla compares its role to frameworks such as LangChain and LlamaIndex, while stressing that Pipecat is vendor neutral even though Daily's work led to it. The framework supports conversational bots, speech-to-speech translation, voice-controlled agents, and real-time vision model applications. He invites developers to use its Discord, contribute service plug-ins, and become maintainers. The purpose is to avoid solving the same audio, model, and orchestration problems repeatedly across projects.

### Better multimodal models will help, but orchestration will remain necessary
[19:38](https://www.youtube.com/watch?v=dRQHikOrH2A&t=1178s)
Kwindla expects highly capable multimodal models to make fast applications easier to build, but says those models are not yet available in the form he wants. Even when they arrive, he expects orchestration layers to remain necessary. His argument is based on the number of separate jobs a real-time application must coordinate, from media transport to tools and model calls. Pipecat is positioned as that coordination layer for current systems, while remaining useful as model capabilities improve.

## Notable quotes
- "Speed really matters and the other is architectural flexibility is really, really important." (01:34)
- "The sort of 500 millisecond Target is pretty important because we hit that uncanny valley pretty quickly when we're above it." (09:22)
- "The most powerful tool we have today for making everything run fast in this domain is actually putting as much together into one compute container as we possibly can." (10:20)
- "If you're trying to make things really, really, really fast you need to figure out how to host stuff yourself and how to host stuff in a way where you can tune and control and combine everything." (11:26)

## Tools & references mentioned
- Daily
- GPT-4
- Deepgram
- Llama 3 8B
- Pipecat
- LangChain
- LlamaIndex
- WebRTC
- Cerebrium AI
- Tri County Health Services

## Who should watch
- You are building a voice agent and need to understand where response time goes across audio, networking, models, and speech generation.
- Your application depends on hosted AI APIs, but you are deciding whether lower latency justifies self-hosting and colocating components.
- You want an open-source framework for real-time voice, speech-to-speech translation, voice-controlled interfaces, or multimodal applications.

## Related talks

- [Realtime Voice AI](https://aietalks.com/talks/realtime-voice-ai) (Mark Backman, Daily, 1:09:41)
- [Building and Scaling an AI Agent Swarm of Low-Latency Real-Time Voice Bots](https://aietalks.com/talks/building-and-scaling-an-ai-agent-swarm-of-low-latency-real-time-voice-bots) (Damien Murphy, Deepgram, 1:07:23)
- [Engineering voice agents: Latency, quality, and scale](https://aietalks.com/talks/engineering-voice-agents-latency-quality-and-scale) (Rishabh Bhargava, Together AI, 24:35)
- [Giving a Voice to AI Agents](https://aietalks.com/talks/giving-a-voice-to-ai-agents) (Scott Stephenson, Deepgram, 13:08)
- [Serving Voice AI at Scale](https://aietalks.com/talks/serving-voice-ai-at-scale) (Arjun Desai, Cartesia & Rohit Talluri, AWS, 17:05)