Voice agents need roughly 800-millisecond voice-to-voice responses, and their pipelines must handle turn detection, interruptions, context, and tool calls.
2
Pipecat gives developers an open-source, vendor-neutral Python framework that can connect different telephony providers, models, and services.
3
Pipecat Cloud handles voice-specific deployment problems such as cold starts, autoscaling, global routing, and real-time networking.
Summary
Kwindla Hultman Kramer explains how to build voice agents with Pipecat, Daily's open-source and vendor-neutral framework. A voice agent has three practical parts: application code, deployment, and a connection to users through the web or telephony. The hard engineering work includes fast responses, turn detection, interruption handling, context management, asynchronous function calls, and connections to existing systems. Pipecat provides these building blocks while allowing developers to choose providers at each layer. Kramer then explains why voice workloads are difficult to deploy. Sessions run for a long time, require low-latency networking, and need fast startup and autoscaling. Pipecat Cloud is Daily's managed layer for these problems, with global infrastructure and support for voice-specific services. In the questions, Kramer compares speech-to-speech with speech-to-text pipelines, discusses Gemini and OpenAI, and gives a cautious view of newer systems such as Moshi, Sesame, and Ultravox.
Voice agents must answer quickly and know when a person has finished speaking
Kramer says people expect about 500 milliseconds in natural conversation, while Pipecat advises targeting 800 milliseconds for voice-to-voice response time. A slower interface can lose ordinary users. The system also has to decide when to speak. Human turn-taking is imperfect, and voice agents are currently worse at detecting the end of a turn. The agent must understand speech, retrieve useful information, sound natural, and respond conversationally. Kramer also warns that the user interface has to account for these timing expectations rather than treating voice like a normal request-response application.
Pipecat packages the hard parts of voice orchestration behind an open interface
Developers use Pipecat so they do not have to implement turn detection, interruption handling, context management, asynchronous function calling, and calls to other tools from scratch. The project is 100 percent open source and vendor neutral. It supports multiple providers at each layer, including telephony providers such as Twilio and Plivo. Pipecat also includes the open-source Smart Turn model. Kramer says the project supports more than 60 models and services, so developers can start without writing integrations while still having room to add substantial application code.
A Pipecat agent is a programmable Python pipeline that can stay simple or become complex
Pipecat agents are built from programmable media-handling elements, written primarily in Python, with some performance-sensitive parts using C. A small pipeline might contain a network input, a processing step, and a network output. Enterprise agents often grow much larger because they connect to legacy systems and other services. Kramer shows that developers can use either OpenAI's chained transcription, text LLM, and voice output approach or its speech-to-speech models by changing only a few lines of code. He also describes a starter example that routes audio through two Gemini multimodal live API instances, one handling conversation and another judging a game.
Voice workloads need deployment systems designed around long sessions and low latency
Kramer says voice agents are difficult to deploy because sessions are long-running and use networking protocols built for low latency. Standard HTTP autoscaling does not solve these workloads. Pipecat users repeatedly asked in Discord how to deploy and scale their agents, often reaching the point of asking how to manage Kubernetes. Pipecat Cloud grew out of that need. Kramer describes it as a thin wrapper around Docker and Kubernetes, optimized for voice AI, so teams can deploy their Pipecat code without building the surrounding operational system themselves.
Pipecat Cloud targets cold starts, autoscaling, routing, and regional data needs
A caller expects the agent to answer quickly, so cold starts matter even more than in many other generative AI systems. Resources also need to expand and contract with traffic, which is hard when demand varies by time or is unpredictable. Kramer compares ordinary HTTP latency with voice latency, where a P95 response above 800, 900, or 1,000 milliseconds can damage the conversation. The full path, including the client, the hosting environment, and the Kubernetes network, must be optimized. Global deployment can reduce latency and meet GDPR or data residency requirements, but it also has to remain affordable.
Turn detection and background noise can cause unwanted interruptions
Kramer calls turn detection one of the main voice AI problems being worked on in 2025 and points developers to Pipecat's open-source Smart Turn model. Background noise creates a separate problem. Transcription models can tolerate many noisy settings, but language models may treat speech-like noise as real speech. That can trigger interruptions and add false content to transcripts, including when a speech-to-speech model is used. Kramer says Crisp currently offers the strongest solution he knows for background noise. Crisp is available inside Pipecat Cloud without a separate charge, while self-hosted users need their own license.
Speech-to-speech models preserve audio information but remain weaker for many enterprise tasks
Speech-to-speech models can avoid information loss during transcription, which matters for mixed-language speech and other audio details. They may also reduce latency when one model handles the whole exchange. Kramer says the practical benefit depends on the APIs and inference stack available today. Audio uses much more context than text, and models have less audio training data than text data. That can reduce performance and cause problems such as replying in the wrong language. He expects speech-to-speech systems to become the default for most voice AI, but says current enterprise applications often still need text-mode models for reliable instruction following and function calling.
Model choice should be tested in the actual Pipecat pipeline
Kramer considers GPT-4o in text mode and Gemini 2.0 Flash in text mode roughly equivalent for the use cases he tests, so he recommends swapping them in the same Pipecat pipeline and running evaluations. He says Gemini is aggressively priced and can be around 10 times cheaper than GPT-4o for a 30-minute conversation, though prices can change. Gemini also handles native audio input well, including a pipeline with audio input and text output. He advises experimenting with speech-to-speech models such as Ultravox, while checking whether the underlying model is capable enough for the application's needs.
"What we came to realize is that there was sort of a middle of the stack that people were asking about a lot in the open source community that boiled down to how do I do my Kubernetes?"09:39
Who should watch
You are building a phone-based or browser-based voice agent and need control over model, telephony, and infrastructure providers.
Your prototype works, but you are running into cold starts, autoscaling, global routing, or real-time latency problems.
You are deciding between a text-based voice pipeline and newer speech-to-speech models for an enterprise application.
