# Your realtime AI is ngmi Sean DuBois, OpenAI and Pion & Kwindla Hultman Kramer, Daily and Pipecat | AI Engineer World's Fair 2025 | 16:30 Source: https://www.youtube.com/watch?v=E71YtNbCFXY Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/your-realtime-ai-is-ngmi Published: 2025-07-31 Tags: latency, voice ## TL;DR - Voice AI has to target roughly 500 milliseconds of response time, because delays above one second lead to low completion rates, low NPS scores, and hang-ups. - WebRTC is the right network layer for edge-to-cloud audio and video because it handles packet loss, jitter, resampling, packetization, bandwidth estimation, and observability. - WebRTC connections can support local peer-to-peer devices, cloud-hosted agents, and multi-party applications such as meetings with LLMs. ## Summary Sean DuBois and Kwindla Hultman Kramer argue that realtime AI applications need to be designed around network behavior before developers build the agent experience. People expect an exchange with an AI to feel close to a human conversation, so voice-to-voice latency should be around 500 milliseconds and generally below one second. They compare WebSockets with WebRTC and recommend WebSockets for server-to-server communication, structured data, and prototypes, while WebRTC is suited to audio and video streams over the internet. WebRTC uses best-effort delivery, buffering, bandwidth estimation, and other machinery that would otherwise have to be built by the application team. The talk then shows several uses of WebRTC, including a Raspberry Pi voice device called Squabbert, a direct local connection, cloud-hosted AI, and multi-party agents. A builder named Yashin also demonstrates a language-learning project for bilingual children. ## Key ideas ### Voice agents have to respond within a human conversational rhythm [01:56](https://www.youtube.com/watch?v=E71YtNbCFXY&t=116s) DuBois says that a person-to-person exchange sounds natural at around 500 milliseconds. Users bring that expectation to AI systems. Voice-to-voice latency measures the time between the human stopping speaking and hearing the first audio from the language model. Once response latency rises much above one second, the speaker says voice agents tend to get very low completion rates and low NPS scores, along with more hang-ups. Every part of the application therefore has to be built for fast response times, rather than adding voice to a system designed around slower text interactions. ### The network can make a voice system slow even when the model is fast [02:35](https://www.youtube.com/watch?v=E71YtNbCFXY&t=155s) The example application takes just under a second from the end of the user's speech to the first returned audio. DuBois calls that good, but not great, and says making it faster can require trade-offs in quality or cost. The model can add delay, and network conditions can make the result worse. Bluetooth receives particular blame. The largest mistake he sees is choosing the wrong way to send and receive audio, because a voice agent's latency is the sum of delays across the whole path from the browser to the cloud service and back. ### WebSockets fit prototypes and structured data, while WebRTC fits media streams [03:27](https://www.youtube.com/watch?v=E71YtNbCFXY&t=207s) The speakers explain that WebSockets are useful for long-lived connections carrying small amounts of data and for prototypes that need to target many platforms. They recommend WebRTC when a web or native application sends audio or video streams across the internet. WebSockets use TCP, which guarantees ordered delivery. That means a lost packet can hold up later data while the operating system keeps retrying. For conversation, old audio is less valuable than timely audio, so WebRTC uses packet handling and buffering that can ignore data arriving outside the latency budget. ### WebRTC removes network and media work that application teams would otherwise build [06:42](https://www.youtube.com/watch?v=E71YtNbCFXY&t=402s) DuBois says WebRTC handles more than packet loss. It provides resampling, packetization, bandwidth estimation for changing networks, and standard APIs for statistics and observability. A WebSocket implementation would need to build these pieces itself. The speakers show code comparing a bidirectional WebRTC audio stream with a WebSocket version. Their advice is to spend development time on the application instead of handling sample rates and network fluctuations. The OpenAI Realtime API supports both connection types, with each suited to different jobs. ### WebRTC already supports many forms of realtime interaction [07:49](https://www.youtube.com/watch?v=E71YtNbCFXY&t=469s) Kramer says developers can put realtime audio into websites, iOS apps, Android apps, and other embedded devices. He points to Facebook Messenger, WhatsApp, Zoom, and Discord as applications that use WebRTC. The same technology has been used for surgery over the internet and for remotely operating vehicles in the field. He expects voice to become a core interface for generative AI, including development environments, brainstorming, personal assistants, coaching, therapy, and research. He compares the current stage to late 2007 for mobile computing, before familiar interaction patterns had been invented. ### Squabbert shows that a small device can connect directly to an AI system [10:06](https://www.youtube.com/watch?v=E71YtNbCFXY&t=606s) Squabbert is a Raspberry Pi voice device built by Chad and named by Chad's daughter, Ella. Its stack includes MLX Whisper, Gemma 3, and custom logic for sampling. During the demo, Squabbert creates a short poem under a two-syllable-word constraint, with mixed success. The technical point is that the Raspberry Pi connects directly to the laptop through a peer-to-peer WebRTC connection on the local network. The device does not need a separate server for this setup, and the demo makes the network connection part of the product design rather than an invisible transport detail. ### The same WebRTC system can move from local peers to cloud agents and group conversations [12:13](https://www.youtube.com/watch?v=E71YtNbCFXY&t=733s) Kramer describes three connection patterns for the demo. Squabbert can connect locally to the laptop, connect to a cloud server that runs the AI, or connect to Pipecat in a multi-party setup. The multi-party option could bring language models into meetings and similar settings. He presents this as a choice that can match the application's needs. The important design point is that WebRTC does not force one deployment shape. The connection can stay close to the device, run through cloud infrastructure, or include several participants. ### Lowering the technical barrier lets non-programmers build voice products [13:16](https://www.youtube.com/watch?v=E71YtNbCFXY&t=796s) The closing example comes from Yashin, a parent raising two bilingual children. She wants language education to feel more natural and fun, and built an early project with guidance from someone in the community despite not being technical. The demo speaks with a child and introduces Mandarin vocabulary. Yashin says she already has parents ready to test it. Kramer connects her example to the broader goal of making voice and WebRTC tools accessible to people with ideas but little coding experience, and points viewers toward Pipecat, community channels, and the project's resources. ## Notable quotes - "If you're talking to a person, around 500 milliseconds sounds natural." (01:56) - "Use websockets for those server to server use cases and small amounts of structured data in places that you want to prototype." (04:38) - "Use WebRTC if you're sending audio and video streams over the internet from your web app, your native." (04:38) - "WebRTC is kind of the standard language of the real-time world." (08:28) - "Voice is like the next bicycle for the mind." (09:25) ## Tools & references mentioned - OpenAI - Pion - Daily - Pipecat - OpenAI Realtime API - WebRTC - WebSockets - MLX Whisper - Gemma 3 - Facebook Messenger - WhatsApp - Zoom - Discord - Chad - Ella - Yashin ## Who should watch - You are building a voice agent and are deciding whether to use WebSockets or WebRTC for the media path. - Your application needs to handle changing networks, packet loss, jitter, audio quality, and connection statistics without implementing all of that infrastructure yourself. - You want to build a voice interface on a device, in a cloud service, or inside a multi-party application and need to understand the available connection patterns. ## Related talks - [Engineering voice agents: Latency, quality, and scale](https://aietalks.com/talks/engineering-voice-agents-latency-quality-and-scale) (Rishabh Bhargava, Together AI, 24:35) - [Building and Scaling an AI Agent Swarm of Low-Latency Real-Time Voice Bots](https://aietalks.com/talks/building-and-scaling-an-ai-agent-swarm-of-low-latency-real-time-voice-bots) (Damien Murphy, Deepgram, 1:07:23) - [Designing Voice Agents for Real Conversations](https://aietalks.com/talks/designing-voice-agents-for-real-conversations) (Chintan Agrawal & Daniel Wirjo, AWS, 32:57) - [Giving a Voice to AI Agents](https://aietalks.com/talks/giving-a-voice-to-ai-agents) (Scott Stephenson, Deepgram, 13:08) - [Your Voice Agent Doesn't Need a Frontier Model](https://aietalks.com/talks/your-voice-agent-doesnt-need-a-frontier-model) (Joel Allou & Ornella Bahidika, Microsoft, 05:45)