# Why ChatGPT Keeps Interrupting You Tom Shapland, LiveKit | AI Engineer World's Fair 2025 | 27:03 Source: https://www.youtube.com/watch?v=1v9zBiZKlIY Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/why-chatgpt-keeps-interrupting-you Published: 2025-07-31 Tags: chatbots, voice ## TL;DR - Voice AI agents interrupt because current turn detection often treats a short silence as the end of a user's turn. - Human listeners predict the meaning and endpoint of an utterance before the speaker finishes, while most voice AI pipelines process speech and response generation in separate stages. - Semantic and acoustic models improve turn detection, but Tom Shapland expects production systems to keep using controllable cascaded pipelines rather than relying entirely on full-duplex models. ## Summary Tom Shapland explains why interruptions remain a serious problem for Voice AI agents. Current systems commonly use a speech-to-text model, voice activity detection, an LLM, and text-to-speech. The VAD often combines a speech detector with a silence rule, such as ending the turn after roughly half a second without speech. Human conversation works differently. Listeners predict the speaker's intended meaning, refine the likely endpoint from semantics and syntax, then use tone and other acoustic cues to finalize it. Shapland describes newer systems that add semantic and acoustic models to VAD, including LiveKit's text-based end-of-utterance model. He also discusses full-duplex models such as Moshi and Meta AI's Sync LLM, which listen and generate continuously. These models handle turn-taking naturally, but they are less capable as general instruction-following models. Shapland expects faster cascaded systems with smarter turn detection to remain the practical choice for commercial agents. ## Key ideas ### Turn-taking failures make voice agents lose users [00:31](https://www.youtube.com/watch?v=1v9zBiZKlIY&t=31s) Shapland calls interruptions the biggest problem in Voice AI agents. An interruption from ChatGPT Advanced Voice Mode is annoying, but the cost is higher in a business workflow. If a dental assistant interrupts a patient, the patient may hang up, and the dentist may stop paying the developer. Turn-taking means deciding who controls the floor during a conversation. It is difficult because responses happen quickly and vary across cultures and people. Shapland contrasts Danish speakers, who may take relatively long to respond, with Japanese speakers, who may respond almost immediately. Even one person changes timing with context, such as responding faster when angry. ### Most production pipelines decide that a turn ended from speech and silence [02:29](https://www.youtube.com/watch?v=1v9zBiZKlIY&t=149s) Shapland gives a simplified voice agent pipeline. Audio chunks from the user go to speech-to-text, then a voice activity detector decides whether the user has finished. If the turn is complete, the transcript goes to an LLM. The streamed response goes to text-to-speech, and the resulting audio returns to the user. The VAD usually has a neural network that classifies speech versus non-speech, followed by a silence algorithm. In his example, the system treats more than half a second without speech as the end of the turn. This process looks backward at silence instead of predicting where the speaker is going. ### Human listeners predict the endpoint before the speaker finishes [04:22](https://www.youtube.com/watch?v=1v9zBiZKlIY&t=262s) Human turn-taking is fast even though producing a response takes longer than the time between turns. Shapland cites research describing a response beginning around 200 milliseconds after the previous speaker finishes, while finding words, generating speech, and articulating it takes about 600 milliseconds. The listener must therefore predict when the turn will end and begin preparing a response early. Semantic content is the strongest input. Syntax, prosody, tone, and visual cues also contribute. The listener first predicts the intended message, then refines the endpoint with meaning and sentence structure, and finally uses acoustic details to finalize the timing. ### Human conversation runs comprehension and response generation in parallel [06:53](https://www.youtube.com/watch?v=1v9zBiZKlIY&t=413s) Shapland describes the human mind as full duplex. A listener is still interpreting the current speaker while beginning to form a response. The comprehension process infers the intended message and predicts when the speaker will finish. The production process starts before the turn ends. Current voice AI systems are much simpler by comparison. They mainly classify speech or non-speech, look at what has already happened, and process the pipeline serially. Shapland connects this difference to interruption failures. A system that waits for silence and then starts generating has less information about whether the speaker was pausing, thinking, or continuing. ### Semantic end-of-utterance models can extend the silence window [09:25](https://www.youtube.com/watch?v=1v9zBiZKlIY&t=565s) LiveKit's approach augments VAD with a text-based semantic model. It takes the last four turns as context: the agent's turn, the user's turn, the next agent turn, and the user's current turn. A transformer predicts an end-of-utterance token from that conversation context. If the model predicts that the user's turn has not ended, the system extends the silence period instead of triggering a response. In the demo, the traditional VAD repeatedly interrupted while a person was explaining how he built a demo. With semantic turn detection enabled, the speaker could pause and think, and the agent waited for the contextual end of the turn. ### Acoustic models add information that text-only detection misses [13:04](https://www.youtube.com/watch?v=1v9zBiZKlIY&t=784s) Another group of systems combines the transcript with the audio signal. These models consume audio tokens and predict the probability that the user has finished speaking. Shapland mentions the Daily team's open-weight Smart Turn model, which combines transformer processing with acoustic features. He also describes AssemblyAI's streaming speech-to-text service, which emits both a transcript and a likelihood that the speaker has finished. These systems use semantic and acoustic information together. A limitation is that a speech-to-text model's built-in end-of-utterance prediction may see only the user's speech, not the agent's current speech, so it lacks the full conversational context. Shapland still says these approaches work remarkably well. ### Full-duplex models imitate continuous human listening and speaking [15:58](https://www.youtube.com/watch?v=1v9zBiZKlIY&t=958s) Full-duplex models process incoming audio and generate speech at the same time. Shapland compares them with computer vision systems that improved when neural networks received raw images instead of relying on hand-written rules. Moshi makes the design concrete: it is always listening and always generating, even when it is not speaking aloud. During the other person's turn, it emits natural silence that cannot be heard. Meta AI's Sync LLM follows a similar full-duplex design and forecasts what the user is saying about five tokens, or roughly 200 milliseconds, ahead. These systems handle turn-taking naturally, but Shapland says they are small, less capable LLMs with weaker instruction following. ### Commercial agents will need controllable cascades alongside smarter turn detection [18:14](https://www.youtube.com/watch?v=1v9zBiZKlIY&t=1094s) Shapland does not expect full-duplex models alone to solve production Voice AI. Commercial systems need control over output details, including how brand names are spoken. He expects smarter VAD augmentations and faster models inside the cascaded speech-to-text, LLM, and text-to-speech pipeline. He also explains that speech-to-speech systems such as OpenAI's Realtime API still use VAD internally, with an optional semantic VAD mode that augments it. ChatGPT Advanced Voice Mode interrupts because it decides the user is finished from the time since the last word or from the preceding content. No current approach has perfected the problem. ## Notable quotes - "Turn taking is this unspoken system we have for who controls the floor between speakers during a conversation." (01:10) - "The primary inputs on creating that prediction are the semantic and this one's the most important one, what the content is of what the person is saying." (05:03) - "The human mind is full duplex. We're both processing input and we're starting to generate output at the same time." (06:53) - "I think we just for real production commercial use cases of voice AI, we need more control." (18:14) - "The industry as a whole doesn't have a good benchmark around turn taking." (24:26) ## Tools & references mentioned - ChatGPT Advanced Voice Mode - LiveKit - OpenAI Realtime API - OpenAI - Smart Turn - Daily - Quinn - AssemblyAI - Moshi - Meta AI - Sync LLM - Canonical AI - Tule - Jordan Deersley ## Who should watch - You are building a phone agent or voice assistant where an interruption can make the user hang up. - You want to understand what VAD, semantic end-of-utterance models, and acoustic turn detectors each contribute. - You are comparing cascaded voice pipelines with full-duplex models and need to weigh natural turn-taking against control and instruction following. ## Related talks - [Designing Voice Agents for Real Conversations](https://aietalks.com/talks/designing-voice-agents-for-real-conversations) (Chintan Agrawal & Daniel Wirjo, AWS, 32:57) - [Voice AI: When Is the "Her" Moment?](https://aietalks.com/talks/voice-ai-when-is-the-her-moment) (Neil Zeghidour, Gradium AI, 19:27) - [ChatGPT is poorly designed. So I fixed it](https://aietalks.com/talks/chatgpt-is-poorly-designed-so-i-fixed-it) (Ben Holmes, Warp.dev, 05:21) - [The Prompt Is Still a Punch Card](https://aietalks.com/talks/the-prompt-is-still-a-punch-card) (Ted Johnson, JoinIn AI, 20:13) - [Giving a Voice to AI Agents](https://aietalks.com/talks/giving-a-voice-to-ai-agents) (Scott Stephenson, Deepgram, 13:08)