# Voice agents in real conversations

A pack of 10 talks from the AI Engineer YouTube channel, in the order to watch them. 3h 14m of video.
Page: https://aietalks.com/packs/voice

A voice agent can have a strong model and still fail because it talks over someone, leaves a dead pause or hands a caller to a person with no useful context. Turn-taking sets the floor: the agent must know what the call should accomplish, when someone has finished speaking, and which interruptions should stop it. Latency then becomes a system problem. Speech recognition, model calls, synthesis and every network hop spend part of the same budget. Smaller models and state machines move decisions off the speaking path; colocated services recover more time. Intercom's support deployment is the final test. Monitoring, escalation and a useful context handoff decide whether the design can answer a real phone line rather than merely impress in a demo.

## This pack is for you if

- Your voice agent sounds capable until a pause, correction, or background noise throws the conversation off.
- You are choosing between a chained pipeline and speech-to-speech, or deciding which work belongs outside the speaking model.
- Your demo is fast enough for one call, but production still needs concurrency, monitoring, and a safe human handoff.

## The talks, in order

### 1. Voice Agents: the good, the bad, and the ugly

Eddie Seagull, Fractional AI | 18:48 | AI Engineer Summit 2025
Video: https://www.youtube.com/watch?v=pzmbleiOfCM
Summary: https://aietalks.com/talks/voice-agents-the-good-the-bad-and-the-ugly.md

Why first: Seagull shows how transcription, streaming, drift and subjective evaluation make ordinary agent problems harder. Caryotakis follows by deciding which of those problems matter to the customer on the other end of a call.

### 2. Voice Agent Engineering

Nik Caryotakis, SuperDial | 19:07 | AI Engineer Summit 2025
Video: https://www.youtube.com/watch?v=2p2ErKRELHM
Summary: https://aietalks.com/talks/voice-agent-engineering.md

Why second: Seagull inventories the ways a voice agent can fail. Caryotakis makes a product choice among them: finish the conversation reliably before chasing a more realistic voice, then read the script aloud to find the awkward parts.

### 3. Why ChatGPT Keeps Interrupting You

Tom Shapland, LiveKit | 27:03 | AI Engineer World's Fair 2025
Video: https://www.youtube.com/watch?v=1v9zBiZKlIY
Summary: https://aietalks.com/talks/why-chatgpt-keeps-interrupting-you.md

Why here: Caryotakis asks teams to design the conversation as spoken behavior. Shapland explains the mechanism that most often ruins that behavior: a serial pipeline that mistakes a short silence for the end of a turn.

### 4. Designing Voice Agents for Real Conversations

Chintan Agrawal & Daniel Wirjo, AWS | 32:57 | AI Engineer World's Fair 2026
Video: https://www.youtube.com/watch?v=hMlLw1LeIK8
Summary: https://aietalks.com/talks/designing-voice-agents-for-real-conversations.md

Why here: Shapland explains why endpoint prediction is hard. Agrawal and Wirjo compare three ways to make that decision in a working pipeline, then show why a cough, an acknowledgement and a correction should not trigger the same response.

### 5. Building Effective Voice Agents

Toki Sherbakov & Anoop Kotha, OpenAI | 17:17 | AI Engineer World's Fair 2025
Video: https://www.youtube.com/watch?v=-OXiljTJxQU
Summary: https://aietalks.com/talks/building-effective-voice-agents.md

Why here: Agrawal and Wirjo settle how the user gets the floor. Sherbakov and Kotha widen the choice to the whole agent, weighing chained and speech-to-speech systems and delegating difficult work away from the frontline voice model.

### 6. Your realtime AI is ngmi

Sean DuBois, OpenAI and Pion & Kwindla Hultman Kramer, Daily and Pipecat | 16:30 | AI Engineer World's Fair 2025
Video: https://www.youtube.com/watch?v=E71YtNbCFXY
Summary: https://aietalks.com/talks/your-realtime-ai-is-ngmi.md

Why here: Sherbakov and Kotha choose what runs inside the agent. DuBois and Kramer turn to the path between user and model, where ordered delivery, packet loss and jitter can make a fast architecture feel slow.

### 7. Engineering voice agents: Latency, quality, and scale

Rishabh Bhargava, Together AI | 24:35 | AI Engineer Europe 2026
Video: https://www.youtube.com/watch?v=N7b1PJc7SFc
Summary: https://aietalks.com/talks/engineering-voice-agents-latency-quality-and-scale.md

Why here: DuBois and Kramer fix the media path from the user's device. Bhargava budgets the delay that remains across speech recognition, the language model and speech generation, then shows what colocation and a thinker-talker split can save.

### 8. Your Voice Agent Doesn't Need a Frontier Model

Joel Allou & Ornella Bahidika, Microsoft | 05:45 | AI Engineer World's Fair 2026
Video: https://www.youtube.com/watch?v=fnLBmfsI_Fg
Summary: https://aietalks.com/talks/your-voice-agent-doesnt-need-a-frontier-model.md

Why here: Bhargava keeps a small model on the conversational path and sends harder work elsewhere. Allou and Bahidika make the bargain concrete by moving lesson state into code so the speaking model can answer in time.

### 9. Optimizing Inference for Voice Models in Production

Philip Kiely, Baseten | 15:13 | AI Engineer World's Fair 2025
Video: https://www.youtube.com/watch?v=gmTHs5T_YAE
Summary: https://aietalks.com/talks/optimizing-inference-for-voice-models-in-production.md

Why here: Allou and Bahidika cut the model's thinking time. Kiely finds the next constraint in speech serving, where first-byte latency and concurrent streams matter more than raw token speed, and careless client code can give the gain back.

### 10. Shipping an Enterprise Voice AI Agent in 100 Days

Peter Bar, Intercom | 17:10 | AI Engineer World's Fair 2025
Video: https://www.youtube.com/watch?v=HOYLZ7IVgJo
Summary: https://aietalks.com/talks/shipping-an-enterprise-voice-ai-agent-in-100-days.md

Why last: Kiely gets the listening, thinking and speaking machinery into its latency budget. Bar puts it on a real support line, where short answers, filler speech, monitoring, escalation and a useful handoff decide whether callers keep using it.