# Voice Agent Engineering

Nik Caryotakis, SuperDial | AI Engineer Summit 2025 | 19:07

Source: https://www.youtube.com/watch?v=2p2ErKRELHM
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/voice-agent-engineering
Published: 2025-04-18
Tags: observability, reliability, testing, voice

## TL;DR
- Production voice agents should favor reliable conversations over realistic voices and speech-to-speech effects.
- A voice product becomes valuable through its conversation design and vertical integrations, rather than its voice or interruption handling.
- Teams can reach reliable voice agents by choosing an extensible stack, tracking latency, preparing fallbacks, and testing full phone interactions.

## Summary
Nik Caryotakis explains how SuperDial builds voice agents for healthcare phone calls, where the customer cares about getting accurate answers in a structured format. SuperDial uses bots for calls, transfers difficult cases to a human fallback team, and learns from office hours and phone-tree traversal. Nik argues that production systems should favor reliability over realistic voices, especially while speech-to-speech models can produce unusable non-speech output. He describes the work of a voice AI engineer, including real-time systems, audio data, asynchronous code, conversation design, and ethical concerns around accents and dialects. SuperDial uses open-source orchestration with a sequenced speech-to-text, LLM, and text-to-speech pipeline. The team also owns model routing, logging, observability, pronunciation handling, fallbacks, and end-to-end phone testing. Nik is direct that the hardest work comes after the MVP, in the last mile of conversation behavior and operational reliability.

## Key ideas
### Reliable conversations matter more than realistic voices in production
[00:54](https://www.youtube.com/watch?v=2p2ErKRELHM&t=54s)
Nik says voice AI in 2025 has fast, affordable language models and realistic text-to-speech, but these systems still create problems such as audio hallucinations, pronunciation errors, and spelling issues. SuperDial has favored reliability over realism because early speech-to-speech models sometimes output things that are not usable speech. He argues that an agent's value does not come mainly from its voice, interruption handling, realistic sound, or turn-taking. The differentiator is the content of its conversations and the vertical integrations around them.

### SuperDial sells completed answers, with humans available when bots fail
[02:14](https://www.youtube.com/watch?v=2p2ErKRELHM&t=134s)
SuperDial automates calls to insurance companies for healthcare administration businesses. Customers provide the people to call and the questions to ask through CSV, an API, or EHR integrations. SuperDial returns the answers in a structured format. Its internal loop waits for offices and call centers to open, attempts the call with a voice bot, and sends the call to a fallback team when the bot needs a human or cannot finish after several attempts. Customers care that the call gets completed and the answers come back reliably, whether a person or a bot handles it.

### Healthcare calls create a learning and audit loop
[03:50](https://www.youtube.com/watch?v=2p2ErKRELHM&t=230s)
SuperDial updates office hours for each number it calls and learns from phone-tree traversal, so future calls can improve. Because the conversations involve sensitive healthcare information, the team randomly removes calls for audits and checks that the system is working. Nik gives a prior authorization call as an example. The bot reaches a human after navigating a phone tree, asks for a member ID, CPT code, case number, and clinical-document details, then returns the information to the customer. A boring call is a successful call for this product.

### The voice AI engineer works across audio, real-time systems, and conversation design
[06:39](https://www.youtube.com/watch?v=2p2ErKRELHM&t=399s)
Nik describes a voice AI engineer as someone working with multimodal data such as MP3 files, audio bytes, transcripts, transcription models, voice models, and speech-to-speech systems. Real-time latency makes asynchronous programming and Python more important. The product constraint is an actual conversation, often inserted into an existing business interaction where callers already have expectations. At SuperDial, the team says, 'say the right thing at the right time' and 'build this plane while we fly it.' Customer-specific scripts sit on top of a shared voice AI stack.

### Conversation design has moved from prescribed paths to descriptions
[10:02](https://www.youtube.com/watch?v=2p2ErKRELHM&t=602s)
Nik contrasts older voice UI, where teams mapped every possible phone-tree direction, with current systems that describe the desired behavior and let the model produce the next action. For SuperDial's existing phone conversations, open-ended questions can work better than forcing a representative to choose from a list. The team tries to adapt to whatever the call-center representative says instead of preventing every unexpected response. Nik recommends hiring a conversation designer when possible. A practical starting exercise is a table read, with one person acting as the bot and another acting as the user.

### An extensible orchestration layer lets the team focus on conversations
[12:36](https://www.youtube.com/watch?v=2p2ErKRELHM&t=756s)
SuperDial replaced its quickly assembled orchestration framework with Pipecat, an open-source framework maintained by Daily. Nik says it is easy to extend and hack on, which matters for transfers and calls that can last an hour and a half. Self-hosting lets SuperDial deploy and scale it as needed. The company owns an OpenAI-compatible endpoint so it can route requests to different models, including models chosen for latency. It routes generative responses through TensorZero, which provides structured and typed LLM endpoints for production experiments, and self-hosts Langfuse for logging and observability because the calls must be HIPAA compliant.

### The last mile includes pronunciation, spelling, fallbacks, and testing
[14:40](https://www.youtube.com/watch?v=2p2ErKRELHM&t=880s)
The model's text is not always the right text to send to speech synthesis, and neither may match the recording that the team wants. Nik shows how pronunciation syntax can make a surname sound correct and how a spelling function can control pauses in long strings such as member IDs. The team reviews recordings as well as transcripts. He also warns teams to choose a phone persona carefully, since SuperDial's former bot name, Billy, caused repeated pronunciation confusion. Other advice includes tracking time to first byte for every processor, keeping upgrade paths for transcription, preparing fallbacks for every stack component, and testing phone behavior end to end.

## Notable quotes
- "A boring call for us is an excellent call because it turns out a lot of work is boring." (06:01)
- "Ultimately it's going to be in the conversational content and the design there and the vertical integrations around it that make your agents work actually valuable." (08:19)
- "The transcript that you may write out by hand immediately the sort of gaps and awkwardness of it comes out when you say these things out loud." (12:18)
- "If your Bot can't talk to an MP3 then you probably have bigger problems." (17:35)
- "What's going to make your Bot unique is the conversation." (16:40)

## Tools & references mentioned
- SuperDial
- OpenAI Realtime API
- Pipecat
- Daily
- TensorZero
- Langfuse
- Deepgram
- Koval
- Vapi
- Kathy Pearl
- Gemini

## Who should watch
- You are building a voice agent prototype and need a practical path from a sequenced STT, LLM, and TTS pipeline to a production phone system.
- Your team handles long, sensitive, or failure-prone calls and needs advice on human fallbacks, structured results, and HIPAA-compliant operations.
- You are deciding whether to build voice infrastructure yourself and want concrete guidance on orchestration, latency, pronunciation, and end-to-end testing.

## Related talks

- [Building Effective Voice Agents](https://aietalks.com/talks/building-effective-voice-agents) (Toki Sherbakov & Anoop Kotha, OpenAI, 17:17)
- [Voice Agents: the good, the bad, and the ugly](https://aietalks.com/talks/voice-agents-the-good-the-bad-and-the-ugly) (Eddie Seagull, Fractional AI, 18:48)
- [Giving a Voice to AI Agents](https://aietalks.com/talks/giving-a-voice-to-ai-agents) (Scott Stephenson, Deepgram, 13:08)
- [Engineering voice agents: Latency, quality, and scale](https://aietalks.com/talks/engineering-voice-agents-latency-quality-and-scale) (Rishabh Bhargava, Together AI, 24:35)
- [Building voice agents with OpenAI](https://aietalks.com/talks/building-voice-agents-with-openai) (Dominik Kundel, OpenAI, 1:25:35)