# Giving a Voice to AI Agents

Scott Stephenson, Deepgram | AI Engineer World's Fair 2024 | 13:08

Source: https://www.youtube.com/watch?v=heYmh_lsX5s
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/giving-a-voice-to-ai-agents
Published: 2025-02-10
Tags: cost, latency, multimodal, voice

## TL;DR
- Voice AI has moved from narrow, slow command systems to open-ended agents built from speech recognition, language models, and speech synthesis.
- The next improvement will come from passing context between every part of the conversation, including tone, background sounds, speaking speed, and prior turns.
- Voice systems should keep separate components when businesses need control over accuracy, behavior, voice, latency, and cost.

## Summary
Scott Stephenson describes the shift from early voice assistants such as Siri to open-ended agents that turn speech into text, process it with a language model, and turn the response back into speech. The current stack is already fast enough for natural turn-taking in some setups, but its components usually lose context between turns. Stephenson argues that future systems will pass context through the whole pipeline. Speech recognition could receive prior conversation, images, documents, or audio, while language and speech models could carry information about emotion, pace, background sound, and vocal style. That context should make agents more accurate and more human-like. Stephenson also argues for keeping components separate when companies need control, such as banks and call centers. Teams can choose different model sizes for each task and reduce cost. Deepgram's planned full-stack voice agent is intended to reduce latency when these pieces are combined.

## Key ideas
### Voice AI has moved from narrow commands to open-ended conversations
[02:34](https://www.youtube.com/watch?v=heYmh_lsX5s&t=154s)
Stephenson contrasts early voice AI, represented by Siri, with the current generation. Earlier systems were slow, less accurate, and limited to specific domains. Current systems can accept and produce arbitrary text because large language models sit between speech recognition and speech synthesis. The basic loop is speech to text, text to text through a language model, and text to speech. The engineering challenge is to make that loop fast, intelligent, and expressive enough for an agent to respond properly in conversation.

### Latency and accuracy have reached the range needed for natural turn-taking
[05:07](https://www.youtube.com/watch?v=heYmh_lsX5s&t=307s)
Stephenson says speech recognition has improved from roughly 75% accuracy five to eight years ago to more than 90% today. Its delay has also fallen from two to five seconds to about 100 or 200 milliseconds for real-time use. He cites a Daily blog post describing a complete voice conversation in less than 500 milliseconds using Deepgram for speech recognition and speech synthesis with Llama in the middle. Human turn-taking usually takes about 400 to 600 milliseconds, so the basic speed problem is becoming manageable.

### The current pipeline loses conversation context between its components
[06:05](https://www.youtube.com/watch?v=heYmh_lsX5s&t=365s)
The usual speech-to-text, text-to-text, and text-to-speech systems work fairly independently. They do not automatically pass the full context of the conversation from one stage to the next. Stephenson says this causes misunderstandings in perhaps 10% or 20% of turns. A speech recognizer may receive only the current audio and a few seconds of context, even though the conversation before that audio could clarify what the person means. The missing information limits both recognition and the response that follows.

### Context can include audio, images, documents, and prior turns
[06:48](https://www.youtube.com/watch?v=heYmh_lsX5s&t=408s)
Stephenson proposes giving speech recognition more than an audio clip. Its input could include a prompt made from other audio, images, documents, or the previous conversation. With the entire conversation available, the model can transcribe more accurately. The model can also produce context for later stages, including readable text, audio, images, or vector embeddings. This shared state lets the system carry information through the conversation instead of rebuilding its understanding on every turn.

### Future agents will carry emotion and delivery instructions through the stack
[08:11](https://www.youtube.com/watch?v=heYmh_lsX5s&t=491s)
Stephenson says a contextual system could recognize whether someone sounds angry or happy, whether the conversation is moving quickly or slowly, and whether light music is playing in the background. The language model could then tell speech synthesis how to deliver the response, such as speaking softly, slowly, or in an authoritative tone. The speech model could report how it sounded, and that information could be passed into the next turn. Stephenson calls this approach contextual AI at Deepgram.

### Separate models give businesses more control than one speech-to-speech model
[09:19](https://www.youtube.com/watch?v=heYmh_lsX5s&t=559s)
Stephenson accepts that several components may eventually be combined into a multimodal or speech-to-speech model, but says a single open-ended model is harder to control. A bank may want highly accurate speech recognition, a smaller language model for tasks such as password resets, and one calm, expressive voice. Keeping the components separate lets a company choose the behavior and compute used at each stage. He says this control matters for businesses such as banks, Spotify, food ordering services, and call centers.

### Choosing the right component size can reduce voice AI costs
[10:32](https://www.youtube.com/watch?v=heYmh_lsX5s&t=632s)
Stephenson addresses cost of goods sold directly. He says AI can feel expensive, though the cost does not have to stay high when teams choose services that focus on operating cost and select an appropriate size for every component. A large model is not automatically needed for each part of a voice agent. Companies can reserve more compute for speech recognition when precision matters, then use a smaller language model for a narrow task and an expressive speech model for the final response.

### A full-stack voice agent can reduce the latency of assembling services
[11:06](https://www.youtube.com/watch?v=heYmh_lsX5s&t=666s)
Deepgram's planned voice AI agent combines the pieces into one full-stack product, while allowing customers to use their own API keys and language models. Stephenson says this can shorten turn-taking to roughly 300, 500, or 600 milliseconds, compared with longer delays that can result when teams connect services themselves. He invites attendees to try the preview and says Deepgram offers $250 in credit for experimentation.

## Notable quotes
- "The bar that we set for ourselves and we set from the beginning, but it takes a long time to get there, is what can a human do." (01:39)
- "It's not just about the accuracy because the next step in that is once you can pass that context along, you can pass the original input context along, but you can also have your speech to text model output context as well." (07:39)
- "The speed part is taken care of, the accuracy part is taken care of, now it's all about context." (09:19)
- "If you just give an open-ended prompt to a speech-to-speech model and just say, 'Go to town,' that's not the kind of experience that a bank wants." (10:00)

## Tools & references mentioned
- Deepgram
- Siri
- Daily
- Llama
- Spotify
- Amazon
- Microsoft
- ElevenLabs

## Who should watch
- You are building a voice agent and need to understand where latency, transcription errors, and unnatural speech come from.
- Your application needs control over recognition, language-model behavior, voice, or operating cost instead of one open-ended speech-to-speech system.
- You are evaluating whether shared context across turns can improve a call center, ordering flow, or other spoken interface.

## Related talks

- [Building Effective Voice Agents](https://aietalks.com/talks/building-effective-voice-agents) (Toki Sherbakov & Anoop Kotha, OpenAI, 17:17)
- [Building voice agents with OpenAI](https://aietalks.com/talks/building-voice-agents-with-openai) (Dominik Kundel, OpenAI, 1:25:35)
- [Engineering voice agents: Latency, quality, and scale](https://aietalks.com/talks/engineering-voice-agents-latency-quality-and-scale) (Rishabh Bhargava, Together AI, 24:35)
- [Serving Voice AI at Scale](https://aietalks.com/talks/serving-voice-ai-at-scale) (Arjun Desai, Cartesia & Rohit Talluri, AWS, 17:05)
- [Voice Agents: the good, the bad, and the ugly](https://aietalks.com/talks/voice-agents-the-good-the-bad-and-the-ugly) (Eddie Seagull, Fractional AI, 18:48)