A single voice agent API can combine speech recognition, an LLM, function calling, and speech synthesis while handling endpointing and streaming audio.
2
Low latency depends on audio chunk size, time to first token, model placement, and regional deployment, while overly fast responses can make an agent interrupt the user.
3
Agent swarms work better when each agent has one task, such as booking, cancellation, technical support, or account verification, and a routing agent connects them.
Summary
Damien Murphy explains how to build a browser-based, real-time voice agent that accepts microphone audio and returns spoken responses. The demo uses a drive-through ordering assistant whose LLM calls backend functions to add items to an order. Murphy compares the traditional speech-to-text, LLM, and text-to-speech pipeline with Deepgram's combined voice agent API, which also handles endpointing and lifecycle events. He covers the JavaScript client, WebSocket audio streaming, downsampling, interruption handling, content moderation, memory, and long-running functions. The talk then moves to production concerns. Murphy argues that voice agents should be split into small, composable agents with a router, rather than putting every business task into one prompt. Regional clusters, redundancy, horizontal scaling, and autoscaling are needed for large or spiky workloads. He is also direct about current limits, including prompt injection, diarization errors, language coverage, and the difficulty of making one agent handle every task.
A combined voice API removes infrastructure work from the usual voice pipeline
The older approach connects speech to text, an LLM, and text to speech as separate services. Murphy says teams can reduce latency by collocating those services, but this turns the project into an infrastructure problem. Deepgram's voice agent API instead accepts audio and returns audio through one interface. It also handles endpointing, which determines when the user has finished speaking. The API remains LLM-agnostic and supports providers and models such as OpenAI, Anthropic, Groq, Llama, and Mixtral. Function calling is part of the flow, so the model can invoke application actions rather than only returning text.
The workshop agent uses the LLM to update an order through backend functions
The live demo connects a browser microphone to a Crusty Crab drive-through agent. Murphy orders a Krabby Patty, a Kelp Shake, and a combo. The LLM decides to call the add-item API, and the frontend displays the resulting order. The application does not parse the model's reply to construct the order. The client gets a call ID, menu data, conversation text, function-call results, assistant text, and audio messages over the connection. The supplied client uses vanilla JavaScript, HTML, CSS, and a WebSocket, while the optional server is a small Express.js API.
Audio chunking and interruption handling directly affect perceived latency
Murphy says a telephone system could send audio in roughly 20-millisecond chunks, while browsers usually send larger chunks. Sending smaller chunks can lower latency, although the service cannot process audio before it arrives. The client downsamples browser audio from a higher sample rate to 16 kHz to reduce bandwidth. When the user starts speaking, the client can clear already scheduled assistant audio. Murphy also recommends voice activity detection for barge-in behavior. More advanced systems can track where the assistant was interrupted, since the model may otherwise assume the user heard its entire response.
Endpointing must account for pauses and the way people speak on calls
Silence-based endpointing waits for a span of quiet to decide that the user is done. Murphy describes semantic endpointing as a way to recognize cases such as, "Hang on a minute, let me get that for you," where the user has paused without finishing the interaction. People also use backchanneling, such as "mhm," while reading out numbers. A voice agent should avoid rushing ahead when someone is giving an email address, account number, or credit card number. Murphy's advice is to model the interaction after what a human operator would do, including spelling out information when audio quality makes it difficult to understand.
Small, composable agents are easier to control than one agent with every business task
Murphy advises against making a single agent handle an entire business through one massive system prompt. A support agent might use retrieval-augmented generation, an account agent might verify ownership, and a booking agent might use payment functions. A routing agent can send a request to booking or cancellation while keeping the same voice on the phone line. Murphy connects this design to three practical goals: lower complexity through single responsibility, lower cost through smaller models where possible, and reuse of sub-agents across different flows.
Production scale requires regional placement, redundancy, and autoscaling
Distance adds latency, so callers in Europe or Asia should not always reach a US server. Murphy describes scaling clusters across regions such as US East, US West, and AMEA, with redundancy for availability. Traffic can also change sharply. A service might need only a small number of agents during normal operations and many more during an outage or disaster. Horizontal scaling within regional clusters and autoscaling are therefore part of the deployment problem. Deepgram supports Kubernetes and provides autoscaling Helm charts, while self-hosting requires distributing the models and managing the supporting infrastructure.
Voice agents still need separate systems for memory, moderation, and long-running work
Murphy treats memory as a separate design problem. For long calls, continuously expanding the system context is undesirable, so a memory system can update old facts when newer information conflicts with them. Content moderation can inspect conversation text before the assistant's output reaches the user, although Murphy says prompt injection remains difficult to prevent completely. Long-running functions should usually be handed to a secondary system. The agent can tell the user that it started the task, then receive a later webhook and prompt the model with the completed result.
Current speech systems have specific limits around languages, voices, and diarization
Deepgram's text-to-speech offering has 12 English voices in the demo, while speech recognition supports 36 languages on Nova 2. Murphy says prompt-based TTS is being developed before expanding the voice system to more languages. The system supports diarization, but a single audio channel is difficult when speakers provide short acknowledgements such as "yeah" or "mhm." Murphy recommends about 30 seconds per speaker to build a solid speaker embedding. Early parts of a call may therefore be less reliable, and speaker identity is not exposed as a definitive person-level identifier.
"Building a model is easy, rolling it out into production at a low price point is hard."01:52
Who should watch
You are building a browser or phone-based voice agent and need to understand the full audio-to-audio path.
Your prototype works, but interruptions, endpointing, function calls, or long-running backend tasks cause awkward conversations.
You are deciding whether to build one general-purpose assistant or several smaller agents that can be routed and reused.
