# Building voice agents with OpenAI

Dominik Kundel, OpenAI | AI Engineer World's Fair 2025 | 1:25:35

Source: https://www.youtube.com/watch?v=iXhba366fQc
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/building-voice-agents-with-openai
Published: 2025-06-29
Tags: agents, guardrails, tool-use, voice

## TL;DR
- Voice agents can make technology more accessible, convey information through speech and emotion, and interact with businesses that have no API.
- Chained voice agents are easier to add to existing text systems, while speech-to-speech agents reduce latency and preserve more audio context.
- A good voice agent starts with a narrow task, a small tool set, early evaluations and guard rails, then delegates complex work to specialized agents.

## Summary
Dominik Kundel explains how to build voice agents with the OpenAI Agents SDK for TypeScript. He defines an agent as a model with instructions, tools and a runtime, then shows how those same primitives support voice interactions, handoffs, tracing, interruptions and human approval. He compares chained systems, which transcribe audio, run a text agent and synthesize the response, with speech-to-speech systems that work directly on audio. Chained systems are easier to connect to existing text agents and offer more visibility into model input and output. Speech-to-speech systems reduce latency and preserve tone, but they are harder to use for complex reasoning. Kundel recommends a focused first task, early evaluations and guard rails, short tool calls and specialist agents for difficult work. The coding session builds a text agent, adds a weather tool, connects a browser agent through WebRTC, displays transcripts and delegates riddle generation to a backend model.

## Key ideas
### An agent combines a model, instructions, tools and a runtime
[00:54](https://www.youtube.com/watch?v=iXhba366fQc&t=54s)
Kundel defines agents as systems that accomplish tasks independently for users. Their basic structure has a model, a set of instructions, access to tools and a runtime that manages the life cycle. He introduces the OpenAI Agents SDK for TypeScript as the TypeScript equivalent of the Python SDK. The SDK includes handoffs, guard rails, streaming input and output, MCP support, tracing and human approval with resumability. Its voice support adds context management, native interruption handling, WebRTC and WebSocket connections. Traces can replay conversations, including user audio and tool calls, which helps explain what an agent did.

### Voice agents can make technology easier to access and connect to businesses without APIs
[03:26](https://www.youtube.com/watch?v=iXhba366fQc&t=206s)
Kundel gives three reasons to build voice agents. Speaking with an agent can make technology more accessible because users can interact with it directly. Speech also carries information through tone, voice and emotion, so a conversation can communicate more than basic text. He describes voice as an API to the real world. For example, an agent could call a business and have a conversation on the user's behalf when that business does not provide an API. The agent can handle an interaction that would otherwise require the user to make the call.

### Chained voice agents are easier to reuse, but each model adds latency
[04:29](https://www.youtube.com/watch?v=iXhba366fQc&t=269s)
The chained architecture takes incoming audio through speech-to-text, sends the resulting text to a conventional text agent, then turns the agent's text response into audio with text-to-speech. Kundel says this is easier to adopt when a text-based agent already exists. It also allows developers to use any language model and inspect the exact text sent to and returned by that model. The costs include difficult turn detection, especially when a user interrupts audio already played to them. Developers must adjust the transcript so the model does not assume the user heard content that was cut off. Chaining also adds latency and loses some audio context.

### Speech-to-speech lowers latency and keeps audio context, but needs help with complex reasoning
[06:16](https://www.youtube.com/watch?v=iXhba366fQc&t=376s)
A speech-to-speech agent uses a model trained to work directly with audio, including tool calls, without an intermediate transcription step. Kundel says this reduces latency by removing the speech-to-text and text-to-speech stages. The model can also use audio context such as tone and voice, which supports more natural conversation. Reusing existing text capabilities is harder, though. Complex state and decision-making can also be more difficult because these models focus heavily on conversational audio. His solution is a delegation pattern: a frontline voice agent handles the conversation and calls more capable reasoning agents such as o4-mini or o3 for specific tasks.

### A small first task makes voice agents easier to evaluate
[12:36](https://www.youtube.com/watch?v=iXhba366fQc&t=756s)
Kundel recommends starting with a small, clear goal and a limited number of tools. Voice agents are harder to measure than text agents, so a narrowly scoped task makes it possible to understand whether the system works before adding more complexity. The Agents SDK allows teams to add tools to other agents and connect them with handoffs. A focused agent can do one use case well and pass other requests to a human or another specialist. He also recommends building evaluations and guard rails early, using the traces dashboard or a custom dashboard such as the one Lemonade built to review and replay customer conversations.

### Voice behavior can be shaped with prompts and conversation states
[14:14](https://www.youtube.com/watch?v=iXhba366fQc&t=854s)
Kundel explains that the speech-to-speech and text-to-speech models are generative, so developers can prompt for tone, emotion, role and personality. He points to openai.fm as a place to experiment with different voice styles. He also describes using a JSON structure for conversation states, similar to giving a human agent a script. This can help the model follow the expected process and steps. Developers can describe what should happen next without relying only on a general personality prompt. A custom GPT is available to help create these conversation scripts.

### The TypeScript SDK can connect browser voice sessions safely through WebRTC
[23:49](https://www.youtube.com/watch?v=iXhba366fQc&t=1429s)
In the coding session, Kundel builds a browser-based real-time agent in a Next.js app. The browser does not receive the main OpenAI API key. Instead, the server creates a short-lived ephemeral key and passes the client secret to the browser, which uses it to connect to the real-time API over WebRTC. The SDK handles microphone and speaker setup in the browser. On a server, developers can send audio buffers or listen for returned audio events. The session also handles interruptions and exposes the underlying transport events for applications that need custom logic.

### Tools, handoffs and approvals keep voice workloads scoped
[54:38](https://www.youtube.com/watch?v=iXhba366fQc&t=3278s)
Kundel reuses the same tool definitions for text and voice agents. With Zod schemas, the tool arguments are described to the model and validated in TypeScript. Long-running tools should start a task, return a task ID and expose a status check, because the voice model is blocked while a tool call runs. Human approval can be required through a function or a simple always-approve setting. Approval happens before the tool executes, so the application can stop a risky action without rolling back later work. Handoffs change the agent's instructions and tools for a specialist task. The voice itself cannot change during a session, although prompts can change accent, pronunciation and tone.

### Guard rails run against transcription and can interrupt unsafe output
[1:08:31](https://www.youtube.com/watch?v=iXhba366fQc&t=4111s)
The Agents SDK applies guard rails to real-time agents by running checks against the ongoing transcription. Developers can choose how often the checks run or wait for a complete transcript. If a policy violation appears early, the agent may say a few words before the guard rail interrupts it. If the text output finishes before the audio has finished playing, the system can stop the remaining audio and correct itself. Kundel gives an example that checks for the word 'Dom'. The model can receive information about why the guard rail fired and what it should do instead. He also says developers can choose among transcription models, even though speech-to-speech itself does not require transcription.

## Notable quotes
- "Voice agents are systems that are going to accomplish tasks independently on behalf of users." (00:54)
- "It can act as like an API to the real world." (04:07)
- "Start with a small and clear goal." (12:36)
- "The model currently does not adjust that transcript and instead it's going to be removed." (37:22)
- "The longer your prompt gets, at one point it increases the likelihood that it gets confused." (45:21)

## Tools & references mentioned
- OpenAI Agents SDK for TypeScript
- OpenAI Agents SDK for Python
- MCP
- WebRTC
- WebSocket
- Twilio
- o4-mini
- o3
- GPT-4.1
- Zod
- Next.js
- Lemonade
- openai.fm
- GPT-4o mini transcribe
- GPT-4o transcribe
- Anoop

## Who should watch
- You have an existing text-based agent and want to add browser or phone-based voice interaction without rebuilding every tool.
- You need to choose between a chained speech-to-text pipeline and a direct speech-to-speech model, and want to understand the trade-offs around latency, visibility and reasoning.
- You are implementing real-time tools, interruptions, handoffs, approvals or guard rails and need concrete TypeScript patterns.

## Related talks

- [Building Effective Voice Agents](https://aietalks.com/talks/building-effective-voice-agents) (Toki Sherbakov & Anoop Kotha, OpenAI, 17:17)
- [Giving a Voice to AI Agents](https://aietalks.com/talks/giving-a-voice-to-ai-agents) (Scott Stephenson, Deepgram, 13:08)
- [Building Conversational AI Agents](https://aietalks.com/talks/building-conversational-ai-agents) (Thor Schaeff, ElevenLabs, 1:01:42)
- [Voice Agents: the good, the bad, and the ugly](https://aietalks.com/talks/voice-agents-the-good-the-bad-and-the-ugly) (Eddie Seagull, Fractional AI, 18:48)
- [Voice Agent Engineering](https://aietalks.com/talks/voice-agent-engineering) (Nik Caryotakis, SuperDial, 19:07)