# Voice Agents Can Just Do Things

Charlie Guo, OpenAI | AI Engineer World's Fair 2026 | 15:50

Source: https://www.youtube.com/watch?v=OpY6MmZFeHo
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/voice-agents-can-just-do-things
Published: 2026-09-15
Tags: computer-use, multimodal, tool-use, voice

## TL;DR
- A voice model can respond by changing software, calling tools, or sending notifications instead of speaking.
- Speech to action could make form filling, creative software, and computer use much easier to control.
- Developers can turn existing API endpoints and React hooks into tools that a model operates through voice.

## Summary
Charlie Guo argues that voice agents do not need to answer with speech. Voice interaction has three modes: speech to speech, speech to action, and event to speech. All existed before current models, but better models make them easier to combine. Guo spends most of the talk on speech to action, where a model uses tools to fill forms, operate creative software, or control a whole computer. He says developers can start with software they already have, since API endpoints and React hooks often map directly to model tools. Voice can also deliver proactive alerts, support hands-free use, and help people who have lost dexterity. Guo describes OpenAI's native-audio approach and GPT realtime 2, which can reason before speaking, call tools in parallel, and use preambles to explain delays. His final advice is to ask what role voice should play in an interaction before deciding to build a voice agent.

## Key ideas
### A voice agent can respond through software instead of speech
[00:01](https://www.youtube.com/watch?v=OpY6MmZFeHo&t=1s)
Guo opens with the assumption that voice agents have to talk back. He argues that speech is only one response mode. A model can show a notification or popup, change a button's state or color, highlight text, or move a ghost cursor across the screen. These interactions communicate what is happening without making the user listen to an explanation. Models are now capable enough to support more of these designs, and Guo says the three modes can be combined rather than treated as separate product categories.

### Speech interaction has three established modes
[01:02](https://www.youtube.com/watch?v=OpY6MmZFeHo&t=62s)
Guo describes speech to speech, speech to action, and event to speech. He points to the Moviefone hotline as an older speech-to-speech example, since callers asked for showtimes and received spoken information. GPS navigation is an example of event to speech because the system reacts to events such as a turn or route change by speaking. The modes are not new, but current models make it possible to build more capable versions and combine them in one product.

### Speech to speech works well for coaching, support, and translation
[02:28](https://www.youtube.com/watch?v=OpY6MmZFeHo&t=148s)
Speech to speech lets a user talk and receive a spoken answer. Guo describes language-learning coaching that can respond to emphasis and emotion. He also discusses customer support agents that could be more pleasant than working through a phone tree. Real-time translation is another example. He says models have become fast enough to translate content dynamically with little to no latency, including the possibility of live events being dubbed in several languages.

### Speech to action is an underexplored way to use voice
[03:58](https://www.youtube.com/watch?v=OpY6MmZFeHo&t=238s)
In speech to action, the model hears the user and uses tools. Guo calls this one of the least explored areas and says he nearly titled the talk 'voice is the next capability overhang.' One example is form filling. A user could describe their information for five minutes, have most of a government document completed, and then check it instead of typing every name and address. Guo says this could improve a large amount of routine internet work.

### Voice can give people access to creative software and the whole computer
[05:11](https://www.youtube.com/watch?v=OpY6MmZFeHo&t=311s)
Guo says his ability to describe software in detail lets him direct Codex, but he does not have the same fluency with music, painting, Photoshop, or Ableton. In those tools, his taste exceeds his ability to operate the software. Voice could let users describe what they want to compose, paint, or create. From there, the broader category is computer use. If models can perform the tasks a person can perform on a computer, Guo asks why users should talk to one application instead of the computer itself.

### Existing application verbs can become voice tools
[06:38](https://www.youtube.com/watch?v=OpY6MmZFeHo&t=398s)
Guo says adding audio intelligence to an existing application is easier than developers may expect. Modern web applications already expose actions through nouns and verbs, including API endpoints and React hooks. Each verb can be converted into a tool that a model can call. The user can then drive existing software through voice. Guo still says developers need guardrails and safety checks, but he wants them to start by considering what it would mean to talk to software they already build.

### Event-driven voice is useful when users cannot watch a screen
[07:41](https://www.youtube.com/watch?v=OpY6MmZFeHo&t=461s)
In event to speech, the model receives an event and talks to the user. Guo sees two main uses. A hands-free or screen-free experience helps when a person's hands or attention are occupied, such as when cooking with a recipe application. Voice can also escalate an important event. Software might first animate something, then show a popup, and finally speak if the user still needs to notice it. Guo says applications should not speak every log, but voice can be another level in an attention system.

### Voice agents can widen access to software
[09:01](https://www.youtube.com/watch?v=OpY6MmZFeHo&t=541s)
Guo connects these interaction modes to accessibility. He knows developers who lost mobility or dexterity in their hands and thought their programming careers were over. Coding agents and voice agents later let them generate much more code than they had before. He says the same combination could unlock access for more people. He also describes how one agent could mix modes in a car or game: it could accept spoken commands, perform actions, report events, and provide information when needed.

### Native audio models can reason, call tools, and explain delays
[10:54](https://www.youtube.com/watch?v=OpY6MmZFeHo&t=654s)
Guo contrasts the traditional voice pipeline, which transcribes speech, sends text to a language model, generates text, and turns it back into audio, with OpenAI's native-audio approach. GPT realtime 2 works directly with audio tokens and can think before speaking. It can also make parallel tool calls. Because reasoning and tool use add latency, preambles let the model tell a user that it is checking something, such as flight prices. Guo says this makes waiting feel more like a phone conversation with a human travel agent.

## Notable quotes
- "Speech is not the only way that a voice model has to respond." (01:02)
- "I would love it if instead of spending an hour filling out a government document, I could just talk for five minutes and it would get 90% of it for me." (04:41)
- "Why am I talking to an app? Why am I talking to a terminal? Why am I not just talking to the entire computer?" (06:05)
- "Start with the question of what is the role of voice and audio in this interaction?" (14:40)
- "I do believe that AGI will be spoken, not typed." (15:18)

## Tools & references mentioned
- OpenAI
- GPT realtime 2
- Codex
- React
- Photoshop
- Ableton
- Moviefone
- CarPlay
- Spotify
- Google Maps

## Who should watch
- You are building a voice interface and have assumed that every response needs to be spoken.
- Your application already has API endpoints or handlers that could become model tools.
- You care about hands-free interaction, accessibility, or letting users control software without mastering its interface.

## Related talks

- [Your Voice Agent is Just a Walkie Talkie](https://aietalks.com/talks/your-voice-agent-is-just-a-walkie-talkie) (Neil Zeghidour, Gradium, 19:14)
- [Building voice agents with OpenAI](https://aietalks.com/talks/building-voice-agents-with-openai) (Dominik Kundel, OpenAI, 1:25:35)
- [Giving a Voice to AI Agents](https://aietalks.com/talks/giving-a-voice-to-ai-agents) (Scott Stephenson, Deepgram, 13:08)
- [Building Effective Voice Agents](https://aietalks.com/talks/building-effective-voice-agents) (Toki Sherbakov & Anoop Kotha, OpenAI, 17:17)
- [Building Conversational AI Agents](https://aietalks.com/talks/building-conversational-ai-agents) (Thor Schaeff, ElevenLabs, 1:01:42)