# From Text to Vision to Voice: Exploring Multimodality with OpenAI Romain Huet, OpenAI | AI Engineer World's Fair 2024 | 23:39 Source: https://www.youtube.com/watch?v=yJHw33cVeHo Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/from-text-to-vision-to-voice-exploring-multimodality-with-openai Published: 2024-07-10 Tags: multimodal, product-strategy, video, voice ## TL;DR - GPT-4o combines audio, video, and text reasoning in one model, which reduces the latency and context loss of stitching separate speech and language systems together. - Live demos show ChatGPT handling voice interruption, emotional tone, camera input, book pages, screen sharing, and coding assistance. - OpenAI is focusing on stronger text reasoning, cheaper and faster models, customized models, asynchronous workloads, and agents that can perceive and act through multiple modalities. ## Summary Romain Huet traces OpenAI's move from GPT-3's text API to GPT-4's reasoning and vision, then to GPT-4o, which handles audio, video, and text in one model. He demonstrates voice conversation, interruption, tone changes, image understanding, screen sharing, and coding help through ChatGPT Desktop. He then combines Sora video frames, GPT-4 vision, and Voice Engine to create and translate narration for a generated tree-frog documentary. The talk also gives OpenAI's direction for developers: improve textual intelligence, offer faster and cheaper models, support batch workloads, customize models, and build agents that can perceive and interact with the world. Huet is direct that Voice Engine was not broadly released because of safety concerns. His practical argument is that developers should start designing software around modalities working together, while expecting model capability, pricing, and customization options to keep changing. ## Key ideas ### OpenAI moved from a text API toward models that work across several modalities [00:13](https://www.youtube.com/watch?v=yJHw33cVeHo&t=13s) Huet begins with OpenAI's developer platform, which launched GPT-3 through an API in 2020. Early uses included coding assistance, copy editing, translation, and text games such as AI Dungeon. GPT-4 added stronger reasoning, creativity, coding, tool use, and data interpretation. Its vision abilities began OpenAI's multimodality work, and GPT-4 Turbo brought vision into the same model as text. GPT-4o goes further by reasoning across audio, video, and text in real time. The name refers to an omni model, with these modalities brought together in one system. ### A single multimodal model reduces the delay and context loss of voice pipelines [04:49](https://www.youtube.com/watch?v=yJHw33cVeHo&t=289s) Huet contrasts GPT-4o with earlier voice assistants, which required a chain of Whisper for transcription, GPT-4 for reasoning, and text-to-speech for the response. Moving those capabilities into one model avoids passing information between separate systems. He says GPT-4o is twice as fast as GPT-4 Turbo, costs half as much, and has higher rate limits. He also describes the model as able to respond with different tones, such as whispering or speaking with more energy, while allowing the user to interrupt before the audio finishes. ### ChatGPT Desktop can hold a natural voice conversation and respond to live direction [06:24](https://www.youtube.com/watch?v=yJHw33cVeHo&t=384s) In the first demo, Huet opens ChatGPT Desktop with Option-Space and asks it to greet the AI Engineer World's Fair audience. He then asks for more energy and for the livestream audience to be included. The model changes its delivery immediately. Huet asks for whispered advice to builders, then requests an even slower, quieter whisper. The demonstration is meant to show low-latency conversation, emotional tone, and interruption rather than a fixed question-and-answer exchange. ### Camera input lets the model interpret drawings, text, and a book page [08:56](https://www.youtube.com/watch?v=yJHw33cVeHo&t=536s) Huet turns on the camera and asks ChatGPT to identify a rough drawing. It recognizes the Golden Gate Bridge and translates the French phrase "bonjour developer" as "hello developers." He then shows Poor Charlie's Almanack, asks for a random page number, and points the camera at page 126. ChatGPT identifies a passage about Coco Chanel's rule for success and gives a short explanation. Huet uses the demo to show that the model can work from objects and printed pages while the conversation continues by voice. ### Screen sharing makes the model useful as a coding and interface assistant [11:35](https://www.youtube.com/watch?v=yJHw33cVeHo&t=695s) Huet shares a travel app and asks ChatGPT why its mobile layout looks unresponsive. The model points out cramped layouts and squished images, then reviews a React component and explains that it manages messages, file uploads, and streamed responses. In the discover component, it suggests adjusting the Tailwind CSS grid for smaller screens. Huet asks which prefix to use, and the model recommends the SM prefix. He presents this as a simple example of reasoning aloud with ChatGPT before asking another coding tool, Cursor, to make the change. ### OpenAI's near-term platform plans combine capability, cost, customization, and agents [14:25](https://www.youtube.com/watch?v=yJHw33cVeHo&t=865s) Huet lists four areas OpenAI is pursuing. The first is stronger textual intelligence, since reasoning improvements still affect what developers can build. The second is faster, cheaper models for use cases that do not need the highest level of intelligence, along with batch processing for asynchronous work. He says the Batch API can reduce pricing by 50 percent for those workloads. The third is customization through fine-tuning and more extensive work with companies such as Harvey. The fourth is agents that can use multiple modalities, access data securely, manage tasks, and interact with the world. ### Sora, vision, and Voice Engine can be combined into one media workflow [18:24](https://www.youtube.com/watch?v=yJHw33cVeHo&t=1104s) Huet shows a Sora-generated tree-frog documentary, then slices six frames from the video and sends them to GPT-4 with vision. A prompt asks the model to narrate what it sees, producing a story from the selected frames. He then previews Voice Engine, which can create a custom voice from a short recording. The generated narration is synthesized in Huet's voice and translated into French and Japanese. He says Voice Engine is not broadly available yet because OpenAI is treating its safety risks seriously. ### Developers should expect software to be designed around interacting modalities [22:07](https://www.youtube.com/watch?v=yJHw33cVeHo&t=1327s) Huet closes by returning to the four areas of work: better textual intelligence, models that are faster and more affordable, customization, and support for multimodal applications and agents. He says OpenAI's goal is for developers to build more with its models, rather than simply spend more on them. His examples show a workflow in which one system generates video, another interprets frames, and another produces speech in several languages. He describes this as an early stage of a fundamental change in how software is built. ## Notable quotes - "GPT-4o is our new flagship model that can reason across audio, video, and text all in real time." (04:05) - "Now we have all of that into a single model." (05:15) - "The goal is not for you guys to spend more with OpenAI, but our goal is for you to build more with OpenAI." (22:25) - "The voice engine model is the ability for us to create custom voices based on very short clips." (19:30) ## Tools & references mentioned - OpenAI - GPT-3 - GPT-4 - GPT-4 Turbo - GPT-4o - ChatGPT - ChatGPT Desktop - Whisper - AI Dungeon - Spotify - Sora - Voice Engine - Cursor - Tailwind CSS - React - Batch API - Harvey - Devin - Cognition Labs - Paul Graham ## Who should watch - You are building a voice, image, or video feature and want to see how one model can handle several input and output types. - You are deciding whether to use screen sharing, camera input, or voice interruption in a developer tool. - You want OpenAI's stated direction on pricing, batch processing, fine-tuning, and agents before committing to a product design. ## Related talks - [See, Hear, Speak, Draw](https://aietalks.com/talks/see-hear-speak-draw) (Logan Kilpatrick & Simón Fishman, OpenAI, 18:43) - [Keynotes & Multimodality Track](https://aietalks.com/talks/keynotes-multimodality-track) (Benjamin Duny & Simon Sturmer, AI Engineer & Chris Lattner, Modular & Auntie Aart & Mike Chambers, AWS & Alex Albert, Anthropic & Harrison Chase, LangChain & Rob Chandra, Substrate & Vic Kapati, M87 Labs & Ben Hilac, Dawn & Karan Goel, Cartesia & Chang She, LanceDB & Noah Schnapp, Character.AI & Steph Duenas & Quinn Dombrowski, Daily & Roman Huet, OpenAI & Brian Bischof, Charles Fry, Hamza Hussein, Jason Liu, Shreya Shankar & Eugene Yan & Thomas Dohmke, GitHub, 8:34:14) - [Giving a Voice to AI Agents](https://aietalks.com/talks/giving-a-voice-to-ai-agents) (Scott Stephenson, Deepgram, 13:08) - [Building voice agents with OpenAI](https://aietalks.com/talks/building-voice-agents-with-openai) (Dominik Kundel, OpenAI, 1:25:35) - [Multi model multimodal and multi agent innovations in Azure AI](https://aietalks.com/talks/multi-model-multimodal-and-multi-agent-innovations-in-azure-ai) (Cedric Vidal, Microsoft, 28:56)