# See, Hear, Speak, Draw

Logan Kilpatrick & Simón Fishman, OpenAI | AI Engineer Summit 2023 | 18:43

Source: https://www.youtube.com/watch?v=bNZV9s3_u44
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/see-hear-speak-draw
Published: 2023-10-24
Tags: image-generation, multimodal, video

## TL;DR
- Multimodal systems currently connect separate models with text, while future systems may handle several input and output types in one model.
- A vision loop can describe an image, generate a new image from that description, compare the result with the original, and revise the prompt.
- Video summarization improves when visual frames are combined with Whisper's transcript instead of summarizing audio alone.

## Summary
Logan Kilpatrick and Simón Fishman describe how developers can build multimodal applications with the models available at the time of the talk. They frame current systems as separate islands: DALL-E generates images, Whisper transcribes audio, and GPT-4 with vision accepts images and text. Text connects these components. Their first demo uses GPT-4 with vision and DALL-E 3 to recreate a photograph, then asks GPT-4 with vision to compare the images and revise the next generation prompt. Their second demo combines sampled video frames with a Whisper transcript to create a richer textual representation and an article about a GPT-4 introduction video. The speakers are clear that these demos are raw and still have limitations. They want developers to design products that can add images, audio, and eventually other modalities as models become more unified.

## Key ideas
### Current multimodal applications connect separate models through text
[02:33](https://www.youtube.com/watch?v=bNZV9s3_u44&t=153s)
Fishman describes today's systems as a set of separate islands. DALL-E takes text and generates images, Whisper takes audio and produces transcripts, and GPT-4 with vision takes images and text and can reason about them. Text acts as the connection between these systems. Developers can already build useful workflows by passing one model's textual output to another model. The speakers distinguish this arrangement from the future they expect, where one model could consume several modalities and produce several kinds of output. They also say that future systems are not available yet, so the demos focus on patterns developers can use with current tools.

### A vision loop can turn a photograph into an iterative image-generation process
[05:18](https://www.youtube.com/watch?v=bNZV9s3_u44&t=318s)
The first demo sends a real photograph to GPT-4 with vision and asks for a detailed, human-readable description. That description becomes the input for DALL-E 3, which generates a synthetic version of the image. In the example, the model notices details from a Halloween-themed Painted Ladies display, including a tombstone, spiders, and an old dogs welcome sign, but the generated image changes several details. The system then sends both images back to GPT-4 with vision. It asks the model to identify differences and rewrite the prompt for another DALL-E generation. The next image changes details such as the marble and spider.

### Image comparison lets models evaluate and revise their own visual output
[07:31](https://www.youtube.com/watch?v=bNZV9s3_u44&t=451s)
Fishman uses the image loop to show a broader interaction pattern. A model can describe an image, compare a current result with a target, and use the differences to decide what to change. The speakers connect this to tasks that normally need a person to inspect a visual output, compare it with a reference, and update the instructions. They say models can now perform part of that loop themselves, although people will remain necessary for some time. Fishman gives an interior-design example: a user could provide an image showing a preferred room style and ask a system to find lamps on Amazon that match it. He describes this as a hard problem today.

### The demos use raw model outputs rather than elaborate prompt engineering
[09:15](https://www.youtube.com/watch?v=bNZV9s3_u44&t=555s)
Kilpatrick stresses that the image demo is a one-hour version with no special prompt engineering. The output from one model goes almost directly into the next model, and the code is about 50 lines. He says people should be able to get better results when they apply techniques discussed elsewhere at the conference. The backstage example follows the same pattern. GPT-4 with vision describes monitors, boxes, cables, and a robotot, DALL-E 3 generates an image, and GPT-4 with vision identifies changes such as vertical lights and a curtain. The speakers present this simplicity as a way to expose the models' capabilities, not as a finished production design.

### Video summaries become richer when they include sampled frames
[13:43](https://www.youtube.com/watch?v=bNZV9s3_u44&t=823s)
The second demo addresses a limitation in transcript-only video summarization. The speakers sample frames from a YouTube video and ask GPT-4 with vision to describe what appears in them. They also use Whisper to transcribe the audio. Combining the visual descriptions and transcript creates a long textual representation that contains information from both channels. For a GPT-4 introduction video, the system uses this representation to produce an article, with DALL-E generating an opening image and selected frames matching the article's context. Kilpatrick says the result captures more of what happens in the video and makes the material more accessible in text form.

### Current systems require developers to assemble the multimodal pipeline
[03:59](https://www.youtube.com/watch?v=bNZV9s3_u44&t=239s)
Kilpatrick says much of the work in making multimodal demos is connecting the different components. Developers have to decide how to move information between image, audio, and text models. He expects developer work to become simpler when a single model can handle text, images, video, or speech directly. Until then, text provides a practical intermediate format. The video demo illustrates this engineering cost: frames must be extracted, images described, audio transcribed, and the resulting text combined before an article can be generated. The speakers expect a unified model to reduce this wiring, but they do not claim that such a model is available in the talk.

### Developers should design products with more modalities in mind
[17:24](https://www.youtube.com/watch?v=bNZV9s3_u44&t=1044s)
The closing advice is to start thinking about multimodal input and output when planning AI products. The speakers expect developers to build applications that use images and other modalities as these capabilities become available. They point to image reasoning as an area with many patterns still to explore. Their examples include visual recreation, comparison, design assistance, and video-to-article workflows. They also mention agents with image input, since much of the internet contains information that text alone cannot represent. The recommendation is practical: consider which product ideas are blocked by today's model boundaries, then plan for systems that can connect more modalities over the coming months.

## Notable quotes
- "Text as a connective tissue between all of these models." (02:54)
- "There are plenty of tasks that we do right now in AI where we need the human in the loop to be able to evaluate a visual output that a model produces." (07:33)
- "This is like the rawest output that you can get." (09:15)
- "Videos have a lot of information in them that is conveyed visually." (14:11)
- "Agents with image input is going to be sick. I can't wait. Feel like so much of the internet requires that." (17:58)

## Tools & references mentioned
- OpenAI
- GPT-4
- GPT-4 with vision
- ChatGPT
- DALL-E 2
- DALL-E 3
- Whisper
- Amazon
- Instagram
- Midjourney
- Apple
- Julia Programming Language

## Who should watch
- You are building an AI product around text and need concrete ways to add images, audio, or video with currently available models.
- You want to understand how to assemble multimodal workflows before unified models can handle several modalities directly.
- You are working on image evaluation, visual search, or video summarization and want examples of model-driven iteration.

## Related talks

- [From Text to Vision to Voice: Exploring Multimodality with OpenAI](https://aietalks.com/talks/from-text-to-vision-to-voice-exploring-multimodality-with-openai) (Romain Huet, OpenAI, 23:39)
- [Keynotes & Multimodality Track](https://aietalks.com/talks/keynotes-multimodality-track) (Benjamin Duny & Simon Sturmer, AI Engineer & Chris Lattner, Modular & Auntie Aart & Mike Chambers, AWS & Alex Albert, Anthropic & Harrison Chase, LangChain & Rob Chandra, Substrate & Vic Kapati, M87 Labs & Ben Hilac, Dawn & Karan Goel, Cartesia & Chang She, LanceDB & Noah Schnapp, Character.AI & Steph Duenas & Quinn Dombrowski, Daily & Roman Huet, OpenAI & Brian Bischof, Charles Fry, Hamza Hussein, Jason Liu, Shreya Shankar & Eugene Yan & Thomas Dohmke, GitHub, 8:34:14)
- [The Intelligent Interface](https://aietalks.com/talks/the-intelligent-interface) (Samantha Whitmore & Jason Yuan, New Computer, 18:56)
- [Voice In, Visuals Out: The Agony and the Ecstasy](https://aietalks.com/talks/voice-in-visuals-out-the-agony-and-the-ecstasy) (Allen Pike, Forestwalk Labs, 13:05)
- [Multi model multimodal and multi agent innovations in Azure AI](https://aietalks.com/talks/multi-model-multimodal-and-multi-agent-innovations-in-azure-ai) (Cedric Vidal, Microsoft, 28:56)