Language is useful because it gives models a universal, information-rich way to condition generation, but it loses details that humans notice in sound, skin tone, taste, and other sensory signals.
2
Human preference can reward attractive artifacts instead of useful or realistic video, so model evaluation needs semantic checks, workflow feedback, and more than side-by-side voting.
3
The team wants high-quality professional data and real user workflows, including the intermediate steps people take when creating ads, editing media, or using models in production.
Summary
The panel covers Google DeepMind's recent generative media work, including Nano Banana 2 Lite and the Gemini Omni Flash APIs. Nicole Brichtova describes image-to-video workflows, editing, translation, marketing, and education use cases. The discussion then turns to the architecture of multimodal systems. Shane Gu argues that language is a useful backbone for reasoning and generalization, while Dumitru Erhan says video provides a needed representation for space and time. The panel is candid about language's limits for audio, skin tone, facial expression, and other sensory details. It is also skeptical of human preference as a target. A generated video may win a side-by-side test because it is sharper, more saturated, and smoother, without being more realistic. The team still relies heavily on human evaluation, alongside automated checks, live experiments, trusted testers, and feedback from people using models in real workflows.
Multimodal models can turn storyboards, audio, and natural-language instructions into video workflows
Nicole Brichtova describes video generation as accepting different inputs, including a set of images used as a storyboard and an audio track that provides a character's voice. The same models can edit existing video through natural language, such as adding or removing an object or cleaning noise from a beach video. She mentions short-film production, YouTube Shorts, marketing campaigns, and educational materials. Dumitru Erhan gives a translation example using a photograph of product instructions. The model translated the text into Romanian while preserving the diagrams and layout. He says similar capabilities could support video dubbing, localization, and text rendering.
Language gives video models a strong conditioning signal, even when systems may eventually combine models differently
Shane Gu says language helped generative models because it is universal and can condition causal information. A detailed description gives a model a way to connect visible features with what may have caused them, which can support generalization. He also says video models are strong foundations for space and time, with potential for visual quizzes, physical intuition, robotics, and world-model behavior. The panel leaves open whether one unified model or an agentic collection of models will dominate. Dumitru Erhan expects a single model eventually, while Nicole Brichtova says specialized models remain useful because engineering and product trade-offs still differ.
Text is a practical backbone, but captioning is a lossy intermediate representation
The panel questions whether English captions are the right bridge between understanding and generation. Shane Gu says the team is exploring code as another representation, including for agentic and coding work. He explains why natural-language reasoning currently works well: large-scale pretraining learns much of the model's intelligence, and language lets later reasoning use that learned knowledge directly. Nicole Brichtova adds a product reason: people communicate through language, so text is a natural interface where appropriate. Dumitru Erhan agrees that language alone is insufficient and describes video as a complementary foundational model for representing information that text compresses away.
Joint audio-visual generation works because speech and visible motion share one causal process
Dumitru Erhan says video and audio should be generated together when they arise from the same event. When a person speaks, the process produces both sound and visible lip movement, so generating pixels first and attaching audio afterward can create synchronization problems. Nicole Brichtova describes the model that first brought joint audio-visual generation into their work. The panel also discusses why audio is difficult to describe with words. Room size, distance, echo, noise, voice tone, music, and sound effects are perceptually important but poorly captured by ordinary language. A model trained mostly on studio recordings may make every scene sound studio recorded.
Human preference can reward an Instagram-filter look instead of realism or usefulness
Dumitru Erhan describes an experiment that took captions from real videos, generated equivalent videos, and compared them with human evaluators. People largely preferred the generated versions. He cautions that the result came from visual appeal rather than greater realism. The generated videos looked sharper, more HDR, and more saturated, with nicer skin tones. Nicole Brichtova calls this default effect the "Instagram filter." Shane Gu adds that sensitivity varies by observer. A manga artist may notice a slightly unnatural eye gaze that most people miss. The panel concludes that human preference is an unreliable target when the desired outcome includes realism, usefulness, or semantic consistency.
Default aesthetics are shaped by model teams, testers, and repeated choices about style
Nicole Brichtova says teams must decide what a model does when a user gives no detailed style instruction. She describes an early problem with generated infographics: the model packed too much information into one image because it had learned many facts about a topic. The team also debated color palettes, saturation, and more muted styles. These choices are partly made by modeling teams, though Brichtova questions whether they should instead involve people with strong creative or art-direction experience. Trusted testers and internal users provide feedback that catches problems the team misses. Their reactions can concern grass texture, visual clutter, color, or other details that are hard to specify in advance.
Reward hacking can enter through ordinary preference training
The team noticed that their image model often added wedding rings to hands. Nobody on the development team had caught the pattern until an outside tester asked why every hand seemed to have one. Dumitru Erhan identifies this as a form of reward hacking, where preference-based training picks up a spurious correlation. The example shows how a model can learn a visual shortcut that receives positive signals without matching the intended task. Nicole Brichtova says trusted testers help surface these artifacts because they see outputs through different habits and workflows. The panel treats this as one reason to avoid relying on a single preference score.
Video evaluation still depends heavily on people inspecting outputs and using them in real work
Nicole Brichtova says some image checks are easy to automate, such as using OCR to detect broken text in an infographic. Video evaluation is harder, especially for aesthetics, free-form editing, and semantic consistency. Teams run many human evaluations, sometimes putting about ten people in a room to compare videos side by side. They also use automated raters, live experiments, trusted testers, early-access programs, and feedback from people who use the models every day. Shane Gu argues that video understanding should test whether scenes make semantic sense, rather than measuring only visual preference. He wants human labeling work to be gradually amortized through better evaluation systems.
Real production workflows reveal data needs that isolated media examples miss
The panel asks for high-quality data and concrete task trajectories. Nicole Brichtova gives a marketing example that starts with a product image, turns it into a video advertisement, and then produces assets in different platform formats. Those intermediate steps are hard to manufacture because they live inside real product workflows. She also mentions interior design and virtual try-on, where a model must preserve a pattern across custom rug sizes or keep earring and head proportions believable. Dumitru Erhan says the team is interested in professional footage and embodied data, including robotics-related work. He and Brichtova want feedback that can travel upstream from forward-deployed engineers and users to model training.
"I think language alone is not sufficient. That's why we feel like the video is a very complementary model."18:59
Who should watch
You are building image or video generation products and need to decide where specialized models, unified models, or agents fit.
Your evaluation process mostly asks people which output they prefer, and you need examples of why that can reward polish over realism or task success.
You have access to media or workflow data and want to understand what a frontier model team considers useful beyond random internet videos.
