Video models have already reached the point where people cannot reliably distinguish generated footage from real footage.
2
Predicting the next frame teaches models about physics, object permanence, gravity, occlusion, and how objects behave in the world.
3
Real-time video models could become interactive world models for robotics, software interfaces, coding agents, and simulated experiences.
Summary
Cristóbal Valenzuela describes Runway's progress from a 128-by-128-pixel AI film in 2019 to video that people cannot reliably distinguish from real footage. He argues that the important capability is not only generating attractive frames. When a model predicts the next frame at scale, it has to learn about gravity, object permanence, occlusion, movement, and the relationships between objects. With enough speed, this can look like a simulation system or physics engine, even without explicitly defined physics laws. Valenzuela then shows how real-time models could respond to clicks and actions, control simulated environments, guide robots, and turn any screen pixel into an interactive world. The talk's main demonstration is Continuum, an early research prototype that converts content into live visual portals. It can support learning, visual thinking for coding agents, and reactive interfaces. Valenzuela says Runway plans to release much of this work over the next 8 to 12 months.
Runway's first AI film shows how quickly generated video has changed
Runway's first AI film, The Orange, was made in 2019. It showed an orange falling from a tree and moving through different parts of the world. The model created 128-by-128-pixel images, which Runway then upsampled to a larger resolution. Valenzuela compares it to watching an old film from the early days of cinema. Runway was trying to find out whether scaling the approach would naturally improve the output. He then contrasts that film with current generated video, where the people and locations are entirely artificial but look real. Even the current state of the art, he says, may look primitive in a few more years.
Next-frame prediction teaches models how the world behaves
Valenzuela explains video as an optical illusion created by showing frames quickly enough to produce movement. Runway applies the same idea through next-frame prediction. Given an image and its previous frames, a sufficiently large network predicts what comes next and produces motion that can look real to people. The process also forces the model to learn more than visual appearance. Valenzuela names physics, object permanence, gravity, occlusion, movement, and the way objects work in the real world. A model generating a consistent scene with a camel on a ball and water in a container has to understand how those objects relate. Runway therefore evaluates video models with tests about physics, geometry, and object behavior.
Scaled video prediction can begin to act like a simulation system
When next-frame prediction is applied in an environment that resembles a game, Valenzuela says the result starts to look like a physics engine or simulation system. The model does not receive a manually written set of physics laws that specifies how each object should behave. It learns the behavior through scale. He shows a simulated world that is not GTA and does not exist as a conventional software environment. The result is a simulation of a simulation. The same approach could produce systems that predict what happens after an action, rather than only generating a fixed video sequence.
Interactive world models let users act inside predicted video
The next step is making the model real-time and interactive. Instead of predicting a fixed sequence, the system predicts the next frame while allowing a person to act at any point. An action can create a different continuation of the world. Valenzuela describes an example where a user clicks on parts of a car image to turn on its lights, rotate the car, or open its hood. There is no physical 3D representation underneath it, and the example does not use Unity, Unreal, or Blender. The video model has learned how clicks and actions affect an interface. Runway calls this type of system a world interface model.
Runway is applying world models to robotics and screen interfaces
Runway has built a robotics lab in its New York office, alongside its studio space. Valenzuela says the models can begin to act as the brains of autonomous and embodied systems as they learn to reason about the world. He then extends the idea beyond video and robotics. Any pixel on a screen could become part of a real-time video model. In one example, an image of a car becomes an interactive application without a prebuilt 3D scene. In another, a person can move objects in an image and see them behave as if they were in the room. The model generates the behavior from its understanding of how objects move through space.
Continuum turns existing content into live visual portals
Continuum is Runway's early research idea for real-time video and world models inside an operating system. Valenzuela shows a page about fog in San Francisco and selects part of its content to create a portal. The portal becomes an interactive video based on the text or other material being viewed. Users can resize it, move it, click through it, and ask questions. Continuum keeps previous portals as a visual memory, including one about batteries. A photosynthesis example changes when the application is resized and lets the user explore parts of a cell. Valenzuela says these experiences do not fit neatly into the categories of video, visualisation, or games.
Visual thinking gives coding agents an interactive space for reasoning
Continuum can connect to a coding agent and help it think through a problem visually. Valenzuela uses the example of asking an agent to create a 3D model of a bicycle. Instead of reasoning only through code, the agent can create an interactive version of Continuum and work through the problem in a visual space. A person can see that process and guide the agent through the visual representation. Valenzuela compares this with people who think through problems visually before deciding what to do. He presents visual thinking as a way to give agents an imagination that both the agent and the user can inspect and interact with.
Future interfaces may generate live experiences around a user's intent
In a final Continuum example, Valenzuela searches for Mars. The system infers the user's intent and creates an interface for a Mars outpost in real time. The interface includes information about robots on Mars and possible robots projected for 2030. It is interactive, so the user can move through the environment as they might in a 3D system. Valenzuela's claim that everything will be simulated refers to this shift from fixed pages and videos toward live experiences generated around the material and the user's actions. He says the work is still at an early stage, while also saying that it works and scales.
"You're learning about physics, you're learning about object permanence, you're learning about gravity, you're learning about occlusion, you're learning about movements, you're learning about how the objects work in the real world."04:44
Who should watch
You are building video, simulation, or robotics systems and want to understand why next-frame prediction could support more than media generation.
You are designing software interfaces and want examples of interfaces generated from pixels, text, clicks, and user intent.
You work on coding agents and are exploring ways for agents and people to reason through problems visually.