Robots Will Be Trained on Video Games

Pim de Witte, General Intuition18:51 · Nov 2026 · 4,280 views
Thumbnail for Robots Will Be Trained on Video Games Watch on YouTube
TL;DR
  1. 1

    Models that act in space and time need continuous visual input and continuous action output, rather than the turn-by-turn structure of an LLM.

  2. 2

    Game controllers connect Medal's large collection of game footage to robot control, so the same pixel-to-action model can operate games, computers, and more than 10 robot types.

  3. 3

    General Intuition plans to release these models through early access, with uses in robotics, interface testing, simulation, and accessibility.

Summary

Pim de Witte argues that models for acting should be built around continuous observation and action. LLMs work well when an interaction unfolds in turns, such as writing code, but they struggle when a system must react to changing pixels, space, and time. General Intuition trains models on Medal game recordings, where frames are paired with controller actions and semantic labels. The company's approach sends frames into a model and produces keyboard, mouse, or controller actions in real time. De Witte says the same primitives can control games and more than 10 robot types because many robots already use game controllers. He also describes world-model environments, high-action-rate game data, accessibility applications, and a planned early-access release. In the Q&A, he contrasts the approach with JEPA and says action labels make his method more practical for action prediction in the short term.

Key ideas
00:58

Acting requires a model to observe while it acts

De Witte uses Olaf from Frozen to explain the difference between predicting and acting. When Olaf rolls down a mountain and temporarily comes apart to avoid a stone, he observes the situation while his behavior unfolds. An LLM predicts tokens in sequence, while an acting system continually receives observations and changes its actions. De Witte says this distinction matters for robotics because space and time keep changing during the task. He argues that purpose-built models fit these settings better than treating an LLM demo as a general solution to every robotics problem.

01:54

Medal turns game recordings into paired visual and action data

General Intuition began with Medal, which De Witte describes as the largest video game recorder. People record themselves playing tens of thousands of games. Medal associates controller or keyboard inputs with the frames on screen, then labels the actions with their semantic meaning. A model can receive frames, optionally receive a text instruction, and choose from actions such as greeting. This gives the model both the visual situation and the action taken in that situation. De Witte presents the setup as a vision-based computer-use model with a larger action space.

02:03

Game controllers provide a common interface between games and robots

De Witte says humans can use the same brain, and often the same controller, to play a game and teleoperate a robot. He claims that many robots below a certain level of freedom already ship with game controllers, so the action space in games often matches the action space used for real-world control. General Intuition therefore sends pixels into a model and gets keyboard, mouse, or controller actions out. The model does not need to predict motor torque directly. It uses the interface people already use to control the machine.

03:24

Continuous frame input separates action models from turn-based LLMs

De Witte describes LLMs as models trained on internet data to predict the next text token. That structure fits work such as coding, where a person and a model can take turns. It is harder to apply when the model must handle environments that unfold in space and time. General Intuition's models continuously receive frames from a robot camera or a screen and predict actions as if the task were a video game. De Witte compares the design choice to Anthropic's focus on coding, saying General Intuition is betting that more problems will be brought into pixel space.

05:21

Game footage supplies unusually dense action data

De Witte says robotics companies often try to predict exact poses or motor values, even though people use a controller to avoid reasoning about motor torque themselves. General Intuition instead uses controller actions as the model's output. Medal sees about a billion uploaded videos per year and was on track for 3 billion in the year of the talk, according to De Witte. He says this produces over 1.5 trillion action tokens per month and compares half a day of the company's collection with the total data Frontier Labs had used to train robotics models at that point.

07:14

World models can create more training environments in the model's native input space

General Intuition also works with world models to generate environments for training. De Witte says an LLM-based robotics pipeline may need to simulate an environment, create a digital twin, deploy a model, and check whether the simulated and real inputs match. Learning directly from pixels avoids those conversions. He wants a robot to observe the same kind of input a human sees while learning. He argues that this can make it easier to train agents in generated environments and to apply the result to real tasks later.

08:55

High action rates make game data useful for fast-moving robot tasks

Games contain many more small, rapid actions than ordinary teleoperation. De Witte uses drone control as an example: a drone may need frequent small adjustments, while a person operating a robot may take far fewer actions. He says the action rate of gamer data can support models that predict many actions at once. The model also learns the semantic meaning of the action associated with a key, rather than only treating the key as an arbitrary symbol. De Witte connects that semantic information to interpretability.

09:52

One model can use the same action primitives across games and robots

De Witte says General Intuition plans to release its models and has already demonstrated them privately. He describes one model that can play video games at a high level and control more than 10 types of robots using the same primitives. The system takes in pixels and produces actions in real time. Early access is planned for the following months, with some customers already using private versions. De Witte expects users to connect the action model to an orchestrator such as Gemini or Astra for long-term planning and other higher-level work.

11:03

The same interface can support accessibility and interface testing

De Witte describes uses beyond direct robot control. A model that operates a computer like a person could help someone who can use only one arm to play a game. The person could control the mouse while the model predicts keyboard actions, and the model can account for the actions the person is already taking. He also suggests using the model to test whether a user interface is intuitive by having an agent explore it. Generated environments could let developers test these interactions without doing all the exploration themselves.

"We just bet on scaling pixel space the same way that Anthropic scaled coding space and betting that people will just as we scale data compute into pixel space that people will bring more problems into pixel space."06:56
Who should watch
  • You are building robot-control systems and want to explore controller-based action models instead of predicting motor values directly.
  • You have game recordings, simulated environments, or computer-use tasks and need a model that reacts continuously to visual input.
  • You work on accessibility or interface testing and want an agent to operate a computer through the same controls a person uses.