Physical Intelligence is building vision-language-action models that use images, language, robot state, and learned actions to control robots.
2
The company says robot data collection is more than half of the work, so it built a teleoperation and annotation pipeline from the ground up.
3
PI05 uses diverse data from many homes and robots to perform long tasks in unfamiliar environments, while cross-robot control suggests that software intelligence may be the main bottleneck.
Summary
Quan Vuong and Jost Tobias Springenberg describe Physical Intelligence's approach to general-purpose robotics. Their target is a model that can control any robot to perform any task, though they say several scientific and engineering problems remain. Springenberg explains why vision-language-action models need more than a standard vision-language model: they must process robot state and produce fast, continuous control actions. The company collects training data through human operators who teleoperate robot arms, then annotates and filters the episodes for training. The scale and variety of this data have grown from static lab scenes to mobile manipulation across many environments. Their PI05 model combines a vision-language model with an action expert and can carry out long cleaning tasks in homes excluded from training. Vuong also shows a coffee-making demonstration on a robot the team had never handled. The talk makes a practical case that data operations and model intelligence are central constraints alongside hardware.
Physical Intelligence wants one model for many robots and tasks
Quan Vuong states the company's mission as making a model that can control any robot to do any task. He is clear that this is not ready today and says multiple scientific breakthroughs are still needed. Physical Intelligence publishes its research, open-sources its models, and discusses its work publicly. He contrasts the target with older industrial robotics, which often works in constrained factory settings with repetitive, structured motions. Robots can now produce complex physical movements, but handling ordinary objects remains difficult because the starting conditions vary. He points to a robot taking shirts from a dryer, placing them in a basket, and later folding them as an example of progress with semi-structured objects.
A vision-language-action model turns scene understanding into robot control
Jost Tobias Springenberg explains that a vision-language model takes text and images, embeds them, and produces a text answer through a transformer. A vision-language-action model adds robot-state inputs such as joint positions and produces actions instead of written answers. Robotics adds difficult engineering requirements. There is no obvious web-scale data source for dextrous robot behavior, the model architecture must support high-frequency control, and there is no standard way to deploy large robot policies across multiple on-premise or on-device locations. Physical Intelligence can reuse pretrained vision-language backbones, but it has to adapt them for direct robot control.
Physical Intelligence treats data collection as a system it must build itself
Springenberg says there is no standard data solution for training highly dextrous policies, so Physical Intelligence is building a data engine from zero. The team chooses an expanding set of tasks, including folding clothes and bagging groceries, then has human operators control leader arms strapped to their own arms. Software transfers those motions to the robot, allowing operators to demonstrate intricate actions. The company schedules collection sessions, tracks each episode, annotates data in the cloud, and filters it into training sets. Springenberg says getting the data pipeline right, including its quality and selection, is more than half of the work.
More data variety takes robot policies beyond static lab scenes
The initial comparison point was the Open X-Embodiment dataset, which Springenberg describes as containing about 3,800 hours of data, largely from static scenes in robot labs. After six months of its own data process, Physical Intelligence had collected about 10,000 hours of successful episodes across hundreds of tasks and tens of environments. Later collection added mobile manipulation and many more scenes. This increased diversity matters because it gives policies exposure to different homes, objects, and layouts. Springenberg connects that scale to the autonomous shirt-folding and other dextrous behaviors shown in the talk, while also acknowledging that broader data creates further engineering problems.
PI0 uses an action-producing architecture for dextrous control
Springenberg places PI0 in the recent progression of vision-language-action models. Earlier systems such as RT2 showed generalization but were limited by the amount of robot data available. Newer multi-robot models adapted their architectures to produce actions through diffusion, which supports the fast generation needed for control. Physical Intelligence's PI0 is presented as a dextrous multi-robot model that is open source. The architectural change matters because a robot policy must output continuous actions at a high rate, rather than generate a slow text response. This is one reason a standard vision-language model cannot simply be used unchanged for frontier robot behavior.
PI05 combines broad multimodal data with a separate action expert
PI05 expands training data to include static and mobile robot data, web data, object-detection data, and language annotations for collected robot episodes. Its model starts with a pretrained transformer and adds an action-expert transformer. The vision-language part answers general questions about a scene and breaks broad instructions into subtasks. For an instruction such as cleaning a bedroom, it can identify subtasks such as picking up a pillow. The action expert attends to the larger model's internal representations and produces continuous actions through a diffusion flow-matching objective at a higher rate. The result is a policy designed for long-horizon control rather than a single isolated movement.
Training across homes improves performance in unseen homes
Springenberg says PI05 can perform difficult episodes lasting up to ten minutes in homes that were entirely absent from training. In one example, the model receives a cleaning instruction in a new home and carries out several tasks in a kitchen. The team also varied the number of homes represented in training while keeping the amount of data fixed, then tested the model in a held-out location. Performance in the test scene generally increased as more locations were added and eventually matched or slightly exceeded training that included the test scene. A bedroom example shows the model breaking cleaning into actions such as throwing trash away and making the bed, then controlling the robot for several minutes.
Cross-robot control makes software intelligence a central test
Vuong shows a robot making coffee even though the Physical Intelligence team had never seen or handled that robot. The robot was far from the company's office, and the team did not know its internal design. Vuong uses the demonstration to test whether one model can run across different hardware without the company spending major effort adapting to each platform. He says hardware and deployment matter, but believes software and model intelligence are one of the main bottlenecks. Physical Intelligence sends model checkpoints to partner companies so they can run inference themselves and discusses low-level technical details with them. He ends by saying scientific, engineering, and operational problems remain unsolved.
"We've never touched this robot before, we don't know how it works internally and yet our model can control that robot to perform a fairly interesting task."16:16
Who should watch
You are building robot policies and need to understand why data collection, annotation, and deployment are separate engineering problems.
You work on multimodal models and want to see how a vision-language model is adapted to produce high-frequency robot actions.
You are evaluating general-purpose robotics claims and want evidence about performance in unseen homes and on unfamiliar hardware.
