Waymo's EMMA uses a multimodal model to turn camera video and routing text into future driving waypoints.
2
Adding intermediate reasoning about objects, their behavior, and the car's driving decision improves planning results and makes the model easier to inspect.
3
Generative video can create controlled sensor conditions for testing end-to-end driving models across weather and time of day.
Summary
Jyh Jing Hwang describes Waymo's move from a modular autonomous-driving stack toward more generalizable end-to-end models. The existing system separates perception, prediction, and planning, while EMMA uses Gemini-style multimodal modeling to process eight camera views and routing text, then predict the car's future waypoints. The initial version is camera-only, self-supervised from driving logs, and does not require a high-definition map. Hwang then adds intermediate outputs for critical objects, predicted behavior, and driving decisions, which improves results on Waymo's larger motion dataset and makes the planner easier to inspect. EMMA is also trained across planning, 3D detection, road-graph estimation, and visual question answering. The talk closes with evaluation. Hwang argues that open-loop replay is limited, so Waymo is testing generative sensor simulation that can vary weather and time of day. He is clear that the work remains research, especially when generated conditions affect camera-only planning.
Waymo's current driving system separates understanding, prediction, and planning
Hwang explains Waymo's deployed system as three connected parts. Perception identifies cars, pedestrians, cyclists, traffic lights, and crossroads. Prediction estimates future world states from the current scene. Planning then decides how the vehicle should drive, including whether to turn and how much to accelerate or steer. He presents this as a complicated system that already operates in San Francisco and other cities. Scaling the service creates rare situations that the system must handle repeatedly, such as a red traffic light being overridden by a traffic controller waving the car through.
Foundation models may help autonomous vehicles handle rare situations
Hwang uses unusual events to explain why general-purpose multimodal models could help with autonomous driving. In one example, a flock of birds suddenly takes off in front of the vehicle. Gemini identifies the flock, describes the expected behavior, and recommends slowing down while remaining alert. In another, a scooter rider slips on a wet road at night. Gemini identifies the event and even notices a distant gas station. Hwang presents these cases as long-tail events that are difficult to cover exhaustively with ordinary driving data, but that foundation models may already understand from broader training.
EMMA predicts near-future waypoints from camera video and route text
The basic EMMA design combines a route instruction with the vehicle's camera views. The route is represented as text, such as turning left or right at the next intersection. Eight surrounding cameras provide 360-degree visual coverage. Built on Gemini, EMMA predicts future waypoints, meaning the locations where the vehicle should be over the next few seconds. Hwang describes three properties of this setup: driving logs provide self-supervision because the vehicle's future locations are known, the model uses cameras rather than lidar, and it does not need a high-definition map apart from the route information.
Intermediate reasoning improves planning and exposes what the model is using
Hwang says the first self-supervised approach has limitations, including the difficulty of using existing labeled data and the lack of explainability in an end-to-end output. EMMA therefore produces intermediate reasoning before its planner output. It identifies critical objects, describes how those objects may behave, and chooses a driving decision. For a scene with a cyclist and another vehicle, the model might decide to maintain speed, yield, or slow down. Hwang reports better results with this chain of reasoning on Waymo's 100k open motion dataset, where the comparison models use specialized architectures and inputs such as oracle perception, road graphs, high-definition maps, and traffic-light states.
More driving data continues to improve the model's planning score
Hwang connects EMMA's scaling behavior to the idea that larger models and larger datasets can keep improving quality. He shows results from training on a dataset much larger than the academic datasets commonly released publicly. In the chart he describes, lower perplexity corresponds to better planner results. As more data is used for training, the measured quality continues to improve. The claim is about the observed trend in this research setup, rather than a finished production system.
One multimodal model can be trained across several autonomous-driving tasks
Because EMMA is a vision-language model, Hwang says its language interface can express many task types. He adds driving, 3D object detection, road-graph estimation, and free-form visual question answering. A prompt determines which answer the model produces, and the outputs are decoded into trajectories, detections, or road-graph visualizations. Hwang reports that EMMA's detection quality is similar to other state-of-the-art models on Waymo's open dataset. The examples are intended to show that several driving tasks can be co-trained in one model instead of being treated as isolated components.
Generative sensor simulation can test planners under controlled conditions
Hwang says open-loop evaluation only replays recorded video and checks the model's output, so it is less faithful than simulation or real-world testing. Waymo's research uses an open-source generative model from Google to create sensor videos, then runs the EMMA planner on them. The conditions can be changed, including weather and time of day. In the reported results, rain and other poor weather reduce the performance of the camera-only planner, while nighttime is generally worse than daytime. Hwang presents this as ongoing research for evaluating end-to-end motion planning models, not as a completed replacement for real-world testing.
"When it rains, when the weather is bad, then usually our planner gets a little bit worse because it's a camera-only model."15:34
Who should watch
You are building an end-to-end driving model and want an example of how camera inputs, route text, and waypoint prediction can fit together.
Your perception and planning systems are separate, and you want to see how intermediate reasoning and multi-task training could connect them.
You need to evaluate motion planning under weather and lighting conditions that are difficult to collect safely in the real world.
