# Agents are Robots Too: What Self-Driving Taught Me About Building Agents Jesse Hu, Abundant | AI Engineer CODE 2025 | 17:37 Source: https://www.youtube.com/watch?v=qqXdLf3wy1E Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/agents-are-robots-too-what-self-driving-taught-me-about-building-agents Published: 2025-11-24 Tags: computer-use, evals, harness-engineering, reinforcement-learning ## TL;DR - The model often does only a small part of an agent system's work; infrastructure for tools, deployment, monitoring, simulation, and feedback takes most of the effort. - Agents need closed-loop interaction because tool calls can fail, processes can continue running, and the agent must observe consequences before choosing what to do next. - Self-driving showed that action models and production infrastructure matter as much as predictive models, and the same problem is now appearing in coding agents. ## Summary Jesse Hu compares digital agents with robots and draws on his work at YouTube, Google, Waymo, and Abundant. An agent has a digital body made from tools, APIs, MCPs, terminals, browsers, VMs, and persistent files. Like a robot, it must observe the world, act, and respond to what happened. Hu argues that many current agents are too open-loop and too turn-based to handle running processes or unexpected events well. He also explains why imitation learning breaks when an agent meets situations outside its demonstrations, such as an unfamiliar browser pop-up. Simulation, reinforcement learning, evaluation, and real-world logs create a feedback loop for improving these systems. His comparison with self-driving is direct: the field spent years assuming perception was the hard part and planning was easy, then discovered that action and execution contained much of the difficulty. Coding agents face a similar gap between impressive reasoning and reliable end-to-end work completion. ## Key ideas ### Agent systems need far more than a capable model [01:13](https://www.youtube.com/watch?v=qqXdLf3wy1E&t=73s) Hu describes a 1% versus 99% problem. Builders may think the model does most of the work, but production systems require hardware or digital tools, integration, deployment, monitoring, retraining, human feedback, and an offline stack. In robotics, that stack includes simulation and training infrastructure. In agents, the equivalent work supports development and operation around the model. Hu says the winning self-driving team would not simply have the best model and online stack. It would have the best offline stack because developers could move faster and ship more reliably. ### An agent's body is made from tools and computing environments [01:52](https://www.youtube.com/watch?v=qqXdLf3wy1E&t=112s) Robots connect a brain to a physical body. Agents connect a model to a digital body made from APIs, MCPs, terminals, browsers, VMs, operating systems, and persistent file systems. These components define what the agent can perceive and what actions it can take. Hu compares them with a robot's hands, arms, and legs. The comparison becomes more important as agents stop producing one-off artifacts and begin operating inside persistent environments with files, processes, and ongoing conversations. ### Closed-loop control lets an agent check what its action actually did [02:56](https://www.youtube.com/watch?v=qqXdLf3wy1E&t=176s) In robotics, turning a wheel is only the start of an action. The system measures how much the car turned, then adjusts if the result differs from the intention. Hu applies this idea to agents. A bash command may run as an open-ended process whose output is not visible in real time. The agent may not know whether it finished, and it may be unable to stop it early. More closed-loop interfaces would let the agent observe progress and respond to the actual result instead of assuming the command worked. ### Turn-based agents trade real-time behavior for simpler reasoning [03:58](https://www.youtube.com/watch?v=qqXdLf3wy1E&t=238s) Robotics systems choose how often to sample the world and replan. They might update once per second or at 50 Hz. Agents usually make an implicit choice to wait for a turn, execute a tool, and wait for the complete response. That makes inputs and outputs easier to reason about, but it prevents immediate responses to pop-ups, changing screens, and long-running processes. Hu points to Terminus from Terminal-Bench as an example of a more flexible interface. It uses a terminal stream that can accept and produce characters, including control commands such as Control-C. ### Persistent environments turn stateless sessions into stateful worlds [07:36](https://www.youtube.com/watch?v=qqXdLf3wy1E&t=456s) A simple agent can start a session, perform work, and return an artifact. Stateful agents running in VMs have to account for what is already running and what remains in persistent storage. Hu says the agent must consider the whole environment, including current Slack messages and the other things it may need to interact with. This changes online operation as well as evaluation and simulation. The starting state matters, and the result of one action can affect later actions in the same environment. ### Imitation learning breaks when the agent leaves the demonstrations [08:38](https://www.youtube.com/watch?v=qqXdLf3wy1E&t=518s) Hu connects browser-agent failures to DAgger and the distribution problem studied in robotics. Imitation learning, like supervised fine-tuning from human demonstrations, works from examples of human behavior. When the agent takes an unfamiliar path, it can quickly move out of distribution. A browser pop-up that was absent from training can confuse the agent, and the resulting mistake can create further mistakes. Reinforcement learning in simulation offers another way to expose the system to consequences and states that are missing from the human examples. ### Agents need action models because plans can fail in the real world [09:29](https://www.youtube.com/watch?v=qqXdLf3wy1E&t=569s) Hu distinguishes prediction from action. An agent predicts, acts, observes the consequence, and then reevaluates what it believed before. A language model may understand text and produce a long reasoning trace, yet its tool calls can fail or it may fail to recover from its own errors. He uses self-driving as a warning. From 2017 to 2020, much of the field focused on perception and assumed that detecting objects would make driving straightforward. The hidden difficulty was building systems that could act reliably in a changing world. ### Simulation and failure logs make iterative improvement more informative [13:33](https://www.youtube.com/watch?v=qqXdLf3wy1E&t=813s) Hu describes hill climbing as repeated experiments against a metric such as a benchmark. The metric may go up or down after each change, so teams need a process for learning from both outcomes. In self-driving, learning feeds simulation, simulation supports deployment, and real-world logs feed back into simulation. Logs are more useful when teams split failures into categories, cities, and specific ways the system can fail. Reviewing individual failures gives engineers more direction than a single aggregate benchmark score. ## Notable quotes - "The model is only doing 1% of the work and 99% of the work goes into other things." (01:13) - "We're dealing with a whole new paradigm in which you predict, you act, and then you deal with the consequences of that action and then re-evaluate everything you've done before." (09:29) - "It turns out that assumption wasn't necessarily true and there's a lot of hidden complexity in creating action models and not just predictive models." (11:29) - "A lot of the reasons are because of the things I brought up before with actions having consequences and the complexity of the real world." (15:37) ## Tools & references mentioned - YouTube - Google - BERT - mixture of experts - Waymo - Abundant - MCPs - Terminus - Terminal-Bench - Dreamer - DAgger - MDPs - offline RL - Remote Labor Benchmark ## Who should watch - You are building a coding agent that operates in terminals, browsers, VMs, or persistent workspaces and need to reason about its interfaces. - Your agent produces good plans but fails during tool execution, loses track of state, or cannot recover after an unexpected event. - You are designing evaluation, simulation, data collection, or deployment infrastructure and want concepts from robotics that apply to digital agents. ## Related talks - [Agent Engineering (Day 2)](https://aietalks.com/talks/agent-engineering-day-2) (swyx, AI Engineer Summit & Shashank Kapoors, Princeton University & Mukun Sudar & Arush, Google & Barry Zhang, Anthropic & Zach Renau, Sierra & Will Brown, Morgan Stanley & John Křizí, Jane Street & Anu Koduri, Bloomberg & Mike Conover, Brightwave & Kevin Hou, Windsurf & Mustafa Ali, Method Financial & Kyle Corbett, OpenPipe & Nick Kotakis, SuperDial & Rahul Sanghvi, Ramp & Karina Nguyen, OpenAI & Stefania Druga, Google & Soumith Chintala, Meta and PyTorch, 8:26:36) - [Why (Senior) Engineers Struggle to Build AI Agents](https://aietalks.com/talks/why-senior-engineers-struggle-to-build-ai-agents) (Philipp Schmid, Google DeepMind, 10:40) - [Building Applications with AI Agents](https://aietalks.com/talks/building-applications-with-ai-agents) (Michael Albada, Microsoft, 15:50) - [How to Build AI Agents that Actually Work](https://aietalks.com/talks/how-to-build-ai-agents-that-actually-work) (Patrick Dougherty, Rosco, 17:44) - [Agents Need More Than a Chat](https://aietalks.com/talks/agents-need-more-than-a-chat) (Jacob Lauritzen, Legora, 14:21)