# Everything Is a Rollout

Alex Shaw & Ryan Marten, Laude Institute | AI Engineer World's Fair 2026 | 21:11

Source: https://www.youtube.com/watch?v=jRCpXUjz4CI
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/everything-is-a-rollout
Published: 2026-07-24
Tags: agents, evals, reinforcement-learning, workflows

## TL;DR
- Agent development has more in common with machine learning than with traditional software engineering because agent behavior must be measured empirically.
- A useful agent environment combines an instruction, a sandbox, a verifier, and a stopping condition, then turns the resulting trajectory into a rollout and reward.
- Harbor connects environment specifications, parallel rollouts, evaluations, and optimization workflows so teams can improve agents against tasks that matter to them.

## Summary
Alex Shaw presents agent development as a machine learning workflow. Traditional software engineering often lets a developer predict what code will do before running it, while model-driven agents produce behavior that must be measured through repeated execution. He describes an environment as an instruction, a virtual computer or sandbox, and a verifier that checks the result. Harbor packages these environments in a shared format and runs agents in parallel across tasks, models, and sandboxes. The resulting trajectories and rewards can support evaluation, supervised fine-tuning, reinforcement learning, or text-based optimization. Shaw argues that companies should build evaluations around work that matters to their own products and internal processes. He also describes production uses such as distributed agentic map-reduce, analyzing coding sessions, processing receipts, searching notes, and reviewing pull requests. The talk is practical about the central loop: define tasks, run rollouts, inspect outcomes, and use what was learned to improve the next batch.

## Key ideas
### Agent development requires empirical measurement because model behavior is uncertain
[02:28](https://www.youtube.com/watch?v=jRCpXUjz4CI&t=148s)
Alex Shaw contrasts a regular expression with a model call that extracts phone numbers. The regular expression is brittle, but its behavior is predictable when run repeatedly. The model may handle unusual formatting better, yet Shaw is not fully confident that repeated runs will produce the same output. The uncertainty grows as tasks become more complex. He generalizes a point from François Chollet: agent performance should be treated as a black-box artifact and managed through empirical evaluation, in the same way that machine learning models are evaluated.

### Agent building maps onto familiar machine learning concepts
[05:44](https://www.youtube.com/watch?v=jRCpXUjz4CI&t=344s)
Shaw maps machine learning ideas to agent development. Training data becomes environments, while test and validation sets become validation environments. Model weights become the combination of skills, prompts, tools, and model choice. A loss function becomes environment rewards and feedback. Backpropagation or an optimizer becomes a text-based method such as GEPA or a coding agent running in a loop. A gradient step becomes a pull request into the repository. He also compares overfitting with reward hacking and other forms of agent overfitting.

### An agent environment needs a task, a computer, and a way to grade the result
[07:56](https://www.youtube.com/watch?v=jRCpXUjz4CI&t=476s)
In Harbor, an environment starts with an instruction that tells the agent what to do. The agent works inside a virtual computer, or sandbox. A verifier then checks whether the task was completed within a time limit or another stopping condition. The verifier can use programmatic tests, rubrics, or another agent. Harbor represents the environment as a file directory with a layout that has become common across the environment space. That shared structure lets environments move between teams and tools.

### A rollout turns one environment run into data for evaluation
[09:20](https://www.youtube.com/watch?v=jRCpXUjz4CI&t=560s)
A Harbor rollout starts with a task and a sandbox. The agent can run outside the sandbox and execute commands into it, or run inside it and call its tools there. After the agent reaches a stopping condition, the run produces a trajectory. The sandbox goes to the verifier, which produces one or more rewards. Harbor aggregates rewards across many rollouts into a dataset that describes how an agent performed. Shaw calls this a simple but universal pattern, while noting that Harbor also supports multi-step runs, separate verification sandboxes, artifact collection, and simulated users.

### Harbor provides a common format and parallel execution layer
[10:34](https://www.youtube.com/watch?v=jRCpXUjz4CI&t=634s)
Shaw describes Harbor as a format for specifying agent environments, an open-source framework for running rollouts in parallel, and a registry of training and validation environment sets. It can run any agent with any model in any sandbox on any task. He says the project is intended to provide a common language for environments, making them interoperable and increasing the speed at which teams can generate data. He also says hundreds of validation sets were already available through Harbor and that new benchmarks were being released with Harbor support.

### Teams should build evaluations around work that matters to them
[11:35](https://www.youtube.com/watch?v=jRCpXUjz4CI&t=695s)
Shaw argues that every company using computers should consider building an evaluation for work it wants to automate. He cites Satya Nadella's advice to start an agentic system with the evaluation that matters and a way to grade its outcome. Once a company has its own evaluation, it can compare models directly instead of relying on a brand, someone else's evaluation, or a public benchmark. Shaw says this lets teams choose their preferred cost and performance tradeoff. He identifies four evaluation targets: agents building a company's products, agents using its products, agents powering product features, and agents automating internal processes.

### Rollouts can run production workflows as well as evaluations
[15:55](https://www.youtube.com/watch?v=jRCpXUjz4CI&t=955s)
Harbor does not have to be used only for evaluation or training. Shaw describes agentic map-reduce, where many agents run on distributed sandboxed compute and their results are aggregated. Examples include reviewing trajectories for reward hacking, processing reimbursement receipts, searching Obsidian files for a note, and asking questions about a collection of pull requests. Harbor Exec grew out of this use case. In his example, coding sessions are processed in parallel, one model writes structured analyses of mistakes, and another model reduces those analyses into recurring failure categories.

### Trajectories and rewards feed several optimization methods
[18:46](https://www.youtube.com/watch?v=jRCpXUjz4CI&t=1126s)
The outputs of rollouts can be reused in multiple ways. Teams can take trajectories and perform supervised fine-tuning. They can use rewards, or trajectories represented as tokens, for reinforcement learning. Shaw mentions an integration between Tinker and Harbor. He also describes text-based optimization and evolutionary methods, where feedback from trajectories and evaluations drives an automatic hill-climbing loop through a harness or scale. This makes the rollout more than a pass-fail test. It becomes source material for changing the agent and creating the next evaluation batch.

## Notable quotes
- "Software engineering was when you knew what the code would do before you ran it." (02:48)
- "Agent performance itself is best treated as a blackbox artifact." (05:05)
- "If you want to build an agentic system start with the aval that matters and your ability to grade the outcome." (11:53)
- "You can do what we call prod rollouts." (15:55)

## Tools & references mentioned
- Harbor
- Terminal-Bench
- OpenThoughts-Agent
- GPT-1
- GPT-5.5
- François Chollet
- GEPA
- Satya Nadella
- Applied Compute podcast
- RampBench
- Ramp
- HubSpot
- Stripe
- Tinker
- Frontier Suite
- Ultra Long Horizon Software Engineering benchmark
- Banker ToolBench
- Handshake
- Cognition
- Rune Bench
- RuneScape
- Scale
- SWE Atlas
- Poolside
- Kevin Gu
- Auto Agent
- AfterQuery
- Frontier Code
- LangChain
- Deep Agents
- Snorkel
- Senior SWE Bench
- Modal
- Cursor CLI
- Obsidian

## Who should watch
- You are building an agent and need a repeatable way to measure whether changes improve its behavior.
- Your company wants to evaluate agents on internal code, product usage, product features, or computer-based operations.
- You need to run many sandboxed agent tasks in parallel and reuse their trajectories for fine-tuning or reinforcement learning.