Reinforcement learning and agent building use the same basic loop of a model interacting with an environment and receiving evaluations.
2
Tools make agent training useful for real tasks, but reward hacking makes evaluation design difficult.
3
Open-source tooling can make multi-turn RL experiments feel closer to writing an ordinary agent, with APIs, environments, parsers, and rubrics as reusable pieces.
Summary
Will Brown argues that reasoning models and agents should be understood through the same interaction loop. A model takes actions, uses tools, changes state, and receives an evaluation. Reinforcement learning improves that behavior by learning from differences between better and worse rollouts. He explains why GRPO has attracted interest as a simpler and more efficient option than PPO, while retaining useful signals from sampled branches. The hard part is often evaluation. Models can exploit a reward signal, so the task must be easier to complete than the evaluation is to game. Brown discusses using smaller subtasks, language models, rubrics, and reward models for ambiguous work. He then describes multi-turn tasks such as coding, games, computer use, and long-horizon planning. His verifiers toolkit gives these environments an OpenAI-compatible interface and supports testing with existing APIs, synthetic data, supervised fine-tuning, and later RL.
Reasoning models and agents follow the same interaction pattern
Brown's central claim is that reasoning and agents are not separate technical categories. An agent has a harness, environment, tools, and an iteration loop. Reinforcement learning describes the same structure with policies, actions, states, rewards, and transition probabilities. In both cases, a model interacts with a system and is evaluated on what happened. Prompt changes, new tools, harness changes, and model swaps are forms of manually improving that loop. RL automates part of this process by updating the model from evaluation scores. This framing applies to agents that do more than chain static API calls.
RL works when the setup and signal are good enough
Brown points to DeepSeek as evidence that reinforcement learning can produce a curve where more RL makes a capable model better, provided the setup and signal are appropriate. He says OpenAI's o3 direction also reflects increased investment in RL and compute after pretraining. In his view, o3's appeal comes from using tools in difficult, interactive tasks, rather than only from being a smarter chat model. RL can help a system that works at small scale stop going off the rails as the task becomes more complex. The method remains a research topic, especially outside large labs.
RL infrastructure has many parts that agent developers cannot ignore forever
Brown shows the architecture of VeRL and the GRPO process from the DeepSeek Math paper to illustrate how much machinery RL involves. Developers who are used to APIs may prefer to ignore these details, but he says people targeting highly capable agents need to understand at least the main pieces. He also argues that startups and individual researchers need simpler ways to use the method. Training an open model for a specific task could create a system with its own learned capability, rather than an application that mainly wraps another model's API.
Advantage estimates identify which sampled decisions led to better outcomes
The general RL loop starts with tasks, which can be prompts, and rollouts, which can contain many tool interactions. Evaluations may happen during or after the sequence. Because language models sample different behavior at nonzero temperature, some rollouts do better than others. Advantage estimation asks what changed and identifies where the model moved onto a better or worse path. Brown describes this as the signal that lets RL increase useful behavior without changing everything. He contrasts this with DPO, where a complete good completion and bad completion may not reveal the fine-grained branching decisions in a complex interaction.
GRPO is a practical middle ground for sampled agent behavior
Brown says PPO contains the kind of branching signal needed for these tasks, but is expensive. GRPO has attracted attention because it is simpler to implement and more computationally efficient while getting a useful forking process from sampling. He advises people not to chase every new paper or react to sensational claims about which model or loss function works. Most developers should focus on what RL is doing overall and decide which implementation details they can leave to software libraries. The important question is which pieces matter for the particular problems they want to solve.
Tools turn RL from benchmark training into agent training
Brown says the defining feature of an agent is the ability to interact with an environment through tools. He connects MCP to this idea, describing it as a way to let a language model change files, make requests, edit code, and run code. Existing RL code often centers on math and code because those tasks are easy to score. Brown's GSM8K example helped make this pattern accessible, but he warns that real software systems will not come from optimizing only the easiest question-answer benchmarks. Multi-turn tool use requires environments and evaluations that match the actual system.
Reward design should make completing the task easier than cheating the evaluation
Brown treats reward hacking as a direct lesson about evaluation design. A good evaluation captures what the developer cares about, and the model should find it easier to do the task than to exploit the scoring mechanism. Models optimize the reward signal, so a poorly aligned signal will lead them toward a shortcut. For ambiguous tasks, Brown describes a spectrum of verification difficulty rather than a simple solvable-versus-checkable split. One approach is to break evaluation into smaller pieces and use language models as evaluation routines. He uses 'rubric' as a broad term for reward models, reward functions, and language-model judges.
Multi-turn RL needs an agent-like programming interface
Brown expects future work to involve agentic search, tool calls, software, games, long-horizon planning, computer use, memory, and larger numbers of tool calls. He proposes treating environments as harnesses, rewards as evaluations, and tasks as prompts, while keeping the policy interface close to an ordinary language-model API. His verifiers repository follows this design. A rollout sets up initial state, loops until the task is done, and passes a client object through each turn. Parsers and rubrics are optional building blocks rather than mandatory abstractions.
Synthetic data and supervised fine-tuning can lower the entry cost to RL
Brown describes a workflow that starts with an existing API for debugging an environment and its reward. Developers can test with models such as Claude, DeepSeek, or OpenAI before training anything. Once the evaluation works, they can generate synthetic data, do supervised fine-tuning, and then begin RL, which is especially useful for smaller models. He says verifiers is designed to run interesting experiments on a couple of GPUs. The repository also addresses engineering problems such as asynchronous training and inference, computation utilization, and slightly off-policy operation, so users do not have to solve all of that first.
"What you really want with an eval is for it to be easier for your model to do the task than to hack the eval."12:15
Who should watch
You are building an agent with several tool calls and want to understand where RL fits beyond prompt and harness changes.
Your team needs to train an open model for a task, but you need a practical path from API-based evaluation to synthetic data, SFT, and RL.
You are designing rewards for messy software or interactive tasks and want a concrete explanation of reward hacking and rubric-based evaluation.
