# Training Agentic Reasoners

Will Brown, Prime Intellect | AI Engineer World's Fair 2025 | 19:17

Source: https://www.youtube.com/watch?v=PbHm2qKnu10
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/training-agentic-reasoners
Published: 2025-07-07
Tags: agent-skills, agents, evals, reinforcement-learning

## TL;DR
- Reinforcement learning and agent building use the same basic loop of a model interacting with an environment and receiving evaluations.
- Tools make agent training useful for real tasks, but reward hacking makes evaluation design difficult.
- Open-source tooling can make multi-turn RL experiments feel closer to writing an ordinary agent, with APIs, environments, parsers, and rubrics as reusable pieces.

## Summary
Will Brown argues that reasoning models and agents should be understood through the same interaction loop. A model takes actions, uses tools, changes state, and receives an evaluation. Reinforcement learning improves that behavior by learning from differences between better and worse rollouts. He explains why GRPO has attracted interest as a simpler and more efficient option than PPO, while retaining useful signals from sampled branches. The hard part is often evaluation. Models can exploit a reward signal, so the task must be easier to complete than the evaluation is to game. Brown discusses using smaller subtasks, language models, rubrics, and reward models for ambiguous work. He then describes multi-turn tasks such as coding, games, computer use, and long-horizon planning. His verifiers toolkit gives these environments an OpenAI-compatible interface and supports testing with existing APIs, synthetic data, supervised fine-tuning, and later RL.

## Key ideas
### Reasoning models and agents follow the same interaction pattern
[00:00](https://www.youtube.com/watch?v=PbHm2qKnu10&t=0s)
Brown's central claim is that reasoning and agents are not separate technical categories. An agent has a harness, environment, tools, and an iteration loop. Reinforcement learning describes the same structure with policies, actions, states, rewards, and transition probabilities. In both cases, a model interacts with a system and is evaluated on what happened. Prompt changes, new tools, harness changes, and model swaps are forms of manually improving that loop. RL automates part of this process by updating the model from evaluation scores. This framing applies to agents that do more than chain static API calls.

### RL works when the setup and signal are good enough
[00:41](https://www.youtube.com/watch?v=PbHm2qKnu10&t=41s)
Brown points to DeepSeek as evidence that reinforcement learning can produce a curve where more RL makes a capable model better, provided the setup and signal are appropriate. He says OpenAI's o3 direction also reflects increased investment in RL and compute after pretraining. In his view, o3's appeal comes from using tools in difficult, interactive tasks, rather than only from being a smarter chat model. RL can help a system that works at small scale stop going off the rails as the task becomes more complex. The method remains a research topic, especially outside large labs.

### RL infrastructure has many parts that agent developers cannot ignore forever
[03:03](https://www.youtube.com/watch?v=PbHm2qKnu10&t=183s)
Brown shows the architecture of VeRL and the GRPO process from the DeepSeek Math paper to illustrate how much machinery RL involves. Developers who are used to APIs may prefer to ignore these details, but he says people targeting highly capable agents need to understand at least the main pieces. He also argues that startups and individual researchers need simpler ways to use the method. Training an open model for a specific task could create a system with its own learned capability, rather than an application that mainly wraps another model's API.

### Advantage estimates identify which sampled decisions led to better outcomes
[07:14](https://www.youtube.com/watch?v=PbHm2qKnu10&t=434s)
The general RL loop starts with tasks, which can be prompts, and rollouts, which can contain many tool interactions. Evaluations may happen during or after the sequence. Because language models sample different behavior at nonzero temperature, some rollouts do better than others. Advantage estimation asks what changed and identifies where the model moved onto a better or worse path. Brown describes this as the signal that lets RL increase useful behavior without changing everything. He contrasts this with DPO, where a complete good completion and bad completion may not reveal the fine-grained branching decisions in a complex interaction.

### GRPO is a practical middle ground for sampled agent behavior
[08:48](https://www.youtube.com/watch?v=PbHm2qKnu10&t=528s)
Brown says PPO contains the kind of branching signal needed for these tasks, but is expensive. GRPO has attracted attention because it is simpler to implement and more computationally efficient while getting a useful forking process from sampling. He advises people not to chase every new paper or react to sensational claims about which model or loss function works. Most developers should focus on what RL is doing overall and decide which implementation details they can leave to software libraries. The important question is which pieces matter for the particular problems they want to solve.

### Tools turn RL from benchmark training into agent training
[10:10](https://www.youtube.com/watch?v=PbHm2qKnu10&t=610s)
Brown says the defining feature of an agent is the ability to interact with an environment through tools. He connects MCP to this idea, describing it as a way to let a language model change files, make requests, edit code, and run code. Existing RL code often centers on math and code because those tasks are easy to score. Brown's GSM8K example helped make this pattern accessible, but he warns that real software systems will not come from optimizing only the easiest question-answer benchmarks. Multi-turn tool use requires environments and evaluations that match the actual system.

### Reward design should make completing the task easier than cheating the evaluation
[12:15](https://www.youtube.com/watch?v=PbHm2qKnu10&t=735s)
Brown treats reward hacking as a direct lesson about evaluation design. A good evaluation captures what the developer cares about, and the model should find it easier to do the task than to exploit the scoring mechanism. Models optimize the reward signal, so a poorly aligned signal will lead them toward a shortcut. For ambiguous tasks, Brown describes a spectrum of verification difficulty rather than a simple solvable-versus-checkable split. One approach is to break evaluation into smaller pieces and use language models as evaluation routines. He uses 'rubric' as a broad term for reward models, reward functions, and language-model judges.

### Multi-turn RL needs an agent-like programming interface
[14:58](https://www.youtube.com/watch?v=PbHm2qKnu10&t=898s)
Brown expects future work to involve agentic search, tool calls, software, games, long-horizon planning, computer use, memory, and larger numbers of tool calls. He proposes treating environments as harnesses, rewards as evaluations, and tasks as prompts, while keeping the policy interface close to an ordinary language-model API. His verifiers repository follows this design. A rollout sets up initial state, loops until the task is done, and passes a client object through each turn. Parsers and rubrics are optional building blocks rather than mandatory abstractions.

### Synthetic data and supervised fine-tuning can lower the entry cost to RL
[16:15](https://www.youtube.com/watch?v=PbHm2qKnu10&t=975s)
Brown describes a workflow that starts with an existing API for debugging an environment and its reward. Developers can test with models such as Claude, DeepSeek, or OpenAI before training anything. Once the evaluation works, they can generate synthetic data, do supervised fine-tuning, and then begin RL, which is especially useful for smaller models. He says verifiers is designed to run interesting experiments on a couple of GPUs. The repository also addresses engineering problems such as asynchronous training and inference, computation utilization, and slightly off-policy operation, so users do not have to solve all of that first.

## Notable quotes
- "And I think the high level thesis of this talk is like no, they're kind of the same thing." (00:28)
- "And RL is simply an algorithm to improve based on the scores of these evaluations." (06:18)
- "What you really want with an eval is for it to be easier for your model to do the task than to hack the eval." (12:15)
- "The thing that makes an agent an agent is tools." (10:20)
- "The barrier of entry is like much lower now than it used to be." (18:40)

## Tools & references mentioned
- Prime Intellect
- DeepSeek
- OpenAI
- o3
- Deep Research
- Claude
- Claude Code
- Devin
- Manus
- VeRL
- GRPO
- PPO
- DPO
- DeepSeek Math
- MCP
- GSM8K
- verifiers

## Who should watch
- You are building an agent with several tool calls and want to understand where RL fits beyond prompt and harness changes.
- Your team needs to train an open model for a task, but you need a practical path from API-based evaluation to synthetic data, SFT, and RL.
- You are designing rewards for messy software or interactive tasks and want a concrete explanation of reward hacking and rubric-based evaluation.

## Related talks

- [Reinforcement Learning for Agents](https://aietalks.com/talks/reinforcement-learning-for-agents) (Will Brown, Morgan Stanley, 18:17)
- [Reasoning + RL](https://aietalks.com/talks/reasoning-rl) (Will Brown & Greg Kamradt, ARC Prize Foundation & Aakanksha Chowdhery, Reflection AI & Ryan Marten, Bespoke Labs & Kyle Corbitt, OpenPipe & Nathan Lambert, AI2 & Christian Szegedy, Former co-founder of xAI, 3:54:58)
- [How to Train Your Agent: Building Reliable Agents with RL](https://aietalks.com/talks/how-to-train-your-agent-building-reliable-agents-with-rl) (Kyle Corbitt, OpenPipe, 19:48)
- [Reinforcement Learning without Verifiable Rewards](https://aietalks.com/talks/reinforcement-learning-without-verifiable-rewards) (Will Brown, Prime Intellect, 19:27)
- [Creating Agents that Co-Create](https://aietalks.com/talks/creating-agents-that-co-create) (Karina Nguyen, OpenAI, 24:22)