# Reasoning + RL

Will Brown & Greg Kamradt, ARC Prize Foundation & Aakanksha Chowdhery, Reflection AI & Ryan Marten, Bespoke Labs & Kyle Corbitt, OpenPipe & Nathan Lambert, AI2 & Christian Szegedy, Former co-founder of xAI | AI Engineer World's Fair 2025 | 3:54:58

Source: https://www.youtube.com/watch?v=-9E9_21tx04
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/reasoning-rl
Published: 2025-06-06
Tags: agents, evals, planning, reinforcement-learning

## TL;DR
- Reinforcement learning becomes more useful for agents when tasks involve tools, multi-turn interaction, and rewards that measure the real task rather than an easy benchmark.
- Interactive benchmarks should test whether an AI can explore unfamiliar environments and acquire skills efficiently, using humans as the comparison point.
- Reasoning models are moving toward planning, calibration, tool use, and independently verified work, while better data recipes and smaller specialized models make training more accessible.

## Summary
This track covers several ways to move reasoning models beyond static math and coding benchmarks. Will Brown explains how RL can train tool-using agents, while warning that reward hacking is usually a failure of evaluation design. Greg Kamradt presents ARC AGI 3, an interactive benchmark based on novel games where agents must explore unfamiliar environments and learn without detailed instructions. Aakanksha Chowdhery argues that autonomous coding is a strong RL domain because code can be checked with tests and execution feedback. Ryan Marten focuses on the data recipe behind OpenThoughts, showing how supervised fine-tuning on carefully filtered reasoning traces can produce strong small models. Kyle Corbitt gives a practical agent-training case study using email search. Nathan Lambert argues that future models need calibration, strategy, and abstraction for long tasks. Christian Szegedy closes with a proposal for self-improving systems built around sandboxing, verification, and alignment checks.

## Key ideas
### Agentic RL needs tools and rewards that match the real task
[11:54](https://www.youtube.com/watch?v=-9E9_21tx04&t=714s)
Will Brown describes RL as comparing different sampled rollouts and learning which tokens led to better outcomes. GRPO is attractive because it keeps this branching signal while being simpler and more computationally efficient than policy optimization methods he discusses. For agents, the important addition is tool access, such as changing files, making requests, editing code, or running code. Brown warns that math and code benchmarks are unusually easy to evaluate, while real-world tasks are messier. Reward hacking happens when a model finds an easier way to improve the score than to do the task. The evaluation must make completing the task easier than exploiting the evaluator.

### Agent training should feel like ordinary agent development
[19:50](https://www.youtube.com/watch?v=-9E9_21tx04&t=1190s)
Brown presents his open-source verifiers toolkit, which makes an RL environment look like a normal agent loop. The rollout sets up state, checks whether the task is done, and calls an OpenAI-compatible client for each turn. Environments, rewards, tasks, and policies map closely to harnesses, evaluations, prompts, and language-model APIs. He uses Wordle as a small example of multi-turn learning, with rewards for solving the game and using fewer turns. Brown also recommends debugging with an API model first, generating synthetic data, and doing supervised fine-tuning before RL. This lets teams test their environment and reward before spending effort on training.

### Interactive benchmarks should measure learning in unfamiliar worlds
[24:00](https://www.youtube.com/watch?v=-9E9_21tx04&t=1440s)
Greg Kamradt argues that general intelligence should be compared with humans because humans are the only known example of general intelligence. ARC AGI 1 and 2 test whether a system can learn a new transformation and apply it, but they are single-turn tasks. ARC AGI 3 is designed to be interactive. It will place agents in games with defined rules and sparse rewards, without telling them what the world contains. The private evaluation games will be unseen by both the AI and its developer. Kamradt says the benchmark should measure skill acquisition efficiency, including how many actions and how much time humans and AI need to explore and complete the same task.

### Autonomous coding is a useful RL target because its outputs can be checked
[45:28](https://www.youtube.com/watch?v=-9E9_21tx04&t=2728s)
Aakanksha Chowdhery traces the move from pretraining to inference-time scaling and then reinforcement learning. Sampling multiple answers or revising a response helps when there is a way to verify correctness, such as a calculator, formal proof, compiler, or unit test. Coding has this advantage because generated code can run against tests. She says RL is difficult to scale because training requires several model copies, inference, and training loops operating together, and reward hacking remains a problem. Reflection AI is using autonomous coding as a starting point for broader software-engineering systems, where code generation is only one part of the end-to-end workflow.

### Reasoning data recipes can make small models strong with ordinary SFT
[1:06:25](https://www.youtube.com/watch?v=-9E9_21tx04&t=3985s)
Ryan Marten says DeepSeek R1's released final model was based on supervised fine-tuning over 800,000 examples, including 600,000 reasoning examples, which left the data recipe as an important missing detail. OpenThoughts 3 studies question sources, mixtures, filtering, teacher models, and answer filtering. Its experiments found that multiple reasoning traces per question work well, synthetic questions can outperform manually written or scraped questions, and question difficulty filtering can help. A stronger evaluation model is not automatically a better teacher. Marten reports that Qwen 32B was a better teacher than DeepSeek R1 for their recipe, and that a distilled 7B model surpassed R1 on a legal reasoning task.

### Reliable agents need realistic environments and carefully shaped rewards
[2:49:16](https://www.youtube.com/watch?v=-9E9_21tx04&t=10156s)
Kyle Corbitt describes ART-E, an email assistant that searches an inbox and answers questions. His first recommendation is to get the best prompted baseline working before training, so tool bugs and environment problems are separated from RL problems. The team used the public Enron email corpus to build realistic inboxes. Gemini 2.5 Pro generated questions and answers grounded in groups of emails, which gave the reward function a verified target. The trained model received extra reward for finding answers in fewer turns and for saying it did not know instead of hallucinating. Corbitt reports that the smaller trained model reached higher accuracy than prompted models while reducing cost, latency, and hallucinations.

### Future reasoning models need calibration, strategy, and abstraction
[3:09:18](https://www.youtube.com/watch?v=-9E9_21tx04&t=11358s)
Nathan Lambert says reasoning models already provide strong skills in math, code, and search, but the next problems involve planning. He separates this into calibration, strategy, and abstraction. Calibration would let a model match token use to task difficulty instead of spending hundreds of tokens on a simple question. Strategy means choosing a promising direction and changing course when it fails. Abstraction means deciding how to divide a hard task into tractable subtasks or when to call another model. Lambert argues that these behaviors will require deliberate data and training, much as backtracking and verification required focused work for earlier reasoning systems. He expects post-training compute to become a much larger part of model development.

### Verified self-improvement requires sandboxing, curriculum generation, and alignment checks
[3:38:12](https://www.youtube.com/watch?v=-9E9_21tx04&t=13092s)
Christian Szegedy argues that human-generated data, tasks, environments, and supervision limit AI self-improvement. He proposes agents that create their own curriculum from problems found on the web, then solve those problems in safe environments. Those environments need branching, snapshots, and rollback so agents can explore freely without damaging systems. Szegedy distinguishes a verifier, which checks whether a solution is correct, from a validator, which checks whether the problem was interpreted in line with human intent. His proposed system has generator, verifier, and validator agents connected to sandboxed compute. The goal is to produce artifacts, such as code, together with proofs or other independently checkable evidence of correctness.

## Notable quotes
- "The thing that makes an agent an agent is tools." (15:06)
- "You want to build a reward signal that actually captures what you care about where gaming it is more difficult than not gaming it." (17:06)
- "If we were able to successfully put a test or put AI to this test, then we could make the capabilities assertion about that AI that we don't currently have in the market right now." (33:19)
- "The next frontier for scaling is reinforcement learning." (56:50)
- "The important thing here is you want to be looking at your rollouts, not just blindly trusting the reward function." (3:04:49)

## Tools & references mentioned
- MCP
- GRPO
- DPO
- PO
- verifiers
- ARC Prize Foundation
- ARC AGI
- ARC AGI 3
- OpenAI
- Claude
- Gemini
- Reflection AI
- PaLM
- DeepSeek R1
- DeepSeek V3
- Qwen
- OpenThoughts
- Curator
- Evalchemy
- OpenPipe
- ART-E
- Enron
- AI2
- AlphaGo
- AlphaZero
- Agent57
- MuZero
- Richard Sutton
- Francois Chollet
- John McCarthy
- David Silver
- Morpheus Cloud

## Who should watch
- You are building an agent with tool calls and need to decide whether RL is justified after prompting and supervised fine-tuning.
- Your evaluation set is easy to game, or your agent must work across unfamiliar environments rather than answer known benchmark questions.
- You want to train a smaller reasoning model and need practical guidance on data filtering, teacher models, reward design, or evaluation.

## Related talks

- [A Taxonomy for Next-gen Reasoning](https://aietalks.com/talks/a-taxonomy-for-next-gen-reasoning) (Nathan Lambert, Allen Institute for AI (AI2) and Interconnects.ai, 19:21)
- [Reinforcement Learning for Agents](https://aietalks.com/talks/reinforcement-learning-for-agents) (Will Brown, Morgan Stanley, 18:17)
- [Training Agentic Reasoners](https://aietalks.com/talks/training-agentic-reasoners) (Will Brown, Prime Intellect, 19:17)
- [Special Topics in Kernels, RL, Reward Hacking in Agents](https://aietalks.com/talks/special-topics-in-kernels-rl-reward-hacking-in-agents) (Daniel Han, Unsloth, 2:20:21)
- [Scaling to Long Horizons](https://aietalks.com/talks/scaling-to-long-horizons) (Ross Taylor & Chengxi Taylor, General Reasoning, 18:07)