# Reinforcement Learning for Agents

Will Brown, Morgan Stanley | AI Engineer Summit 2025 | 18:17

Source: https://www.youtube.com/watch?v=JIsgyk0Paic
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/reinforcement-learning-for-agents
Published: 2025-03-07
Tags: agents, evals, reinforcement-learning, tool-use

## TL;DR
- Reinforcement learning gives agents a way to improve through repeated interaction with an environment, using rewards to reinforce successful strategies.
- Reasoning behaviors such as long chains of thought can emerge from reinforcement learning when models are rewarded for solving problems, rather than being shown manually written reasoning traces.
- Building agentic reinforcement learning systems will require environments, rubrics, verification, monitoring, and infrastructure that can support multi-step tool use.

## Summary
Will Brown presents reinforcement learning as a possible next step for agent engineering. Current systems often combine chatbot or reasoning models with prompts, tools, evaluations, and hand-built workflows. These systems can work well, but they usually have limited autonomy. Brown explains how reinforcement learning lets a model try actions, receive scores, and repeat strategies that lead to better results. He connects this process to DeepSeek R1 and OpenAI's reasoning models, where longer reasoning behavior emerged through training rather than being explicitly written into the data. He then discusses the harder problem of applying the same approach to agents that use tools across multiple steps. His examples include a small Llama model trained on math problems and an open-source framework for reinforcement learning inside multi-step environments. Brown is careful about the limits. Rewards can be hacked, generalization is uncertain, and API models may not be enough for every task. AI engineers may need to build environments and rubrics alongside prompts and evaluations.

## Key ideas
### Most current agents are engineered pipelines with limited autonomy
[01:42](https://www.youtube.com/watch?v=JIsgyk0Paic&t=102s)
Brown places current language-model systems between chatbots, reasoners, and more autonomous agents. Models such as o1, o3, R1, Grok 3, and Gemini can handle questions and interactive problem solving, while engineers build longer tasks by chaining model calls. Prompt engineering, tool calling, evaluations, and human review fill in much of the behavior. He distinguishes these pipelines from agents that act for long periods with less supervision. Cursor, Windsurf, Replit, and search products usually have tight feedback loops. Devin, Operator, and OpenAI's deep research feel more autonomous because they can work for longer, though they still do not handle every kind of task.

### Reinforcement learning gives a system a path beyond manual prompt tuning
[03:58](https://www.youtube.com/watch?v=JIsgyk0Paic&t=238s)
Brown describes reinforcement learning as an agent interacting with an environment while pursuing a goal and improving through repeated attempts. This addresses a common engineering situation: a system works at roughly its current level, perhaps after extensive prompt tuning, but the team has no clear way to raise its performance. Reinforcement learning supplies that path through trial, feedback, and repetition. The model tries actions, receives a numerical signal, and learns to repeat strategies that produced better outcomes. Brown says this may help agents acquire skills instead of relying only on better general-purpose models.

### Reasoning behavior can emerge when models are rewarded for solving problems
[04:45](https://www.youtube.com/watch?v=JIsgyk0Paic&t=285s)
Brown connects recent reasoning models to reinforcement learning. He says DeepSeek's R1 paper helped explain how systems like o1 can learn to reason: give the model questions, check whether the answers are correct, and reinforce behavior associated with successful solutions. Long chains of thought were not manually inserted as thousands of-token examples. They emerged because extended reasoning became a useful strategy. Brown describes reinforcement learning as a process for finding good ways to solve problems. Synthetic data becomes more useful when verification, rejection sampling, or another feedback mechanism tells the training process which attempts worked.

### Exploration and exploitation turn rollouts into training feedback
[07:02](https://www.youtube.com/watch?v=JIsgyk0Paic&t=422s)
The basic loop is to try different approaches, keep more of what works, and use less of what fails. Brown illustrates this with code generation, where a model can receive rewards for formatting, using the requested language, and passing test cases. Instead of curating every training example in advance, the system creates rollouts and scores them. He explains GRPO as a simple version of this idea: for one prompt, sample several completions, score them, and push the model toward the higher-scoring ones. The algorithm is useful as a clear mental model, even though Brown does not present it as a major algorithmic breakthrough.

### Agentic reinforcement learning is harder because tasks span tools and steps
[08:08](https://www.youtube.com/watch?v=JIsgyk0Paic&t=488s)
Brown points to OpenAI's deep research as evidence that end-to-end reinforcement learning can support a more autonomous system. The system may make many browsing or internet-query tool calls before producing a report. Brown also gives limits from observed behavior: it is not a general software-engineering agent, and it can struggle with out-of-distribution work or repetitive manual calculations. Reinforcement learning can teach particular skills and increase autonomy when combined with environments, tools, and verification. It has not produced a system that can solve every type of problem.

### Rubrics can train behavior, but poorly designed rewards invite cheating
[10:42](https://www.youtube.com/watch?v=JIsgyk0Paic&t=642s)
Brown calls his work on reward design 'rubric engineering'. A reward can check the final answer, but it can also give partial credit for following an XML structure or producing an integer in the requested format, even when the integer is wrong. These intermediate signals can help a model learn the behavior needed to reach the goal. Brown suggests experimenting with language models that design rubrics, automatic rubric tuning, DSPy-based prompt tuning, and language-model judges. He warns about reward hacking: a model may find a shortcut that produces a high score without performing the intended task.

### Multi-step reinforcement learning needs reusable environments and protocols
[14:52](https://www.youtube.com/watch?v=JIsgyk0Paic&t=892s)
Brown describes an open-source framework for running reinforcement learning inside multi-step environments. The aim is to reuse agent frameworks that already connect models to tools and APIs. Engineers define an environment and an interaction protocol, while the trainer handles the model interaction. Rewards then let the model improve through repeated runs. Brown presents this as research code rather than a finished solution. The design separates environment construction from concerns such as model weights and token handling, so existing agent systems may become training environments instead of remaining fixed application workflows.

### AI engineers may need to build skills through trial and error
[15:50](https://www.youtube.com/watch?v=JIsgyk0Paic&t=950s)
Brown argues that prompts can provide knowledge, but they have difficulty including a learned skill. People often need several attempts before they can perform a task well, and he says models appear to improve in a similar way through trial and error. Fine-tuning may become more useful as open models approach closed models and as reinforcement learning becomes part of systems such as R1 and deep research. The engineering work carries over from today's practice: teams still need good evaluations, prompts, environments, rubrics, and monitoring. The difference is that these components may feed a training loop rather than only guide a fixed model.

## Notable quotes
- "Reinforcement learning at the core is really about identifying good strategies for solving problems." (06:30)
- "What was catching people's imagination was that it was one file of code." (12:31)
- "It is really hard to include a skill in a prompt." (15:50)
- "We still need good monitoring tools." (17:01)

## Tools & references mentioned
- Morgan Stanley
- Columbia University
- Christos Papadimitriou
- Tim Roughgarden
- OpenAI
- o1
- o3
- DeepSeek R1
- Grok 3
- Gemini
- Cursor
- Windsurf
- Replit
- Devin
- Operator
- OpenAI deep research
- Hugging Face
- Llama 1B
- GRPO
- DSPy
- AWS
- Two Sigma
- MongoDB
- AmFam

## Who should watch
- You are building agents from chained model calls and want a possible route beyond manual prompt tuning.
- Your team is considering reinforcement fine-tuning or multi-step tool use and needs to understand the open design questions around rewards and environments.
- You are building evaluation, prompt, or agent infrastructure and want to see how that work could carry into reinforcement learning systems.

## Related talks

- [Reasoning + RL](https://aietalks.com/talks/reasoning-rl) (Will Brown & Greg Kamradt, ARC Prize Foundation & Aakanksha Chowdhery, Reflection AI & Ryan Marten, Bespoke Labs & Kyle Corbitt, OpenPipe & Nathan Lambert, AI2 & Christian Szegedy, Former co-founder of xAI, 3:54:58)
- [Training Agentic Reasoners](https://aietalks.com/talks/training-agentic-reasoners) (Will Brown, Prime Intellect, 19:17)
- [Reinforcement Learning without Verifiable Rewards](https://aietalks.com/talks/reinforcement-learning-without-verifiable-rewards) (Will Brown, Prime Intellect, 19:27)
- [How to Train Your Agent: Building Reliable Agents with RL](https://aietalks.com/talks/how-to-train-your-agent-building-reliable-agents-with-rl) (Kyle Corbitt, OpenPipe, 19:48)
- [RL Environments at Scale](https://aietalks.com/talks/rl-environments-at-scale) (Will Brown, Prime Intellect, 18:30)