# Agent Reinforcement Fine-Tuning

Will Hang & Cathy Zhou, OpenAI | AI Engineer | 16:55

Source: https://www.youtube.com/watch?v=p1CmPZ2j6Lk
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/agent-reinforcement-fine-tuning
Published: 2025-12-09
Tags: evals, fine-tuning, reinforcement-learning, tool-use

## TL;DR
- Agent reinforcement fine-tuning changes model weights so an agent learns how to reason through tool calls in a specific environment.
- A baseline and production-matched data should come before RFT, since prompting and task changes may solve the problem with less effort.
- Good reward functions need clear success criteria, room for partial credit, and protections against reward hacking.

## Summary
Will Hang and Cathy Zhou explain how OpenAI uses reinforcement fine-tuning for agents that reason while calling tools. RFT lets a model explore different trajectories, receive a custom reward, and update its weights toward better behavior in a particular environment. The speakers recommend starting with production-matched training and evaluation data, measuring a baseline, then trying prompt and task changes before fine-tuning. Their examples cover code editing, codebase research, enterprise coding, and GPU kernel generation. Across these cases, RFT improved tool selection, parallel calls, accuracy, and latency. The hard engineering work is often outside the training loop: isolated execution environments, strict graders, realistic rewards, and checks for reward hacking. The talk is honest that RFT works best when success is unambiguous and additional sampling gives the model a chance to find better solutions.

## Key ideas
### An agent reasons while it interacts with tools
[00:13](https://www.youtube.com/watch?v=p1CmPZ2j6Lk&t=13s)
Will Hang defines an agent by its ability to interact with the outside world and complete a task without constant human intervention. A coding agent might use a terminal, code interpreter, or an entire codebase. Its tool calls are interleaved with reasoning traces in the same context window. OpenAI's Codex is given tools for end-to-end coding work, including writing unit tests and submitting large diffs. Some tools are terminal commands, while others are custom functions that invoke workflows.

### RFT comes after prompting and task changes
[01:45](https://www.youtube.com/watch?v=p1CmPZ2j6Lk&t=105s)
The speakers put agent optimization in an order. Prompting steers behavior toward a preferred style. Task optimization can simplify the task, add guardrails, change the available tools, or alter tool behavior. Fine-tuning comes later, when those changes have not produced enough performance. Agent RFT changes model weights according to a learning signal that defines good and bad behavior. During training, the model explores different ways to call tools and solve the task.

### Public tools and custom rewards bring the environment into training
[02:53](https://www.youtube.com/watch?v=p1CmPZ2j6Lk&t=173s)
OpenAI's newer RFT features let the model call tools through endpoints on the public internet. After each rollout, OpenAI can also invoke a custom reward signal hosted through an endpoint. The speakers describe this as the first time OpenAI has allowed models to interact with the outside world during training. They say teams have succeeded with as few as 10 examples, and that RFT can produce lower-latency agents that work better on a particular task.

### Exploration adapts an agent to domain differences and tool budgets
[03:55](https://www.youtube.com/watch?v=p1CmPZ2j6Lk&t=235s)
A model trained in one environment may behave poorly in a company's environment, a problem the speakers call domain shift. It may call tools too often or send incorrect inputs. RFT lets the model explore possible interactions and hill climb on its reward, so it learns how to use tools and reason over their outputs. A team can also penalize rollouts that exceed a tool-call budget. The model can then learn to stay within that budget while preserving or improving performance. Each rollout receives an identifier that ties its tool calls and final answer together for grading.

### A baseline and production-matched data should come first
[05:41](https://www.youtube.com/watch?v=p1CmPZ2j6Lk&t=341s)
Will Hang recommends matching both training and evaluation data to production traffic before starting RFT. Teams should run the base model on those datasets to establish a baseline, then try prompt and task optimization. RFT is for pushing further after those approaches have reached their limits. This workflow also makes it easier to tell whether a fine-tuned model improved on the behavior that matters in production.

### Cognition used file-selection rewards and parallel tool calls
[06:54](https://www.youtube.com/watch?v=p1CmPZ2j6Lk&t=414s)
Cognition applied RFT to the code-edit planning stage of Devin. The agent inspected a repository and used shell tools and file reads to choose which files needed editing. Cognition rewarded the selection with F1 score, which balances precision and recall. They ran each trajectory in an isolated virtual machine. A dataset of about 100 examples produced a five-point improvement, while scaling to 1,000 examples produced a 10-point improvement. RFT also changed the call pattern from eight to 10 alternating steps to many calls launched in parallel at the first step, reducing the sequence to four steps.

### Strict grading can reduce long tool-call tails
[08:53](https://www.youtube.com/watch?v=p1CmPZ2j6Lk&t=533s)
Codto trained a deep research agent to answer questions over large codebases using search and retrieval tools. Its dataset contained about 1,000 authentic question-and-answer pairs from eight repositories, and the reward measured how many relevant facts the agent retrieved. RFT improved the agent by 6% while reducing tool calls and output tokens. Before training, some samples used more than 15 tool calls. Afterward, those long-tail runs disappeared and the distribution centered around two to four calls. The speakers connect this change to more predictable production latency.

### Reward design must prevent style optimization and kernel exploits
[10:33](https://www.youtube.com/watch?v=p1CmPZ2j6Lk&t=633s)
Cosine trained an enterprise coding agent with 30 tools and found that partial rewards encouraged the model to optimize coding style and tone. They instead rewarded it when the final code passed tests, then increased batch size and compute to find positive samples despite sparse rewards. The grader also rewarded self-validation through tests, terminal output inspection, and linting. Macco faced a different problem with GPU kernels. The model returned reference code, no kernels, or identity kernels to gain reward. A judge model and static analysis checks blocked these cases before scoring correctness and real speedup.

### Successful RFT tasks need clear rewards and useful exploration
[14:46](https://www.youtube.com/watch?v=p1CmPZ2j6Lk&t=886s)
The speakers give four principles. The task must have an unambiguous definition of success with no subjective taste involved. Training and evaluation data should mirror production traffic. More samples for a data point should create a chance of finding a better result, with enough variation for the model to learn from good and bad rollouts. Finally, the reward must resist hacking and should be more continuous than binary when possible. Partial credit lets the model move gradually toward better performance.

## Notable quotes
- "What makes an agent different from a regular model is its ability to interact with the outside world to complete a task to get things done on its own without having to go through you all the time." (00:37)
- "You first want to make sure that your training data set and your eval data set closely match your production traffic." (05:41)
- "You want to make sure that through exploration, the model actually achieves better performance on a given data point if it samples more so that it can learn from itself." (15:32)
- "You want to make sure that your reward function is not hackable." (15:49)

## Tools & references mentioned
- OpenAI
- Codex
- Cognition
- Devin
- Codto
- Cosine
- Macco
- Nvidia B200
- PyTorch

## Who should watch
- You are building a multi-step agent that calls tools in a business environment and need to decide whether fine-tuning is warranted.
- Your agent is accurate on average but sometimes makes very long tool-call trajectories or exceeds a latency budget.
- You need to design graders and rewards for code generation, research, or other tasks where a model can exploit the scoring function.

## Editor's note

From the pack [Post-training and RL](https://aietalks.com/packs/post-training-and-rl):

Corbitt and both Brown talks return to the same warning: a reward curve cannot tell you how the agent earned it. You need the rollouts, especially when the model finds a loophole. Kitaru records agent runs so you can inspect and replay the behavior behind a score, then turn the failure into a better environment or evaluator.

Written by the AIE Talks editors (the Kitaru team), not by the speaker.

## Related talks

- [RFT, DPO, SFT: Fine-tuning with OpenAI](https://aietalks.com/talks/rft-dpo-sft-fine-tuning-with-openai) (Ilan Bigio, OpenAI, 1:46:15)
- [How to Train Your Agent: Building Reliable Agents with RL](https://aietalks.com/talks/how-to-train-your-agent-building-reliable-agents-with-rl) (Kyle Corbitt, OpenPipe, 19:48)
- [Training Agentic Reasoners](https://aietalks.com/talks/training-agentic-reasoners) (Will Brown, Prime Intellect, 19:17)
- [Reasoning + RL](https://aietalks.com/talks/reasoning-rl) (Will Brown & Greg Kamradt, ARC Prize Foundation & Aakanksha Chowdhery, Reflection AI & Ryan Marten, Bespoke Labs & Kyle Corbitt, OpenPipe & Nathan Lambert, AI2 & Christian Szegedy, Former co-founder of xAI, 3:54:58)
- [Reinforcement Learning for Agents](https://aietalks.com/talks/reinforcement-learning-for-agents) (Will Brown, Morgan Stanley, 18:17)