Agent reinforcement fine-tuning changes model weights so an agent learns how to reason through tool calls in a specific environment.
2
A baseline and production-matched data should come before RFT, since prompting and task changes may solve the problem with less effort.
3
Good reward functions need clear success criteria, room for partial credit, and protections against reward hacking.
Summary
Will Hang and Cathy Zhou explain how OpenAI uses reinforcement fine-tuning for agents that reason while calling tools. RFT lets a model explore different trajectories, receive a custom reward, and update its weights toward better behavior in a particular environment. The speakers recommend starting with production-matched training and evaluation data, measuring a baseline, then trying prompt and task changes before fine-tuning. Their examples cover code editing, codebase research, enterprise coding, and GPU kernel generation. Across these cases, RFT improved tool selection, parallel calls, accuracy, and latency. The hard engineering work is often outside the training loop: isolated execution environments, strict graders, realistic rewards, and checks for reward hacking. The talk is honest that RFT works best when success is unambiguous and additional sampling gives the model a chance to find better solutions.
Will Hang defines an agent by its ability to interact with the outside world and complete a task without constant human intervention. A coding agent might use a terminal, code interpreter, or an entire codebase. Its tool calls are interleaved with reasoning traces in the same context window. OpenAI's Codex is given tools for end-to-end coding work, including writing unit tests and submitting large diffs. Some tools are terminal commands, while others are custom functions that invoke workflows.
The speakers put agent optimization in an order. Prompting steers behavior toward a preferred style. Task optimization can simplify the task, add guardrails, change the available tools, or alter tool behavior. Fine-tuning comes later, when those changes have not produced enough performance. Agent RFT changes model weights according to a learning signal that defines good and bad behavior. During training, the model explores different ways to call tools and solve the task.
Public tools and custom rewards bring the environment into training
OpenAI's newer RFT features let the model call tools through endpoints on the public internet. After each rollout, OpenAI can also invoke a custom reward signal hosted through an endpoint. The speakers describe this as the first time OpenAI has allowed models to interact with the outside world during training. They say teams have succeeded with as few as 10 examples, and that RFT can produce lower-latency agents that work better on a particular task.
Exploration adapts an agent to domain differences and tool budgets
A model trained in one environment may behave poorly in a company's environment, a problem the speakers call domain shift. It may call tools too often or send incorrect inputs. RFT lets the model explore possible interactions and hill climb on its reward, so it learns how to use tools and reason over their outputs. A team can also penalize rollouts that exceed a tool-call budget. The model can then learn to stay within that budget while preserving or improving performance. Each rollout receives an identifier that ties its tool calls and final answer together for grading.
A baseline and production-matched data should come first
Will Hang recommends matching both training and evaluation data to production traffic before starting RFT. Teams should run the base model on those datasets to establish a baseline, then try prompt and task optimization. RFT is for pushing further after those approaches have reached their limits. This workflow also makes it easier to tell whether a fine-tuned model improved on the behavior that matters in production.
Cognition used file-selection rewards and parallel tool calls
Cognition applied RFT to the code-edit planning stage of Devin. The agent inspected a repository and used shell tools and file reads to choose which files needed editing. Cognition rewarded the selection with F1 score, which balances precision and recall. They ran each trajectory in an isolated virtual machine. A dataset of about 100 examples produced a five-point improvement, while scaling to 1,000 examples produced a 10-point improvement. RFT also changed the call pattern from eight to 10 alternating steps to many calls launched in parallel at the first step, reducing the sequence to four steps.
Codto trained a deep research agent to answer questions over large codebases using search and retrieval tools. Its dataset contained about 1,000 authentic question-and-answer pairs from eight repositories, and the reward measured how many relevant facts the agent retrieved. RFT improved the agent by 6% while reducing tool calls and output tokens. Before training, some samples used more than 15 tool calls. Afterward, those long-tail runs disappeared and the distribution centered around two to four calls. The speakers connect this change to more predictable production latency.
Reward design must prevent style optimization and kernel exploits
Cosine trained an enterprise coding agent with 30 tools and found that partial rewards encouraged the model to optimize coding style and tone. They instead rewarded it when the final code passed tests, then increased batch size and compute to find positive samples despite sparse rewards. The grader also rewarded self-validation through tests, terminal output inspection, and linting. Macco faced a different problem with GPU kernels. The model returned reference code, no kernels, or identity kernels to gain reward. A judge model and static analysis checks blocked these cases before scoring correctness and real speedup.
Successful RFT tasks need clear rewards and useful exploration
The speakers give four principles. The task must have an unambiguous definition of success with no subjective taste involved. Training and evaluation data should mirror production traffic. More samples for a data point should create a chance of finding a better result, with enough variation for the model to learn from good and bad rollouts. Finally, the reward must resist hacking and should be more continuous than binary when possible. Partial credit lets the model move gradually toward better performance.
"What makes an agent different from a regular model is its ability to interact with the outside world to complete a task to get things done on its own without having to go through you all the time."00:37
Who should watch
You are building a multi-step agent that calls tools in a business environment and need to decide whether fine-tuning is warranted.
Your agent is accurate on average but sometimes makes very long tool-call trajectories or exceeds a latency budget.
You need to design graders and rewards for code generation, research, or other tasks where a model can exploit the scoring function.
