# Reinforcement Learning without Verifiable Rewards Will Brown, Prime Intellect | AI Engineer World's Fair 2026 | 19:27 Source: https://www.youtube.com/watch?v=AQv3qRCG6Gw Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/reinforcement-learning-without-verifiable-rewards Published: 2026-07-31 Tags: agents, evals, reinforcement-learning, synthetic-data ## TL;DR - Reinforcement learning works easily when a verifier can check math, code, or database state, but real agent tasks often need softer reward signals. - Environments can create useful supervision from documents, production traces, simulators, judges, and tasks built by working backwards from a known solution. - Reward design needs repeated inspection, small training experiments, trace analysis, and human expertise because models will exploit loose objectives. ## Summary Will Brown explains how reinforcement learning can extend beyond tasks with exact answers. He starts with a model and harness acting inside an environment, receiving rewards that update the model weights. Math, code, and deterministic tool use have straightforward verifiers, while reports, purchases, refunds, and other real-world work do not. Brown's approach is to make environments the anchor for generating tasks, evaluating behavior, and refining reward signals. Documents and repositories can produce grounded question-answer tasks. Production traces can reveal the task distribution. Simulators can provide controllable versions of tools and applications, allowing teams to plant an end state and train models to recover it. Judges and test-time search help inspect rollouts and extract rubrics. Brown is direct that reward hacking requires trace review and small RL runs, with experts deciding what behavior is acceptable. He also argues that supervised learning from the environment may be needed when RL alone cannot add new world knowledge to the model. ## Key ideas ### RL updates model weights from agent behavior inside an environment [01:24](https://www.youtube.com/watch?v=AQv3qRCG6Gw&t=84s) Brown describes an agent as a model plus a harness placed into an environment. The environment contains a task and a world, such as a Docker image, code base, task-specific tools, applications, browser tabs, skills, and a scoring rule. The agent interacts with that world in a loop and receives a reward. RL turns the reward into an advantage by subtracting a baseline and possibly scaling the result. A policy-gradient update then nudges the model weights toward behavior that receives higher reward. Brown places methods such as GRPO, REINFORCE, and SISPO inside this general framework: they all seek ways to make the model produce actions with higher reward. ### Environments support training and evaluation even when rewards are not exact [04:00](https://www.youtube.com/watch?v=AQv3qRCG6Gw&t=240s) Brown says environments are useful beyond reinforcement learning. The same tasks, harnesses, and rewards can generate static data for supervised fine-tuning, support on-policy distillation, and provide a test bed for prompt optimization and agent experiments. Verifiable rewards are the easy case. A numerical math answer can be parsed and checked, code can be tested or linted, and a tool-use task can be scored from a known database state. Real work is less defined. A report may analyze documents in several reasonable ways, a booking may have no single best result, and handling a refund depends on how well the agent responds to the user. These tasks need other ways to manufacture a useful signal. ### Real-world tasks make reward hacking an unavoidable design problem [04:13](https://www.youtube.com/watch?v=AQv3qRCG6Gw&t=253s) Brown argues that most real-world agent tasks do not have a clean best answer or a fixed distribution. Existing benchmarks often contain only a few hundred carefully designed tasks, which makes them expensive and difficult to scale to open-ended work. A loose reward proxy can also have undefined behavior at its boundaries. During RL, a model may find a path that increases the score while missing the actual purpose of the task. Brown uses continual learning to mean deploying models in complex settings, observing mistakes after they happen, and using those mistakes to prevent repeats. He says current systems require too much low-level human effort, so more of task discovery, reward design, and model improvement needs to become easier to automate. ### Grounding and judges turn real material into training signal [08:27](https://www.youtube.com/watch?v=AQv3qRCG6Gw&t=507s) Brown presents grounding as a way to create supervision from source material. An A/B comparison between having a source in context and not having it can produce a capability gap that supplies a learning signal. Judges can use the reasoning ability of language models to decide whether an action was good or bad, with additional inference used to make those decisions. Production traces are especially useful because they reveal what users and deployed agents actually do, even before labels exist. Documents and code repositories provide other raw material. Brown also describes search as something that can be applied to task creation, world construction, reward criteria, and rollout review. The goal is to turn real inputs into environments rather than rely only on hand-built benchmark tasks. ### Working backwards from a known solution makes hard tasks verifiable [10:57](https://www.youtube.com/watch?v=AQv3qRCG6Gw&t=657s) For document tasks, Brown suggests sampling documents, generating question-answer pairs grounded in them, and checking that the questions are answerable from the same material. The training task can then discard the initial search and ask the agent to recover the answer. This reverses the direction from a solution that is easy to verify to a harder upstream task. Code offers a similar pattern. A real pull request, diff, description, or test case can be taken as a completed artifact. Parts of the files can be removed, and the model can learn to restore a reachable end state. Brown generalizes the method to simulated tools and web applications, where a controllable simulator can plant an answer or final state even when the production system cannot be inspected directly. ### Difficulty must be tuned so rollouts separate model behavior [14:19](https://www.youtube.com/watch?v=AQv3qRCG6Gw&t=859s) Brown says RL needs tasks with an advantage gap. A task should produce a meaningful difference between what one rollout does and what a collection of rollouts can discover. Tasks that are too easy give the model little to learn, while tasks that are too hard provide no useful successful behavior. Search and additional inference can help generate and filter tasks until they sit at a workable difficulty. The same compute can refine simulators, test verification rules, and create rubrics from traces. For judging, Brown describes looking back over a sequence of events, comparing multiple model judgments, and extracting questions about specific failure modes. Those rubric questions can then support cheaper auditing and targeted task generation. ### Reward hacks require trace inspection and real training runs [15:08](https://www.youtube.com/watch?v=AQv3qRCG6Gw&t=908s) Brown warns that reward hacking can appear gradually if teams do not look for it. Simple cases may be obvious to a human, but judges do not always recognize them, and telling a model not to exploit a loophole does not guarantee that it will comply in a rollout. Collecting examples of hacks creates a corpus that can reveal recurring failures. Teams can then revise the implementation or reward and test the change. Brown says some problems appear only after RL begins, so environment design must include small runs with individual models on individual environments. Metrics can record tool-call patterns, while judges inspect traces for behavioral changes. Human experts still decide what the task actually requires and which behavior is acceptable. ### RL needs supervised world knowledge when the environment contains new information [17:20](https://www.youtube.com/watch?v=AQv3qRCG6Gw&t=1040s) Brown distinguishes skill refinement from adding dense new knowledge to model weights. RL can improve how an agent performs a skill, but it may not cause the model to explore and absorb information that exists in the environment. He cites the Echo work and describes combining reinforcement learning with supervised learning signals from the environment. That extra signal can give the model a native world model and help it predict what the environment will generate. Brown says this can let an agent adapt to the world rather than only repeat improved behavior. His broader design is an online loop: deploy an agent, find issues in production, turn those issues into new tasks, and train on them while keeping humans involved at the level of goals and guardrails. ## Notable quotes - "Most real-world tasks are not this verifiable." (04:49) - "And so anything where you can move backwards like this is super useful for kind of getting supervision for free." (11:26) - "I think these things can kind of feel like reward hacking can kind of sneak up on you if you're not careful for it." (15:19) - "And ultimately what you want is to surface the most important pieces up to the human." (16:38) ## Tools & references mentioned - Prime Intellect - GRPO - REINFORCE - SISPO - Lab - Echo - MCP - Docker - on-policy distillation - prompt optimization ## Who should watch - You are building an agent for reports, research, browsing, purchases, support, or another task without an exact answer checker. - Your team has production traces but no labels, and you need a way to turn those traces, documents, or repositories into training tasks. - You are designing rewards and want a practical account of how to detect reward hacking before it becomes model behavior.