Reinforcement learning environments give language models a dynamic task, actions, and verifiable rewards, so they can improve through trial and error.
2
Verifiers packages environments as reusable Python software for evaluation and training, including single-turn, multi-turn, and tool-using setups.
3
A carefully designed Tic-Tac-Toe environment turned a weak small model into a player that drew 85% of its games against an optimal opponent and later beat GPT-5 Mini in that test.
Summary
Stefano Fiorucci explains how reinforcement learning environments translate classic agent-environment interaction into language-model training. An environment supplies the task data, interaction loop, tools, and scoring rules. The model explores possible trajectories, while verifiable rewards favor outcomes such as correct answers, won games, or successful tool calls. Fiorucci introduces Verifiers, an open-source library from Prime Intellect that packages these environments as Python artifacts and handles model serving, rollouts, and training integrations. His main example is Tic-Tac-Toe. He improves a small model with supervised fine-tuning, then uses GRPO-style reinforcement learning, controlled opponent difficulty, deterministic randomness, and stratified sampling. The model learns to avoid invalid moves, dominate random opponents, and draw 85% of games against an optimal opponent. Fiorucci is candid about failures caused by small batch sizes, biased opponents, excessive reasoning length, and premature training changes. The talk is a practical guide to building and debugging environments rather than treating RL as a black box.
Verifiable rewards let models learn from outcomes instead of imitating examples
Fiorucci contrasts supervised fine-tuning with reinforcement learning using verifiable rewards. In supervised fine-tuning, a model imitates curated prompt-and-response examples, so its completions tend to remain close to the example distribution. With reinforcement learning, the model can generate different reasoning traces and answers, then receive a reward when the outcome is correct. The same pattern applies beyond question answering. A correct answer, a won game, or a successful tool call can be checked automatically and used as a training signal. This gives the model room to discover strategies that were not explicitly written into human examples.
An LLM environment contains the task machinery around the model
When Fiorucci maps classic reinforcement learning to language models, the language model becomes the agent. The environment includes the data, harnesses, and scoring rules needed to check and possibly train the model. In Tic-Tac-Toe, the model produces a move, while the environment prompts it, tracks the board, generates the opponent's move, checks whether the game has ended, and returns a reward. Tools make the environment more involved. A weather API or terminal can become part of the model's action space, with the environment executing calls and returning their results.
Verifiers turns reinforcement learning environments into reusable Python packages
Verifiers, an open-source library by Prime Intellect, provides modular components for building environments as installable Python packages. It supports single-turn tasks, multi-turn interactions, and tool environments. Developers define response parsers and reward functions while the library handles model serving through an OpenAI-compatible API, parallel trajectories, and result collection. It also includes a trainer and integrations with Prime RL, Tinker, and Sky RL. Fiorucci's point is practical: the library lets engineers spend their time defining the task and its rewards instead of rebuilding rollout infrastructure for every experiment.
Multi-turn and tool environments are loops over shared state
Fiorucci shows how a multi-turn environment adds a state dictionary and a stopping condition to the basic interaction pattern. In the double-check example, the model answers a math question and the environment responds with 'Are you sure?' The loop continues until the stopping method returns true. Tool environments build on the same foundation. Tools are Python functions that the model can call, after which the environment executes them and returns results. Verifiers also includes support for MCP servers, persistent per-rollout state, recursive language models, and shared environments through the environments hub.
Reward design and opponent control determine what a game model can learn
Fiorucci begins Tic-Tac-Toe with a simple setup: the model plays X, moves first, and faces a random opponent. The environment checks move syntax, applies moves, and assigns winner and format rewards. He then makes the task harder by allowing the model to play first or second and by mixing random and minimax opponent behavior. A probability range controls opponent difficulty, since a perfect opponent can prevent the model from seeing wins early in training. Invalid moves also receive a flat minus 0.1 penalty instead of ending the game immediately, giving smaller models more chances to learn.
Deterministic randomness and stratified batches reduce misleading training signals
For group-based reinforcement learning, Fiorucci wants reward differences to reflect the model's choices rather than unrelated environment randomness. He assigns each dataset example a seed for selecting the starting player, then derives turn seeds from the example seed and board state. If two rollouts reach the same board, the opponent responds the same way. He also uses stratified sampling so each training batch contains a balanced mix of opponent difficulties. Without that balance, a small batch might contain mostly easy or mostly hard games, causing the average reward to fluctuate and making training unstable.
A warm-up with synthetic supervised data prepares a weak model for RL
Fiorucci evaluates GPT-5 Mini and Liquid AI's LFM-2 in the Tic-Tac-Toe environment. LFM-2 struggles with the required format and often makes invalid moves, so he first uses supervised fine-tuning to teach the syntax and valid move behavior. GPT-5 Mini generates synthetic games, and Fiorucci creates 200 examples while filtering out losing games. The fine-tuned model follows the format almost perfectly, makes fewer invalid moves, and plays better. Reinforcement learning then has a model that can participate in the environment, leaving RL to improve game strategy rather than spending all its effort on basic output formatting.
After an initial successful run, Fiorucci raises opponent skill and increases sampling temperature to push the model beyond strategies it has already learned. The change causes a sharp early drop in winning and total reward, which he interprets as exploration. The model tries random or unfamiliar strategies before recovering and reaching higher scores. Temperature is risky because excessive values can produce gibberish, but too little exploration can leave the model stuck. In the final evaluation, the trained model becomes a Tic-Tac-Toe master and performs better than GPT-5 Mini against an optimal opponent.
Environment bugs and training choices can create false confidence
Fiorucci describes several failure modes from the project. Small batch sizes led to unstable training and model collapse because the model learned from too few games and opponent types at once. A minimax implementation that always selected the first equally good move created a hidden opponent bias, allowing the model to memorize one predictable player while appearing strong on benchmarks. He also warns that starting with a reasoning model may waste limited GPU capacity on long traces that get truncated. Engineers should inspect rollouts, test the model in the real task after programmatic evaluation, watch logs for instability, and avoid changing a slow run prematurely.
"For this to work, differences in rewards should come from how the model plays, not from environment randomness."23:55
Who should watch
You are building post-training systems for language models and need environments that can run evaluation and training with the same task logic.
Your model can produce plausible text but struggles with multi-step interaction, tool calls, game state, or outcomes that can be checked automatically.
You want a practical example of debugging RL training, including opponent design, reward shaping, batch size, randomness, and exploration.
