Long-horizon AI needs better algorithms, environments, and compute management, rather than a larger context window alone.
2
Value models reduce variance and provide earlier learning signals, but they add bias and the cost of training another model.
3
Current agents struggle with open-ended, uncertain tasks because many benchmarks do not simulate other actors or realistic environments.
Summary
Ross Taylor opens with Galactica, arguing that curated data, repeated training, and internal thinking tokens anticipated later work on reasoning. His account of early Llama 2 experiments shows that reinforcement learning with verifiable rewards produced strong math results, but better base models, more compute, and larger context windows were needed before reflective behavior emerged. Chengxi Taylor frames long-horizon work as a mindset as well as an engineering problem. Long tasks create gradient variance, sparse rewards, credit-assignment problems, and variable trajectory lengths. Value models, bootstrapping, compaction, and external memory tools can help. A football-trading benchmark exposed the limits of current systems: frontier models were given real money and lost it. Chengxi argues that agents need open-ended environments with uncertainty, competing actors, and better simulation. Long rollouts also force a choice between GPU utilization and off-policy staleness. General Reasoning is building environments through openreward.ai.
Galactica showed that a strong base model still needs post-training
Ross Taylor contrasts Galactica with ChatGPT, which arrived about two weeks later. Galactica was released as a base-model demo, so users encountered hallucinations and strange behavior. ChatGPT had reinforcement learning from human feedback, which Taylor says helped language models become useful products. He also points to InstructGPT, where a 1-billion-parameter model with RLHF outperformed a 175-billion-parameter model. Galactica's reaction overshadowed much of its research, although Taylor says it outperformed PaLM, Chinchilla, and GPT-3.5 in scientific domains with less compute. The lesson he takes from the episode is that a good base model is insufficient without the right post-training objective.
Curated data and internal thinking tokens were early routes to better reasoning
Taylor says Galactica used a 105-billion-token corpus rather than the trillions of tokens used in Chinchilla, and that curated, high-quality data drove its results. The project also provided an early empirical case for multi-epoch training, at a time when the common view was to avoid training beyond one epoch. More importantly, Galactica treated intermediate reasoning as internal working memory inside special tags. Taylor distinguishes this from prompting for a visible chain of thought or using a numerical scratchpad. The goal was to spend inference compute on the work between the question and the answer. He later became interested in applying reinforcement-learning pressure directly to those thinking tokens.
Early Llama 2 reasoning experiments had the objective but lacked the conditions for reflection
Taylor describes an unpublished Meta recipe that continued pretraining Llama 2 on mathematics and science data, then used PPO with verifiable rewards. The setup included a strong outcome reward model to initialize the value model. He says it reached state-of-the-art math and reasoning results internally. It did not produce the inference-time scaling and reflective behaviors later associated with R1 and O1, such as backtracking. Taylor's explanation is that the base models were weaker, the math corpus was poor, and the context window was only 4,000 tokens. Better base models, more reinforcement-learning compute, and larger context windows eventually made the behavior emerge.
Long-horizon work requires patience because useful reasoning can exceed the context window
Chengxi Taylor says long-horizon tasks are a mindset because problems such as proving major theorems can involve years of reading, writing, and reflection. He uses Fermat's Last Theorem to illustrate how a mathematician's work could translate into tens or hundreds of billions of tokens, far beyond a 1-million-token context window. One response is compaction: generate until the context fills, summarize the work, and continue from the summary. Reinforcement learning can train both the compaction behavior and the task itself. Taylor also describes file-system tools as scratchpads, self-search tools for revisiting earlier trajectories, and archive tools for building on previous results. These tools can create shortcuts, so agents must not simply retrieve an answer without doing the required reasoning.
Value models make sparse, variable-length long tasks easier to train
Long trajectories create several optimization problems, according to Chengxi Taylor. Gradient variance grows with sequence length, rewards may arrive only at the end, credit assignment becomes difficult, and trajectories can have different lengths. Critics, or value models, reduce variance and can provide trajectory-level signals that fit compaction. Taylor also says they can encourage batch diversity and support bootstrapping before an episode ends. The cost is added complexity compared with GRPO because a second model must be trained alongside the policy. The value model can also introduce bias, so it is a useful approximation with a real trade-off rather than a complete solution.
A football-trading benchmark exposed the gap between coding skill and open-ended reasoning
General Reasoning built a benchmark in which agents created machine-learning models to trade football matches over a one-year Premier League horizon. The setup used real money, with each frontier model receiving 100,000 to start. Chengxi Taylor says all of them lost money. He connects this result to the industry's focus on coding and procedural tasks, where the solution space is often tightly specified. Real-world work contains uncertainty, competing actors, and many possible strategies. Current benchmarks do not capture enough of that complexity, and they provide too little simulation of environments where other participants have their own goals and beliefs.
Pipeline reinforcement learning trades stale policies for better GPU use
Long inference runs make infrastructure choices part of the learning problem. In a traditional setup, training waits until inference has produced the full sequence. Pipeline RL starts training while more sequences are still being generated, which keeps GPUs busier but makes the data increasingly off-policy. Chengxi Taylor says their experience suggested that up to eight steps of off-policy behavior was acceptable. A rollout that takes weeks can exceed that limit, leaving GPUs idle if training waits for fresh data. A value model allows bootstrapping before the episode ends, so training can continue earlier. That improves utilization while adding value-model bias. Long-horizon systems therefore need an explicit balance between policy freshness and hardware use.
Long-horizon progress depends on environments as much as algorithms
Taylor closes by grouping the remaining work into algorithms, environments, and compute. He says environment infrastructure matters especially for long-horizon reinforcement learning, and points to openreward.ai, General Reasoning's platform with more than 350 environments behind a single API endpoint. The company uses it internally, and Taylor says frontier and newer labs also use it. His broader claim is that realistic environments must include open-ended goals, uncertainty, and other agents with their own behavior. Larger context windows do not solve those missing pieces. The talk treats long-horizon learning as a set of practical trade-offs, including memory, credit assignment, simulation quality, rollout length, and hardware scheduling.
"So, essentially, we made a trade-off between the off-policy and the GPU utilization."15:22
Who should watch
You are designing reinforcement-learning systems for agents that need to work across long sequences instead of completing short benchmark tasks.
Your rollouts are long enough that sparse rewards, stale policies, idle GPUs, or context limits are already affecting training.
You want a concrete account of why current agents can perform coding procedures yet fail in open-ended environments with uncertainty and competing actors.
