Scaling model size produced broad improvements and emergent reasoning abilities, but pre-training gains are becoming harder to find.
2
Inference-time scaling works best when outputs can be checked, as with mathematics and code, although majority voting may require too many samples.
3
Reinforcement learning with automatically verified rewards could train coding models to generate correct solutions, while the systems and reward design remain difficult to scale.
Summary
Aakanksha Chowdhery traces the path from scaling laws and emergent reasoning to reinforcement learning for autonomous coding. Larger language models learned to solve more tasks, especially when prompted to produce reasoning chains. RL from human feedback then helped models follow instructions in chat and code. As pre-training became more expensive and its gains less certain, researchers turned to inference-time scaling, using multiple samples, revision, and verification. This works especially well for mathematics and programming because answers can be checked with formal tools, compilers, or unit tests. Chowdhery argues that reinforcement learning can move this capability into training, so models learn to produce correct answers without requiring enormous numbers of samples at inference time. She is direct about the engineering cost: RL systems need several model copies, training and inference loops, careful GPU placement, and reward functions that avoid hacking. Reflection AI is applying this approach to end-to-end autonomous software engineering workflows.
Scaling language models improved performance across tasks and produced new capabilities
Chowdhery begins with the scaling-laws result that test loss follows a power-law relationship with compute, data, and model parameters. Larger transformer models improved across benchmarks and generalized beyond their training domain. She then describes emergent behavior in PaLM, where asking for reasoning chains changed answers to math word problems from wrong to correct. The effect was not limited to mathematics. It appeared in question answering across languages, puzzles, and multitask language understanding. This led to prompting methods such as asking models to think step by step. PaLM was scaled to 540 billion parameters, although Chowdhery notes that current model makers generally do not publish parameter counts.
Human-feedback reinforcement learning taught models to follow instructions
Once models could reason more effectively, instruction following made chatbot applications possible. Chowdhery explains that reinforcement learning from human feedback uses preference data: humans choose which of two answers they prefer, and the model is trained to favor those responses. The same approach also improved code-generation performance. This history matters because it provides an earlier example of reinforcement learning changing the behavior of a language model. The question she raises is whether a similar training loop can teach models to produce correct solutions in domains where correctness can be checked automatically.
Inference-time scaling gains come from sampling, revision, and verification
Because pre-training can cost tens of millions of dollars while inference is relatively cheap, Chowdhery describes several ways to spend more compute when a model answers. One is self-consistency: generate multiple independent answers and use majority voting. Another is sequential revision, where the model evaluates and fixes an earlier response. These methods work especially well when there is a verifier. Mathematics can use a calculator or formal proof, while programming can use unit tests, compilers, or execution feedback. Chowdhery points to pass@k and coverage results showing that taking more samples improves accuracy for open models on verifiable coding benchmarks.
Verification makes coding a strong domain for training correct generation
Inference-time scaling needs a way to identify a correct output. Chowdhery says this is straightforward for equations, formal proofs, and code that passes unit tests. In coding, a compiler or a framework such as PyTorch can also provide feedback. Domains without automatic verification get smaller gains from generating many answers and voting. Autonomous coding therefore has an advantage: software can be executed and tested. That gives a training system a concrete signal about whether a generated patch works, instead of relying only on a human preference judgment.
Majority voting can require impractical numbers of samples
Chowdhery gives a limit to simple inference-time scaling. If correct generations are rare, the system may need to sample thousands of times before finding one. Majority voting and longer reasoning chains can show that a correct solution exists somewhere in the sample set, but this does not make the method practical across all tasks. The next step is to train the model itself to generate correct answers more often. Her proposal is to apply reinforcement learning where automated verification can provide the reward, rather than paying the inference cost repeatedly at deployment time.
Reinforcement learning is the next scaling direction for verifiable tasks
Chowdhery presents Reflection AI's view that reinforcement learning is the next frontier after scaling pre-training and inference. She connects this to the progression described by David Silver and Saton, from simulation-based systems such as AlphaGo and AlphaZero, through language models trained with human data, toward systems that learn from experience. She cites mathematical results showing that test-time compute improves accuracy and that applying reinforcement learning can make training-time compute improve accuracy as well. The argument is strongest in tasks with a reliable verifier, where experience can be converted into a useful reward.
RL systems are difficult to scale because they need several models and careful infrastructure
The systems challenge is larger than simply training a bigger language model. For RL methods used with human feedback, the training setup may require four copies of a large model. DPO can remove the value model and reduce this to three copies, as shown by DeepSeek Math, but the infrastructure remains difficult. Teams must place the model copies across GPUs, keep training and inference loops running, and use the cluster efficiently. Chowdhery also warns about reward hacking when a neural reward model is used. Autonomous coding offers a better route because unit tests, execution feedback, and benchmark evaluations can support more direct reward functions.
Autonomous coding requires an end-to-end software engineering system
Generating code is only one part of software engineering. Chowdhery says an autonomous system must generalize across the wider workflow, which creates a harder problem than producing isolated code snippets. Reflection AI is starting with autonomous coding as the root problem for its broader mission to build superintelligence. She says the company has about 35 people with backgrounds in language models and reinforcement learning. In the question period, she adds that coding agents need multiple capabilities and will require multiple language-model calls. Whether those calls use one model or several remains an open design choice.
"Scaling up RL is even more challenging than scaling up LLMs because you have multiple copies of the model and you have a training loop and an inference loop."14:32
Who should watch
You are building coding agents and need to decide where reinforcement learning or inference-time sampling can help.
Your tasks have unit tests, compiler checks, or other automated signals that could support training rewards.
You are designing large-scale RL infrastructure and want a candid account of model copies, GPU placement, reward hacking, and end-to-end workflow problems.
