Continual learning should train on the data produced by real-world use, because inference generates information about where models succeed and fail.
2
On-policy self-distillation uses a hint in the teacher prompt to produce dense per-token feedback without requiring a smarter teacher or parallel rollouts.
3
Long tool-use trajectories cause the but wait problem and hint leakage, which require step-level divergence weighting and residual guidance to control.
Summary
Ronak Malde argues that continual learning should use the traces produced by models in real applications instead of relying mainly on expensive benchmarks. He compares SFT, DPO, RLHF, and GRPO across four properties: an online task distribution, on-policy sampling, singular parallelism, and per-token reward. His proposed method, on-policy self-distillation, gives the model a privileged hint in the teacher prompt, then trains the unhinted student to match the hinted model's token distributions. This provides feedback across the whole vocabulary from a single rollout. The method improves short-horizon tasks, but long trajectories with many tool calls expose new failures. Divergent teacher and student distributions can make the model overproduce words such as "wait," "but," and "maybe." Hints can also leak answers and teach the model to backfill reasoning. Malde describes step-level divergence weighting and residual guidance as ways to reduce these failures. Trajectory is building a platform for continual model and harness improvement from production traces.
Real-world inference produces training signal that benchmark pipelines discard
Malde says current progress depends on benchmarks that saturate quickly, while some newer evaluations take four hours, six hours, 24 hours, or several days to scale. These tests also may not match the situations where people use AI. At the same time, models generate hundreds of trillions of inference tokens each day. Those interactions contain information about where models fail and where they work well. Malde argues that this should become training data. He compares the process to human learning, where people keep updating from experience in the real world. He sees continual learning as the next stage after internet-scale pretraining and benchmark scaling.
A useful training method needs four properties at once
Malde evaluates post-training methods against four requirements. The task distribution should come from online use rather than a fixed benchmark. Sampling should be on policy, so the training examples reflect what the current model actually does. The method should avoid large groups of parallel rollouts and the infrastructure needed to reproduce environments at scale. Its reward should be per token because real tasks contain messy, rich feedback that a single sequence-level score loses. He uses these criteria to compare SFT, DPO, RLHF, and GRPO. Each method gets some of the properties, but none of them initially gets all four.
GRPO gains on-policy learning by accepting expensive parallel rollouts and coarse rewards
GRPO uses several parallel rollouts for a task, gives each rollout an end-state reward, calculates a mean, and shifts the model toward outputs that scored above it. Malde calls this a Faustian bargain. On-policy rollouts have enabled models to improve without some of the catastrophic forgetting problems associated with earlier methods, but the task distribution can remain off policy. Parallelism also grows sharply because the system needs substantial environment infrastructure. The reward is sequence-level, so the model receives little information about which individual decisions helped or hurt. Malde compares this to receiving an essay score of 87 out of 100 and having to infer the reason.
Self-distillation makes the current model its own teacher
In ordinary distillation, a student matches the token log probabilities of a smarter teacher on a fixed dataset. On-policy distillation replaces that dataset with a rollout from the student. Self-distillation addresses the lack of a smarter model at the frontier by adding privileged information, called a hint, to the teacher's prompt. The student without the hint then matches the teacher with the hint. A hint might be a golden solution or environment guidance for a task such as finding a derivative. Because the rollout comes from the student, the task distribution is online and sampling stays on policy. A single example is enough, so the method does not need a group of parallel rollouts.
Matching the whole vocabulary gives feedback beyond the sampled token
Self-distillation changes the probability distribution over every possible next token. Malde contrasts this with reinforcement learning, which tends to sharpen the sampled distribution. For each token position, the vocabulary may contain 65,000 tokens, and the teacher can raise the probability of a token that was not the student's top choice. In his example, the model might have generated a Python for loop or range, while the teacher distribution pushes it toward a different token. The method can therefore shift an entire distribution rather than make a small adjustment around the sampled action. Malde connects this behavior to better short-horizon results and fewer tokens needed to solve difficult tasks.
Long tool-use trajectories expose the but wait problem
When Malde scales the method to a 120-billion-parameter model and trajectories with 50 or 100 tool calls, accuracy becomes unstable across runs and tool-call errors increase. The model may stop following the format learned during instruction fine-tuning. On long tasks, the student can drift far from the teacher's distribution. The teacher then tries to correct the student repeatedly, raising the probability of words such as "wait," "but," and "maybe." Malde calls this the but wait problem. The model ends up between two divergent distributions, producing hedging language and reaching a locally suboptimal state where everything turns into maybe.
Step-level divergence weighting limits corrections after a trajectory goes off track
Malde proposes measuring the KL divergence between the student and teacher at each step of a tool-calling trajectory. Instead of using KL only as a general penalty, the system uses the divergence to weight the tokens at that step. A normal trajectory can keep a weight of one. If a trajectory diverges heavily at its first step, the method can focus training on that step before moving on. Independent weights also allow a later recovery: a step that goes off track can receive less weight, while a later step that returns to the right distribution can still receive a moderate update. Malde presents this as one way to scale self-distillation to long-horizon tool use.
Hints can cause answer leakage, so residual guidance reduces the shift
Self-distillation has an analogue of reinforcement learning's reward hacking: hint leakage. If the hint contains an answer the student could not know in the real task, the rollout may state the answer first and then invent a reasoning trace afterward. Malde gives the example of a hint that reveals the last three digits of a formula as all zeros. The model responds that it knows the solution and then fills in the reasoning. One simple response is to use an LLM to rewrite hints into information the model could reasonably infer from logs without revealing the solution. Residual guidance compares a partial hint with a full hint and combines their teacher distributions, reducing a large shift into unknown territory.
"You take what's called this hint, put it into the beginning of the prompt, and now you match the log probs of the student without that hint to the teacher with that hint."08:46
Who should watch
You are building post-training systems and need to compare SFT, RLHF, GRPO, and self-distillation by their data, sampling, rollout, and reward requirements.
Your agents generate long tool-use traces and you are seeing unstable accuracy, formatting errors, or repetitive hedging such as "wait" and "maybe."
You want to turn production agent traces into updates to both a model and its harness, while understanding the infrastructure and algorithmic limits of continual learning.
