Continual learning can start with a one-time dump of enterprise production traces, without a replayable environment or a golden answer.
2
Online hints built from each model rollout can adapt to unusual production behaviors and raise the ceiling for continuous improvement.
3
Per-step hinting and relevance-masked self-distillation help the student learn the intended behavior without copying irrelevant teacher preferences or degrading existing abilities.
Summary
Samuel Denton presents continual learning as two connected spectrums. Traces range from a one-time offline dump to a system where serving and training share one loop. Hints can come from static priors or from the model's current rollout. Applied Compute focuses on offline hints with offline traces, which can improve an enterprise agent on day one, and online hints with online traces, which can adapt to changing behavior in production. In SWE-bench, an offline hint raised the task-completion tool-call rate from about 22% to 60% while the test pass rate stayed flat. For a customer with unusual hyperlink formatting, rollout-specific online hints raised correct formatting from about 15% to around 80%, while reward-based training and supervised fine-tuning had degraded coding performance. Denton also explains why hints should target a particular step and why learning should mask irrelevant tokens.
Continual learning ranges from trace dumps to a shared serving and training loop
Denton describes offline distillation as learning from a single batch of production traces. A less offline setup collects traces in daily batches. At the online extreme, the model serves a request, creates a trace, learns from it, updates its weights, and then serves the next request. Enterprises sit at different points on this spectrum, so Applied Compute wants to provide value before a fully online system is available. Offline traces can be used immediately, while a model that updates during production can keep improving as new requests arrive. The distinction is about how close training is to live serving, not about a separate kind of model.
Hints can come from static behavior goals or from the current rollout
The second axis is where the teacher's extra information comes from. An offline hint can encode a known rubric, a general behavior change, or a pattern found in production data. Denton's example is a support agent that gives refunds too readily. An online hint is built from the rollout that the on-policy model just produced. It can refer to what the model actually did and tell the next training target how to change. Crossing trace timing with hint timing creates four broad areas: offline hints with offline traces, offline hints with online traces, an offline trace with one on-policy step, and online hints with online traces.
The practical focus is offline improvement now and online adaptation later
Applied Compute researches all four areas but concentrates on the offline-hint and offline-trace corner and the online-hint and online-trace corner. The first lets a customer provide production traces and receive a targeted behavior change without replaying the original environment. The second can use a replayable environment or live production traffic. Since its hint is constructed from each rollout, it can respond to different behaviors instead of applying the same instruction everywhere. Denton frames these as two stages of value: use existing data to improve an agent on day one, then update the model continuously once online serving and training infrastructure is available.
The method is designed to work without a golden answer
Denton keeps returning to the absence of a fully specified target. Much distillation work assumes that every task has a golden answer or a polished rubric that can be copied into the student. Enterprise production behavior often lacks that kind of reference. The proposed teacher is created from behavior information, hints, and the trace being learned from. In the offline case, the system can use a production dump and a general behavior goal. In the online case, it can inspect what the serving model did and construct a more specific lesson. The aim is continual learning without needing a manually authored ideal answer for every request.
An offline hint increased SWE-bench submissions without lowering test performance
The SWE-bench experiment used a Qwen thinking model that could take up to 80 turns before submitting. Applied Compute wanted it to finish by turn 40. Denton separates three measurements: whether the agent called the submission tool, whether the environment's tests passed, and the intersection of those outcomes as the SWE-bench task pass rate. The hint told the model that it was near the turn limit, asked it to finalize and verify the fix, and instructed it to call the tool. The task-completion call rate rose from about 22% to 60%, while the test pass rate stayed roughly constant and even rose slightly.
The teacher moved the reasoning toward a tool call without changing tool-call tokens
The offline SWE-bench traces had not called the task-completion tool. Because the rollout was conditioned on those earlier traces, the teacher did not directly alter the tool-call tokens. Instead, it pushed the reasoning path toward deciding to finish, after which the tool call followed. Denton describes this as surprising because the desired action appeared even though the teacher never explicitly trained those action tokens in the original setup. Adding one on-policy step changed the situation slightly. The current model could roll out one step from the old trace, and the teacher could then encourage the actual tool token. This improved SWE-bench task pass rate more than the fully offline version.
Rollout-specific hints fixed unusual hyperlink formatting where other methods hurt coding
A customer needed a coding agent to format hyperlinks in an unusual way. The format was far outside what earlier post-trained models had seen, and the agent still had to retain its general coding ability. Adding a reward for the format caused a degradation in coding performance. Supervised fine-tuning on correctly formatted traces caused the same type of problem. Applied Compute instead let the model complete a rollout, then created a hint tied to that rollout. The hint described the previous formatting and stated how the next attempt should format hyperlinks. Correct formatting rose from about 15% to around 80%. A fixed offline hint improved the behavior less than the rollout-specific online hint.
Learning should focus on the step after the hint and mask irrelevant tokens
Denton says per-step hinting is important because a hint at the start of a long rollout is less connected to the behavior being taught. A judge chooses where the hint belongs, and distillation is applied to the next step or a few steps after it rather than to the entire rollout. The learning signal falls as the distance from the hint increases. Relevance-masked self-distillation addresses a second problem. The teacher may prefer connector words that do not matter to the intended behavior. An LLM judge selects which teacher tokens the student should learn from. Denton says this improved learning of an out-of-distribution behavior while reducing catastrophic degradation.
"So, in general, we want to think about how we can do continual learning and distillation without having some beautifully golden rubric to accompany every task."09:55
Who should watch
You have production traces from an enterprise agent but cannot replay the environment or write a golden answer for every task.
Your model needs to adopt a narrow production behavior, such as an exact output format or earlier tool use, without losing its existing coding ability.
You are building infrastructure that connects live serving with continual training and want guidance on where to place hints and which tokens to distill.
