Current language-model benchmarks reset the model between independent tasks, so they measure point capability while ignoring learning from prior experience.
2
A continual-learning benchmark needs headroom, shared structure across instances, and a learning signal such as reward, error messages, or textual feedback.
3
On Continual Learning Bench 1.0, vanilla in-context learning outperformed more elaborate context-management systems on reward, gain, and cost, although longer-horizon tasks may change the result.
Summary
Parth Asawa argues that standard language-model evaluation quietly assumes the system forgets everything after each task. That setup measures isolated capability, while continual learning asks whether a system improves from experience, retains useful information, and adapts when the environment changes. He proposes three requirements for evaluating it: tasks must have headroom beyond pretraining, shared latent structure across instances, and a learning signal. His gain metric compares a stateful run with the same system reset between instances, separating learning from the base model's initial strength. Continual Learning Bench 1.0 applies this approach across six domains, including database exploration with schema changes. The first results were surprising: plain in-context learning beat more complex memory systems on reward, gain, and cost. Asawa also describes failures involving both excessive forgetting and failure to update. He argues that future models may need training architectures designed for continual learning from the start, rather than methods added to frozen checkpoints.
Standard benchmarks erase the experience that would reveal learning
Language models are usually asked to complete one task, then another independent task, across many benchmarks. Their aggregate scores produce familiar leaderboards, but the setup effectively tells the model to forget its memory after every activity. Asawa contrasts this with an evaluation where performance improves as a function of prior experience. He argues that this distinction matters because an agent deployed in the world can encounter related tasks over time, while independent benchmark instances provide no opportunity to benefit from what came before. A score on one isolated task cannot show whether the system learned from earlier interactions.
Continual learning requires retention and adaptation over long horizons
Asawa defines continual learning as sample-efficient online learning that remains stable over long horizons. A system must retain useful prior information without forgetting, while updating its beliefs when it receives new data. Current language-model training usually produces a frozen checkpoint after offline pretraining, supervised fine-tuning, and later training stages. The deployed weights do not normally change as the model interacts with the world. Possible routes to continual learning include putting experience into the context, writing it to an external memory such as a notepad or key-value store, or updating the model's parameters online.
Existing continual-learning evaluations miss sample efficiency and frontier-model headroom
Older continual-learning studies often train on task A, then task B, then task C, and measure whether earlier performance survives while the model learns the new task. Language-model evaluations also test factual recall across long conversations. Asawa says these approaches do not make sample efficiency a first-order requirement and do not always measure learning. Frontier models have already seen much of the internet and many economically valuable tasks during pretraining, so finding tasks where online experience produces improvement is difficult. A benchmark needs tasks that cannot be solved equally well through offline training alone.
Chaining independent benchmarks cannot create a learning problem
Taking existing benchmark instances and putting them into a sequence does not make them suitable for continual learning. Traditional instances are designed to be independent, so they do not share structure that a model can discover and exploit later. Without that shared structure, there is no meaningful reason for performance on a future instance to improve because of earlier experience. Asawa's benchmark instead constructs sequences with a shared latent structure in the environment. The system can learn properties that recur across instances and use them to work more effectively over time.
Gain separates learning from the strength of the initial system
Cumulative reward can confuse a strong starting model with a model that actually learns. In Asawa's example, one system has higher reward throughout the sequence but does not improve much relative to its stateless version. The benchmark therefore measures gain, defined as stateful reward minus stateless reward. The stateful run can retain notes, update a policy, or grow its context. The stateless run resets between every instance. Comparing the two estimates how much prior experience helped on a later task, while reward still measures base performance and cost measures the resources spent.
Database exploration tests both accumulated knowledge and forgetting after change
The database exploration task gives an agent natural-language questions that it must answer through SQL queries. At the beginning, the agent has to explore unfamiliar schemas, discover how tables connect, and learn data-specific quirks. A continually learning system should need fewer queries on later questions, such as around the tenth instance, because it has learned the database. The benchmark then introduces concept drift through a migration: columns can be dropped, new columns can appear under different names, and formats can change. The system must discard stale knowledge while preserving information that remains useful.
Vanilla in-context learning led the first benchmark results
Continual Learning Bench 1.0 covers blind spectrum monitoring, codebase adaptation, cohort studies and epidemiology, exploitable poker, database exploration, and sales prediction. The tasks use sequences of instances with per-instance reward metrics, and domain experts check whether the environments and changes are realistic. In the initial results, vanilla in-context learning, which simply places experience in the context, ranked above more elaborate context-management systems. This held for aggregate reward and also across reward-versus-cost and gain-versus-cost Pareto frontiers. Asawa cautions that the tasks had medium horizons and may not have pushed in-context learning far enough.
Continual-learning failures fall between stability and plasticity
Asawa groups failures around stability and plasticity. Stability means retaining information and using it later. Plasticity means learning from new information and changing when circumstances require it. In the sales-prediction task, a model first overpredicts, then lowers its forecast after correction, then receives feedback that it underpredicted. Instead of moving toward a middle estimate, it jumps back to its original overprediction, suggesting that it lost the intermediate correction. In an epidemiology example, a system's notepad records that a cohort definition belongs to another schema even though it applies to the current study. The system cannot update its prior when new evidence matters.
Models may need to be designed for continual learning before deployment
Asawa is excited by parametric approaches that jointly reconsider architecture, data, and algorithms for continual learning. He says much current work starts with an already trained model and asks how to add learning after the frozen checkpoint exists. That approach may reflect a sunk-cost fallacy, since the models were not designed to learn continually in the first place. If continual learning were treated as a first-order requirement, training might consist of a single learning phase followed by deployment in an environment where the model updates its weights. The benchmark roadmap includes longer-horizon and domain-specific tasks, more open-source and parametric systems, and simulations outside deterministic settings.
"It was still surprising that these more expensive context management systems perform a lot poorly compared to just vanilla in context learning on these sets of tasks where you have to do real learning."14:29
Who should watch
You build or use language-model evaluations that score independent tasks and want to know what those scores omit.
Your agent needs to learn from repeated interactions, maintain useful memory, or adapt after a schema or environment change.
You are choosing between context growth, external memory, context management, and parameter updates for an online system.
