Models learn public knowledge well, but they cannot acquire deep, private knowledge about a person or company after training.
2
Training directly on a fixed private corpus can produce near-zero loss while causing generation to collapse, so memorizing documents is not enough.
3
The desired approach lets compute keep improving a model by making its own training problems harder, similar to AlphaGo.
Summary
Jack Morris frames private knowledge acquisition as a scaling problem. Public models improve by increasing data, training compute, and model size, but those axes mostly operate on data that can be made public. A model therefore knows GitHub and Wikipedia while knowing nothing about a company's emails, meetings, or working habits. Starting with a pretrained model and a fixed private corpus leaves compute as the main axis to push. Morris reviews several approaches, including KV compaction, on-policy distillation, self-study, synthetic-data continuation, and unsupervised reinforcement learning. Each can transfer some information, but each eventually reaches a data wall because the training set is defined in advance. Engram is pursuing a system where improving the model generates harder training material, allowing additional compute to produce more depth instead of simply fitting the same corpus again.
Private context is where current models lack depth
Morris contrasts the breadth of models with the depth gained through focused practice. Terence Tao has described AI as knowing public mathematical topics and making surprising connections, while lacking the intuition of someone who has spent years working in one area. Morris applies the same distinction to private data. Models struggle with rare skills such as writing AMD kernels because few good examples are public. They also cannot know a person's emails, writing preferences, travel habits, or company relationships unless those details entered public training data. He describes scaling compute on context as the pursuit of depth and personalized knowledge after training.
The usual scaling axes mostly operate on public data
Deep learning progress has come from scaling three things: the amount of data, the training compute, and the size of the model. Morris says these methods work well on Wikipedia, Reddit, arXiv, GitHub, and post-training data gathered from experts. That information is public by definition because a model must be able to expose it to a user. The same scaling does not give a model more knowledge of an individual's life or a company's work. His question is how to apply the same pressure to a private corpus, rather than accepting that the benefits of scale stop at public data.
A private corpus leaves compute as the practical scaling axis
For a company such as Engram, the data budget appears fixed. The company cannot create enough new private examples to use data scaling in the same way as a public model, and training from scratch would discard useful knowledge about the outside world. Morris therefore assumes a pretrained model plus an unstructured corpus, such as company emails or meeting transcripts. In that setup, compute is the main remaining axis. He adds that practical data access can expand through related books, internet searches, or conversations with knowledgeable people, but the core research question keeps the original corpus fixed: how can more compute produce a model that knows it more deeply?
Directly training on documents memorizes them without useful generalization
Morris gives a financial-report example. A model trained on 10,000 reports can reach a loss of 0.0001 and appear to know the documents perfectly. Generation then collapses. Next-token training on the available corpus transfers information into the weights, but it does not teach the model to answer questions whose answers are not explicitly encoded in the same form. The method also has a hard limit: once the model has absorbed the available information, more training does not provide a continuing source of improvement. A private model needs behaviors and generalization around the corpus, not just a low training loss.
Context compression and distillation transfer information with limits
KV compaction compresses a long corpus into key-value states so the model can behave as if the material remains in context. Morris describes learned approaches and a greedy method for approximating the compaction. The method is useful when the data fits into context, but it does not address larger collections and does not use gradients to alter the model. On-policy distillation instead shows text to the model and trains it to act as if the text were in context, updating the model during training. The hard question is what to distill. Raw documents are unsuitable, so self-study methods generate question-and-answer pairs conditioned on the corpus.
Synthetic continuation can imitate pretraining but may overwrite useful knowledge
Morris considers generating synthetic data from the private corpus and continuing pretraining on that material. He sees this as promising because pretraining is unusually effective at knowledge acquisition. The approach still has costs. Continued training can overwrite part of the original pretraining, and scaling it is difficult. It also assumes access to a good pretrained base model, while many teams begin with a post-trained model. Unsupervised reinforcement learning offers another version of the idea, using an environment and an RL objective such as GRPO instead of distillation. Morris says both directions are promising, but neither yet provides unlimited improvement.
Every fixed synthetic dataset eventually creates a data wall
The approaches Morris reviews all define some training set, whether it contains compacted information, generated questions, synthetic documents, or RL tasks. Classical machine learning then imposes a ceiling: unless the model is underparameterized, it eventually learns the defined data. Synthetic data does not remove that ceiling. The model eventually absorbs the generated examples, reaches an upper bound, and requires another generation stage. This produces a curve that flattens instead of the sustained scaling behavior associated with pretraining. The unresolved problem is how to create the next stage of training after the first dataset has been fitted.
Self-improvement makes additional compute buy harder training problems
Morris points to AlphaGo as the property he wants. AlphaGo improves by generating harder questions for itself as its policy gets better. Applied to private context, a model would first learn from a corpus, then use its improved abilities to generate better training material, and repeat the process. Engram is looking for curves that keep rising rather than plateauing after one pass over data. In his example, the goal is a model that builds an increasingly accurate internal model of a person's identity, work, goals, and preferences. The training process must make itself harder as the model improves.
"You can get to a loss of 0.0001, and you can end up with a model that knows the data perfectly well. And then when you generate from it, it basically collapses."11:05
Who should watch
You are building an AI system over private company documents and need more than retrieval or memorized text.
You want to understand why continued pretraining, distillation, KV compaction, and synthetic data each reach a limit.
You are researching continual learning or self-improving training methods and want a clear statement of the unresolved problem.
