# The Base Model Is Dead

Varun Singh, Arcee AI | AI Engineer World's Fair 2026 | 17:45

Source: https://www.youtube.com/watch?v=xbPriQWXtWM
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/the-base-model-is-dead
Published: 2026-07-31
Tags: open-models, reinforcement-learning, synthetic-data

## TL;DR
- Modern base models contain much less general web text and much more code, STEM, instruction data, and synthetic data than older models such as GPT-3.
- Reinforcement learning now adds major capabilities, so supervised training must prepare the model with the atomic skills and representations that RL will compose.
- A base model is increasingly trained for reasoning and agentic behavior rather than only for broad human knowledge and web-text prediction.

## Summary
Varun Singh argues that the old base-model recipe has changed. Earlier models were trained mostly on web text, books, and Wikipedia, with post-training used to turn accumulated knowledge into a useful chat interface. Reinforcement learning was relatively small. After OpenAI o1 and DeepSeek R1, RL became a major source of reasoning and software-use capability. That changes what the base model must provide. Training data now includes more code, STEM, instruction examples, synthetic rephrasings, reasoning traces, and longer agentic data. Singh also describes mid-training as a stage that prepares models for post-training and RL distributions. From Arcee AI's Trinity work, he discusses practical issues with synthetic data, mixture coefficients, mixture-of-experts load balancing, and learning stable representations early. His conclusion is that a base model should encode the prior needed by the next training stage, which may be reasoning or agentic behavior rather than general web knowledge alone.

## Key ideas
### Older base models learned a broad web-text view of human knowledge
[00:12](https://www.youtube.com/watch?v=xbPriQWXtWM&t=12s)
Singh describes the older base model as a reflection of the human internet. GPT-3 used Common Crawl, WebText-2, books, and Wikipedia, with web text and Wikipedia making up roughly 85% of the mix. Llama 3 still used general-knowledge web text for about half of its tokens. Pre-training accumulated world knowledge and useful representations through next-token prediction. Post-training then shaped that knowledge into a chat interface, with question-answer formatting and interaction conventions. Reinforcement learning was mostly a small final adjustment, which Singh calls a "cherry on top," rather than a major source of new capability.

### Instruction data is moving into the earliest stages of training
[02:49](https://www.youtube.com/watch?v=xbPriQWXtWM&t=169s)
Singh contrasts newer recipes with the older web-heavy mix. In Nemotron 3 Ultra, several of the largest data categories are marked SFT, meaning supervised fine-tuning data with question-and-answer or chat structure. Pulling this data into training early lets the model learn the shape of conversations and the tasks it may need to perform downstream. The change is visible in the reduced share of web text and the growing role of code. Singh says code has become the dominant data subset in some modern pre-training recipes, even though GPT-3 had no dedicated code datasets.

### Synthetic data can repeat and reshape useful information
[09:41](https://www.youtube.com/watch?v=xbPriQWXtWM&t=581s)
Singh explains synthetic data through Arcee AI's Trinity Large. The team used web-scale synthetic data mainly by rephrasing seed items, so the model encountered the same information in multiple forms. He also points to Kimi K2 and earlier datasets such as Swallow Code and Swallow Math. In his account, synthetic data can increase the token supply, clean up low-quality material, and make examples look more like instruction or agentic tasks. Those examples let the model learn task representations during pre-training. Singh acknowledges the concern that careless synthetic data can cause model collapse, while describing rephrasing as one way teams are using it at scale.

### Mixture-of-experts models need post-training distributions earlier
[11:01](https://www.youtube.com/watch?v=xbPriQWXtWM&t=661s)
A mixture-of-experts model can develop experts that specialize during training, while load-balancing objectives try to keep expert use broadly equal. Singh says a major failure mode appears when the model sees a very different distribution during post-training than it saw during pre-training. That shift can create large expert imbalances. He describes the MAI Thinking 1 report as an example where the team had to raise the load-balancing coefficient substantially during SFT. Singh's preference is to expose the model to relevant post-training data earlier, so it develops stable representations before those later stages.

### Mid-training prepares models for longer and more agentic data
[12:30](https://www.youtube.com/watch?v=xbPriQWXtWM&t=750s)
Singh describes mid-training as a newer stage that exposes a model to the distributions it will encounter in post-training and RL, often at longer context lengths. That makes room for agentic traces and other long datapoints. He notes that many models already use longer contexts during pre-training, so these datasets could be introduced earlier to help create stable representations. Rather than treating pre-training, mid-training, post-training, and RL as completely separate categories, Singh proposes two broad training paradigms: supervised next-token prediction and reinforcement learning.

### Supervised training should supply the skills that RL will compose
[14:55](https://www.youtube.com/watch?v=xbPriQWXtWM&t=895s)
As RL takes a larger share of compute, Singh says supervised learning can be understood as preparation for RL. He cites work arguing that the base model needs exposure to the atomic skills required during reinforcement learning. Given an environment with enough difficulty, RL can learn to compose those skills. This also changes how teams think about training data that the model would not naturally see in human output, including reasoning traces and examples designed for test-time compute. Singh says such data can warm the model up to behaviors that later help it explore effectively during RL.

### The base model now carries priors for reasoning and agentic behavior
[16:45](https://www.youtube.com/watch?v=xbPriQWXtWM&t=1005s)
Singh's conclusion is that base models have moved from carrying general human knowledge and world priors toward carrying reasoning and agentic behavior priors. He treats that description as specific to the current way chatbots are used, rather than as a permanent definition. If another interaction paradigm becomes important, the base model should be trained to provide the prior for that paradigm. The practical implication is that teams cannot assume a large web-text corpus remains the right foundation. They need to decide what later training will require the model to already know how to represent.

## Notable quotes
- "RL was mostly just a cherry on top, shaping the flavor of the interactions more than conferring extra knowledge or quality onto the base model itself." (02:58)
- "The trend seems to be that synthetic data not only allows you to get more and more tokens, but also clean up tokens, get higher quality tokens, and have tokens that are shaped more like instruct or agentic tasks all the way back in pre-training." (10:55)
- "The base model needs to have some exposure to the atomic skills that it would need to compose during RL." (14:55)
- "Base models have kind of moved from general human knowledge and world priors to reasoning and agentic behavior priors." (16:45)

## Tools & references mentioned
- Arcee AI
- Trinity
- Trinity Large
- GPT-3
- Common Crawl
- WebText-2
- Wikipedia
- Llama 3
- GLM-4.5
- GLM-5
- OpenAI o1
- DeepSeek R1
- Claude Code
- MAI Thinking 1
- Nemotron 3 Ultra
- Kimi K2
- Swallow Code
- Swallow Math
- AlphaGo

## Who should watch
- You are choosing between a web-heavy pre-training mix and data that prepares a model for reasoning or agent use.
- Your mixture-of-experts model develops unstable expert usage when the post-training distribution differs sharply from pre-training.
- You are designing the boundary between pre-training, mid-training, supervised fine-tuning, and reinforcement learning.