# The Messy Reality of Scale: Synthetic Data and Pre-Training Marah Abdin & Robert McHardy, poolside | AI Engineer World's Fair 2026 | 17:31 Source: https://www.youtube.com/watch?v=KhYifX22yhE Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/the-messy-reality-of-scale-synthetic-data-and-pre-training Published: 2026-07-26 Tags: data-pipelines, open-models, reliability, synthetic-data ## TL;DR - Synthetic data complements organic data by making hidden rationale, planning, and structure more explicit while reducing repetition in the training mix. - Poolside checks distributed training replicas against each other and stops runs when their weights diverge, which exposes failures such as broken GPUs and numerical precision bugs. - A 118 billion parameter model with 8 billion active parameters scaled the recipe successfully and showed strong early coding and agentic results against larger open-weight models. ## Summary Marah Abdin describes synthetic data as a way to complement organic data. Poolside uses it to expose implicit rationale, planning, and structure, fill gaps, reduce repetition, and change how examples are presented. Its pipelines range from cheap rephrasing to orchestrated educational data generation, with task difficulty adjusted so the model can still produce correct and diverse results. Robert McHardy then explains the training checks used at larger scales. Multiple replicas periodically hash their weights, and training stops when the hashes differ. This caught silent corruption from a broken GPU and helped diagnose a tensor-parallel accumulation that lost precision in BF16. A race condition in FP8 kernels later corrupted gradients without being visible to the replica hash check. The team applied these lessons to a 118 billion parameter model with 8 billion active parameters, trained on 30 trillion tokens across 4,000 GPUs. Its early base-model evaluations were especially strong for coding and agentic performance, though McHardy is clear that post-training could change the final results. ## Key ideas ### Synthetic data exposes teaching signals that organic data leaves implicit [01:58](https://www.youtube.com/watch?v=KhYifX22yhE&t=118s) Marah Abdin says poolside does not view synthetic data as a replacement for organic data. Organic examples contain useful information that is often hidden or presented in a way that does not teach the model well. Synthetic generation can extract implicit rationale, planning, and structure, then project them into new forms. It can also fill gaps and regularize both the token presentation and the teaching process. For one model, synthetic data made up 13% of the pre-training mix before post-training. The team has continued generating data in different directions, building a corpus of six trillion tokens that keeps growing. ### Scaling the training budget made repetition a model problem [03:11](https://www.youtube.com/watch?v=KhYifX22yhE&t=191s) The team initially focused more on quality than quantity because it was working at a smaller scale. As the models and training budget grew, high-quality sources began repeating in inefficient ways. That repetition saturated the model too early and created a token-uniqueness problem. Poolside used scalable rephrasing to replace repeated tokens and reduce the repetition. It also built specialized pipelines for moving material between code and text and for handling scanned documents. Abdin describes the reported ablation as something to interpret cautiously, while saying that the difference between repeated seeds and reduced repetition persisted consistently. ### Synthetic pipelines trade cheap scale for more orchestration [04:44](https://www.youtube.com/watch?v=KhYifX22yhE&t=284s) Abdin breaks synthetic data pipelines into reusable parts: seeds, primary inputs, metadata, secondary inputs, a generator, and supplementary functions such as filters and validators. Cheap pipelines use smaller models and depend heavily on their seeds. More expensive pipelines add orchestration when the source material is valuable. The team tunes task difficulty because a task that is too hard makes the model lose correctness and diversity. A difficult generation can be broken into smaller steps. For a novel, that might mean generating the setting, characters, styles, plot, and twists before producing chapters one at a time. ### Porting and multi-turn generation change the shape of existing data [06:37](https://www.youtube.com/watch?v=KhYifX22yhE&t=397s) Poolside uses several ways to transform data. Cross-domain porting moves material from one mode to another, such as translating code or converting math problems into code. Multi-turn workflows add iteration, including two agents talking to each other or a judge and an evolver repeatedly changing a task. The team also uses multistage workflows that build an output gradually instead of asking for the full result in one prompt. These transformations let the pipeline create new training shapes from existing material while keeping the generation process modular. ### An orchestrator filters generations and controls which agent acts next [07:43](https://www.youtube.com/watch?v=KhYifX22yhE&t=463s) Poolside's Hive infrastructure lets the team define a queue of agents, with each agent's prompt, parameters, model, inputs, and outputs configured separately. The system can control when an agent enters or leaves the queue and how often it runs. Orchestrators sit between agents and can change instructions, select the next agent, or skip an agent. A supervisor has a broader view and monitors the orchestrator. Abdin describes this arrangement as a way to police generations while still allowing the agents some creativity and allowing instructions to change during the workflow. ### Replica hashes turn training divergence into a hard failure [09:03](https://www.youtube.com/watch?v=KhYifX22yhE&t=543s) Robert McHardy argues that data and training implementation have to be treated together. A good data mix cannot rescue faulty training code, and correct code cannot rescue poor data. His team therefore trusts nothing at large scale. Distributed replicas of the same model should have identical weights, so the team periodically calculates a hash over each replica and compares the results. Matching hashes allow training to continue. A mismatch means something serious has happened, so the run is crashed rather than allowed to continue with unknown state. ### Large runs exposed hardware and precision failures that looked like training behavior [10:41](https://www.youtube.com/watch?v=KhYifX22yhE&t=641s) One run had a broken GPU that caused silent data corruption. Its loss curve became spiky and its gradient norms grew unusually large even though the model, data, and implementation matched another run. In a separate run, activations grew before the language-model head, where tensor parallelism required accumulation. That accumulation happened in BF16, which eventually lacked enough precision. The loss flattened and the model stopped learning. Moving the accumulation to FP32 allowed convergence to resume from the earlier checkpoint. McHardy presents both cases as failures that become visible only when training is large enough. ### FP8 introduced a silent gradient-corruption blind spot [13:15](https://www.youtube.com/watch?v=KhYifX22yhE&t=795s) When poolside scaled to a 118 billion parameter model with 8 billion active parameters, trained on 30 trillion tokens across 4,000 GPUs, the team found a race condition in its FP8 training kernels. Illegal memory accesses and NaNs helped reveal the problem, but the more concerning effect was silent: about 0.5% of the gradient could be replaced by random values. Replica weight hashes could not detect this because real training does not provide redundant replicas with identical weights and identical data for checking forward and backward computations. McHardy says the team was working on a dry-run hash checker for this kind of failure. ### The scaled recipe was strongest on coding and agentic evaluations [15:07](https://www.youtube.com/watch?v=KhYifX22yhE&t=907s) The larger model's results were base-model evaluations, so McHardy warns that they will not map perfectly to the post-trained model. On coding evaluations including multiple versions of CodeBench and BigCodeBench, it outperformed the team's previous smaller model and several larger or newer open-weight models, including GLM 4.5 Air. It was competitive on BigBench Hard and EvalPlus. Its strongest result was on SWE-bench Agentless Multilingual, which poolside uses as a proxy for agentic performance during pre-training. McHardy also acknowledges that the model lagged on MMLU Pro because the team had chosen to prioritize coding data. ## Notable quotes - "Synthetic data gives us a track to extract some of these features and project them on some new planes." (02:17) - "If task is too hard for your model, then your model will start to fall on its face." (06:03) - "We don't trust anything." (09:44) - "That broken GPU caused silent data corruption and therefore made the training behave the way it did." (11:03) - "The recipe held, it scaled, and we will continue scaling it from here." (16:41) ## Tools & references mentioned - poolside - Hugging Face - Hive - GLM 4.5 Air - DeepSeek V4 Flash Max - SWE-bench Agentless Multilingual - BigCodeBench - BigBench Hard - EvalPlus - MMLU Pro ## Who should watch - You are building synthetic data pipelines and need practical ways to control task difficulty, repetition, and generation quality. - Your distributed training runs sometimes fail in ways that look like ordinary optimization problems, such as a spiky loss curve or stalled convergence. - You are evaluating large coding models and want to understand which base-model results may indicate agentic performance before post-training.