# Data Quality Is the Compute Multiplier Ari Morcos, DatologyAI | AI Engineer World's Fair 2026 | 19:05 Source: https://www.youtube.com/watch?v=_PdK6x7PQNM Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/data-quality-is-the-compute-multiplier Published: 2026-07-31 Tags: cost, multimodal, open-models, synthetic-data ## TL;DR - Better data steepens the scaling curve, so the same compute budget can produce a stronger model or the same model can be trained with less compute. - Useful training data must be selected for the target task, balanced for diversity and redundancy, and sequenced across training stages. - Curated data improved model quality, inference efficiency, multilingual performance, and domain adaptation in the examples Ari Morcos shared. ## Summary Ari Morcos argues that data quality is the most underused way to get more from limited compute. Better data increases the information each token gives the model, which changes the scaling curve. He describes DatologyAI's process as an oil refinery for existing public, proprietary, and licensed data: clean it, curate it, create synthetic variations, then compose and sequence it across training stages. The examples include vision-language models that gained 14 percentage points from curation, matched a much larger model with far less training compute, and produced cheaper correct answers at inference. Curated multilingual data also improved performance with relatively little non-English data. Morcos explains why synthetic rephrasing works best when applied to high-quality documents. Customer examples from Thomson Reuters and Arcee show gains in legal adaptation, post-training, and open model development. His conclusion is direct: when compute is scarce, improving signal per token can be cheaper than buying more hardware. ## Key ideas ### Data quality changes the scaling curve [01:56](https://www.youtube.com/watch?v=_PdK6x7PQNM&t=116s) Morcos treats data, compute, time, and dollars as different labels for the same underlying scaling axis. If better data turns the original performance curve into a steeper one, a fixed compute budget produces a stronger model. The same quality target can also be reached with much less compute. He gives the intuition in terms of signal per token and marginal information gain per data point: training data should teach the model as much as possible when it is shown. The right dataset depends on the target task. Legal models need relevant legal data, while healthcare models need healthcare data. Diversity also matters because narrow data can make a model brittle when questions are phrased differently. ### Data preparation is a pipeline with four stages [03:48](https://www.youtube.com/watch?v=_PdK6x7PQNM&t=228s) DatologyAI starts with existing public, proprietary, and licensed tokens rather than sourcing a new stream of tokens. Morcos compares this to an oil refinery and describes four stages: clean, curate, create, and compose. Cleaning removes unusable documents and decontaminates training data against downstream benchmarks. Curation uses quality classifiers, topic taxonomies, redundancy reduction, quality and relevance weighting, and task-distribution matching. Creation uses synthetic rephrasing to expand a high-quality dataset and add format diversity. Composition determines how datasets are combined and sequenced across training stages. Morcos says large models are generally trained through at least three data phases, so the transitions between stages matter. ### Curation can make smaller models beat larger-model frontiers [06:54](https://www.youtube.com/watch?v=_PdK6x7PQNM&t=414s) In a vision-language example, DatologyAI curated the Mammoth dataset, described as containing about 25 billion tokens, for training the fusion adapter between a text model and a vision model. The curated models moved beyond the frontier defined by several strong public vision-language models. Holding the other factors constant, curation produced about a 14 percentage point absolute improvement. Morcos also says one curated model roughly matched Qwen 3.5 4B while using 145 times less training compute. He connects this result to the compute-multiplier idea: choosing data well can make a smaller training run behave as if it had received far more compute. The result did not require post-training. ### Better data can reduce inference cost as well as training cost [08:58](https://www.youtube.com/watch?v=_PdK6x7PQNM&t=538s) Morcos says curation changes how concisely some models answer. In his comparison, models trained on DatologyAI's data produced substantially shorter responses. He then measures inference efficiency with the compute used per response rather than the compute used during training. The curated models remained beyond the comparison frontier and reached performance similar to Qwen 3.5 with 35 times fewer flops per correct answer. This matters because reasoning models are already using far more tokens than non-reasoning models. If data helps a model reach the same answer with fewer output tokens or fewer inference flops, the benefit continues after training and affects the cost of serving the model. ### Curating language data produces cross-lingual gains [09:24](https://www.youtube.com/watch?v=_PdK6x7PQNM&t=564s) Morcos argues that internet data gives an uneven view of the world, with strong implications for model access outside English-speaking and developed markets. In the multilingual MMLU example, only 8 percent of the training data was multilingual, and most languages had at most 6 billion tokens. Even with that limited amount, the curated model performed beyond the comparison frontier and was slightly better than Qwen 3 while using roughly eight times less compute. Curating English data also improved non-English accuracy through cross-lingual transfer. The size of that benefit correlated with how similar a language was to English. Curating non-English data helped English performance too, although the effect was smaller. ### Synthetic data works when it expands high-quality source material [12:16](https://www.youtube.com/watch?v=_PdK6x7PQNM&t=736s) DatologyAI's synthetic-data approach is called rephrasing. A source document, such as one about a corporate takeover, can be converted into many templates, including true-or-false questions. Since the information comes from the source document, this avoids relying on a model to invent the underlying facts and avoids the model-collapse problem Morcos describes. The rephrasing model has an easier task: it transforms a document into a new format rather than learning all of the concepts itself. Repeating this across many formats increases diversity and extracts more training value from strong documents. The selection step matters. Random documents are poor candidates for rephrasing, while high-quality documents can produce much better results. ### Domain data works best when mixed with the original distribution [14:10](https://www.youtube.com/watch?v=_PdK6x7PQNM&t=850s) Thomson Reuters used proprietary legal data with DatologyAI to mid-train an open model. After 100 billion tokens, legal benchmark performance rose by about five percentage points, while general capabilities also improved instead of suffering catastrophic forgetting. Morcos attributes this to mixing domain-specific data with data representative of the original pretraining distribution. The legal data was only part of the mix. He also reports that Thomson Reuters' post-training process produced almost three times more gain on the mid-trained model than on the default instruction-tuned model. The model began post-training with a more accurate policy, so the same post-training data became more effective. Morcos uses this to argue that pretraining, mid-training, and post-training should be planned together. ### Public curated data can support frontier-level open models [16:23](https://www.youtube.com/watch?v=_PdK6x7PQNM&t=983s) Morcos describes Arcee's Trinity Large, trained on 17 trillion curated tokens from public datasets. The run used no proprietary data and no closed-model-generated data. He says the model was competitive with the open frontier, matched GLM-5 and Kimi on many tasks, and outperformed Claude on a couple of tasks. Arcee had not trained a model before beginning this work in the middle of the previous year. Across salaries, compute, research and development, repetitions, and other costs for this and several other models, Morcos says the total was less than $20 million. He presents the example as evidence that high-quality data can reduce the cost of building a powerful model, including for teams without a long prior history of model training. ### Repeating high-quality data can beat adding low-quality tokens [17:43](https://www.youtube.com/watch?v=_PdK6x7PQNM&t=1063s) Morcos closes by returning to signal per token. He says it is often better to repeat high-quality data than to add low-quality data, at least up to a threshold. The practical problem is difficult because a useful system must score data across many different axes and apply those judgments to petabytes of material. He describes data quality as a research and engineering problem that combines understanding which examples matter with scaling that process to very large datasets. His advice for teams constrained by compute is to invest in data curation rather than simply seeking more tokens. The aim is to spend each training token where it provides the most information for the intended model and tasks. ## Notable quotes - "Data quality is a compute multiplier because what it does is it makes the learning curve steeper." (01:56) - "We don't source new tokens like many data providers. Rather we take existing tokens coming from public data sets, proprietary data sets and licensed data sets and make them way better." (03:50) - "There's no one golden data set to rule them all that's good for everything no matter what you want to do." (03:04) - "If you use your data correctly and you simulate token scarcity appropriately, you can also get very predictable scaling to much larger models." (10:43) - "Data quality remains the single most underleveraged compute multiplier." (17:50) ## Tools & references mentioned - DatologyAI - Ari Morcos - Beyond Scaling Laws - NeurIPS - H100 - Google - Meta - Gemini - OpenAI - Mammoth - Qwen 3 - Qwen 3.5 4B - Qwen 3.5 - InternVL - GLM-5 - Kimi - Claude - Thomson Reuters - Arcee - Trinity Large - LegalBench - multilingual MMLU ## Who should watch - You are training or customizing a model and your hardware budget is limiting the size of the run. - Your team has large public, licensed, or proprietary datasets but lacks a clear process for filtering, weighting, deduplicating, and sequencing them. - You care about inference cost, multilingual performance, or domain adaptation and want examples of how data choices affect those outcomes.