# State of Data Sean Cai, Independent | AI Engineer World's Fair 2026 | 18:22 Source: https://www.youtube.com/watch?v=ZyIoTOAbRfs Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/state-of-data Published: 2026-07-26 Tags: benchmarks, data-pipelines, enterprise, evals ## TL;DR - Process data from live work is more valuable than static datasets because it captures decisions, state changes, failures, recoveries, and outcomes. - Benchmark scores can hide opposite model failures when a single task, scaffold, or leaderboard number stands in for a wider distribution of performance. - Data companies are moving into enterprise infrastructure because durable value comes from pipelines into real work and systems that retrain models as the underlying model changes. ## Summary Sean Cai argues that the valuable part of the data market is moving beyond image labels and other static records. Models need process data from real work, including reasoning traces, decision sequences, state changes, failures, recoveries, and verified outcomes. He separates naturally captured workflows from expert-created examples, and says vendors often sell the latter as if it were the former. Cai uses three forms of verifiability to explain why coding matured before harder fields such as healthcare, law, and taste. He is sharply critical of benchmarks that test isolated questions, reuse vendor-created data, or report one noisy score. His finance examples show models can achieve similar scores while failing in opposite ways. Cai expects data companies to become enterprise businesses, building model-routing, reinforcement-learning data, and retraining infrastructure. His final claim is that the durable asset is a live connection to real work, not a frozen dataset. ## Key ideas ### The data market is fragmenting because quality does not scale with quantity [00:12](https://www.youtube.com/watch?v=ZyIoTOAbRfs&t=12s) Cai says the familiar image-annotation market is the least interesting part of data. The harder problem is data that moves a model from general competence to professional expertise. Large companies once had to source people, build environments, design rewards, and run evaluations themselves because the market was immature. Specialists now outperform them in many of those steps. Cai expects the fragmentation to persist, with labs mandating diversification across roughly 20 to 30 vendors because they do not trust any single supplier to preserve quality at scale. He describes bespoke systems that translate messy business context into evaluations as increasingly important. ### Process data teaches how work gets done [04:19](https://www.youtube.com/watch?v=ZyIoTOAbRfs&t=259s) Cai distinguishes state-based data from process-based data. State data is the final row in an ERP system or the saved output of a task. Process data records the trajectory from a blank page to finished work, including the reasoning trace and sequence of decisions. He calls naturally captured workflow data type one, with GitHub commits and session replays as examples. Type two data is manufactured by placing experts in an artificial setting and asking them to create examples. Cai says type two helped when models were weak, but realism inherited from actual work is what moves systems toward professional performance. ### A live business is a more durable data source than a static dataset [05:15](https://www.youtube.com/watch?v=ZyIoTOAbRfs&t=315s) Static datasets lose value as the frontier moves. Cai says the durable supply of process data is a live business that a model or data company partners with. A dead startup's codebase may contain useful records, but it does not keep producing new examples as the work changes. He also gives a blunt account of the market's incentives: vendors commonly sell type two, contrived data while presenting it as type one, real workflow data. That distinction matters because the acquisition method determines whether the examples continue to match changing tools, practices, and model capabilities. ### Task verifiability predicts which application markets mature first [06:11](https://www.youtube.com/watch?v=ZyIoTOAbRfs&t=371s) Cai builds on Jason Wei's Verifier's Law, which says that the ease of training a model for a task is proportional to how verifiable the task is. He separates verifiability into asymmetry, the difficulty of breaking work into checkable steps; veracity, the level of agreement about what correct means; and proliferation, how often the real world produces fresh verified examples. Coding scores highly on all three because unit tests provide decomposable correctness, developers share a broad definition of working code, and GitHub provides many public examples with commit messages. Cai uses this framework to explain the sequence from coding to search, finance, healthcare, law, cyber, biology, and scientific discovery. ### Many benchmarks measure a narrow scaffold instead of sustained expertise [08:13](https://www.youtube.com/watch?v=ZyIoTOAbRfs&t=493s) Cai says a common benchmark recipe hires domain experts, asks them to use a chat system to create plausible tasks, selects cases where the model diverges, and sells the resulting data to improve the same benchmark. He calls this Goodhart's law with a profit motive. Contrived tests usually ask one isolated, in-distribution question. They do not show whether a model can sustain correct reasoning across a dependent episode. Differences between harnesses and infrastructure also create false positives and false negatives. In his view, one leaderboard score under one scaffold is only one sample from a distribution that nobody measured. ### Similar finance scores can conceal opposite failures [10:18](https://www.youtube.com/watch?v=ZyIoTOAbRfs&t=618s) Cai describes three finance tasks from real-world vendors: an ARR waterfall reconciliation, an LBO valuation memo, and a long-short pair trade for hedge-fund trading. These are long-horizon tasks with deterministic checks paired with an LLM judge. In his example, GPT 5.5 and Opus 4.8 scored within three points on the same task, but failed in opposite directions. GPT handled the arithmetic while Opus handled the methodology. He also says Opus 4.8 performed worse than Opus 4.7 on several rubrics, with over-engineered self-reflection contributing to the difference. Rubric-level analysis reveals post-training choices that a single score hides. ### Data-market spending can indicate the next AI application area [11:57](https://www.youtube.com/watch?v=ZyIoTOAbRfs&t=717s) Cai says data markets can act as an upstream signal for which application-layer products labs will release next. He points to Anthropic's spending on cybersecurity data in January, followed by spending on biological data in March and April, and connects those purchases with later products including Metis, Cybor, and Claude bio or life sciences. His checklist starts by classifying tasks along the three verification axes. For long-horizon work, he looks for enforced step length, varied tool calls, state transitions that constrain later actions, and required failure recovery. Raw data should show sequential decisions, expert actions that can be inferred at each step, and outcomes recorded by independent parties. ### Robotics data vendors cannot ignore unresolved research choices [13:21](https://www.youtube.com/watch?v=ZyIoTOAbRfs&t=801s) Robotics is Cai's counterexample because the right data modality is still unsettled. He names ego data, teleoperation, and UMI as competing possibilities, then criticizes vendors that offer large quantities of poorly specified video. A vendor's choice is entangled with the research question of which modality will work. Cai says environment companies that succeed will look more like research accelerators than generic data suppliers. Their infrastructure needs to help enterprise applications across changing research assumptions. His advice is not to build a business around one robotics modality before the field has settled the underlying question. ### The durable moat is the pipeline into real work [14:22](https://www.youtube.com/watch?v=ZyIoTOAbRfs&t=862s) Cai maps white-collar work against task horizon. Short-horizon work is already addressable, while deep, dependent, long-horizon work contains much of the remaining economic value and data opportunity. The boundary moves as companies build real-world data pipelines. He says successful data companies are moving into enterprise because customers will want to own their model intelligence rather than rent it from a lab. That creates demand for infrastructure that can route work among models, manage reinforcement-learning datasets across base-model changes, and automatically rerun post-training after a model swap. His final distinction is direct: the moat is the pipeline into real work plus retraining infrastructure, not the dataset alone. ## Notable quotes - "The models work now what scares and badly priced is the sort of data that takes some from generalist competence into real expertise." (00:49) - "The only durable supply of it technically is a live business you partner with, not a dead startup's code bases like so many data companies out there are buying today." (05:32) - "A single benchmark number under a single scaffold is like basically one sample from a distribution who's basically with nobody measured." (09:41) - "If you're a builder, your moat's not the data, it's the sort of pipeline into real-world work." (17:26) ## Tools & references mentioned - Scale AI - Remarque - Surge - Antikythera mechanism - GPQA - Jason Wei - Verifier's Law - GitHub - Anthropic - Mercari - Opus 4.7 - Opus 4.8 - GPT 5.5 - Metis - Cybor - Claude bio / life sciences - OpenAI - AWS - Google - GLM 5.2 ## Who should watch - You are building evaluation or reinforcement-learning data and need to distinguish real workflow capture from expert-created examples. - Your team relies on leaderboard scores and wants to understand why isolated benchmarks can miss long-horizon failures. - You are starting a data company or enterprise AI platform and need a view on model portability, retraining, and access to live work.