# The Art & Science of Benchmarking Agents Vincent Chen, Snorkel AI | AI Engineer Europe 2026 | 23:25 Source: https://www.youtube.com/watch?v=iNkFlCiij0U Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/the-art-science-of-benchmarking-agents Published: 2026-06-04 Tags: agents, benchmarks, evals ## TL;DR - Agent capabilities are improving faster than the field's ability to measure how they behave in real deployments. - Useful benchmarks need carefully validated tasks, deliberate task distributions, enough difficulty to leave model headroom, and evaluation methods that measure the dimensions that matter. - The next generation of benchmarks should test environment complexity, autonomy horizon, and outputs that go beyond plain text. ## Summary Vincent Chen argues that benchmarks have fallen behind agent capabilities. Enterprises may be impressed by coding agents, yet still hesitate to deploy them in high-stakes settings because measurement does not capture real-world behavior well enough. Drawing on more than 120 applications to Snorkel AI's $3 million Open Benchmarks Grant, he presents a framework for building useful benchmarks. The science includes expert-validated tasks, intentional coverage of real-world distributions and rare failure modes, unsaturated difficulty, and evaluation methods that measure cost, latency, tool use, policy adherence, and other relevant dimensions. The art is choosing a thesis about where capabilities are going, creating research roadmaps, and making the benchmark easy for researchers and builders to run, extend, and use for training. Chen's proposed frontier is more complex environments, longer autonomy horizons, and richer outputs such as artifacts, uncertainty signals, and requests for help. ## Key ideas ### Agent measurement is lagging behind agent capability [01:52](https://www.youtube.com/watch?v=iNkFlCiij0U&t=112s) Chen describes an asymmetry in real deployments. Models are improving, especially in coding, and their behavior feels better as teams compare model cards. Enterprises still hesitate to release agents into high-stakes areas such as finance, insurance, and healthcare. The problem is not simply that capabilities are missing. Chen says the ability to measure agents in practice is falling behind the capabilities themselves. He argues that field deployments, red teaming, private human evaluations, crowdsourced labeling, and open benchmarks all belong in the measurement toolkit. Open benchmarks matter because the strongest ones do more than record past progress. They define goals and influence what researchers build next. ### Individual tasks need expert validation and clear solutions [06:35](https://www.youtube.com/watch?v=iNkFlCiij0U&t=395s) A useful benchmark starts with high-quality individual tasks. Chen says tasks should represent real-world complexity, have well-structured instructions, and have solutions that can be checked. Domain experts should validate those solutions. He points to GPQA, whose quality process used an original author, reviewers, adjudicators, and opportunities for revision. The task authors had to show that other experts could solve the questions, which was difficult because the questions tested graduate-level and professional knowledge. Chen also praises the benchmark's incentive design, where payouts depended partly on agreement. He treats this multi-expert process as a foundation for trustworthy measurement. ### Benchmarks should represent the important parts of a task distribution [08:20](https://www.youtube.com/watch?v=iNkFlCiij0U&t=500s) Chen says benchmark builders should define a taxonomy for the domain and distribute tasks intentionally across it. They can model the normal stream of work an agent encounters, or focus on failure modes that are rare but costly in production. His self-driving example is yellow lights, pedestrians, and motorcyclists. These cases may appear less often than ordinary scenarios, yet errors on them matter greatly. He cites MMLU's coverage of 57 academic and professional domains as an example of deliberate taxonomy design. The point is to make the benchmark's coverage explainable rather than letting the task mix emerge accidentally. ### Difficulty must leave room above current model performance [09:37](https://www.youtube.com/watch?v=iNkFlCiij0U&t=577s) A benchmark loses value when models saturate it. Chen wants tasks that expose weaknesses and separate systems near the frontier. He uses ARC-AGI 2 as an example of a benchmark that remained unsaturated for a long period, then showed a large capability change during the reasoning push. Its tasks targeted a form of efficiency that humans could demonstrate while models struggled. Chen also mentions ARC-AGI 3, which launched with every task solvable by humans and frontier models below 1 percent. For him, this gap creates useful headroom. A difficult score is meaningful when it reflects a capability the benchmark was designed to isolate. ### Evaluation should measure the dimensions that determine real success [11:21](https://www.youtube.com/watch?v=iNkFlCiij0U&t=681s) Accuracy alone is often too narrow for agent evaluation. Chen names cost, latency, reasoning traces, intermediate steps, and tool use as possible dimensions, depending on the capability being tested. The method must measure what it claims to measure and do so in a reproducible way. He cites Tau Bench, which evaluates multi-turn task completion with a user simulator and also checks policy constraints. An agent that books the correct flight but violates fare-class rules still fails. The example shows why benchmark designers need to decide which outcomes matter before choosing a score. A completion that breaks the surrounding rules is not a successful task. ### A strong benchmark makes a bet about where the field is going [13:03](https://www.youtube.com/watch?v=iNkFlCiij0U&t=783s) Chen distinguishes benchmarks that measure well from those that shape research. The latter have a thesis about a capability subspace or a future direction. He describes Terminal Bench as a bet on the command-line interface for coding agents and broader computer use. The benchmark assumed the CLI would become an important way for agents to interact with the world, before that direction was obvious. Chen believes the bet has proved consequential as teams build enterprise capabilities around tools such as Claude and Codex. A benchmark with a thesis gives labs a target to improve against and can influence how they organize future systems. ### Researcher experience affects whether a benchmark gets adopted [16:02](https://www.youtube.com/watch?v=iNkFlCiij0U&t=962s) Chen says benchmark builders often underestimate the experience of the people who will use the benchmark. Running models and agents should be simple. Adding tasks should be simple. The signals should also be usable for reinforcement learning or later tuning. He points to HELM from Stanford's CRFM as an early standardized, modular harness for reproducible evaluations. He also cites Harbor, shipped with Terminal Bench 2.0, as evaluation infrastructure used by teams building agents. Interfaces and extension points are part of the benchmark's practical value. If researchers cannot run, extend, or reuse it easily, the benchmark is less likely to become a common target. ### Future benchmarks need harder environments, longer horizons, and richer outputs [18:19](https://www.youtube.com/watch?v=iNkFlCiij0U&t=1099s) Chen proposes three dimensions for the next generation of benchmarks. Environment complexity means including the changing, messy conditions of real work. A real codebase may include organization-specific policies, Slack context, screenshots, flaky tools, distributed CI, human preferences, and parallel contributors. Autonomy horizon measures how long an agent can operate before reliability breaks down, including changes in context, integrations, requirements, and organizational priorities. Output complexity goes beyond chat or documents. Chen wants artifacts, nuanced judgments, uncertainty signals, and behaviors such as stopping or asking for more information. These outputs could support evaluation as well as training rewards. ## Notable quotes - "Our ability to actually measure these agents in practice is falling behind of where the capabilities actually are." (02:28) - "The best open benchmarks aren't just about taking a snapshot of progress looking backwards. They're actually about defining progress and shaping the field and setting a goalpost about where capabilities need to go." (03:22) - "A great benchmark, one that really shapes where all of us are going, is producing new roadmaps." (14:33) - "The axes for the next great benchmarks are threefold: environment complexity, autonomy horizon, and capturing the wide range of output complexity." (18:37) ## Tools & references mentioned - Snorkel AI - Open Benchmarks Grant - GPQA - MMLU - ARC-AGI 2 - ARC-AGI 3 - Tau Bench - Terminal Bench - Terminal Bench 2.0 - SWE-bench - HELM - Stanford CRFM - Harbor - Claude - Codex - ARC Prize Foundation ## Who should watch - You are designing an agent benchmark and need a concrete checklist for task quality, coverage, difficulty, and scoring. - Your benchmark has users, but running it or adding tasks is still difficult. Chen explains why researcher and builder experience affects adoption. - You want to evaluate agents in realistic settings with longer runs, changing environments, policy constraints, or outputs that need human judgment.