# Agentic Evaluations at Scale, For Everybody

Nicholas Kang & Michael Aaron, Google DeepMind | AI Engineer Europe 2026 | 20:03

Source: https://www.youtube.com/watch?v=Ubwb6NzegyA
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/agentic-evaluations-at-scale-for-everybody
Published: 2026-05-25
Tags: agents, benchmarks, evals, open-source

## TL;DR
- AI benchmarks become hard to trust when their setup, model configuration, and harness are not transparent or consistent.
- Kaggle is building open tools for community-created evaluations, including hackathons, standardized agent exams, Game Arena, and a benchmark platform.
- Agent evaluations are especially difficult because the harness, model endpoint, task ambiguity, cost, and changing model versions can all affect the result.

## Summary
Nicholas Kang and Michael Aaron argue that AI evaluations are too scattered, opaque, and concentrated among a small group of researchers. Published leaderboards go stale, model reports often omit important configuration details, and changing the harness can alter an agent's performance substantially. They describe Kaggle's efforts to open the process through hackathons, standardized agent exams, Game Arena, and a platform where anyone can build, run, and share evaluations. The examples range from a wastewater treatment safety benchmark created by an engineer in Turkey to PvP games such as poker, chess, and Werewolf. The talk is honest about the engineering costs. Game Arena needed about 400,000 poker hands for statistical significance, while community benchmarks need incentives, accessible tools, expert review, and ways to compare changing models over time. For agentic systems, the central problem is deciding whether an evaluation measures the model, the harness, or both.

## Key ideas
### AI benchmarks are scattered and become stale after publication
[02:01](https://www.youtube.com/watch?v=Ubwb6NzegyA&t=121s)
Nicholas Kang says more than 10 AI benchmarks can appear every day, leaving people to search arXiv for hours to find them. Once a paper is published, its leaderboard often stops receiving attention because the authors move on to another benchmark. The result is a collection of scores that quickly loses relevance. Kaggle's team wants evaluations to remain available and useful to a wider community instead of disappearing into individual papers. This problem is separate from model quality: even a carefully designed benchmark can become outdated when the field changes and no one maintains it.

### Benchmark scores can change when the evaluation setup changes
[02:50](https://www.youtube.com/watch?v=Ubwb6NzegyA&t=170s)
Kang describes a benchmark Kaggle published with an AI lab. A competing lab reran it and reported much better results after optimizing the setup for its own model. The difference was compaction provided through that lab's API, which Kaggle had not used for all the models in the original run. Kang's point is that the score depends on model settings, orchestration, and the harness, not only on the underlying model. Without those details, a chart in a model release note does not reveal exactly what was tested or allow a fair comparison.

### Open evaluation needs contributions from people outside AI research
[03:44](https://www.youtube.com/watch?v=Ubwb6NzegyA&t=224s)
Kang argues that a small population of AI researchers and technical professionals is creating evaluations for systems expected to help much more of humanity. Areas that nobody benchmarks cannot improve through measurement. He gives the example of a wastewater treatment plant engineer in Turkey who has spent 20 years in the field and built a safety benchmark from his own experience. The dataset reflects incidents where people died after safety protocols were not followed, and it does not exist elsewhere on the web. Kaggle sees open contributions as a way to bring this kind of specialized knowledge into AI evaluation.

### Hackathons can turn specialized knowledge into open benchmarks
[05:52](https://www.youtube.com/watch?v=Ubwb6NzegyA&t=352s)
Kaggle provides a platform where anyone can host a hackathon. Kang says the format can focus energy and expertise on a defined problem while leaving room for creative solutions. The outputs are intended to be open source. He describes a current hackathon with the Google DeepMind AGI team, based on a paper about measuring the cognitive faculties of AGI. Participants are working on five of the paper's ten faculties. The platform also has to provide practical support, such as dataset hosting and access to AI models, because some participants cannot afford several API keys. Human experts are still needed to judge creativity and innovation, and agreement between experts is difficult.

### A standardized agent exam gives ordinary users a quick baseline
[09:45](https://www.youtube.com/watch?v=Ubwb6NzegyA&t=585s)
Kang describes standardized agent exams as an experimental product launched the previous week. A user pastes a one-line prompt for an agent, and the system runs an exam and returns a leaderboard score. He sees a gap between sophisticated evaluation systems used by research labs and enterprises and consumer agents that are released without much testing. A safety-focused exam could give users a baseline before an agent gets access to an inbox or an Amazon account. The exam must be difficult enough to produce a useful signal without taking so long that people abandon it. More than 500 agents had already been evaluated without much promotion.

### PvP games keep a benchmark moving after fixed tests saturate
[12:19](https://www.youtube.com/watch?v=Ubwb6NzegyA&t=739s)
Michael Aaron presents Game Arena as a way to reduce saturation in benchmarks. Models play against one another in games such as Werewolf, poker, and chess, producing ELO-style ratings. A PvP system always has a winner and a loser, so a model can continue competing even after it has mastered a fixed test. The games are chosen to probe different abilities. Werewolf examines deception, poker introduces randomization and deception, and chess provides a familiar test of planning. Aaron says model personalities appear in poker, with some models going all in more often and newer models sometimes acting more risk-averse.

### Game Arena trades statistical confidence against high operating costs
[15:03](https://www.youtube.com/watch?v=Ubwb6NzegyA&t=903s)
The Game Arena system designs and iterates on games, checks that models can play them, tunes prompts for fairness, runs simulations, and publishes conversations and scores. It uses OpenSpiel games, a model proxy available in Colab, Kaggle's simulation platform, and Bradley-Terry pairwise scheduling to reduce the number of games required. Even so, poker needed about 400,000 hands to reach statistical significance, with many turns inside each hand. Aaron wants methods that preserve confidence without requiring millions of games. The team also has to keep the experience interesting for Kaggle users and deal with old models disappearing, new models arriving, and endpoints that may not clearly identify the model running behind them.

### Agent benchmarks must separate model ability from harness behavior
[18:39](https://www.youtube.com/watch?v=Ubwb6NzegyA&t=1119s)
Aaron says Kaggle's benchmark platform is aimed at community involvement rather than production evaluation. Users define assertions or use LLM judging, group checks into tasks, run them against selected models, and combine tasks into a benchmark. Agentic evaluation makes this harder because it is unclear what the test measures. He cites a claim about SWE-Bench Pro where six frontier models are within a few percentage points, while the harness changes performance by 22 percent. That creates a basic measurement problem: a result may reflect the harness rather than the model. Ambiguous tasks, rapid model releases, and deprecations make comparisons over time harder as well.

## Notable quotes
- "If something's not being evaluated, not being benchmarked, we cannot hill climb on it, we cannot know how good we are at those things." (04:02)
- "So, this is like a proprietary novel data set that he's created from his own experience. Doesn't live anywhere else on the web." (05:15)
- "You just paste a one-line prompt your agent, and essentially it takes an exam, and we return a score for you on a leaderboard that you can compare its performance against." (09:46)
- "So, for poker in order to get statistical significance, we had to run about 400,000 poker hands." (15:19)
- "Are you testing the harness? Are you checking the model?" (19:02)

## Tools & references mentioned
- Kaggle
- Google DeepMind
- Google DeepMind AGI team
- SWE-Bench Pro
- OpenSpiel
- Colab
- Bradley-Terry
- ELO
- Werewolf
- poker
- chess
- Mopbook
- Morph LLM
- XKCD

## Who should watch
- You are publishing model or agent scores and need to understand how configuration and harness choices affect comparisons.
- You want to contribute a benchmark based on specialist knowledge that is not represented in existing AI datasets.
- You are building consumer agents and want a lightweight exam or safety baseline before giving them access to real accounts.

## Related talks

- [Building and evaluating AI Agents](https://aietalks.com/talks/building-and-evaluating-ai-agents) (Sayash Kapoor, AI Snake Oil, 20:00)
- [The Art & Science of Benchmarking Agents](https://aietalks.com/talks/the-art-science-of-benchmarking-agents) (Vincent Chen, Snorkel AI, 23:25)
- [Evals Are Broken, Use Them Anyway](https://aietalks.com/talks/evals-are-broken-use-them-anyway) (Ara Khan, Cline, 19:04)
- [Einstein Arena: Harnessing Collective Agent Intelligence for Open Science](https://aietalks.com/talks/einstein-arena-harnessing-collective-agent-intelligence-for-open-science) (James Zou, Together AI, 16:56)
- [Agentic Excellence: Mastering AI Agent Evals with Azure AI Evaluation SDK](https://aietalks.com/talks/agentic-excellence-mastering-ai-agent-evals-with-azure-ai-evaluation-sdk) (Cedric Vidal, Microsoft, 20:01)