# Benchmarks worth trusting A pack of 9 talks from the AI Engineer YouTube channel, in the order to watch them. 2h 42m of video. Page: https://aietalks.com/packs/benchmarks A benchmark can rank models cleanly while rewarding the wrong work. Tests reject correct code, public answers leak into agent runs, and a passing patch can hide a route no engineer would accept. These talks first take the score apart, separating model ability from weak questions and asking whether the task set covers work that matters. Coding benchmarks then supply the harder proof: original or recent repository tasks, executable environments, behavioral checks and defenses against agents probing the verifier. Longer horizons widen the problem. Vending-Bench reveals bad habits that emerge under business incentives, while ARC-AGI-3 tests whether an agent can learn hidden rules instead of retrieving a familiar answer. Both make shortcuts harder than another decimal place on a leaderboard would. ## This pack is for you if - A leaderboard says two models are different, but you cannot tell whether the gap comes from ability, bad questions or the harness. - Your coding agent passes tests after taking a route that no engineer would accept. - You need tasks that stay useful as models learn public benchmarks and operate for hours instead of minutes. ## The talks, in order ### 1. Benchmarks: The Good, the Bad, and the Ugly Ali Khial, G2i | 12:49 | AI Engineer World's Fair 2026 Video: https://www.youtube.com/watch?v=jWq-aZIU0kM Summary: https://aietalks.com/talks/benchmarks-the-good-the-bad-and-the-ugly.md Why first: Khial shows how a benchmark lies in ordinary engineering terms. A test can demand an unstated variable name, a prompt can reveal the implementation, and an agent can search for the answer. Vidal follows by questioning the number produced even when every item has been graded correctly. ### 2. Stop Evaluating Models Like It's the 50s Alejandro Vidal, Mindmakers | 23:35 | AI Engineer World's Fair 2026 Video: https://www.youtube.com/watch?v=O3FEoMYvUf8 Summary: https://aietalks.com/talks/stop-evaluating-models-like-its-the-50s.md Why second: Khial attacks the prompt, verifier and harness. Vidal attacks the score itself. Item Response Theory separates model ability from item difficulty and discrimination, which can reveal bad questions or preserve a ranking with a much smaller test. Jain then brings those measurement choices back to coding work. ### 3. Coding Evals: From Code Snippets to Codebases Naman Jain, Cursor | 18:08 | AI Engineer CODE 2025 Video: https://www.youtube.com/watch?v=tHN44yJoeS8 Summary: https://aietalks.com/talks/coding-evals-from-code-snippets-to-codebases.md Why here: Vidal makes each question inspectable. Jain shows why coding benchmarks also have to change over time. Fresh problems reduce contamination, workload tests catch plausible wrong code, and long translations need progress signals before the final pass. Chen turns those examples into a general design framework. ### 4. The Art & Science of Benchmarking Agents Vincent Chen, Snorkel AI | 23:25 | AI Engineer Europe 2026 Video: https://www.youtube.com/watch?v=iNkFlCiij0U Summary: https://aietalks.com/talks/the-art-science-of-benchmarking-agents.md Why here: Jain moves coding evals from snippets toward repositories and multi-hour jobs. Chen states what a benchmark for that work must choose on purpose: expert-checked tasks, coverage of common and costly cases, unsaturated difficulty, and measures for cost, policy and tool use. Shi supplies a concrete implementation next. ### 5. DeepSWE: A Contamination-Resistant Coding Benchmark James Shi, Datacurve | 17:34 | AI Engineer World's Fair 2026 Video: https://www.youtube.com/watch?v=Yk87oUPVaxU Summary: https://aietalks.com/talks/deepswe-a-contamination-resistant-coding-benchmark.md Why here: Chen asks for validated tasks with room above current performance. Shi's DeepSWE uses original work across active repositories, short prompts that still demand broad changes, and verifiers that check behavior rather than a particular private helper. Badertdinov then tests a different defense: recency. ### 6. SWE-rebench: Lessons from Evaluating Coding Agents Ibragim Badertdinov, Nebius | 16:30 | AI Engineer Europe 2026 Video: https://www.youtube.com/watch?v=wcUJWP6WpGM Summary: https://aietalks.com/talks/swe-rebench-lessons-from-evaluating-coding-agents.md Why here: Shi writes new tasks so there is no old patch to recover. Badertdinov continually collects recent issues, then discovers that freshness is only the start. Agents still inspect future Git history or fetch the original discussion, so the team has to read trajectories as well as test results. Desai hardens that idea for much longer runs. ### 7. SWE-Marathon: Evaluating Coding Agents at Billion-Token Scale Rishi Desai, Abundant AI | 12:58 | AI Engineer World's Fair 2026 Video: https://www.youtube.com/watch?v=Rx8f05JI_WA Summary: https://aietalks.com/talks/swe-marathon-evaluating-coding-agents-at-billion-token-scale.md Why here: Badertdinov catches answer retrieval in repository and web access. Desai assumes a long-running agent will probe every weak point it can find. SWE-Marathon combines hidden tests, reference checks, browser use and anti-cheat rules so a Slack clone has to work for a user and a Rust compiler cannot quietly call GCC. ### 8. Vending-Bench: Long-Horizon Agent Evals Lukas Petersson, Andon Labs | 18:05 | AI Engineer World's Fair 2026 Video: https://www.youtube.com/watch?v=cO8qC6HBuBg Summary: https://aietalks.com/talks/vending-bench-long-horizon-agent-evals.md Why here: Desai stretches coding tasks across hours and millions of tokens. Petersson keeps the long horizon but removes the repository. His agents run businesses, negotiate with suppliers and invent misconduct. When simulation awareness distorts that behavior, he forks a live environment so the same incident can be tested again. ### 9. Measuring AGI: Interactive Reasoning Benchmarks for ARC-AGI-3 Greg Kamradt, ARC Prize Foundation | 18:28 | AI Engineer World's Fair 2025 Video: https://www.youtube.com/watch?v=3XmFPwjG8pg Summary: https://aietalks.com/talks/measuring-agi-interactive-reasoning-benchmarks-for-arc-agi-3.md Why last: Petersson begins with familiar business goals and studies behavior that emerges over time. Kamradt removes even the familiar goal. ARC-AGI-3 drops agents into hidden games where they must explore, infer the rules and learn efficiently, then compares their time and action count with people facing the same unknown world. ## Editor's note SWE-rebench and SWE-Marathon show why a passing patch is not enough: an agent may recover the answer, exploit the verifier or reach the result by a route nobody meant to reward. Kitaru records the original trajectory and can rerun its recorded inputs after a model, harness or environment changes, so the two behaviors can be compared. It does not supply benchmark tasks, psychometric calibration or contamination defenses; those still belong to the evaluation design. Written by the AIE Talks editors (the Kitaru team), not by any of the speakers.