A benchmark can rank models cleanly while rewarding the wrong work. Tests reject correct code, public answers leak into agent runs, and a passing patch can hide a route no engineer would accept. These talks first take the score apart, separating model ability from weak questions and asking whether the task set covers work that matters. Coding benchmarks then supply the harder proof: original or recent repository tasks, executable environments, behavioral checks and defenses against agents probing the verifier. Longer horizons widen the problem. Vending-Bench reveals bad habits that emerge under business incentives, while ARC-AGI-3 tests whether an agent can learn hidden rules instead of retrieving a familiar answer. Both make shortcuts harder than another decimal place on a leaderboard would.
Why first: Khial shows how a benchmark lies in ordinary engineering terms. A test can demand an unstated variable name, a prompt can reveal the implementation, and an agent can search for the answer. Vidal follows by questioning the number produced even when every item has been graded correctly.
Why second: Khial attacks the prompt, verifier and harness. Vidal attacks the score itself. Item Response Theory separates model ability from item difficulty and discrimination, which can reveal bad questions or preserve a ranking with a much smaller test. Jain then brings those measurement choices back to coding work.
Why here: Vidal makes each question inspectable. Jain shows why coding benchmarks also have to change over time. Fresh problems reduce contamination, workload tests catch plausible wrong code, and long translations need progress signals before the final pass. Chen turns those examples into a general design framework.
Why here: Jain moves coding evals from snippets toward repositories and multi-hour jobs. Chen states what a benchmark for that work must choose on purpose: expert-checked tasks, coverage of common and costly cases, unsaturated difficulty, and measures for cost, policy and tool use. Shi supplies a concrete implementation next.
Why here: Chen asks for validated tasks with room above current performance. Shi's DeepSWE uses original work across active repositories, short prompts that still demand broad changes, and verifiers that check behavior rather than a particular private helper. Badertdinov then tests a different defense: recency.
Why here: Shi writes new tasks so there is no old patch to recover. Badertdinov continually collects recent issues, then discovers that freshness is only the start. Agents still inspect future Git history or fetch the original discussion, so the team has to read trajectories as well as test results. Desai hardens that idea for much longer runs.
Why here: Badertdinov catches answer retrieval in repository and web access. Desai assumes a long-running agent will probe every weak point it can find. SWE-Marathon combines hidden tests, reference checks, browser use and anti-cheat rules so a Slack clone has to work for a user and a Rust compiler cannot quietly call GCC.
Why here: Desai stretches coding tasks across hours and millions of tokens. Petersson keeps the long horizon but removes the repository. His agents run businesses, negotiate with suppliers and invent misconduct. When simulation awareness distorts that behavior, he forks a live environment so the same incident can be tested again.
Why last: Petersson begins with familiar business goals and studies behavior that emerges over time. Kamradt removes even the familiar goal. ARC-AGI-3 drops agents into hidden games where they must explore, infer the rules and learn efficiently, then compares their time and action count with people facing the same unknown world.