Public benchmarks can be manipulated through unfair comparisons, privileged access to test data, and model tuning for persuasive style.
2
Goodhart's Law makes benchmark scores unreliable when companies have billions of dollars and market reputation tied to the results.
3
Teams should build continuous evaluations from real production data, then test quality, cost, and latency before deployment.
Summary
Darius Emrani argues that public AI benchmarks have become unreliable because their scores influence investment, acquisitions, enterprise contracts, and developer attention. He describes three ways companies can gain an advantage: comparing an expensive or specially configured model with ordinary configurations from competitors, gaining privileged access to benchmark questions, and tuning models to win human preference judgments through style. Goodhart's Law explains why this happens once scores become targets. Emrani proposes improving public benchmarks with fair computational comparisons, open test data, independent governance, rotating questions, style controls, and public reporting of all attempts. For product teams, his stronger recommendation is to stop relying on generic leaderboards. They should collect real user queries, define application-specific metrics, test several models, systematize the process, and repeat evaluations as models and requirements change. A pre-deployment loop should block releases until they meet a quality bar, then continue monitoring after launch.
Emrani defines a benchmark as a model, a test set, and a metric. Its value comes from applying the same questions and scoring system to different models, like the SAT for different test takers. Those scores now influence billions of dollars in investment, enterprise contracts, developer attention, and acquisitions. He points to Sona acquiring AutoRover after its strong SWE results as an example of how one benchmark can affect a company's position. Once a single number helps define market leaders and competitors, companies have strong reasons to find ways to improve that number.
Selective comparisons can make a model look better than it is
The first manipulation is comparing a company's best setup with competitors' standard setups. Emrani uses xAI's Grok-3 results as an example. The published graphs showed Grok-3 beating other models, but did not include OpenAI o3's high-performance consensus-64 configuration. Consensus-64 runs the model 64 times and uses the consensus answer, so it costs more. That cost difference should be visible. A fair comparison should match the best configuration against the best configuration, or compare standard configurations consistently. Without that rule, a benchmark graph can present an uneven contest as a leadership result.
Benchmark funding can create a trust problem around test access
FrontierMath was presented as a highly protected benchmark for advanced mathematics. Emrani says OpenAI funded it and received access to the full dataset. There was a verbal agreement that OpenAI would not train on the questions, and OpenAI employees described it as a strongly held-out evaluation set. Even if those conditions were followed, the arrangement creates a conflict in perception. The funder could inspect the questions, evaluate models internally, and announce a score before independent verification. Emrani's point is that financial ties between benchmark creators and model companies weaken confidence in the resulting score.
Human preference rankings can reward style instead of correctness
Emrani describes Meta entering 27 Llama 4 Maverick variants into LM Arena, with versions tuned for appeal. In one example, a private version answered a riddle whose answer was 3.145 with a long, emoji-filled and flattering response that made no sense, yet it beat Claude's correct answer. Researchers showed that filtering out length, formatting, and personality effects changed the rankings. GPT-4o Mini and Grok-2 fell, while Claude 3.5 Sonnet rose and tied for first. Emrani compares this with choosing a surgeon for bedside manner instead of surgical skill. The reported 39% SAT score variance from essay length gives a parallel example of style affecting evaluation.
Goodhart's Law explains why high-stakes benchmarks decay
Emrani applies Goodhart's Law to public AI evaluations: when a measure becomes a target, it stops being a good measure. Benchmarks now carry enough financial and reputational weight to become targets worth optimizing. He quotes Andrej Karpathy saying, "I don't really know what metrics to look at right now," John Yang describing SWE-bench as something they "kind of just made up," and Martin Szegedy saying the yardsticks are fundamentally broken. His argument is that benchmark failure follows from the incentives around the scores, rather than from one isolated bad test.
Public benchmarks need fair comparisons, open data, and style controls
Emrani proposes changes to each part of a public benchmark. Model comparisons should use the same computational budget and constraints, with cost-performance trade-offs shown clearly. Test sets should be open source, with methods and code available, while benchmark creators should have no financial ties to the companies being evaluated. Questions should rotate regularly to reduce overfitting. Metrics should control for style effects so engagement does not substitute for substance. Every attempt should be public, which removes the ability to report only a model's best run. He points to style-controlled LM Arena rankings and domain-specific work such as LegalBench, MedQA, and FinTech benchmarks.
Product teams should evaluate models on their own real data
Emrani's practical alternative is a five-step evaluation process. First, collect real production queries. He says five actual queries can be more useful than 100 academic questions. Second, choose metrics that fit the application, such as quality, cost, and latency. A chatbot and a medical diagnosis system need different measures. Third, test several leading models on the team's own data instead of trusting a leaderboard. Fourth, make the evaluation consistent and repeatable, either by building it internally or using a platform such as Scorecard. Fifth, keep iterating because models and product needs change.
Pre-deployment evaluation should continue after release
Scorecard's workflow starts by identifying issues, building improvements, and running evaluations before deployment. The team repeats the cycle, uses the results as feedback, and deploys only after the system reaches its quality bar. Monitoring starts after deployment, so new issues feed into the next evaluation cycle. Emrani says this process separates teams that ship reliable AI from teams that repeatedly fight production problems. It requires more work than checking a leaderboard, but it measures the behavior users actually need. His final rule is to use public benchmarks selectively and judge them by whether they are useful for the decision at hand.
"We're really not measuring which model is most accurate. We're measuring which model is most charming."05:29
Who should watch
You are choosing a model for a production system and need evidence that generic leaderboard results transfer to your own data.
Your team publishes or consumes benchmarks and needs to detect uneven configurations, test-set access, or preference bias.
You are building an evaluation pipeline that must catch quality problems before deployment and continue checking them after release.
