Counting correct answers treats every benchmark item as equally important, even when items differ in difficulty and quality.
2
Item Response Theory estimates both model ability and item properties, producing calibrated scores with likelihood intervals.
3
Adaptive testing and residual analysis can shrink benchmarks, detect bad items and contamination, and reveal how models differ.
Summary
Alejandro Vidal argues that LLM evaluation still relies too heavily on Classical Test Theory: count correct answers, divide by the number of questions, and treat the result as a useful score. He proposes borrowing Item Response Theory from psychometrics. IRT estimates an item's difficulty and discrimination, then places each model on a shared ability scale with an uncertainty interval. This makes it possible to audit benchmark items, remove noisy or mislabeled questions, and select a much smaller set that preserves the original ranking. Vidal also uses residuals, the gap between expected and observed answers, to find unexpected model behavior, possible inference problems, and benchmark contamination. Adaptive testing can give each organization a private fingerprint set while retaining a shared anchor set. He closes with ideas involving group bias, model relationship fingerprints, multidimensional ability, merged benchmarks, and alignment measurement.
Raw accuracy assumes that every benchmark item carries the same information
Vidal says the common approach is Classical Test Theory, which counts the number of right answers. Splitting a score into its individual questions exposes the assumption underneath it: every question has the same weight. That is a poor fit for real benchmarks because some questions are harder, some are more useful, and some may be mislabeled. A benchmark can therefore produce a misleading ranking even when it contains many questions. Vidal starts from real data from Epoch AI and argues that evaluation should model each question as its own variable instead of collapsing all responses into one percentage.
IRT puts item difficulty and model ability on one shared scale
Item Response Theory assigns each item a difficulty parameter, B, and maps model ability, theta, to the probability of a correct answer. The B value is the point where the curve crosses a 50% chance of success. Vidal also introduces A, the item's discrimination parameter. A steep curve separates models with different ability levels, while a flat or negatively correlated curve provides little useful signal or may behave in the wrong direction. Theta estimates a model's ability across all item curves, and the resulting distribution gives a likelihood interval rather than only a single score.
Calibrated scores can reverse a ranking based on correct-answer counts
Vidal compares Claude Opus 4.1, with 245 right answers, and Gemini 3 Pro, with 247, across 337 questions. Their raw scores are close, but the IRT estimates differ by almost one standard deviation. His explanation is that the models may be answering different parts of the difficulty range. One model could get more easy questions while another gets fewer but harder questions. IRT captures that distinction through theta and the item parameters. The example shows why a percentage can hide what a model actually managed to answer.
Once every item has difficulty and discrimination values, Vidal can inspect the benchmark itself. Highly discriminative items provide useful signal. Items near zero are noisy, and items below zero may correlate negatively with model ability. He uses those results to flag questions for review by another LLM. One example has an incorrect gold answer. Another asks for the number of passengers but labels the total number of passengers and crew as correct. These checks can find mislabeled questions and identify items that should be rewritten or removed.
A smaller item set can preserve a benchmark's model ranking
Vidal describes selecting items in order of discrimination to reduce the cost of an evaluation. In one real benchmark, he targets a 99% correlation with the original ranking and obtains about 97 items instead of 484, while keeping nearly the same ranking. Random selection performs much worse in his comparison. He cautions that this result depends on the benchmark. GPQA has items that are individually discriminative and do not overlap much, so random subsets perform similarly. A reduced or rotating subset is useful when many items repeat the same information.
Residuals expose unexpected answers and possible system failures
IRT predicts how likely a model is to answer each item correctly. Vidal compares those predictions with the actual answers and calculates residuals, or errors, for individual questions. Gemini 3 Pro gives a wrong answer on an item where the model predicts an 86% chance of success. DeepSeek R1 gives a correct answer where success is not expected. A single outlier does not prove leakage or overfitting, since repeated sampling could change the result, but residuals give evaluators a way to find unusual behavior. Inference errors, incorrect quantization, or a malfunctioning platform could create broader inconsistent patterns.
Adaptive testing can protect private benchmarks from contamination
Vidal proposes an adaptive setup with a shared anchor set and private fingerprint sets. The anchor items are representative and appear for every organization. Each organization also receives a different private set, with especially difficult items kept private. When new models are evaluated later, unusually large residuals on one organization's fingerprint set can indicate that the organization has seen or trained on those items. Vidal calls the method imperfect, but says it can help protect an expensive benchmark from leakage while still allowing repeated evaluation.
Model error patterns can reveal group bias and model lineage
The same item-level analysis can compare groups of models, such as open-weight and closed-weight systems. Vidal fits separate response curves and looks for gaps between them. A gap near zero suggests similar behavior, while larger gaps identify items that favor one group. He also correlates residual patterns to create a model fingerprint. Models from the same lab, different versions of one model, distillations, and different effort levels can have related error patterns. Vidal gives a correlation of .38 between two models with the same base and suggests this could help detect distillation without consent.
"This is not bulletproof, but this is a very good technique that you can use to protect your benchmarks."17:51
Who should watch
You maintain an internal benchmark and need to cut evaluation cost without losing its model ranking.
Your benchmark contains questionable labels, noisy items, or repeated questions that may be hiding useful differences.
You need to investigate contamination, inference problems, open-weight versus closed-weight behavior, or possible model distillation.
