# Rethinking What State-of-the-Art Means Bertrand Charpentier, Pruna AI | AI Engineer Europe 2026 | 19:36 Source: https://www.youtube.com/watch?v=hqHC6Z_lXyo Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/rethinking-what-state-of-the-art-means Published: 2026-06-01 Tags: benchmarks, cost, evals, latency ## TL;DR - A model's leaderboard position depends on the leaderboard, the task, and the evaluation set, so one ranking cannot identify the best model for every use case. - Model evaluation should include many samples, human preferences, task-specific metrics, and the compute required to produce the results. - Pareto plots reveal several state-of-the-art models with similar quality and large differences in latency or price, often favoring smaller specialized models. ## Summary Bertrand Charpentier argues that "state of the art" should not mean the model at the top of one public leaderboard. Different leaderboards rank image editing models differently, and rankings also change by task. Manual inspection is shaped by the evaluator's preferences and by the small set of samples they happen to see. Automated metrics can disagree or show tiny differences that are hard to interpret. Charpentier also argues that quality scores hide the cost of evaluation and inference. Design Arena's 26,000 battles for ChatGPT image took 20 days of compute and cost $5,000, while a compressed model could run the same evaluation in 7 hours for $265. He recommends plotting quality against latency or price and selecting from the Pareto front. This approach can identify several models with similar quality but very different efficiency, including small models specialized for a particular task. ## Key ideas ### A public leaderboard gives a quick choice, not a universal answer [00:14](https://www.youtube.com/watch?v=hqHC6Z_lXyo&t=14s) Charpentier begins with the common practice of selecting the top model on a public leaderboard. For image editing, he uses Design Arena as an example and says choosing ChatGPT image is a reasonable low-effort decision. The limitation is that the leaderboard does not capture how a particular product's users will interact with the model. Different public leaderboards also disagree. He names LM Arena, Design Arena, and Artificial Analysis, and notes that one model called Human moves from rank 10 on Artificial Analysis to rank 5 on Arena. Some models appear on one leaderboard and not another, while Elo scores use different ranges. A top position therefore does not establish overall superiority. ### The relevant ranking depends on the task the product performs [04:05](https://www.youtube.com/watch?v=hqHC6Z_lXyo&t=245s) Aggregated scores combine tasks such as object removal, background changes, and text editing, but a product may need only one of them. Charpentier says task-specific leaderboards produce very different rankings. ChatGPT image is not consistently first, and different models perform well at different jobs. He attributes this to differences in training, since some models have been trained more heavily on particular tasks. His advice is to evaluate the operation the product will actually perform. A team focused on object removal should use an object-removal evaluation rather than relying on a general image-editing score. ### Small leaderboard samples can miss the conditions of a real service [05:45](https://www.youtube.com/watch?v=hqHC6Z_lXyo&t=345s) Charpentier questions whether public leaderboards have enough samples to say much about a specific application. He points to Artificial Analysis evaluations built from a few thousand samples for each item, which may be small compared with the inference volume of a deployed service. He says some Pruna AI models handle millions of inferences per day, so a team may learn more from its own API traffic than from a public leaderboard. Win rates create another problem. Models rarely approach a 100 percent win rate, and most lose at least 40 percent of their battles. If a product's use cases fall into those losses, selecting the highest-ranked model means selecting the wrong one for that portion of the work. ### Manual comparisons are shaped by personal preference and sample selection [07:33](https://www.youtube.com/watch?v=hqHC6Z_lXyo&t=453s) Internal evaluation often means looking at a few prompts and images, then forming an intuitive opinion. Charpentier demonstrates why this can mislead. Audience members prefer different images for the same prompt, and some people change their preference when shown another set. He says manual inspection is biased by the evaluator's own taste and by the particular samples selected. Human evaluation can still be useful, but it needs to be scaled and involve many people. A handful of examples can give a feeling for a model, yet it cannot establish which model will be preferred across a wider set of users and prompts. ### Generic metrics can disagree and hide meaningful differences [10:23](https://www.youtube.com/watch?v=hqHC6Z_lXyo&t=623s) Charpentier describes an automated comparison of eight models using CLIP score across different datasets. The rankings change between metrics and datasets, while the numerical differences between models remain small. That makes it difficult to tell which model is actually better from the score alone. He recommends understanding what a metric measures and using several metrics. A task-specific metric can be more useful. For text rendering, he shows an evaluation where the ranking is more consistent and the gaps between models are clearer. The metric should match the behavior the application needs, rather than being selected simply because it is a familiar benchmark. ### Evaluation itself has a compute and energy cost [12:21](https://www.youtube.com/watch?v=hqHC6Z_lXyo&t=741s) Charpentier argues that model quality is often connected to the compute used to generate evaluation samples. He describes 26,000 Design Arena battles for ChatGPT image, with each image taking 62 seconds to generate. The full evaluation takes 20 days of compute, costs $5,000, and uses about 556 kilowatt-hours, which he compares with the energy from running 400 marathons. A faster compressed model can run the same evaluation in 7 hours for $265 and with energy comparable to four marathons. The difference changes what can be evaluated and how often. Efficiency therefore belongs in the comparison rather than being treated as a separate implementation detail. ### Pareto fronts show several valid state-of-the-art choices [14:21](https://www.youtube.com/watch?v=hqHC6Z_lXyo&t=861s) Charpentier's preferred tool is a Pareto plot. The vertical axis contains a quality score, while the horizontal axis contains an efficiency measure such as image-generation latency or price. The Pareto front identifies models for which improving quality would require giving up efficiency, or vice versa. He says the plots commonly contain three or four models with quality scores in a narrow range from 1,100 to 1,200, while efficiency can differ by as much as 20 times. A team can then choose according to its latency or cost limit. The same method can use a task-specific score, such as text-rendering quality, instead of a general capability score. ### Compression produces specialized models that can sit on the frontier [16:09](https://www.youtube.com/watch?v=hqHC6Z_lXyo&t=969s) Charpentier says benchmarking is useful when teams evaluate many samples, match the conditions of the intended use case, use multiple benchmarks, and include efficiency. With this approach, evaluations often surface small performance models rather than one large foundation model. He describes Pruna AI's work on performance models served through endpoints and its open-source material for compressing models. In the question period, he names quantization, pruning, distillation, and caching as compression methods. For image and video generation, reducing denoising steps can cut repeated backbone computations from roughly 20 to 50 steps to far fewer, depending on how aggressively the model is compressed. ## Notable quotes - "It's not because ChatGPT image is ranked top one on one leaderboard that it means that it's the best overall." (04:05) - "Most of the models they lose at least 40% of their battles." (06:46) - "People tend to just look at quality, but it's important not to look only at quality, but also at efficiency." (14:21) - "There is not one single state of the art model, but there are actually multiple of them." (14:43) - "Benchmarking is not dead. We can do it properly and get a lot of useful information out of it." (16:09) ## Tools & references mentioned - Pruna AI - Bertrand Charpentier - Design Arena - LM Arena - Arena - Artificial Analysis - ChatGPT image - CLIP score - Flux - BFL - quantization - pruning - distillation - caching ## Who should watch - You are choosing an image or video model from public leaderboards and need an evaluation that matches one product task. - Your team compares generative models manually or with a generic metric, but the results change across prompts, datasets, or evaluators. - You need to balance output quality against inference latency, evaluation cost, or energy use. ## Related talks - [Benchmarks: The Good, the Bad, and the Ugly](https://aietalks.com/talks/benchmarks-the-good-the-bad-and-the-ugly) (Ali Khial, G2i, 12:49) - [Trends Across the AI Frontier](https://aietalks.com/talks/trends-across-the-ai-frontier) (George Cameron, ArtificialAnalysis.ai, 17:52) - [How to Evaluate a Model for Your Use Case](https://aietalks.com/talks/how-to-evaluate-a-model-for-your-use-case) (Emmanuel Turlay, Sematic, 07:32) - [What Do Models Still Suck At?](https://aietalks.com/talks/what-do-models-still-suck-at) (Peter Gostev, Arena.ai, 20:24) - [Stop Evaluating Models Like It's the 50s](https://aietalks.com/talks/stop-evaluating-models-like-its-the-50s) (Alejandro Vidal, Mindmakers, 23:35)