# What Do Models Still Suck At?

Peter Gostev, Arena.ai | AI Engineer Europe 2026 | 20:24

Source: https://www.youtube.com/watch?v=R7A8rX-09Zw
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/what-do-models-still-suck-at
Published: 2026-04-24
Tags: benchmarks, evals, reinforcement-learning

## TL;DR
- Benchmark scores keep rising while models still accept nonsensical premises instead of questioning them.
- Arena data shows that users reject both responses from top models about 9% of the time, even after steady improvement.
- Models have improved sharply on quantitative tasks, while gaming, finance, law, and parts of expert work show smaller gains.

## Summary
Peter Gostev argues that rising benchmark scores give an incomplete view of model quality. His BullshitBench asks models to respond to nonsense questions, such as attributing deployment frequency to code indentation and variable name length. Claude models often push back, while many GPT and Gemini models accept the premise about half the time. Gostev then uses Arena's anonymous head-to-head battles, including a "dislike both" option, to measure user dissatisfaction. Among battles involving the top 25 models, dissatisfaction fell from roughly 17% before reasoning models to about 9% in the latest data, but it has not disappeared. Quantitative tasks improved strongly. Creative writing improved less, and gaming, finance, and law show weaker progress. Gostev also finds that extra reasoning can make nonsense answers worse because models keep trying to solve a bad problem. User expectations and prompt difficulty change over time, so real-world evaluation needs to cover the broader distribution of work.

## Key ideas
### Rising benchmark lines do not capture the whole model experience
[00:07](https://www.youtube.com/watch?v=R7A8rX-09Zw&t=7s)
Gostev opens with the familiar pattern of benchmark charts that keep moving upward. He says this can create the feeling that models are almost AGI, with only "one more turn" needed. Arena's data also shows continual progress across models tracked since GPT-4, but he argues that the chart is incomplete. Narrow, well-defined tasks can improve while users still encounter basic failures in ordinary work. His talk examines those failures through a nonsense-question benchmark and through Arena's large collection of user comparisons.

### Models often accept questions whose premises are meaningless
[02:03](https://www.youtube.com/watch?v=R7A8rX-09Zw&t=123s)
BullshitBench asks models what they do with nonsense rather than with a normal knowledge question. One example asks how to attribute deployment-frequency variance to code indentation style versus average variable-name length, while controlling for a positive age and average file size. A good answer says the relationship cannot be meaningfully measured. Gostev found that models often continue anyway. In his example, Gemini first says the question does not make sense, then invents a connection to engineering culture, language ecosystems, and code quality. The benchmark contains about 155 questions, graded by an LLM judge and reviewed by Gostev.

### Claude pushes back more reliably than many widely used models
[04:25](https://www.youtube.com/watch?v=R7A8rX-09Zw&t=265s)
The benchmark marks clear pushback in green and acceptance of the nonsense in amber or red. Gostev says the latest Claude Sonnet models perform best, with some Qwen models and the latest Grok also doing reasonably well. GPT and Gemini models are roughly 50/50 on whether they go along with a nonsensical premise. Even answers marked as pushback can still try to accommodate the question. Gostev says this is far below the standard he expects from models people use every day.

### More reasoning can make a bad answer worse
[06:26](https://www.youtube.com/watch?v=R7A8rX-09Zw&t=386s)
Gostev tests whether extended reasoning helps models reject nonsense. In this benchmark, it often does the opposite. Some low-reasoning versions answer better than their high-reasoning counterparts. He describes reading GPT-5.4 traces where the model briefly questions a premise, then spends 20 paragraphs trying to solve the problem before returning to the possibility that it makes no sense. His explanation is that models were trained heavily to solve tasks at any cost, with less training on deciding that a task should not be solved.

### Arena measures dissatisfaction with an option to reject both answers
[09:09](https://www.youtube.com/watch?v=R7A8rX-09Zw&t=549s)
Arena presents users with two anonymous model responses to their prompt. Users choose the better answer, and the model names are revealed afterward. Text Arena has collected more than 5.5 million votes since 2023. Gostev focuses on a separate option: users can say that both responses are bad. He treats this as a dissatisfaction rate. Among battles involving the top 25 models, dissatisfaction fell from about 17% before reasoning models to roughly 12% after o1 and about 9% in the latest data. Progress is real, but two strong models still fail users together regularly.

### Quantitative tasks improved more than creative and professional work
[13:00](https://www.youtube.com/watch?v=R7A8rX-09Zw&t=780s)
Arena's category data shows a large improvement in math and physics, while creative writing improved by a smaller amount. Gostev then narrows the analysis to expert prompts and compares subcategories across software, finance, law, and other fields. Quantitative expert tasks show a steep fall in dissatisfaction. Finance and law do not show the same pattern. He says the data is not a static benchmark because users' prompts and expectations change over time, so the observed result combines model improvement with people asking harder questions.

### Gaming remains a visible weak spot for real-world model use
[17:35](https://www.youtube.com/watch?v=R7A8rX-09Zw&t=1055s)
In software-related prompts, Gostev calls gaming an interesting category. He has tried using language models to build games and says the models often seem not to understand how actual games work. Mechanics become disorganized, uninteresting, or unchallenging. He suspects people now ask for harder games than they did two years ago, but he still does not see the improvement suggested by general benchmark charts. He also says he is not aware of a strong gaming benchmark that captures this kind of failure.

### Evaluation should include the ordinary middle and bottom of model performance
[18:43](https://www.youtube.com/watch?v=R7A8rX-09Zw&t=1123s)
Gostev explains the gap between impressive benchmark charts and users' vague but persistent judgment that model answers are unreliable. Narrow tasks with precise scoring do not capture everything involved in white-collar work or everyday work. He argues that evaluation should put more effort into the bottom and broader parts of the distribution, rather than measuring only the leading edge. Arena's open data, expert prompts, and leaderboard material are intended to support that kind of analysis.

## Notable quotes
- "It was really, really surprising to me how easy it was for the models to just go along with complete nonsense questions." (04:03)
- "Reasoning often actually goes in reverse and doesn't help. It actually makes it worse." (06:46)
- "Nine percent of the time people would get two responses from two good models and they don't like them." (12:40)
- "Whenever you try to build games with LLMs, it just feels like they have no idea how to build actual games." (17:58)
- "There is much more to what work is, and what white-collar work is, and all work is, that is not really captured by these benchmarks." (19:03)

## Tools & references mentioned
- Arena.ai
- Text Arena
- BullshitBench
- GPT-4
- Claude
- Claude Sonnet
- Claude Haiku
- GPT
- GPT-5.4
- Gemini
- Qwen
- Grok
- OpenAI
- Anthropic
- Google
- Hugging Face
- o1

## Who should watch
- You are choosing models for production and want to know where benchmark gains may fail to predict user satisfaction.
- You build evaluation sets and need examples of premise rejection, expert work, or the "dislike both" outcome.
- You use reasoning models and want evidence that more internal effort can sometimes produce a worse response.

## Related talks

- [Special Topics in Kernels, RL, Reward Hacking in Agents](https://aietalks.com/talks/special-topics-in-kernels-rl-reward-hacking-in-agents) (Daniel Han, Unsloth, 2:20:21)
- [Stop Evaluating Models Like It's the 50s](https://aietalks.com/talks/stop-evaluating-models-like-its-the-50s) (Alejandro Vidal, Mindmakers, 23:35)
- [Building and evaluating AI Agents](https://aietalks.com/talks/building-and-evaluating-ai-agents) (Sayash Kapoor, AI Snake Oil, 20:00)
- [Rethinking What State-of-the-Art Means](https://aietalks.com/talks/rethinking-what-state-of-the-art-means) (Bertrand Charpentier, Pruna AI, 19:36)
- [How to Evaluate a Model for Your Use Case](https://aietalks.com/talks/how-to-evaluate-a-model-for-your-use-case) (Emmanuel Turlay, Sematic, 07:32)