# RAG Evaluation Is Broken! Here's Why (And How to Fix It)

Yuval Belfer & Niv Granot, A21 Labs | AI Engineer World's Fair 2025 | 10:58

Source: https://www.youtube.com/watch?v=Ywl4LsvHKzU
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/rag-evaluation-is-broken-heres-why-and-how-to-fix-it
Published: 2025-06-03
Tags: evals, rag, structured-outputs

## TL;DR
- Most RAG benchmarks use questions whose answers fit inside one or a few chunks, so they do not reflect messy real-world data.
- Aggregation questions expose the limits of top-k retrieval, because relevant evidence may be spread across many documents.
- For corpora with stable structure, converting documents into schemas and querying them with text-to-SQL can work better than standard RAG.

## Summary
Yuval Belfer and Niv Granot argue that RAG evaluation rewards systems for solving artificial local questions. Typical benchmarks assume that an answer is contained in one chunk, then test retrieval or generation separately. This misses parsing, chunking, aggregation, and questions whose answers span a corpus. The speakers show the problem with a FIFA World Cup corpus containing 22 Wikipedia documents. Standard RAG pipelines answered only 5% and 11% of their questions correctly. Their proposed approach is to identify structured corpora, derive a schema during ingestion, populate it from each document, and use text-to-SQL at query time. This fits questions involving counts, maximums, minimums, and calculations. The approach has limits. Some corpora do not have a useful relational structure, and issues such as normalization, ambiguity, schema complexity, and abstention remain. Their broader advice is to evaluate each client's data and use methods beyond chunking, embedding, retrieval, and reranking when the questions require it.

## Key ideas
### Local benchmark questions hide the hard parts of RAG
[00:00](https://www.youtube.com/watch?v=Ywl4LsvHKzU&t=0s)
The speakers say most RAG benchmarks use local questions with local answers. They start from a long document, find a question whose answer appears somewhere inside it, and treat that passage as the gold answer. Some benchmarks use complicated multi-hop questions, but the speakers call examples such as matching a future wife's name to the mother of a US first lady manufactured and unrealistic. These tests also leave out chunking, parsing, and cases where a question does not point to one location in the data. Real data is messier and differs across datasets, so benchmark performance does not generalize reliably.

### Benchmark scores create a cycle of disappointment
[02:20](https://www.youtube.com/watch?v=Ywl4LsvHKzU&t=140s)
Teams need something to optimize, so they build RAG systems around flawed benchmarks. After getting high scores, they celebrate the result in places such as Slack, LinkedIn, or Twitter. When users try the system or the team tests it on customer data, the system struggles. The team then creates another benchmark and optimizes for that one, even though it may have the same flaws. The speakers describe this as a repeated cycle built around pipelines with local questions and local answers.

### Aggregation questions defeat top-k retrieval
[03:25](https://www.youtube.com/watch?v=Ywl4LsvHKzU&t=205s)
Financial and other large corpora contain questions that require aggregation, such as which company reported the highest quarterly revenue most often or how many years Apple exceeded $100 billion in annual revenue. A top-k retriever can return ten chunks while the relevant company or document appears in the eleventh or twelfth result. The same problem occurs with questions about all Fortune 500 companies. The answer requires considering the corpus as a whole, rather than compiling an answer from a small fixed set of retrieved chunks.

### A small FIFA corpus exposes the gap
[04:40](https://www.youtube.com/watch?v=Ywl4LsvHKzU&t=280s)
The speakers built a corpus of 22 historical FIFA World Cup Wikipedia pages and asked questions such as which team won the tournament most often and how many World Cups Brazil entered. They used Wikidata and Kaggle knowledge bases for the answers. Common RAG pipelines, including examples associated with LangChain, LlamaIndex, and Open Responses, answered only 5% and 11% of the questions correctly. These questions had answers available in local segments, yet the systems still performed very poorly.

### Structured corpora can move the work into a database
[05:30](https://www.youtube.com/watch?v=Ywl4LsvHKzU&t=330s)
For corpora with a regular structure, the speakers propose converting unstructured documents into data rather than answering every question from page chunks. During ingestion, the system clusters documents into subcorpora, identifies a schema for each one, populates that schema from every document, and stores the result in a database. For the FIFA corpus, the schema can include the year, winner, top three teams, top scorer, and other attributes. At query time, the system identifies the relevant schema and turns the question into SQL. A question about the team with the most wins then becomes a database query.

### Schema construction brings its own ambiguities
[07:58](https://www.youtube.com/watch?v=Ywl4LsvHKzU&t=478s)
The database approach does not fit every corpus or query. Even the FIFA example raises normalization questions. If a host country is listed as West Germany, should it count as Germany when a user asks about Germany? Should the host field be singular when South Korea and Japan hosted together? The system also needs a way to handle questions outside the corpus, such as whether Real Madrid won a 2006 final. The speakers warn that language models tend to please users, which can lead to answers when the data should prompt abstention.

### The ingestion and query tradeoffs remain real
[09:17](https://www.youtube.com/watch?v=Ywl4LsvHKzU&t=557s)
Clustering documents and inferring schemas introduces a tradeoff between a fine-grained schema and the compute spent during ingestion. A complex schema also makes text-to-SQL harder. The speakers present their approach as a fit for particular structured settings, rather than a general replacement for RAG. Their conclusion is that each client and corpus needs separate treatment, and that the standard sequence of chunking, embedding, retrieving, and reranking is insufficient for many questions.

## Notable quotes
- "The answer is not as simple and really that's the main problem here that benchmarks do not correlate with real world data that real world data is messier." (01:55)
- "So we create new benchmarks, optimize for them, and we're showing that we're the best. But those benchmarks probably have the same problems." (02:57)
- "It's just like how many times or it's counting questions or max and min questions or calculation questions and you just it does just doesn't make sense to try to answer these questions by going over the pages or specific chunks of the pages." (05:56)
- "RAG is not a one-size-fits-all system and you have to account for every client separately." (09:45)

## Tools & references mentioned
- A21 Labs
- LangChain
- LlamaIndex
- Open Responses
- Wikidata
- Kaggle
- FIFA World Cup
- SQL

## Who should watch
- You are evaluating a RAG system with high retrieval or generation scores, but users still report that answers are incomplete or wrong.
- Your data contains financial, historical, or other corpus-wide questions involving counts, comparisons, maximums, or minimums.
- You are deciding whether to keep improving chunking and retrieval or move part of the workload into a structured database.

## Related talks

- [Your RAG is Tripping, Here's the Real Reason Why](https://aietalks.com/talks/your-rag-is-tripping-heres-the-real-reason-why) (Benjamin, EyeLevel.ai, 06:13)
- [Building Production-Ready RAG Applications](https://aietalks.com/talks/building-production-ready-rag-applications) (Jerry Liu, LlamaIndex, 18:35)
- [Navigating RAG Optimization with an Evaluation Driven Compass](https://aietalks.com/talks/navigating-rag-optimization-with-an-evaluation-driven-compass) (Atita Arora, Qdrant & Deanna Emery, Quotient AI, 18:14)
- [The RAG Stack We Landed On After 37 Fails](https://aietalks.com/talks/the-rag-stack-we-landed-on-after-37-fails) (Jonathan Fernandes, 18:52)
- [open-rag-eval: RAG Evaluation without "golden" answers](https://aietalks.com/talks/open-rag-eval-rag-evaluation-without-golden-answers) (Ofer Mendelevitch, Vectara, 05:03)