Most RAG benchmarks use questions whose answers fit inside one or a few chunks, so they do not reflect messy real-world data.
2
Aggregation questions expose the limits of top-k retrieval, because relevant evidence may be spread across many documents.
3
For corpora with stable structure, converting documents into schemas and querying them with text-to-SQL can work better than standard RAG.
Summary
Yuval Belfer and Niv Granot argue that RAG evaluation rewards systems for solving artificial local questions. Typical benchmarks assume that an answer is contained in one chunk, then test retrieval or generation separately. This misses parsing, chunking, aggregation, and questions whose answers span a corpus. The speakers show the problem with a FIFA World Cup corpus containing 22 Wikipedia documents. Standard RAG pipelines answered only 5% and 11% of their questions correctly. Their proposed approach is to identify structured corpora, derive a schema during ingestion, populate it from each document, and use text-to-SQL at query time. This fits questions involving counts, maximums, minimums, and calculations. The approach has limits. Some corpora do not have a useful relational structure, and issues such as normalization, ambiguity, schema complexity, and abstention remain. Their broader advice is to evaluate each client's data and use methods beyond chunking, embedding, retrieval, and reranking when the questions require it.
Local benchmark questions hide the hard parts of RAG
The speakers say most RAG benchmarks use local questions with local answers. They start from a long document, find a question whose answer appears somewhere inside it, and treat that passage as the gold answer. Some benchmarks use complicated multi-hop questions, but the speakers call examples such as matching a future wife's name to the mother of a US first lady manufactured and unrealistic. These tests also leave out chunking, parsing, and cases where a question does not point to one location in the data. Real data is messier and differs across datasets, so benchmark performance does not generalize reliably.
Teams need something to optimize, so they build RAG systems around flawed benchmarks. After getting high scores, they celebrate the result in places such as Slack, LinkedIn, or Twitter. When users try the system or the team tests it on customer data, the system struggles. The team then creates another benchmark and optimizes for that one, even though it may have the same flaws. The speakers describe this as a repeated cycle built around pipelines with local questions and local answers.
Financial and other large corpora contain questions that require aggregation, such as which company reported the highest quarterly revenue most often or how many years Apple exceeded $100 billion in annual revenue. A top-k retriever can return ten chunks while the relevant company or document appears in the eleventh or twelfth result. The same problem occurs with questions about all Fortune 500 companies. The answer requires considering the corpus as a whole, rather than compiling an answer from a small fixed set of retrieved chunks.
The speakers built a corpus of 22 historical FIFA World Cup Wikipedia pages and asked questions such as which team won the tournament most often and how many World Cups Brazil entered. They used Wikidata and Kaggle knowledge bases for the answers. Common RAG pipelines, including examples associated with LangChain, LlamaIndex, and Open Responses, answered only 5% and 11% of the questions correctly. These questions had answers available in local segments, yet the systems still performed very poorly.
Structured corpora can move the work into a database
For corpora with a regular structure, the speakers propose converting unstructured documents into data rather than answering every question from page chunks. During ingestion, the system clusters documents into subcorpora, identifies a schema for each one, populates that schema from every document, and stores the result in a database. For the FIFA corpus, the schema can include the year, winner, top three teams, top scorer, and other attributes. At query time, the system identifies the relevant schema and turns the question into SQL. A question about the team with the most wins then becomes a database query.
The database approach does not fit every corpus or query. Even the FIFA example raises normalization questions. If a host country is listed as West Germany, should it count as Germany when a user asks about Germany? Should the host field be singular when South Korea and Japan hosted together? The system also needs a way to handle questions outside the corpus, such as whether Real Madrid won a 2006 final. The speakers warn that language models tend to please users, which can lead to answers when the data should prompt abstention.
Clustering documents and inferring schemas introduces a tradeoff between a fine-grained schema and the compute spent during ingestion. A complex schema also makes text-to-SQL harder. The speakers present their approach as a fit for particular structured settings, rather than a general replacement for RAG. Their conclusion is that each client and corpus needs separate treatment, and that the standard sequence of chunking, embedding, retrieving, and reranking is insufficient for many questions.
"It's just like how many times or it's counting questions or max and min questions or calculation questions and you just it does just doesn't make sense to try to answer these questions by going over the pages or specific chunks of the pages."05:56
Who should watch
You are evaluating a RAG system with high retrieval or generation scores, but users still report that answers are incomplete or wrong.
Your data contains financial, historical, or other corpus-wide questions involving counts, comparisons, maximums, or minimums.
You are deciding whether to keep improving chunking and retrieval or move part of the workload into a structured database.
