# Stop Chunking Like It's 2022

Yuval Belfer, AI21 Labs | AI Engineer World's Fair 2026 | 18:01

Source: https://www.youtube.com/watch?v=r9OwPx_HoV0
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/stop-chunking-like-its-2022
Published: 2026-09-16
Tags: rag, search

## TL;DR
- Chunk size depends on the query, so no single fixed size can work best for every question in a corpus.
- Multiscale indexing stores several chunked copies of the same data, queries each copy, and combines document rankings with reciprocal rank fusion.
- Across four datasets, the method improved recall by roughly 20 to 40 percent over fixed chunk sizes, at the cost of two to five times the memory.

## Summary
Yuval Belfer argues that chunking remains a retrieval problem even as teams adopt agentic search. Fixed chunking loses information: large chunks preserve context but weaken focused matches, while small chunks miss answers spread across a scene or document. His experiments duplicate datasets across six window sizes and show that different questions prefer different sizes. An oracle that selects the best size per query beats every fixed choice by 20 to 40 percent of recall. Belfer's practical method indexes every window size, retrieves from all copies, converts chunk results into document rankings, and merges them with reciprocal rank fusion. He reports better recall across QM-Sum, Narrative QA, Seinfeld, and FinanceBench. The tradeoff is extra storage, estimated at two to five times the memory, while parallel retrieval and rank fusion add little latency. The talk is honest that the window sizes and number of copies still need better selection methods.

## Key ideas
### Agentic search does not remove retrieval infrastructure
[00:01](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=1s)
Belfer opens by pushing back on claims that chunking, retrieval-augmented generation, or retrieval itself is dead. Agentic search can use tools such as search and find, but it becomes expensive when the system has a large amount of data and many possible queries. If the underlying data is poorly organized, agents still have to inspect too much material. He compares this with a World Cup directory: a question about which team won the most tournaments requires checking every year and aggregating the answers. The answer cannot be found by opening one folder.

### Chunking is lossy compression
[04:17](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=257s)
Every fixed chunk size throws away some useful information. Large chunks preserve the broader picture, but their embeddings may lose the details needed for a focused question. Small chunks preserve local detail, but the surrounding context disappears and retrieval becomes less efficient. Belfer calls this process lossy compression because no choice keeps everything. He rejects the idea that a corpus has one universally correct size. The best size depends on the question, which creates a problem because indexing happens before the system knows the future queries.

### Different questions prefer different window sizes
[06:13](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=373s)
Belfer's team duplicated several datasets and indexed each copy with a different chunk size. The datasets included QM-Sum, Narrative QA, and an in-house Seinfeld trivia dataset. Two Seinfeld questions illustrate the difference. The question asking for Jerry's favorite church is narrow and has an answer contained in a small passage, so a 100-token chunk places the answer at rank one while larger windows push it below rank 50. The question about Jerry's nemesis and pure evil needs information spread across a scene, so small chunks fail to retrieve the answer.

### The oracle experiment measures the cost of choosing one size
[08:12](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=492s)
The team created an oracle that chooses the best chunk size separately for each query. This does not describe a deployable system because it uses the answer after the fact. It measures how much performance is available if the system could choose correctly. The oracle curve beats the curves for every fixed size. Across datasets, the gap is about 20 to 40 percent in recall. Belfer says this is the cost of choosing an arbitrary value such as 512 or 1,000 tokens when the query is unknown at indexing time.

### Indexing and retrieval have opposite information constraints
[09:59](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=599s)
At indexing time, the system controls chunk size but does not know what users will ask. It can estimate likely queries, but it cannot adapt to every future question. At retrieval time, the system has the query but the chunk size has already been fixed, and rebuilding the index for each query is impractical. Belfer frames this as an information problem at both ends of the pipeline. He mentions contextual retrieval and related approaches that improve each chunk while retaining a fixed window, then proposes keeping multiple window sizes instead.

### Multiscale indexing compares document rankings instead of chunk rankings
[11:02](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=662s)
The proposed method creates several indexed copies of the corpus, each with a different window size, and queries all of them for every user question. Chunk rankings from different window sizes cannot be compared directly because the chunks cover different amounts of text. The system therefore retrieves the whole document associated with each matching chunk. Each index then produces a ranking of the same documents. Reciprocal rank fusion combines those rankings into one list. Belfer describes the implementation as a simple script rather than a model.

### The method improves recall across four datasets
[13:53](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=833s)
Belfer reports tests on QM-Sum, Narrative QA, Seinfeld, and FinanceBench. The combined method matches or beats the best fixed chunk size across recall-at-one through recall-at-ten measurements. He describes improvements in the range of 20 to 40 percent on many results, with additional MTEB results reported elsewhere at roughly 10 to 40 percent depending on the dataset. The result matters because no individual fixed size dominates across all datasets and recall levels. Combining several sizes avoids requiring the system to predict the right size in advance.

### The tradeoff is storage rather than much higher latency
[15:24](https://www.youtube.com/watch?v=r9OwPx_HoV0&t=924s)
Multiscale indexing keeps several copies of the indexed database. Belfer estimates the extra memory at roughly two to five times the original amount. He says retrieval can run partly in parallel, and reciprocal rank fusion takes little time, so latency does not change much. The current choice of sizes, such as 50, 100, 200, and larger windows, was arbitrary. Future work includes deciding how many sizes to keep, selecting the sizes systematically, and testing methods beyond reciprocal rank fusion.

## Notable quotes
- "Chunking is essentially a lossy compression." (04:58)
- "There is no right chunk size." (04:58)
- "The gap which you can see between the orange line and all the blue lines is big." (09:31)
- "Why commit to one where we can commit to several?" (11:13)
- "Agents didn't kill retrieval. Nothing died. Come on. It's just infrastructure." (17:04)

## Tools & references mentioned
- AI21 Labs
- agentic search
- MCP
- RAG
- LlamaIndex
- QM-Sum
- Narrative QA
- Seinfeld
- FinanceBench
- Anthropic Contextual Retrieval
- MTEB
- reciprocal rank fusion

## Who should watch
- You are choosing a fixed chunk size for a RAG index and want evidence about the cost of that decision.
- Your queries vary between narrow fact lookups and questions that require context spread across a scene or document.
- You can spend more memory on indexing but want to avoid adding a model or much query latency.

## Related talks

- [Retrieval Augmented Generation in the Wild](https://aietalks.com/talks/retrieval-augmented-generation-in-the-wild) (Anton Troynikov, Chroma, 12:20)
- [When All Context Matters: Extended Cache Augmented Generation](https://aietalks.com/talks/when-all-context-matters-extended-cache-augmented-generation) (Luis Romero-Sevilla, Orbis Operations, 05:52)
- [Stop Fine-Tuning to Fix Retrieval Problems](https://aietalks.com/talks/stop-fine-tuning-to-fix-retrieval-problems) (Anant Srivastava, Oracle, 20:12)
- [CrabRAG: Why Automated Assistants Need Graph Memory, Not More Tokens](https://aietalks.com/talks/crabrag-why-automated-assistants-need-graph-memory-not-more-tokens) (Stephen Chin, Neo4j, 20:42)
- [GraphRAG methods to create optimized LLM context windows for Retrieval](https://aietalks.com/talks/graphrag-methods-to-create-optimized-llm-context-windows-for-retrieval) (Jonathan Larson, Microsoft, 15:09)