Chunk size depends on the query, so no single fixed size can work best for every question in a corpus.
2
Multiscale indexing stores several chunked copies of the same data, queries each copy, and combines document rankings with reciprocal rank fusion.
3
Across four datasets, the method improved recall by roughly 20 to 40 percent over fixed chunk sizes, at the cost of two to five times the memory.
Summary
Yuval Belfer argues that chunking remains a retrieval problem even as teams adopt agentic search. Fixed chunking loses information: large chunks preserve context but weaken focused matches, while small chunks miss answers spread across a scene or document. His experiments duplicate datasets across six window sizes and show that different questions prefer different sizes. An oracle that selects the best size per query beats every fixed choice by 20 to 40 percent of recall. Belfer's practical method indexes every window size, retrieves from all copies, converts chunk results into document rankings, and merges them with reciprocal rank fusion. He reports better recall across QM-Sum, Narrative QA, Seinfeld, and FinanceBench. The tradeoff is extra storage, estimated at two to five times the memory, while parallel retrieval and rank fusion add little latency. The talk is honest that the window sizes and number of copies still need better selection methods.
Agentic search does not remove retrieval infrastructure
Belfer opens by pushing back on claims that chunking, retrieval-augmented generation, or retrieval itself is dead. Agentic search can use tools such as search and find, but it becomes expensive when the system has a large amount of data and many possible queries. If the underlying data is poorly organized, agents still have to inspect too much material. He compares this with a World Cup directory: a question about which team won the most tournaments requires checking every year and aggregating the answers. The answer cannot be found by opening one folder.
Every fixed chunk size throws away some useful information. Large chunks preserve the broader picture, but their embeddings may lose the details needed for a focused question. Small chunks preserve local detail, but the surrounding context disappears and retrieval becomes less efficient. Belfer calls this process lossy compression because no choice keeps everything. He rejects the idea that a corpus has one universally correct size. The best size depends on the question, which creates a problem because indexing happens before the system knows the future queries.
Belfer's team duplicated several datasets and indexed each copy with a different chunk size. The datasets included QM-Sum, Narrative QA, and an in-house Seinfeld trivia dataset. Two Seinfeld questions illustrate the difference. The question asking for Jerry's favorite church is narrow and has an answer contained in a small passage, so a 100-token chunk places the answer at rank one while larger windows push it below rank 50. The question about Jerry's nemesis and pure evil needs information spread across a scene, so small chunks fail to retrieve the answer.
The oracle experiment measures the cost of choosing one size
The team created an oracle that chooses the best chunk size separately for each query. This does not describe a deployable system because it uses the answer after the fact. It measures how much performance is available if the system could choose correctly. The oracle curve beats the curves for every fixed size. Across datasets, the gap is about 20 to 40 percent in recall. Belfer says this is the cost of choosing an arbitrary value such as 512 or 1,000 tokens when the query is unknown at indexing time.
Indexing and retrieval have opposite information constraints
At indexing time, the system controls chunk size but does not know what users will ask. It can estimate likely queries, but it cannot adapt to every future question. At retrieval time, the system has the query but the chunk size has already been fixed, and rebuilding the index for each query is impractical. Belfer frames this as an information problem at both ends of the pipeline. He mentions contextual retrieval and related approaches that improve each chunk while retaining a fixed window, then proposes keeping multiple window sizes instead.
Multiscale indexing compares document rankings instead of chunk rankings
The proposed method creates several indexed copies of the corpus, each with a different window size, and queries all of them for every user question. Chunk rankings from different window sizes cannot be compared directly because the chunks cover different amounts of text. The system therefore retrieves the whole document associated with each matching chunk. Each index then produces a ranking of the same documents. Reciprocal rank fusion combines those rankings into one list. Belfer describes the implementation as a simple script rather than a model.
Belfer reports tests on QM-Sum, Narrative QA, Seinfeld, and FinanceBench. The combined method matches or beats the best fixed chunk size across recall-at-one through recall-at-ten measurements. He describes improvements in the range of 20 to 40 percent on many results, with additional MTEB results reported elsewhere at roughly 10 to 40 percent depending on the dataset. The result matters because no individual fixed size dominates across all datasets and recall levels. Combining several sizes avoids requiring the system to predict the right size in advance.
The tradeoff is storage rather than much higher latency
Multiscale indexing keeps several copies of the indexed database. Belfer estimates the extra memory at roughly two to five times the original amount. He says retrieval can run partly in parallel, and reciprocal rank fusion takes little time, so latency does not change much. The current choice of sizes, such as 50, 100, 200, and larger windows, was arbitrary. Future work includes deciding how many sizes to keep, selecting the sizes systematically, and testing methods beyond reciprocal rank fusion.
