# The Unreasonable Effectiveness of BM25 for Agentic Search

Jo Kristian Bergum, Hornet.dev | AI Engineer World's Fair 2026 | 18:29

Source: https://www.youtube.com/watch?v=fZH97QHHYjY
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/the-unreasonable-effectiveness-of-bm25-for-agentic-search
Published: 2026-09-16
Tags: agents, evals, rag, search

## TL;DR
- BM25 has not changed, but language models are much more capable search users because they know entities and can write long, specific queries.
- On the BrowseComp benchmark, giving the model the evidence directly produces high accuracy, while making it retrieve that evidence causes accuracy to fall.
- A file-system workspace lets agents inspect retrieved documents with familiar tools such as grep, while BM25 supplies a simple and explainable retrieval primitive.

## Summary
Jo Kristian Bergum argues that BM25 is becoming more useful because the user has changed. A language model knows names, companies, dates, postal codes, and product identifiers, so it can write longer and more exact queries than a person. It can also issue several queries, inspect the results, and reformulate them. Bergum uses the BrowseComp benchmark of 830 riddle-like questions to separate reasoning from retrieval. When the needed evidence is placed directly in the model's context, accuracy is high. When the model must find it through a search tool, accuracy drops because query formulation and retrieval quality become limiting factors. He also argues that the choice of BM25 parameters and implementation matters. His preferred design places retrieved documents in a file-system workspace, where the model can use familiar tools and decide when to read more. Traditional single-query ranking metrics fit this workflow poorly, so evaluation should measure whether the agent completes its task.

## Key ideas
### Agentic search puts retrieval inside an agent loop
[00:55](https://www.youtube.com/watch?v=fZH97QHHYjY&t=55s)
Bergum defines agentic search as search inside an agent loop. The agent is trying to complete a task such as coding or deep research, and it needs information during that process. A working system needs a capable model that can use tools and formulate queries, a harness that exposes retrieval through tool calling or code mode, and a retrieval engine that can search large document collections efficiently.

### BM25 is a scoring function whose name came from experiment number 25
[02:27](https://www.youtube.com/watch?v=fZH97QHHYjY&t=147s)
BM25 means Best Match 25. Bergum explains that researchers ran many scoring experiments, and the twenty-fifth one worked best. The function scores a query and a document by looking at interactions between their terms. Retrieval systems then use that score to find the top documents. The scoring function itself has stayed the same, while search engines have continued to develop ways to accelerate top-k retrieval.

### Language models have made the old search user more powerful
[03:32](https://www.youtube.com/watch?v=fZH97QHHYjY&t=212s)
Bergum's central claim is that BM25 has not changed, but its user has. Language models contain general knowledge about entities, companies, and dates. They can use that knowledge to write longer, more precise queries than people usually write. They can also issue many queries and reformulate them after reading results. Exact matching remains useful when the model knows names, postal codes, SKUs, and other identifiers that are difficult to represent in an embedding.

### BrowseComp separates reasoning ability from retrieval quality
[04:57](https://www.youtube.com/watch?v=fZH97QHHYjY&t=297s)
BrowseComp contains about 830 long, riddle-like questions over roughly 100,000 web documents. The model receives a simple search tool that accepts a query string and returns snippets. Each question has a reference answer for checking end-to-end accuracy. Bergum uses the benchmark to show that the model can often answer correctly when the evidence documents are placed directly in its context, while accuracy falls when it has to retrieve those documents.

### Context windows still require a system to choose the evidence
[06:01](https://www.youtube.com/watch?v=fZH97QHHYjY&t=361s)
Bergum compares a model's usable context window to a floppy disc. A floppy disc held about 1.4 megabytes, while he estimates that current models begin to lose quality at around 350,000 tokens. Even a much better model cannot place every document into that space. Retrieval must select the information that enters the context window, so it remains relevant even when model reasoning improves.

### Query formulation creates a new search workload
[08:37](https://www.youtube.com/watch?v=fZH97QHHYjY&t=517s)
Bergum compares human query logs, including the old AOL sample, with queries written by GPT-5. People still tend to search with only a few terms. GPT-5 can write much longer queries, use phrases and operators such as site, and draw on its general knowledge. A BrowseComp question can become a search trajectory in which the model issues a query, reads the response, reformulates the query, and continues until it finds the answer or fills its context.

### The choice of BM25 parameters and implementation changes results
[09:44](https://www.youtube.com/watch?v=fZH97QHHYjY&t=584s)
Bergum warns that 'BM25' does not identify one identical configuration. The function has hyperparameters, and the settings used in one BrowseComp baseline were not suitable for long documents. That baseline made more advanced retrieval methods look much better than BM25. He argues that implementations also differ in performance. Exact matching, low operating cost, available tooling, and readable results make BM25 useful to an agent that needs to understand why a result was returned.

### A file-system workspace gives agents more control over retrieved documents
[12:06](https://www.youtube.com/watch?v=fZH97QHHYjY&t=726s)
Bergum describes research from Jimmy Lin's group on dynamic workspace expansion. Retrieved documents can be placed in a file-system workspace instead of being dumped directly into the context. The agent can first see titles and short snippets, then read more when needed. It can use familiar primitives such as grep, bash, and other coding-agent tools. This combines retrieval infrastructure with the sandbox tools that current models are trained to use.

### Agentic search needs task-level evaluation
[14:42](https://www.youtube.com/watch?v=fZH97QHHYjY&t=882s)
Traditional information retrieval evaluates one query and one ranked list with metrics such as NDCG. Bergum says that setup fits poorly when an agent can reformulate queries, expand searches, and follow a trajectory. Evaluation should instead ask whether the model completes its assigned task, such as answering a question correctly. He also describes Hornet's work on efficient BM25 evaluation over large web-document collections.

## Notable quotes
- "BM25 hasn't changed. It's the same scoring function, but the change here is really that we got a more powerful user." (03:32)
- "If you artificially just stuff the evidence documents that is needed to answer this question into the context window of the model the accuracy is really high, right? So reasoning is not the bottleneck." (07:07)
- "GPD5 on the other hand it's a much more powerful user. It has the general knowledge and it can like bam write out very long queries." (09:14)
- "Which BM25 do you mean? There are differences in implementation, in performance, in parameters." (17:08)
- "A lot of the classical information retrieval evaluation is now kind of dead." (14:56)

## Tools & references mentioned
- Hornet.dev
- BM25
- BrowseComp+
- GPT-4
- GPT-5
- AOL
- Jimmy Lin
- Waterloo
- NDCG
- grep
- bash

## Who should watch
- You are building search or retrieval for an agent and need to decide whether lexical search still belongs in the system.
- You are comparing BM25 with embeddings and want to understand why query length, exact identifiers, and parameter settings affect the comparison.
- You need an evaluation approach for agents that issue several queries instead of one fixed ranked-list request.

## Editor's note

Jo Kristian Bergum argues that traditional ranked-list metrics fit poorly when an agent can reformulate queries and follow a search trajectory. Kitaru records each real agent run, including model calls and tool results, so a person can judge completed runs and turn those judgments into repeatable evaluation cohorts. That gives task-level checks actual agent behavior rather than only one retrieval query.

Written by the AIE Talks editors (the Kitaru team), not by the speaker.

## Related talks

- [How We Taught Agents to Use Good Retrieval](https://aietalks.com/talks/how-we-taught-agents-to-use-good-retrieval) (Hanna Lichtenberg, Mixedbread, 14:28)
- [The Search Engine for the Agentic Web](https://aietalks.com/talks/the-search-engine-for-the-agentic-web) (Will Bryk, Exa, 17:49)
- [Grep or Embeddings? Agentic Search Over Company Documents](https://aietalks.com/talks/grep-or-embeddings-agentic-search-over-company-documents) (George He, LlamaIndex, 23:22)
- [Agentic Search for Context Engineering](https://aietalks.com/talks/agentic-search-for-context-engineering) (Leonie Monigatti, Elastic, 1:03:13)
- [Evaluating AI Search: A Practical Framework for Augmented AI Systems](https://aietalks.com/talks/evaluating-ai-search-a-practical-framework-for-augmented-ai-systems) (Julia Neagu, Deanna Emery & Mara Sher, Quotient AI, 20:33)