Watch on YouTubeBM25 has not changed, but language models are much more capable search users because they know entities and can write long, specific queries.
On the BrowseComp benchmark, giving the model the evidence directly produces high accuracy, while making it retrieve that evidence causes accuracy to fall.
A file-system workspace lets agents inspect retrieved documents with familiar tools such as grep, while BM25 supplies a simple and explainable retrieval primitive.
Jo Kristian Bergum argues that BM25 is becoming more useful because the user has changed. A language model knows names, companies, dates, postal codes, and product identifiers, so it can write longer and more exact queries than a person. It can also issue several queries, inspect the results, and reformulate them. Bergum uses the BrowseComp benchmark of 830 riddle-like questions to separate reasoning from retrieval. When the needed evidence is placed directly in the model's context, accuracy is high. When the model must find it through a search tool, accuracy drops because query formulation and retrieval quality become limiting factors. He also argues that the choice of BM25 parameters and implementation matters. His preferred design places retrieved documents in a file-system workspace, where the model can use familiar tools and decide when to read more. Traditional single-query ranking metrics fit this workflow poorly, so evaluation should measure whether the agent completes its task.
Bergum defines agentic search as search inside an agent loop. The agent is trying to complete a task such as coding or deep research, and it needs information during that process. A working system needs a capable model that can use tools and formulate queries, a harness that exposes retrieval through tool calling or code mode, and a retrieval engine that can search large document collections efficiently.
BM25 means Best Match 25. Bergum explains that researchers ran many scoring experiments, and the twenty-fifth one worked best. The function scores a query and a document by looking at interactions between their terms. Retrieval systems then use that score to find the top documents. The scoring function itself has stayed the same, while search engines have continued to develop ways to accelerate top-k retrieval.
Bergum's central claim is that BM25 has not changed, but its user has. Language models contain general knowledge about entities, companies, and dates. They can use that knowledge to write longer, more precise queries than people usually write. They can also issue many queries and reformulate them after reading results. Exact matching remains useful when the model knows names, postal codes, SKUs, and other identifiers that are difficult to represent in an embedding.
BrowseComp contains about 830 long, riddle-like questions over roughly 100,000 web documents. The model receives a simple search tool that accepts a query string and returns snippets. Each question has a reference answer for checking end-to-end accuracy. Bergum uses the benchmark to show that the model can often answer correctly when the evidence documents are placed directly in its context, while accuracy falls when it has to retrieve those documents.
Bergum compares a model's usable context window to a floppy disc. A floppy disc held about 1.4 megabytes, while he estimates that current models begin to lose quality at around 350,000 tokens. Even a much better model cannot place every document into that space. Retrieval must select the information that enters the context window, so it remains relevant even when model reasoning improves.
Bergum compares human query logs, including the old AOL sample, with queries written by GPT-5. People still tend to search with only a few terms. GPT-5 can write much longer queries, use phrases and operators such as site, and draw on its general knowledge. A BrowseComp question can become a search trajectory in which the model issues a query, reads the response, reformulates the query, and continues until it finds the answer or fills its context.
Bergum warns that 'BM25' does not identify one identical configuration. The function has hyperparameters, and the settings used in one BrowseComp baseline were not suitable for long documents. That baseline made more advanced retrieval methods look much better than BM25. He argues that implementations also differ in performance. Exact matching, low operating cost, available tooling, and readable results make BM25 useful to an agent that needs to understand why a result was returned.
Bergum describes research from Jimmy Lin's group on dynamic workspace expansion. Retrieved documents can be placed in a file-system workspace instead of being dumped directly into the context. The agent can first see titles and short snippets, then read more when needed. It can use familiar primitives such as grep, bash, and other coding-agent tools. This combines retrieval infrastructure with the sandbox tools that current models are trained to use.
Traditional information retrieval evaluates one query and one ranked list with metrics such as NDCG. Bergum says that setup fits poorly when an agent can reformulate queries, expand searches, and follow a trajectory. Evaluation should instead ask whether the model completes its assigned task, such as answering a question correctly. He also describes Hornet's work on efficient BM25 evaluation over large web-document collections.
"If you artificially just stuff the evidence documents that is needed to answer this question into the context window of the model the accuracy is really high, right? So reasoning is not the bottleneck."07:07