The main limit on knowledge agents is often finding the right documents, because models reason better than current retrieval tools search.
2
Agents write poor semantic-search queries because training has taught them keyword patterns from coding tools, web search, and benchmark design.
3
A search agent can learn better retrieval by choosing among overview, semantic, metadata-filtering, and grep tools, then training on retrieval and trajectory rewards.
Summary
Hanna Lichtenberg argues that knowledge agents are often limited by retrieval rather than reasoning. On BrowseComp Plus and Office QA Pro, models perform far below their Oracle results when they must search noisy corpora with default tools. Better retrieval from Mixedbread recovers much of that lost performance. The problem is partly how agents write queries. Training from coding agents, web search, and lexical benchmarks pushes them toward long keyword strings, which work poorly for semantic search. Mixedbread built a search-agent harness with overview search, semantic search, metadata filtering, and grep. The agent can run parallel searches across up to four rounds, divide the user's intent into aspects, and rank the evidence it finds. Its training uses supervised fine-tuning followed by on-policy reinforcement learning, with rewards for retrieval quality and search trajectories. Hanna reports strong intermediate results, including a beta agent that reaches 93.4% accuracy on Snowflake's MatchQA benchmark with Gemini 3.5 Flash.
Hanna says LLM reasoning has improved rapidly while search has improved slowly over roughly the last 20 years. Retrieval tools are still the main way a model gets knowledge for legal, financial, and other work beyond code. Mixedbread calls the resulting difference between reasoning and search the knowledge gap. She says the gap appears in real tasks, not only in theory. The model may be capable of answering a question, but it cannot do so when the right documents are not retrieved.
Oracle results show that document access is the bottleneck
The talk compares BrowseComp Plus and Office QA Pro. BrowseComp Plus uses a corpus of 100,000 documents, while Office QA Pro asks complex questions about US Treasury material from roughly the past 100 years. The Oracle result estimates performance when the model receives the right documents. Hanna gives Oracle performance of 93% for BrowseComp and 64% for Office QA Pro. With Codex's default tools, performance drops sharply, by nine points on BrowseComp and eight points on Office QA Pro. Better Mixedbread search reduces the remaining gap to three points on BrowseComp and nearly closes it on Office QA Pro.
Hanna shows a benchmark query made of fragments such as "senator woman questions billionaires not a company" and calls it gibberish for a semantic search system. Models learn this style from coding agents that explore repositories with grep, where they try to list expressions likely to occur in a file. They also imitate keyword-heavy web-search patterns. Benchmark design adds another pressure: benchmarks such as BEIR and NanoBEIR use entity-based queries that favor BM25. The agent therefore guesses document keywords instead of describing the meaning it needs to find.
The search harness gives the agent different tools
Mixedbread built a search agent with four tools. Overview search performs wide semantic search and returns summaries of up to 50 chunks, giving the agent a view of the corpus without filling its context. Main semantic search returns the full payload of the top 10 chunks. A filter-chunks tool sorts and finds chunks through metadata facets. Grep handles exact keyword matching. The agent has up to four search rounds and can run parallel searches within a round. It receives the user query, an initial semantic-search result, and available metadata hints before planning.
The harness asks the agent to state what evidence it needs before writing a query. It divides the user's intent into up to four separate aspects and chooses the best tool for each one. The prompt asks for one concise sentence describing what the agent wants to find, rather than asking it to write a search query. Hanna says this helps avoid the model's learned BM25-style pattern. Prompt examples show good queries and ways to split an input across aspects. The initial semantic results also expose the corpus language and suggest where deeper searches should go.
Training combines a teacher with search-specific rewards
Mixedbread trains a small language model so the agent can search quickly. The training goal includes tool choice, semantic-query quality, exploration, ranking, and efficiency. It starts with supervised fine-tuning from a larger teacher model, followed by on-policy reinforcement learning. The search reward combines retrieval and trajectory rewards. Retrieval scoring includes NDCG and an LLM judge that checks relevance, whether all returned chunks are relevant, and whether the ranking is plausible. A trajectory judge evaluates whether queries are natural sentences, whether exploration is sufficient, and whether the agent searched too much or too little.
The trained agent uses semantic sentences and grep patterns differently
An example trajectory begins with an initial search and metadata hints, followed by four parallel searches in the first round and a grep search in the second. For semantic search, the trained agent writes a sentence describing what it wants to find. Its grep queries retain keyword patterns because exact matching is the purpose of that tool. Hanna says the trained agent was not yet released, but she presents intermediate results. On the Oblique Congress benchmark, it reached an NDCG@10 of 0.4, compared with 0.18 for the best model reported in that benchmark's paper, the GPT multi-hop agent.
The beta search agent performs strongly with a smaller model
Mixedbread's beta agent, called Mixedbread Agentic Search, is already in production. Hanna says it ranks first on Snowflake's MatchQA benchmark, reaching 93.4% accuracy when Gemini 3.5 Flash uses it as a search tool. She also says it achieved that result with less effort than comparable language models using other search agents. The trained agent itself was not released at the time of the talk, so the production result concerns the beta version rather than the unreleased model.
"We kind of trick the model into not thinking it has to write the typical BM25 based query by just instructing it to write one concise sentence describing what it wants to find."09:12
Who should watch
You are building a knowledge agent for legal, financial, research, or other document-heavy work and need to understand why capable models still miss evidence.
Your agent produces long keyword queries for a semantic search backend, and you want a concrete tool and prompt design to change that behavior.
You are training a small search agent and need examples of retrieval rewards, trajectory rewards, parallel search, and tool selection.
