# Better Search Took This AI Agent From 36% to 98% Accuracy

Aamir Shakir, Mixedbread | AI Engineer NYC 2026 | 16:30

Source: https://www.youtube.com/watch?v=3Ad73Onx3Aw
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/better-search-took-this-ai-agent-from-36-to-98-accuracy
Published: 2026-11-03
Tags: embeddings, multimodal, rag, reranking

## TL;DR
- Aamir Shakir argues that agent quality is often limited by access to the right data rather than by the intelligence model itself.
- The usual retrieval pipeline loses information during OCR, chunking, embedding, vector search, and reranking, especially in complex documents.
- A Mixedbread customer increased answer accuracy from 36% to 98% after replacing that pipeline with multimodal, late-interaction retrieval.

## Summary
Aamir Shakir argues that many agent failures begin before the language model sees any context. OCR can lose tables and diagrams, chunking can split information at the wrong boundary, and a single embedding can flatten a complex financial report into one point. As collections grow, vector search struggles to distinguish similar reports from different years. Agents respond by issuing more searches and using more subagents, which increases cost without fixing retrieval. Mixedbread uses multimodal indexing and late-interaction retrieval, with multiple contextualized embeddings for each document and query. In a demo with Intel earnings reports, the system searches page visuals, layout information, and OCR, then references source locations with bounding boxes. Shakir says a hedge fund with more than one billion documents improved answer accuracy from 36% to 98% after moving from its existing OCR, chunking, embedding, and vector database stack to Mixedbread.

## Key ideas
### Agents become expensive when search returns too many possible documents
[01:30](https://www.youtube.com/watch?v=3Ad73Onx3Aw&t=90s)
Shakir describes an equity analyst asking why a new Intel GPU uses LPDDR5 memory instead of HBM. A local agent might use lexical search to find terms such as Intel, GPUs, or LPDDR5, but a large document collection produces many hits without useful relevance. The agent may then read every document or start more searches and subagents. Shakir says this can push the cost of a basic question to 10 or 20 dollars, while filling the context window with material the agent does not need.

### The standard retrieval pipeline loses information at every stage
[03:54](https://www.youtube.com/watch?v=3Ad73Onx3Aw&t=234s)
The pipeline Shakir calls the 'Frankenstein stack' starts with parsing, transcription, or OCR. Tables, diagrams, and characters can be damaged during that step. Chunking then introduces another decision about where a page, figure, or table should be split. A single embedding compresses the remaining content into one point, and vector search adds retrieval and reranking on top. Teams must separately choose vendors and methods for OCR, chunking, embeddings, databases, and ranking. Shakir says customers can spend six or eight months evaluating these pieces and assign four or five people to the work.

### Single-vector search struggles with similar financial reports
[06:02](https://www.youtube.com/watch?v=3Ad73Onx3Aw&t=362s)
Shakir uses a simple sentence such as 'I love bread' to explain how an embedding places text somewhere in a vector space. A financial filing contains much richer information, including revenue, net income, dates, companies, and tables. A single point may identify a company's earnings report without preserving enough detail to distinguish reports from 1996, 2000, and 2026. He says search quality can look good with tens or hundreds of documents, then degrade significantly as a collection grows because the system cannot separate fine-grained facts.

### The main production bottleneck is access to data
[08:26](https://www.youtube.com/watch?v=3Ad73Onx3Aw&t=506s)
Shakir's conclusion is that the intelligence needed by agents is already available, while access to the relevant company knowledge is the limiting factor. He expects intelligence and data access to become separate parts of the system. Teams can choose among intelligence models, but those models still need accurate, detailed access to the underlying documents. Mixedbread was built around this retrieval problem, with ingestion for PDFs, spreadsheets, images, video, and audio in more than 100 languages.

### Late interaction keeps several pieces of context for each document
[09:04](https://www.youtube.com/watch?v=3Ad73Onx3Aw&t=544s)
Shakir contrasts older keyword search, such as BM25, with newer embedding methods. Mixedbread's newer approach is a late-interaction model. Instead of one embedding for a document and one for a query, it creates multiple embeddings for each. The embeddings can capture details such as net income and net revenue while retaining the context that they belong to a particular company. Shakir says the combination of this method, larger scale, and better training data produces a large improvement in retrieval quality.

### Multimodal indexing preserves information from page visuals
[11:08](https://www.youtube.com/watch?v=3Ad73Onx3Aw&t=668s)
In the demo, Shakir searches Intel earnings reports for the foundry segment operating margin in the second quarter of 2025. OCR alone would not capture the meaning of a chart or table reliably. Mixedbread creates an embedding from the visual appearance of the page, while also exposing layout data and OCR for downstream use. The search result identifies the relevant page and can provide bounding boxes so an analyst or agent can trace an answer back to its source.

### Complex questions can trigger multi-query retrieval
[13:43](https://www.youtube.com/watch?v=3Ad73Onx3Aw&t=823s)
For a question about whether Intel's foundry profitability is improving and what drives it, Shakir shows an agentic search mode. The system decomposes the question and issues multiple searches before returning relevant data to a language model for explanation. He says the agent can choose between ordinary search and agentic search based on query complexity. Mixedbread exposes these search modes as primitives for an application or another agent.

### A hedge fund reported a large accuracy increase after changing search
[14:57](https://www.youtube.com/watch?v=3Ad73Onx3Aw&t=897s)
Shakir describes a hedge fund with more than one billion documents and an internal agent platform for analysts and fund managers. Its earlier stack used OCR, chunking, an embedding model, and a vector database. Human experts graded the resulting answers as correct 36% of the time. After switching to Mixedbread, the reported answer accuracy rose to 98%. Shakir also says multimodal retrieval improved results and that language-model spending fell by three times because less context was needed to find and read the answer.

## Notable quotes
- "The real real bottleneck is actually access to your data and to your knowledge." (08:35)
- "So this is a super lossy process from going from this really rich information to hey here's just a point." (04:50)
- "Instead of having just one embedding, you have multiple embeddings for your documents and you have multiple embeddings for your queries." (10:07)
- "The issue is not intelligence, it's again getting the right data." (15:06)

## Tools & references mentioned
- Mixedbread
- Claude Code
- Claude Cowork
- BM25
- Elasticsearch
- retrieval-augmented generation
- Intel
- NVIDIA
- Astra
- Fable

## Who should watch
- You are building an agent over financial reports, research files, or other large document collections and the agent keeps returning plausible but poorly supported answers.
- Your retrieval stack has separate OCR, chunking, embedding, vector database, and reranking components that are difficult to evaluate and maintain.
- You need agents to search tables, charts, scanned pages, or other multimodal documents and to cite the exact source location.

## Related talks

- [Building a Smarter AI Agent with Neural RAG](https://aietalks.com/talks/building-a-smarter-ai-agent-with-neural-rag) (Will Bryk, Exa, 18:42)
- [Vector Isn't Enough: Hybrid Search & Retrieval](https://aietalks.com/talks/vector-isnt-enough-hybrid-search-retrieval) (Jeff Vestal & James Williams, Elastic, 1:16:20)
- [How We Taught Agents to Use Good Retrieval](https://aietalks.com/talks/how-we-taught-agents-to-use-good-retrieval) (Hanna Lichtenberg, Mixedbread, 14:28)
- [How We Took an AI Agent From 41% to 97% Accurate](https://aietalks.com/talks/how-we-took-an-ai-agent-from-41-to-97-accurate) (Milind Sunil Shah, Brainbase Labs, 13:23)
- [From Raw Documents to AI-Ready Data](https://aietalks.com/talks/from-raw-documents-to-ai-ready-data) (Leo Platzer & Jeff Koss, Deasy Labs (Collibra), 21:37)
