# Retrieval + Search

Jerry Liu, LlamaIndex & Chong, LanceDB & Calvin, Harvey.ai & Julia & Danna Emmery, Quotient AI & Tony Ma, MongoDB & Sherwood & Sautwik, 11x & Will Bryk, Exa & David, Pybabs | AI Engineer World's Fair 2025 | 4:00:34

Source: https://www.youtube.com/watch?v=a0TyTMDh1is
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/retrieval-search
Published: 2025-06-05
Tags: evals, multimodal, rag, search

## TL;DR
- Document agents need specialized parsing and validation before retrieval can work on complex PDFs, spreadsheets, images, and videos.
- Enterprise search quality depends on domain-aware evaluation, flexible multimodal infrastructure, and retrieval systems that combine semantic, lexical, and structured signals.
- The practical way to improve a RAG system is to inspect failing queries, then add the simplest technique that addresses the observed failure.

## Summary
This track covers how teams build retrieval systems for messy documents, legal data, live web search, sales knowledge bases, and large enterprise corpora. LlamaIndex describes a document toolbox that combines traditional parsers with LLM and vision-language model validation, including an Excel agent for irregular spreadsheets. LanceDB and Harvey.ai explain why legal RAG requires domain experts, layered evaluation, privacy controls, and infrastructure for multimodal data at very large scale. Quotient AI argues that static benchmarks miss the changing web and presents dynamic evaluation sets with reference-free metrics. Tony Ma makes the case for RAG as a modular way to access proprietary information, while Exa describes search designed for AI queries rather than human keyword searches. The 11x team walks through a production knowledge base for an AI sales representative. Pybabs closes with a practical framework: start with context windows or BM25, inspect failures, then add embeddings, reranking, query fan-out, domain models, or product-level fallbacks as needed.

## Key ideas
### Complex documents must be understood before they can be retrieved
[12:17](https://www.youtube.com/watch?v=a0TyTMDh1is&t=737s)
Jerry Liu argues that complicated PDFs contain embedded tables, charts, images, irregular layouts, headers, and footers that were designed for people rather than machines. If parsing fails, a better language model cannot recover the missing structure. LlamaIndex combines traditional parsing with LLMs and vision-language models, then adds validation and reasoning at test time. The goal is a general document-understanding layer rather than a hand-tuned parser for every document class. Liu also describes an Excel agent that converts unnormalized spreadsheets into normalized two-dimensional tables and lets an agent answer questions over both versions.

### Document agents split into assistant and automation workflows
[17:45](https://www.youtube.com/watch?v=a0TyTMDh1is&t=1065s)
LlamaIndex distinguishes assistant interfaces from automation interfaces. Assistants usually take natural-language questions, run a relatively unconstrained tool loop, and keep a human involved while the agent works. Automation systems process batches in the background, use more constrained control flow, and return structured results for review or downstream integrations. Examples include financial data normalization, invoice reconciliation, contract review, and technical data-sheet ingestion. Liu describes automation agents as a backend that structures data and exposes useful tools, while assistant agents provide the user-facing research or question-answering experience.

### Legal RAG needs evaluation at several levels of fidelity
[29:39](https://www.youtube.com/watch?v=a0TyTMDh1is&t=1779s)
Chong and Calvin describe legal retrieval across uploads, project vaults, and large legal corpora. Queries can combine dates, regulations, abbreviations, domain terminology, implicit filters, and multiple parts of a question. They use expert review for high-fidelity evaluation, expert-labeled criteria for a middle layer, and automated measures such as precision, recall, folder matching, and section matching for rapid iteration. No single evaluation answers every question. Their system also needs privacy and retention controls, online query performance, offline ingestion and experiments, and flexible combinations of vector, keyword, and filtered retrieval.

### Dynamic evaluation is needed when the web and user questions keep changing
[49:43](https://www.youtube.com/watch?v=a0TyTMDh1is&t=2983s)
Julia and Danna argue that static benchmarks such as SimpleQA and HotpotQA are useful starting points, but they do not fully represent live web search. They built an open-source agent that creates refreshed evaluation sets by searching targeted domains through multiple real-time search providers, collecting grounding documents, and generating evidence-based question-and-answer pairs. They use answer completeness, document relevance, and hallucination detection when ground-truth answers are unavailable. These metrics expose different failure modes. For example, retrieving relevant documents while producing an incomplete answer suggests a need for more retrieval, while detailed answers can increase hallucination opportunities.

### RAG remains a modular way to use proprietary information
[1:07:48](https://www.youtube.com/watch?v=a0TyTMDh1is&t=4068s)
Tony Ma compares long context, fine-tuning, and RAG as ways to give models access to enterprise data. Long context can mean sending an entire library for every question, while fine-tuning moves knowledge into model parameters and makes updates and access control harder. RAG retrieves a smaller, relevant subset when the question arrives. He describes the retrieval stack as embedding models, a vector database, and a language model, then discusses hybrid search, rerankers, query decomposition, document enrichment, domain-specific embeddings, and compression. His view is that model quality will absorb more of today's retrieval tricks over time, although application-specific signals will still matter.

### AI search needs a different interface from human web search
[3:12:04](https://www.youtube.com/watch?v=a0TyTMDh1is&t=11524s)
Will Bryk explains that traditional search engines were built for people who type short keyword queries and open a few links. AI systems can send long, highly specific queries, process many results, and ask for comprehensive knowledge. Exa's neural search represents documents with embeddings so a query such as finding personal sites of San Francisco engineers interested in information retrieval can match meaning rather than only shared words. Bryk also shows why AI systems need controls for result count, domains, dates, and search modes. An agent may combine a neural search for candidates with keyword searches for each candidate's GitHub page.

### A production knowledge base can replace manual sales-context entry
[2:57:44](https://www.youtube.com/watch?v=a0TyTMDh1is&t=10664s)
The 11x team describes how Alice, its AI sales development representative, moved from a manually maintained library to a knowledge base that proactively ingests seller information. Users can provide documents, images, websites, audio, and video. The pipeline stores uploads, sends them to parsing vendors, receives webhooks, saves the parsed artifacts, embeds them in Pinecone, and makes them available to the agent. The team converts everything to Markdown, then chunks first by Markdown headers, then sentences, then tokens. It also projects vectors into three dimensions so customers can inspect the chunks Alice uses, which gives the sales team a way to see what the agent knows.

### RAG improvements should follow observed query failures
[3:41:05](https://www.youtube.com/watch?v=a0TyTMDh1is&t=13265s)
David presents a practical sequence for improving retrieval. Start with the product outcome and a set of queries, then inspect failures before choosing techniques. A full-context approach may be enough for a small document set. BM25 helps with exact terms, embeddings help with semantic wording, and rerankers resolve conflicts among candidates. Harder systems may need custom embeddings, domain signals such as price or popularity, click feedback, query fan-out, supplementary retrieval, or smaller distilled models for latency. When quality still cannot be guaranteed, the product should degrade gracefully or involve a human instead of pretending that the system is certain.

## Notable quotes
- "If the documents are not processed correctly, no matter how good your LLM is, it will fail." (12:34)
- "You can answer today's questions with yesterday data." (52:52)
- "The practical way to think about techniques is: always look at what's broken and if it's not broken don't fix it." (3:42:54)
- "Traditional search engines were built for humans." (3:27:17)
- "RAG is complex. It was a lot harder than we thought it was going to be." (3:15:15)

## Tools & references mentioned
- LlamaIndex
- Llama Parse
- LanceDB
- Harvey.ai
- Quotient AI
- Tavily
- MongoDB
- Voyage AI
- Pinecone
- Exa
- 11x
- Alice
- Julian
- Firecrawl
- Cloud Glue
- Letta
- SimpleQA
- HotpotQA
- LangGraph
- Weights & Biases
- BM25
- PageRank
- OpenAI
- Anthropic
- Gemini
- Google

## Who should watch
- You are building a RAG system over PDFs, spreadsheets, legal files, media, or other data that does not fit cleanly into plain text.
- Your search product uses live web data and needs evaluations that keep working as sources and user questions change.
- You have added retrieval techniques without a clear improvement loop and need a way to connect failing queries to concrete engineering work.

## Related talks

- [Building Production-Ready RAG Applications](https://aietalks.com/talks/building-production-ready-rag-applications) (Jerry Liu, LlamaIndex, 18:35)
- [Information Retrieval from the Ground Up](https://aietalks.com/talks/information-retrieval-from-the-ground-up) (Philipp Krenn, Elastic, 1:48:07)
- [RAG is dead, right??](https://aietalks.com/talks/rag-is-dead-right) (Kuba Rogut, Turbopuffer, 11:13)
- [Layering Every Technique in RAG, One Query at a Time](https://aietalks.com/talks/layering-every-technique-in-rag-one-query-at-a-time) (David Karam, Pi Labs, 20:22)
- [Scaling Enterprise-Grade RAG: Lessons from Legal Frontier](https://aietalks.com/talks/scaling-enterprise-grade-rag-lessons-from-legal-frontier) (Calvin Qi, Harvey & Chang She, LanceDB, 16:40)