# Layering Every Technique in RAG, One Query at a Time

David Karam, Pi Labs | AI Engineer World's Fair 2025 | 20:22

Source: https://www.youtube.com/watch?v=w9u11ioHGA0
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/layering-every-technique-in-rag-one-query-at-a-time
Published: 2025-07-29
Tags: evals, rag, reranking, search

## TL;DR
- RAG quality work should start with query sets, loss analysis, and a target quality bar before anyone chooses a retrieval technique.
- Each retrieval layer fixes particular failures, from BM25 and embeddings through rerankers, custom signals, query fan-out, and supplementary retrieval.
- Some failures require product decisions, such as handing work to a person or showing a simpler interface when the system lacks enough understanding.

## Summary
David Karam presents RAG as a sequence of increasingly expensive techniques, chosen from observed query failures. He starts with putting all documents into the LLM, then moves through BM25, relevance embeddings, cross-encoder rerankers, custom embeddings, domain and user signals, query fan-out, supplementary retrieval, and distillation. Each layer addresses a different problem. BM25 handles exact terms, embeddings handle semantic phrasing, and rerankers combine query and document information at higher cost. Domain-specific vocabulary, prices, popularity, and clicks require signals beyond semantic relevance. At larger scale, fan-out and extra backend calls improve recall but create latency and GPU costs. Karam argues that teams should baseline against easy, medium, and hard query sets, inspect failures, and select the simplest technique with useful impact. When engineering cannot reach perfect quality, the product should degrade gracefully or include a human handoff. The talk is practical because it treats every choice as empirical rather than ideological.

## Key ideas
### Quality work starts with outcomes and failing queries
[01:45](https://www.youtube.com/watch?v=w9u11ioHGA0&t=105s)
Karam says teams should begin with the product outcome and a quality bar, such as the point where a CRM agent can be launched. They should create easy, medium, and hard query sets, establish a baseline, and perform loss analysis to find what is broken. Techniques belong after that diagnosis. He recommends a "complexity adjusted impact" approach: try easy methods first, and do not fix something that is not broken. BM25 is often worth trying because it is easy and can improve quality, while custom embeddings demand much more work and only make sense when the query and data failures justify them.

### In-memory context is the simplest retrieval system
[04:09](https://www.youtube.com/watch?v=w9u11ioHGA0&t=249s)
The first technique is to put every document into the LLM context. Karam uses NotebookLM and a small set of five documents as an example of a system that may need no retrieval layer at all. This approach fails when the documents do not fit in the context window or when the LLM does not attend properly across too much material. Those failures tell the team why it needs another layer. The point is to begin with the simplest possible system and move on only when the observed query cases show that the context approach is insufficient.

### BM25 and embeddings solve different query shapes
[04:50](https://www.youtube.com/watch?v=w9u11ioHGA0&t=290s)
BM25 uses query terms, their frequency, document length, and the location of terms. It works well for keyword-shaped searches and is simple to test. Relevance embeddings move the query and documents into vector space, which handles more semantic variation. Karam contrasts "iPhone battery life," which may work well with term matching, with a question about how long an iPhone lasts before charging, where vector search is more useful. Both methods have failure modes. Teams should inspect their query sets to see which shape dominates before investing in vector retrieval.

### Cross-encoder rerankers improve a smaller candidate set
[06:18](https://www.youtube.com/watch?v=w9u11ioHGA0&t=378s)
When BM25 and vector search produce a conflicted candidate set, a reranker can score the candidates again. Karam describes cross-encoders as models that attend to the query and document together before producing a score. This is more powerful than comparing separate query and document vectors, but it is too expensive to run across every document. The usual pattern is therefore broad retrieval followed by reranking of a smaller set. Rerankers still measure a form of relevance, so they fail when the application needs signals that semantic similarity does not capture.

### Custom embeddings are justified by domain vocabulary
[07:59](https://www.youtube.com/watch?v=w9u11ioHGA0&t=479s)
Standard relevance models can fail when words have specialized meanings. Karam discusses legal search, where terms such as "regime," "moot," and "material" carry domain-specific meanings, and where laws and regulations must be retrieved accurately. A custom embedding model can represent that domain in its own vector space. He does not present custom embeddings as a default upgrade. The decision should come from evaluations and loss analysis. If failures cluster around vocabulary that is out of distribution for a standard relevance model, custom modeling may be warranted.

### Ranking needs signals about the corpus and the user
[09:00](https://www.youtube.com/watch?v=w9u11ioHGA0&t=540s)
Semantic relevance is only one part of ranking. Karam gives shopping examples where a price constraint is ignored, and notes that merchant information, popularity, and podcast listen counts can matter even though they have no semantic relationship to the query. PageRank is another example, since it measures prominence and links in the web corpus. These are signals about the data's structure. User behavior adds another layer through clicks, thumbs-up, and thumbs-down feedback. A ranking function may combine relevance, structured domain signals, and predicted user preferences.

### Fan-out gives the search backend queries it can handle
[12:17](https://www.youtube.com/watch?v=w9u11ioHGA0&t=737s)
Complex assistant requests can be difficult for a search engine because the LLM does not know the backend's exact query language or capabilities. Karam recommends taking a large request and creating multiple narrower queries, a process he calls fan-out. He points to Google AI Mode issuing many queries for one request. Application-specific orchestration can identify searches that matter to the domain and avoid sending an overly broad request that overwhelms the backend. The boundary between the LLM and search engine remains difficult, so he argues that teams need control over orchestration rather than relying only on prompts.

### More backends can resolve ambiguous intent, at a cost
[14:26](https://www.youtube.com/watch?v=w9u11ioHGA0&t=866s)
Supplementary retrieval means calling additional search backends when the first search is not enough. Karam uses "falafel" as an example of an ambiguous query. The user might want food, restaurants, or images, so different backends provide different evidence. His advice is to search more when recall matters, unless cost has become a real constraint. At scale, many backends and queries can create GPU and latency pressure. Distillation can then reduce model size while holding the quality bar constant, especially when a model only needs to perform a narrow task such as question answering.

### Product design must absorb the failures of stochastic systems
[17:17](https://www.youtube.com/watch?v=w9u11ioHGA0&t=1037s)
Karam says quality engineering will never reach 100 percent because these systems are stochastic. When engineering cannot solve every case, the problem has to move into product design. A customer-support bot may hand difficult cases to a human. Google Shopping can show a rich, filterable interface when its understanding is strong, then fall back to a simpler list when understanding is weak. The product should upgrade or degrade according to its confidence and available information. This lets the interface match what the system actually knows instead of pretending that every query can receive the same treatment.

## Notable quotes
- "If it's not broken don't fix it and if it is broken do fix it." (03:01)
- "Relevance is not ranking." (07:14)
- "At sufficient complexity, things will keep breaking." (10:59)
- "Everything is empirical in this domain." (19:18)

## Tools & references mentioned
- Google Search
- Pi Labs
- NotebookLM
- BM25
- cross encoders
- PageRank
- Perplexity
- MCP
- Google AI Mode
- ChatGPT

## Who should watch
- You are deciding whether your RAG system needs BM25, vector search, reranking, or custom models, and you need a way to choose based on actual query failures.
- Your search system combines structured data, user behavior, or multiple backends, and semantic relevance alone is producing poor rankings.
- You are facing latency, cost, or reliability limits and need product-level fallbacks rather than another retrieval layer.

## Related talks

- [Building Production-Ready RAG Applications](https://aietalks.com/talks/building-production-ready-rag-applications) (Jerry Liu, LlamaIndex, 18:35)
- [The RAG Stack We Landed On After 37 Fails](https://aietalks.com/talks/the-rag-stack-we-landed-on-after-37-fails) (Jonathan Fernandes, 18:52)
- [Scaling Enterprise-Grade RAG: Lessons from Legal Frontier](https://aietalks.com/talks/scaling-enterprise-grade-rag-lessons-from-legal-frontier) (Calvin Qi, Harvey & Chang She, LanceDB, 16:40)
- [Retrieval + Search](https://aietalks.com/talks/retrieval-search) (Jerry Liu, LlamaIndex & Chong, LanceDB & Calvin, Harvey.ai & Julia & Danna Emmery, Quotient AI & Tony Ma, MongoDB & Sherwood & Sautwik, 11x & Will Bryk, Exa & David, Pybabs, 4:00:34)
- [RAG in 2025: State of the Art and the Road Forward](https://aietalks.com/talks/rag-in-2025-state-of-the-art-and-the-road-forward) (Tengyu Ma, MongoDB, 18:48)