Production RAG quality depends on improving the entire pipeline, including data preparation, embeddings, retrieval, synthesis, and evaluation.
2
Teams should measure retrieval and final answers separately before choosing more advanced techniques, because evaluation shows which part of the system needs work.
3
Small-to-big retrieval, metadata filters, agent-based document tools, and fine-tuning can improve results when basic chunking and search changes are not enough.
Summary
Jerry Liu explains why a simple retrieval-augmented generation application is easy to build but difficult to run well in production. A RAG system has data ingestion, retrieval, and response synthesis stages, and failures can come from any of them. Poor retrieval can cause irrelevant context, hallucinations, low recall, or too much text in the model context. Liu recommends building evaluation datasets and measuring retrieval separately from end-to-end answer quality before changing the system. He then moves from basic changes, such as chunk-size tuning, hybrid search, and metadata filters, to more advanced methods. Small-to-big retrieval uses small chunks for matching and larger windows for synthesis. Agent architectures can model documents as tools for question answering and summarization. Fine-tuning can target embeddings, adapters, or language models, including distilling behavior from a larger model into a smaller one.
A RAG application has separate ingestion, retrieval, and synthesis stages
Liu describes the basic RAG stack as data ingestion followed by data querying, where querying includes retrieval and synthesis. LlamaIndex can hide much of this in about five lines of code, but he encourages engineers to understand the lower-level parts. Those parts include loading data, retrieving from a vector database, and combining retrieved context with a language model. The same pattern appears in applications such as chat over PDFs and other unstructured data. Production problems become easier to diagnose when the system is treated as a set of components rather than one opaque prompt.
Naive RAG fails when retrieval returns the wrong context or too much context
Liu says response quality is often poor because retrieval does not return the right chunks. Low precision means that some retrieved chunks are irrelevant, while low recall means that the needed information is missing from the top-k results. These failures can lead to hallucinations, irrelevant answers, and too much fluff in the response. He also mentions outdated information and language-model problems such as toxicity and bias. Improving a RAG system can therefore involve the stored data, chunk sizes, embedding model, retrieval algorithm, or synthesis process.
Evaluation should identify whether retrieval or synthesis is causing the problem
Before trying advanced techniques, Liu says teams need a task-specific way to measure performance. Retrieval evaluation can use human-labeled data, user feedback, or synthetic queries paired with relevant document IDs. Ranking metrics include success rate, hit rate, MRR, and NDCG. He describes this part as a traditional information-retrieval problem. End-to-end evaluation instead measures the final answer for a query, using human annotations, user feedback, reference answers, or synthetic data generated with GPT-4. Language-model-based evaluators can then compare outputs against the evaluation benchmark.
Basic retrieval changes should come before more complex RAG architectures
Liu recommends starting with what he calls table-stakes RAG techniques rather than immediately using the hardest methods. These include better parsing, chunk-size experiments, hybrid search, and metadata filters. More advanced retrieval can add reranking, recursive retrieval, embedded-table handling, and small-to-big retrieval. Agents and fine-tuning come later because they may be harder to implement and can add latency or cost. This ordering gives teams a way to improve a system with simpler changes before adding more moving parts.
More retrieved tokens do not automatically produce better answers
Chunk-size tuning can have a large effect on system performance, but retrieving more tokens is not always helpful. Liu connects this to the lost-in-the-middle problem, where information in the middle of a language model's context is easier to miss than information near the end. Reranking retrieved tokens also does not guarantee a better final answer. He describes experiments where a dataset had an optimal chunk size, and where reranking could increase error metrics. The amount and position of context both matter to synthesis quality.
Metadata filters combine structured constraints with semantic search
Metadata gives each text chunk structured information, such as a page number, document title, or summary of adjacent chunks. It can also contain generated questions that a chunk answers. Liu uses a question about risk factors in a 2021 SEC 10-K as an example. Raw semantic search might return material from other years or unrelated sections. A system can infer a filter such as year equals 2021, similar to a SQL WHERE clause, and combine it with semantic search. This narrows the candidate set and improves retrieval precision.
Small-to-big retrieval matches on precise text and synthesizes from a wider window
Embedding a large text chunk can dilute the useful signal when the chunk contains irrelevant material. Liu proposes embedding smaller units, such as sentences, then expanding the surrounding context during synthesis. Smaller units are more likely to match a specific query, while the larger returned window gives the language model enough information to form a complete answer. He says this approach can reduce lost-in-the-middle problems and allow a smaller top-k value. In his example, k equal to 2 can be preferable to k equal to 5 when naive retrieval returns large chunks and too much context.
Document agents can answer questions that one top-k retrieval step cannot
A single retrieval-and-generation step is limited by the retriever and works best for straightforward question answering. Some tasks require several reasoning steps, such as summarizing one document and comparing it with others. Liu describes an architecture where each document is modeled as a set of tools. Those tools can summarize the document or answer questions about specific facts. For large collections, the system can retrieve among these tools before using them. The language model then acts on selected document capabilities rather than receiving all raw text directly in its context.
Fine-tuning can target embeddings, adapters, or response synthesis
Liu describes several fine-tuning options for RAG. Embedding models can be adapted to a domain so relevant queries retrieve the right material instead of relying on a general pretrained representation. Teams can generate synthetic query datasets from raw text chunks with language models and use them to tune embeddings. An adapter can be tuned without changing the base model weights, and tuning the query side can avoid reindexing the full document corpus. For synthesis, a larger model such as GPT-4 can generate synthetic training data that is distilled into a smaller model such as GPT-3.5 Turbo.
"A very related idea here is just like embedding a reference to the parent chunk."13:57
Who should watch
You have a RAG prototype that retrieves plausible text but produces inconsistent or padded answers.
You need to separate retrieval errors from synthesis errors before spending time on model or agent changes.
You are choosing between chunking, metadata filters, reranking, agent workflows, and fine-tuning for a production system.
