RAG pipelines have several independent failure points, from missing source data and poor chunking to irrelevant retrieval and hallucinated answers.
2
Evaluation metrics can show whether a change improves retrieval, reduces noise, or lowers hallucinations, instead of relying on intuition about which pipeline setting to change.
3
In the demo, smaller chunks with a larger retrieval window, reranking, and hybrid search raised faithfulness from 76 to just under 85 across 10 experiments.
Summary
Atita Arora and Deanna Emery explain how to improve a retrieval augmented generation system by treating evaluation as the guide for every experiment. They begin with the parts of a RAG pipeline that can fail: source extraction, chunking, embeddings, retrieval, context size, reranking, prompting, and answer generation. Emery then walks through a question-answering system built on Qdrant documentation. The team measures context relevance, chunk relevance, faithfulness, and text quality, with faithfulness as the main concern because inaccurate answers could mislead users. Their experiments show why aggregate scores need inspection at the individual-question level. Larger chunks retrieved more necessary information but also added noise. Smaller chunks with a larger retrieval window performed better. Reranking improved retrieval further, and hybrid search helped with Qdrant's specialist terminology. Across 10 experiments, faithfulness rose from 76 to just under 85 without changing the generic question-answering prompt.
Atita describes a basic RAG flow that splits documents, embeds and stores them in a vector database, retrieves relevant chunks, and places those chunks in an LLM prompt. More advanced systems can add query rewriting, reranking, query routing, agents, and self-improvement. Each stage introduces possible failures. Source information may be missing or extracted incorrectly. Chunking and embeddings may not fit the domain. Retrieval may return the wrong documents or the wrong number of documents. Generation may produce incomplete answers or stray beyond the supplied context. The system also has to handle vague questions, conversation history, concurrent users, security, and compliance.
Domain data quality and representation shape retrieval quality
The speakers recommend improving the information that enters the pipeline before changing the language model. Data cleaning and better extraction can prevent missing facts from reaching retrieval. A general-purpose embedding model is a reasonable starting point, but an embedding model that understands the domain's terminology can improve results. Metadata can add information about documents and support filtering during retrieval. Chunking also needs testing. Reducing chunk size or using semantic chunking may help, while context size matters because relevant information can be lost when too much material is placed in the prompt.
Evaluation tells teams which RAG setting to change
A RAG pipeline has many settings that can be adjusted, so it is difficult to know what is wrong without measurements. Atita argues that evaluation provides a guided way to measure progress and make informed changes. Deanna explains that evaluation must match the underlying documents and domain. Quotient's evaluation workflow uses realistic inputs and expected outputs, then handles prompt formatting, LLM execution, and metric computation. If a team does not already have an evaluation set, Quotient can help generate one. The point is to create a repeatable loop instead of changing pipeline components based on isolated impressions.
Faithfulness matters more than answer quality when hallucinations mislead users
The demo uses Qdrant documentation as the source for a question-answering system. The first decision is what to optimize. Helpful answers matter, but avoiding inaccurate information matters more because a wrong answer could misguide users. The evaluation therefore focuses on faithfulness as the hallucination metric. Context relevance measures whether the retrieved documents contain the information needed to answer the question. Chunk relevance measures how much of the retrieved material is useful rather than noise. Together, these metrics distinguish a retrieval failure from an answer-generation failure.
More retrieved information can improve recall while lowering faithfulness
The first experiments use a naive RAG setup with an embedding model, chunking parameters, a retrieval window, and a Mistral instruct model. Increasing chunk size produces small gains in text-quality metrics, but faithfulness drops considerably. Context relevance rises because the system retrieves more of the information needed for answers, while chunk relevance falls because more of the retrieved text is irrelevant. The next experiment keeps the smaller chunks and expands the retrieval window. That combination achieves the best relevance and faithfulness scores among the early tests, showing why chunk size and retrieval count should be tested together.
Reranking removes some of the noise returned by vector search
The team examines poorly performing questions and finds many unrelated documents that repeat words such as "Qdrant," "support," and "search." Simply expanding the retrieval window might increase the chance of finding the right document, but it would also add more noise to the LLM context. Reranking offers another option. The system retrieves a broader set, scores those documents with a second model, and returns only the top few. Experiments with Mixedbread, Cohere, and Jina's ColBERT model improve context relevance and faithfulness overall. Cohere gives the best scores in this comparison, although some individual questions still fail.
Hybrid search helps when a domain depends on exact terminology
The remaining failures involve Qdrant documentation filled with specialist terms, acronyms, and jargon. General-purpose embedding models may miss those exact relationships, while training or fine-tuning a domain model could take substantial time and money. The team therefore tests hybrid search, which combines dense and sparse vectors. Sparse vectors help retrieve documents that share terminology with the query. Hybrid search with the Cohere reranker performs best across the measured metrics except chunk relevance. On the two difficult examples, context relevance and faithfulness move close to one, showing the value of matching the retrieval method to the language used in the source material.
RAG improvement is an iterative loop of measurement and hypotheses
Across 10 experiments, the demo raises faithfulness from 76 to just under 85 without changing the generic question-answering prompt. The process starts with a baseline, compares configurations using several metrics, inspects the questions and documents behind poor results, forms a hypothesis, and tests the next change. The speakers stress that evaluation data must stay current as the application changes. They also warn against adding complex RAG machinery without first choosing metrics that reveal the problem. Domain understanding and careful experiments produce the gains in this example.
"Starting from scratch you can improve your RAG system by making incremental changes evaluating using a combination of metrics that together can help you identify underlying issues then observing patterns in your data forming a hypothesis and repeating the process."16:46
Who should watch
You are building a RAG application and need to decide whether to change chunking, embeddings, retrieval size, reranking, or the LLM.
Your evaluation set reports a single score, but you need to understand which questions and retrieved documents cause failures.
Your source material uses specialist terminology and dense retrieval is missing exact matches.
