# The RAG Stack We Landed On After 37 Fails

Jonathan Fernandes | AI Engineer World's Fair 2025 | 18:52

Source: https://www.youtube.com/watch?v=2CXn-CByNoo
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/the-rag-stack-we-landed-on-after-37-fails
Published: 2025-06-03
Tags: evals, observability, rag, reranking

## TL;DR
- Jonathan Fernandes uses LlamaIndex for orchestration, Qdrant for vector search, open embedding models in production, and Docker for deployments that may need to run on-premise.
- A naive RAG system produced irrelevant help information, while adding a Cohere reranker returned a specific answer pointing to booth five at London St. Pancras International.
- Tracing with Arize Phoenix or LangSmith and testing with Ragas are needed to find slow or weak components across a larger evaluation set.

## Summary
Jonathan Fernandes walks through the RAG stack he settled on after 37 unsuccessful attempts. He starts with a small HTML knowledge base for a London railway operator and tests the question, "Where can I get help in London? I'm at the station." The initial LlamaIndex pipeline retrieves poor source material and gives an unhelpful answer. He then changes the embedding model, language model, and vector database, but the answer remains weak. A reranking step using a Cohere cross-encoder selects better documents and produces a useful answer about booth five at London St. Pancras International. Fernandes explains where bi-encoders and cross-encoders fit, why the latter is slower, and how tracing and evaluation expose problems. His production setup uses Docker containers for ingestion, Qdrant, model serving, Phoenix, and Ragas, with open models where on-premise processing is required.

## Key ideas
### The stack changes between a fast prototype and an on-premise production system
[00:40](https://www.youtube.com/watch?v=2CXn-CByNoo&t=40s)
Fernandes separates his workflow into prototyping and production. He usually prototypes in Google Colab because it provides access to a free hardware accelerator. Since his financial-institution work often requires data and processing to stay on premise, he uses Docker for production, either on-premise or in the cloud. His prototype choices include LlamaIndex or LangGraph for orchestration, while he usually uses LlamaIndex in production. He prefers closed APIs for simplicity during development, then moves toward open models from BAAI or NVIDIA when deploying.

### A naive RAG pipeline can retrieve plausible information that does not answer the question
[06:46](https://www.youtube.com/watch?v=2CXn-CByNoo&t=406s)
Fernandes describes retrieval-augmented generation as three connected steps. The system embeds the user query, searches a vector database for relevant documents, and combines the retrieved information with the query before sending it to a language model. In his example, the question is, "Where can I get help in London? I'm at the station." The naive LlamaIndex pipeline returns an answer about approaching London black cabs because most have ramps. The answer sounds plausible, but it does not identify the right help point from the railway knowledge base.

### Query processing and post-retrieval processing target different failure points
[08:49](https://www.youtube.com/watch?v=2CXn-CByNoo&t=529s)
Fernandes adds two places for improvement around retrieval. Query processing can remove information such as personally identifiable data before the request enters the RAG system. Post-retrieval processing can improve the accuracy of the documents selected from the vector database. He uses this distinction to explain why a RAG pipeline needs more than embedding a query and taking the nearest documents. Each stage can introduce a different kind of error, so the fix depends on where the weak result appears.

### Bi-encoders make retrieval fast, while cross-encoders improve a smaller candidate set
[09:28](https://www.youtube.com/watch?v=2CXn-CByNoo&t=568s)
A cross-encoder sends both the query and a document to a BERT model, then produces a score from 0 to 1 for their semantic similarity. Fernandes says this can improve accuracy, but it becomes slow and difficult to scale as documents get larger. A bi-encoder uses separate encoders for the query and documents, creates embeddings for each, and compares them with cosine similarity. That separation makes it fast and scalable for information retrieval. He places the bi-encoder in the vector database stage and the cross-encoder after retrieval, where it acts as a reranker over a smaller set of candidates.

### Changing the embedding and language models alone does not fix the retrieved context
[11:57](https://www.youtube.com/watch?v=2CXn-CByNoo&t=717s)
Fernandes replaces the in-memory vector store with Qdrant and changes the default OpenAI embedding from text-embedding-ada-002 to an open BGE small model from BAAI downloaded through Hugging Face. He also switches the language model from GPT-3.5 Turbo to GPT-4 and sets the temperature to zero. The answer improves from the black-cab response to advice about arriving 75 minutes before departure and telling a team member. He then inspects the source nodes and finds that the system used two HTML files that still did not contain the best answer. The changes have not solved document selection.

### Reranking the retrieved candidates produces the first useful answer
[14:49](https://www.youtube.com/watch?v=2CXn-CByNoo&t=889s)
Fernandes adds Cohere's reranking model as a node postprocessor. The pipeline retrieves five results from Qdrant, then reranks those candidates against the original question. This produces the best answer in the demonstration: at London St. Pancras International, the user can get help at booth number five next to the Eurostar ticket gates. The example shows the role of reranking clearly. Vector search narrows the document set quickly, while a more expensive comparison chooses the most relevant items from that smaller group.

### Tracing and evaluation are needed beyond a single successful query
[16:29](https://www.youtube.com/watch?v=2CXn-CByNoo&t=989s)
Fernandes says a production RAG system needs monitoring and tracing tools such as Arize Phoenix or LangSmith. These tools show how long each component takes, which helps with troubleshooting and finding whether the language model or another stage dominates runtime. He also warns that testing one question is not enough. A RAG evaluation framework such as Ragas can test many documents and assess the quality of the full solution. In production, he expects to use open reranking models from NVIDIA and open embedding or language models where the deployment requires them.

### Docker Compose ties the production components into separate containers
[17:17](https://www.youtube.com/watch?v=2CXn-CByNoo&t=1037s)
His production environment uses a Compose YAML file and separate images for data ingestion, Qdrant, the front-end application, model serving, Phoenix tracing, and Ragas evaluation. The ingestion image pulls HTML files from the knowledge base. Qdrant can be pulled from Docker Hub. Fernandes often uses Ollama to serve models, with Hugging Face Text Generation Inference as another option. The Compose setup lets these pieces run as containers, while the configuration records the embedding, reranking, and language models used by the system.

## Notable quotes
- "The first and most important step is the retrieval step." (06:46)
- "This solution is excellent for additional accuracy, but it's slow and not scalable." (10:04)
- "At London St. Pancras International. You can get help by going to booth number five next to the Eurostar ticket gates." (16:12)
- "What you'll want to do when working with a RAG solution is to have a RAG evaluation framework that will allow you to test on a whole load more documents." (16:45)

## Tools & references mentioned
- LlamaIndex
- LangGraph
- Google Colab
- Docker
- Qdrant
- OpenAI
- GPT-3.5 Turbo
- GPT-4
- BAAI
- BGE small
- Hugging Face
- Qwen
- Alibaba Cloud
- Meta
- Ollama
- Hugging Face Text Generation Inference
- Arize Phoenix
- LangSmith
- Cohere
- NVIDIA
- Ragas
- BERT
- Eurostar

## Who should watch
- You are building a RAG prototype and need a concrete path from a notebook experiment to a Docker-based deployment.
- Your vector search returns plausible but irrelevant documents, and you need to understand where reranking fits.
- You need to measure latency and answer quality across a test set instead of judging the system from one query.

## Related talks

- [The Hidden Costs of Building Your Own RAG Stack](https://aietalks.com/talks/the-hidden-costs-of-building-your-own-rag-stack) (Ofer, Vectara, 15:14)
- [Layering Every Technique in RAG, One Query at a Time](https://aietalks.com/talks/layering-every-technique-in-rag-one-query-at-a-time) (David Karam, Pi Labs, 20:22)
- [Navigating RAG Optimization with an Evaluation Driven Compass](https://aietalks.com/talks/navigating-rag-optimization-with-an-evaluation-driven-compass) (Atita Arora, Qdrant & Deanna Emery, Quotient AI, 18:14)
- [OpenRAG: An open-source stack for RAG](https://aietalks.com/talks/openrag-an-open-source-stack-for-rag) (Phil Nash, IBM, 15:52)
- [RAG Evaluation Is Broken! Here's Why (And How to Fix It)](https://aietalks.com/talks/rag-evaluation-is-broken-heres-why-and-how-to-fix-it) (Yuval Belfer & Niv Granot, A21 Labs, 10:58)