RAG gives enterprise AI access to proprietary information without retraining a language model or sending an entire library into its context.
2
Retrieval quality has improved through better embedding models, hybrid search, reranking, query and document enrichment, domain-specific models, and vector compression.
3
RAG systems should gradually require fewer hand-built retrieval tricks as embedding models, rerankers, and language models absorb more of that work.
Summary
Tengyu Ma compares RAG with fine-tuning and long-context approaches for giving AI systems access to enterprise data. He argues that RAG is easier to update, govern, and use economically because it retrieves only the information needed for each query. He explains the retrieval stack, including embedding models, vector databases, rerankers, hybrid search, query decomposition, document enrichment, domain-specific embeddings, and compression. Ma then describes how RAG may evolve. Today, users assemble many parsing, chunking, graph, and recursive-search techniques because the underlying models have limits. He expects models to take over more of these improvements. He also discusses multimodal embeddings that can process screenshots of PDFs, slides, tables, and video frames, along with an embedding system that automatically chunks long documents and includes information from other chunks. The talk is practical and commercially grounded, while Ma is honest that retrieval quality remains uneven across different datasets.
Enterprise AI needs a way to access proprietary data
Ma says language models and agent systems cannot know a company's private information out of the box. If a model already knew MongoDB's internal data, that information could have been leaked. Enterprise applications therefore need a way to ingest and use proprietary documents. He compares three options: long-context systems place the documents directly in the model context, fine-tuning updates model parameters so the model can answer from learned information, and RAG retrieves a small subset of documents for each query. He focuses on RAG because it addresses private data without requiring the model to memorize the whole collection.
RAG is easier to update and govern than memorization or full-context reading
Ma uses a library analogy. Long context is like scanning an entire library for every question, while fine-tuning is like reading and memorizing the library in advance. Memorization makes it hard to decide what to retain or forget, and it complicates access control when different users can access different books. RAG keeps the information in a library and retrieves the relevant chapters when needed. That makes the system modular, and Ma describes it as reliable, fast, and cheap. Its basic flow is to embed documents and queries, search a vector database, provide the retrieved documents to a language model, and generate an answer from that context.
Retrieval quality has improved, but average scores hide large differences
Ma says retrieval models have improved substantially over the previous two years through work across data curation, data selection, architecture, loss functions, and evaluation. He describes better quality at the same model size, or smaller models at the same quality and cost. His evaluation averages results across about 100 datasets and reaches roughly 80% accuracy, leaving room for improvement. That average is uneven: around half of the datasets may reach 90% or 95%, while others score 60%, 30%, or 20%. Common retrieval tasks can already perform very well, but difficult domains still need better models and system design.
Embedding compression can cut storage costs with a measured accuracy tradeoff
Ma discusses Matryoshka learning and quantization as ways to reduce vector storage. With Matryoshka learning, an embedding remains useful when only an initial subset of its dimensions is stored. A 256-dimensional prefix of a 2,048-dimensional vector may retain nearly the same accuracy, with a small loss. Quantization lowers the precision of the vector. He says storage can be reduced by at least 10 times without much performance loss, while a 100-times reduction may cost around 5% to 10% in accuracy. Domain-specific models can improve this tradeoff further.
Several retrieval layers improve results beyond the embedding model
Ma lists practical ways to improve RAG after choosing an embedding model. Hybrid search combines lexical and other search methods, then uses a reranker to order the results. Query decomposition turns a complex question into smaller subqueries that can search different document subsets. Query expansion uses a language model to make the query more informative. Document enrichment adds titles, headers, categories, authors, dates, and other global information to chunks. He also mentions domain-specific embedding models, fine-tuned embedding models, graph-based retrieval, and iterative retrieval. These methods add layers around embeddings because no single retrieval or language model captures every useful signal.
RAG matches how people retrieve information from external memory
Ma expects RAG to remain a lasting part of AI systems because it resembles how people use large external collections. People keep information in libraries, retrieve a relevant subset, and use that subset to answer a question or take an action. This is more efficient than putting all information into the brain or reading every book for every question. In an enterprise, the same pattern lets an application use a small part of a much larger data store. Ma treats hierarchical selection as the basic reason RAG can remain useful even as language models improve.
RAG should move more work from application tricks into models
Ma compares the future of RAG with the evolution of language-model applications. Earlier systems required users to build many steps around a base model, while current models provide more capability out of the box. He expects the same change in retrieval. Current applications rely on parsing strategies, chunking methods, recursive search, contextual chunks, and graph RAG because embedding models, rerankers, and language models are imperfect. Over time, he expects the model layer to absorb more of the gains provided by these techniques. Some customization will remain for information that general-purpose models do not understand, including domain-specific definitions of similarity.
Multimodal and automatic chunking systems can simplify the retrieval pipeline
Ma describes multimodal embeddings that accept screenshots directly. A system can embed a PDF, PowerPoint slide, table, or figure without separately extracting text, identifying table headers, or creating separate image and text embeddings. Video can be handled by embedding screenshots from consecutive frames, although he does not present this as a perfect solution. He also describes an upcoming context-aware, automatic chunking system. It takes a long document, creates chunks and vectors, and includes information from other chunks in each vector. Each vector therefore contains details from its own chunk along with broader document context, which can improve focused retrieval without passing an entire long document to the language model.
"I do believe that RAG will be there forever because this is very similar to how humans are using additional large amounts of data."10:41
Who should watch
You are choosing between RAG, fine-tuning, and long-context methods for an application that must use private company data.
Your retrieval pipeline depends on manual chunking, query rewriting, reranking, or graph-based methods and you want to understand where model improvements may replace that work.
You need to reduce vector storage costs or retrieve information from PDFs, slides, tables, and other multimodal documents.
