# When All Context Matters: Extended Cache Augmented Generation Luis Romero-Sevilla, Orbis Operations | AI Engineer World's Fair 2026 | 05:52 Source: https://www.youtube.com/watch?v=XovaGv4f39A Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/when-all-context-matters-extended-cache-augmented-generation Published: 2026-06-28 Tags: caching, cost, long-context, rag ## TL;DR - Simple RAG can replace an obsolete document collection quickly, but it cannot pass every relevant document to the language model when the whole collection matters. - GraphRAG captures relationships across documents, but rebuilding its knowledge graph whenever rapidly changing data is replaced is computationally expensive and slow. - Extended Cache Augmented Generation distributes documents across parallel cached context buckets and uses a supervisor model to explore them and synthesize an answer. ## Summary Luis Romero-Sevilla describes a document collection where every document contributes to answering a question, while the documents are replaced frequently. Simple RAG can refresh its vector database quickly, but retrieving only similar chunks misses the wider context, and passing the whole collection to a language model is impractical. GraphRAG can model relationships between entities and documents, yet rebuilding the graph after each data replacement takes too much computation and time. His proposed approach, Extended Cache Augmented Generation, loads documents into several large-context model caches. A supervisor model queries these buckets, asks follow-up questions when needed, and combines their answers. Documents are distributed without domain labels because a supervisor may ignore a domain that looks irrelevant even when its documents contain useful connections. Parallel cache loading makes the process faster than GraphRAG, while the broader context can produce more accurate answers than Simple RAG. Cache lifetime remains a cost-control decision. ## Key ideas ### Simple RAG refreshes changing collections quickly, but it loses whole-collection context [00:03](https://www.youtube.com/watch?v=XovaGv4f39A&t=3s) Luis Romero-Sevilla begins with a collection where many documents describe one event, every document may matter to the user's question, and the contents become obsolete quickly. Simple RAG uses an embedding model to turn documents into vectors and stores them in a vector database. A question is also converted into a vector, and documents within a similarity threshold are retrieved for the language model. Replacing an obsolete collection is relatively fast because the vectors can be inserted into a new database. The limitation appears when all documents are relevant. Retrieving only similar documents cannot capture the full set, while sending the entire collection to the language model is impractical. ### GraphRAG models relationships across a collection by reading every document [01:58](https://www.youtube.com/watch?v=XovaGv4f39A&t=118s) Luis explains that a collection in which every document contributes to a global answer must contain relationships between details. GraphRAG uses a language model to read the documents, extract entities and relationships, and construct a knowledge graph. When a user asks a question, the system navigates that graph and synthesizes an answer from across the collection. This works well when the collection changes infrequently because the graph can preserve relationships that ordinary similarity search may miss. In the scenario he describes, however, the documents are deeply interconnected and replaced often. Recomputing the graph after each replacement is computationally expensive and takes a long time. ### Cache Augmented Generation avoids rebuilding a graph by putting documents into model context [02:59](https://www.youtube.com/watch?v=XovaGv4f39A&t=179s) Luis proposes continuing from the processing GraphRAG already requires. Since each document must pass through a language model for entity and relationship extraction, the system could instead place the documents directly into the context of a model with a large context window. This is Cache Augmented Generation, or CAG. The system loads the documents into context and caches that context by storing the model's KB matrix. The approach avoids creating an explicit knowledge graph, but it has a direct limit. The context window has a fixed capacity, and filling it too heavily degrades answer quality. A single cache therefore cannot hold an unlimited collection. ### ECAG spreads documents across parallel context buckets and gives each cache a local question-answering role [03:37](https://www.youtube.com/watch?v=XovaGv4f39A&t=217s) Extended Cache Augmented Generation addresses the context limit by using multiple CAG systems in parallel. Documents are distributed across separate context buckets, and each cache answers questions about its own contents. A smarter supervisor model decides which buckets to query and eventually synthesizes the result. Luis says the documents should not be organized by domain. With dense relationships, the supervisor may ignore a domain that initially looks irrelevant, even though it contains useful information. Instead, documents are distributed in no particular order, with the requirement that the number of documents be balanced so the system needs the smallest practical amount of material. ### The supervisor builds understanding progressively and can ask buckets follow-up questions [04:40](https://www.youtube.com/watch?v=XovaGv4f39A&t=280s) The supervisor does not have to resolve the entire collection in one pass. It explores the buckets, builds an internal understanding over time, and asks a particular bucket follow-up questions when it finds something interesting. This gives the system a way to connect evidence held in separate caches without requiring one cache to contain every document. Since the caches can be loaded in parallel, Luis says the knowledge-building process is significantly faster than GraphRAG. He also presents the approach as more accurate than Simple RAG for this particular problem because it can draw on a wider portion of the collection. ### Cache lifetime is the main cost control, and ECAG still has trade-offs [05:02](https://www.youtube.com/watch?v=XovaGv4f39A&t=302s) Luis acknowledges that KV caches can be expensive. He points to cache lifetime as one way to reduce the cost, by optimizing how long each cache remains active. He does not present ECAG as a universal replacement for other retrieval systems. His closing point is that retrieval strategies trade off compute, cost, and speed, so the appropriate design depends on the specific problem. For a rapidly changing collection with dense relationships, ECAG avoids repeatedly rebuilding a graph and avoids forcing the entire collection into one context window. The remaining design question is how many caches to use and how long to retain them. ## Notable quotes - "All the documents in the collection are relevant for us to answer the question." (01:36) - "Recomputing a knowledge graph every time the data gets replaced is computationally very expensive, and it takes relatively long time." (02:59) - "For this reason, all documents are distributed in no particular order." (04:19) - "Because all caches can be loaded in parallel, the knowledge building process is significantly faster than graph rag while providing more accurate answers than a simple rag." (04:40) - "Currently, there is no one-solution-fits-all." (05:26) ## Tools & references mentioned - Simple RAG - GraphRAG - Cache Augmented Generation - Extended Cache Augmented Generation - Orbis Operations - Luis Romero-Sevilla ## Who should watch - You are building retrieval for a dataset whose documents are replaced frequently, and ordinary top-similarity retrieval leaves out information that may matter. - Your system needs relationships across many documents, but rebuilding a GraphRAG knowledge graph after every update is too slow or expensive. - You are evaluating large-context caching and want a design that uses several context buckets with a supervisory model.