# When Vectors Break Down: Graph-Based RAG for Dense Enterprise Knowledge

Sam Julien, Writer | AI Engineer World's Fair 2025 | 15:47

Source: https://www.youtube.com/watch?v=XlAIgmi_Vow
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/when-vectors-break-down-graph-based-rag-for-dense-enterprise-knowledge
Published: 2025-07-22
Tags: embeddings, rag, search

## TL;DR
- Vector retrieval can return the wrong answer when documents use highly concentrated, repeated language or when naive chunking combines related facts.
- Writer combined context-aware splitting, a specialized text-to-graph model, JSON stored in a Lucene-based search engine, knowledge graphs, and Fusion-in-Decoder.
- Writer reported higher accuracy and faster responses than seven vector search systems on Amazon's RobustQA dataset, while graph relationships enabled multihop answers and exposed retrieval sources.

## Summary
Sam Julien explains why Writer moved beyond standard vector retrieval for enterprise RAG. Large companies often have terabytes of documents with repeated terms, similar product specifications, and facts spread across pages. Nearest-neighbor search can select the wrong passage or lose the relationship between facts. Writer's approach converts context-aware chunks into graph nodes and edges with a specialized model, stores the resulting data as JSON in a Lucene-based search engine, and combines graph retrieval with Fusion-in-Decoder. Julien presents this as a historical account rather than a production blueprint. The team changed its design when graph databases created scaling, cost, and query challenges. He reports that the resulting system performed best on Amazon's RobustQA benchmark against seven vector search systems, with the fastest response time in that comparison. The graph also supports multihop questions, source display, subqueries, and answers involving complex document layouts.

## Key ideas
### Enterprise documents become difficult for vectors when their language is dense and repetitive
[03:35](https://www.youtube.com/watch?v=XlAIgmi_Vow&t=215s)
Julien says enterprise data is often measured in terabytes and uses specialized language with little variation between documents. A mobile phone company might have thousands of documents that all mention megapixels, cameras, and battery life. When a user asks the system to compare two phone models, vector retrieval can return many plausible passages without preserving enough structure to tell them apart. This is different from a broad collection where documents discuss clearly unrelated subjects. Writer's retrieval work focused on these concentrated patterns because they made ordinary similarity search unreliable for enterprise questions.

### Naive chunking can separate or confuse facts that humans connect easily
[05:29](https://www.youtube.com/watch?v=XlAIgmi_Vow&t=329s)
Writer first tried regular search, then chunking documents, embedding the chunks, and using nearest-neighbor search. Julien gives an example about Apple's founding timeline: a human can identify that the Macintosh was created in 1984, but naive chunking places that fact near the Lisa's introduction in 1983. The vector search can then associate the wrong year with the Macintosh. Julien's point is that matching a chunk by similarity does not guarantee that the answer preserves the right relationship between facts. Context-aware splitting became part of Writer's later design.

### A graph preserves relationships while retrieval selects evidence
[06:44](https://www.youtube.com/watch?v=XlAIgmi_Vow&t=404s)
Writer moved to graph-based RAG because a graph could preserve relationships in the source text and provide more context to the model. The system queried a graph structure to retrieve relevant documents and then generated an answer. Julien says this could be combined with full-text search and similarity search rather than used alone. The graph approach was especially useful for concentrated enterprise data, where many passages contain the same terms. The team's decision came from trying to solve customer problems rather than following the retrieval technique that was most popular at the time.

### Writer replaced difficult graph infrastructure with components that matched its team's expertise
[07:42](https://www.youtube.com/watch?v=XlAIgmi_Vow&t=462s)
The team ran into four practical issues with graph databases at the time: converting data into a structured graph was costly at scale, database growth exposed expertise and cost limits, Cypher struggled with the required similarity matching, and language models handled text-based queries better than complex graph structures. Writer responded by building a specialized model that mapped data into graph nodes and edges and could run on CPUs or smaller GPUs. It also used context-aware splitting and chunking to preserve semantic relationships. Instead of keeping the graph in a graph database, Writer converted the graph structure to JSON and stored it in a Lucene-based search engine.

### Fusion-in-Decoder lets graph relationships improve evidence aggregation
[10:27](https://www.youtube.com/watch?v=XlAIgmi_Vow&t=627s)
Julien revisits the original RAG paper, which describes a retriever and a generator without the prompt-and-context pattern that later became common. He describes Fusion-in-Decoder as an alternate path built from that architecture. Passages are processed independently in the encoder for linear scaling, then combined in the decoder for evidence aggregation. Writer applied knowledge graphs to Fusion-in-Decoder so the system could understand relationships among retrieved passages. Julien says the graph-based version improved efficiency and reduced cost, and Writer built its own implementation because it develops its own models. The team later published a white paper with its findings.

### The reported benchmark result came from combining several retrieval techniques
[13:19](https://www.youtube.com/watch?v=XlAIgmi_Vow&t=799s)
Writer compared its retrieval system, which combined a knowledge graph with Fusion-in-Decoder, against seven vector search systems using Amazon's RobustQA dataset. Julien says Writer achieved the best accuracy and the fastest response time in that comparison. The talk does not present the benchmark as a single magic component. He describes the system as an assembly of techniques chosen through research and repeated problem solving. The product description gives the reported accuracy as 86.31 percent, while the talk itself focuses on the comparison's accuracy and response-time ranking.

### Graph structure adds product capabilities that vector retrieval struggles to provide
[13:55](https://www.youtube.com/watch?v=XlAIgmi_Vow&t=835s)
The graph lets Writer expose snippets, subqueries, and sources that show how an answer was retrieved. Those details can appear in the product and in the API for developers. The system can also answer multihop questions by reasoning across multiple documents and topics. Julien says the approach helps when an answer is split across pages, or when a similar term does not exactly match the user's wording. His practical advice is to use whatever architecture preserves useful relationships, including a graph database, a search engine, or another design that stores graph-derived data.

## Notable quotes
- "The market is starting to catch up that vector search is just not enough for rag at scale." (00:35)
- "Enterprise data is really dense, specialized and massive." (07:35)
- "The main takeaway being as you're seeing in several of these talks like the first talk about hybrid search there are many different ways that you can get the benefits of knowledge graphs in rag." (02:28)
- "So rather than staying focused on the solution, you know, step back, look at the research and figure out what you can do to solve the challenges for your customers." (10:52)

## Tools & references mentioned
- Writer
- Sam Julien
- Joe Christian Bergam
- Latent Space
- Palmyra X5
- Palmyra Med
- Palmyra Fin
- Failsafe QA
- Amazon RobustQA
- Fusion-in-Decoder
- Facebook
- Cypher
- Lucene

## Who should watch
- You are building RAG for product documentation, financial records, technical specifications, or another corpus where many documents repeat the same terms.
- Your vector system retrieves plausible passages but loses the relationship between facts, mixes versions, or cannot answer questions spanning multiple documents.
- You are deciding whether to add a graph database and want an example of using graph-derived data with a search engine instead.

## Related talks

- [HybridRAG: A Fusion of Graph and Vector Retrieval](https://aietalks.com/talks/hybridrag-a-fusion-of-graph-and-vector-retrieval) (Mitesh Patel, NVIDIA, 20:24)
- [Intro to GraphRAG](https://aietalks.com/talks/intro-to-graphrag) (Zach Blumenfeld, Neo4j, 1:18:35)
- [Agentic GraphRAG: Simplifying Retrieval Across Structured and Unstructured Data](https://aietalks.com/talks/agentic-graphrag-simplifying-retrieval-across-structured-and-unstructured-data) (Zach Blumenfeld, Neo4j, 15:25)
- [Graph Intelligence: Enhance Reasoning and Retrieval Using Graph Analytics](https://aietalks.com/talks/graph-intelligence-enhance-reasoning-and-retrieval-using-graph-analytics) (Alison Cossette & Andreas Kollegger, Neo4j, 1:41:16)
- [GraphRAG: The Marriage of Knowledge Graphs and RAG](https://aietalks.com/talks/graphrag-the-marriage-of-knowledge-graphs-and-rag) (Emil Eifrem, Neo4j, 19:15)