Knowledge graphs capture relationships between entities that semantic vector retrieval can miss.
2
A hybrid RAG system combines an offline graph and vector-building process with online retrieval and answer generation.
3
Most of the tuning effort goes into ontology design, triplet quality, retrieval depth, latency, and evaluation.
Summary
Mitesh Patel explains how to combine GraphRAG with vector retrieval for questions that depend on relationships between entities. A knowledge graph stores triplets such as entity, relationship, and entity, while a vector database stores embeddings from document chunks. The graph can preserve connections that ordinary semantic retrieval may lose, especially when answering questions that require multiple hops across related entities. Patel describes the system as four stages: data, data processing, graph or vector database creation, and inference. He spends particular attention on ontology design and LLM prompting, since noisy triplets produce noisy retrieval. He also covers chunk overlap, multi-hop search, latency, graph acceleration, and end-to-end evaluation with Ragas. In one experiment, cleaning text, shortening outputs, and fine-tuning a Llama model improved triplet accuracy from 71% to 87% on 100 documents. Patel's recommendation is conditional: use graph retrieval when the data and question require complex relationships, and consider a hybrid system when vector similarity alone is insufficient.
Knowledge graphs preserve relationships between entities
Patel describes a knowledge graph as a network of entities connected by relationships. The entities might be people, places, concepts, or events. His example is a speaker connected to a conference by the relationship "speaker at", and an attendee connected to the session by the relationship "attended". These edges let a retrieval system use information about how entities relate. Patel argues that this is the main difference from a semantic RAG system, which may find related text without explicitly representing the connection between the entities.
The system has four parts: data, data processing, graph or vector database creation, and inference. Patel divides these into offline and online stages. Offline processing turns documents into vector embeddings stored in a vector database and extracts graph triplets consisting of entities and their relationships. Once those stores exist, the online path answers a question by querying them and turning the retrieved information into readable prose. The answer should not expose raw relationships that the user must interpret themselves.
Ontology design determines the quality of graph retrieval
Extracting triplets from unstructured documents is difficult, so Patel recommends using an LLM with a defined ontology. The ontology describes which entities and relationships matter for the particular use case. A prompt can then ask the model to extract only that structured information and save it as triplets. Patel says teams should expect repeated iteration here. If the ontology is wrong or the triplets are noisy, retrieval will also be noisy. In his estimate, this work can consume 80% of the effort spent building the system.
Vector retrieval depends on chunk size and overlap
The vector side starts by splitting documents into chunks, converting each chunk into an embedding, and storing the vectors. Chunk size affects the amount of information represented in each retrieval unit. Overlap helps preserve context across chunk boundaries. Without overlap, a relationship that crosses from one chunk into the next can be lost. Patel presents this as a well-studied and relatively straightforward part of the system, while graph construction requires more application-specific decisions.
Multi-hop graph retrieval trades context for latency
A single-hop graph search can find an entity and its immediate relationships, but it may miss the wider context available through several connected nodes. Patel recommends testing retrieval depth, including how far the search should move from the first node to the second and third. Deeper searches can provide more context, but they take longer. Production systems therefore need a practical balance between the number of hops and the latency the application can tolerate. Graph search acceleration can make deeper retrieval more feasible.
Patel recommends Ragas for testing a RAG workflow from end to end. It evaluates the query, the retrieval process, and the generated response. The measures he names include faithfulness, answer relevancy, precision, recall, helpfulness, coherence, complexity, and verbosity. Ragas uses an LLM under the hood, with GPT integrated by default, but it can also be connected to another model through an API. This lets a team inspect which part of the pipeline is producing a weak final answer.
Small data-processing changes improved triplet accuracy
Patel describes experiments that changed the quality of generated triplets. The team cleaned text by removing characters such as apostrophes and other material they considered unimportant for triplet generation. They also reduced overly long outputs and fine-tuned a Llama model with LoRA. On a test of 100 documents, the reported accuracy rose from 71% with the model used as-is to 87% after fine-tuning, with additional cleaning and output changes improving the result further. Patel cautions that the score could fall as the document pool grows.
The right retrieval design depends on the data and question
Patel does not prescribe graph retrieval for every application. Structured datasets such as retail data, financial-services data, and employee databases can be good graph use cases because their relationships are already defined. Unstructured data can also fit if a useful knowledge graph can be created from it. The deciding factor is whether the application needs to understand complex relationships to answer questions. Graph systems require more computation, so Patel recommends testing that need before adding the extra machinery.
"The more deeper you go, the better context you'll get. But there's a disadvantage of that. The more deeper you go, the more time you're going to spend on retrieving that information."09:50
Who should watch
You are deciding whether a vector-only RAG system can answer questions that depend on entity relationships.
You are building GraphRAG and need practical guidance on ontology design, triplet extraction, hop depth, and latency.
You want an evaluation workflow for retrieval and generated answers, or examples of tuning graph construction quality.
