Open-RAG-Eval evaluates retrieval and generation without requiring golden answers or golden chunks.
2
UMBRELA scores retrieved passages from zero to three, while AutoNuggetizer checks whether important answer nuggets are supported.
3
The project collects RAG outputs through connectors and presents evaluation results in files or a web interface.
Summary
Ofer Mendelevitch presents Open-RAG-Eval, an open-source framework for evaluating RAG systems without manually prepared golden answers or golden chunks. Users provide important queries, then a connector collects the retrieved chunks and generated answers from a RAG pipeline. Evaluators calculate retrieval, generation, citation, and hallucination metrics. UMBRELA scores each retrieved passage from zero to three and is based on University of Waterloo research showing a correlation with human judgment. AutoNuggetizer creates atomic facts called nuggets, selects important ones, and uses an LLM judge to determine whether the response supports them fully or partially. Other metrics check citation support and whether the full response matches retrieved content. Results can be inspected as files or uploaded to open-evaluation.ai for comparison across queries. The framework includes connectors for Vectara, LangChain, and LlamaIndex, and supports contributions for other RAG pipelines.
Open-RAG-Eval removes the need for manually prepared golden data
Mendelevitch frames the main problem as the need for golden answers or golden chunks in RAG evaluation. Preparing those references does not scale, especially when a system has many important queries. Open-RAG-Eval uses an open-source, research-backed approach developed with the University of Waterloo and the Jimmy Lin lab. The goal is to evaluate a RAG pipeline using the system's queries, retrieved content, and generated answers without first creating a reference answer for every case.
Connectors turn a set of queries into evaluation data
The workflow starts with a collection of queries, which might contain 10, 100, or 1,000 items that matter to a RAG system. A RAG connector gathers the actual retrieved chunks and generated answers from the pipeline. Mendelevitch mentions connectors for Vectara, LangChain, and LlamaIndex, with more possible through contributions. The connector outputs are passed to evaluators, which calculate metrics and produce RAG evaluation files containing the information needed to assess the pipeline.
UMBRELA evaluates each retrieved chunk against a query on a scale from zero to three. A zero means the passage has nothing to do with the query. A three means it is dedicated to the query and contains the exact answer. The value of the method is that it does not need a known-good chunk for comparison. Mendelevitch says research from the University of Waterloo's Jimmy Lin lab found that this scoring approach correlates well with human judgment.
AutoNuggetizer checks generation through atomic facts
AutoNuggetizer evaluates a generated answer without requiring a golden answer. It first creates atomic units called nuggets, assigns each nugget a vital or okay rating, and sorts them to select the top 20. An LLM judge then examines the RAG response and determines whether each selected nugget is fully supported or partially supported by the answer. This gives the evaluation a set of specific claims to check instead of relying on a single reference response.
Citation faithfulness checks whether cited passages support the response
The citation faithfulness metric examines whether a cited passage supports the claim made in the response. It classifies the citation as fully supported, partially supported, or unsupported. This evaluates the connection between the answer and its cited evidence, rather than only checking whether the response contains a citation. The framework also includes a hallucination detector from Vectara that checks whether the entire response aligns with the retrieved content.
The evaluation UI makes result files easier to compare
After an evaluation runs, Open-RAG-Eval produces files that can be inspected directly, although Mendelevitch says this can be complex. Users can drag and drop those files onto open-evaluation.ai. The interface shows the queries that were run and lets users compare retrieval scores, generation scores, and other evaluation results. The UI provides a view over the metrics without requiring users to read the underlying files manually.
Open connectors and visible metrics make the framework inspectable
Mendelevitch presents the project as fully open source, so users can inspect how the metrics work. He connects this to transparency: the evaluation methods are visible rather than hidden behind an opaque service. The existing connectors cover Vectara, LangChain, and LlamaIndex. Teams using another RAG pipeline are invited to contribute connectors through issues or pull requests.
"Open-RAG-Eval is an open source project that is aiming to solve one of the major problems of RAG evaluation, the fact that you require golden answers or golden chunks to do anything which is really nonscalable."00:22
Who should watch
You are evaluating a RAG system and do not have the time or staff to create golden answers and golden chunks for every query.
Your pipeline needs separate signals for retrieval quality, answer support, citation faithfulness, and hallucination.
You maintain a RAG stack outside the listed connectors and want to inspect or contribute an evaluation integration.
