# Evaluating AI Search: A Practical Framework for Augmented AI Systems

Julia Neagu, Deanna Emery & Mara Sher, Quotient AI | AI Engineer World's Fair 2025 | 20:33

Source: https://www.youtube.com/watch?v=wRJD0inpmjU
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/evaluating-ai-search-a-practical-framework-for-augmented-ai-systems
Published: 2025-07-29
Tags: debugging, evals, rag, search

## TL;DR
- AI search needs dynamic evaluation because web content and user questions change over time.
- Answer completeness, document relevance, and hallucination detection measure different parts of search quality and expose trade-offs between them.
- Evaluation is most useful when it explains why a response failed and points to a fix, rather than only ranking providers.

## Summary
Julia Neagu, Deanna Emery, and Mara Sher describe an evaluation framework for AI search systems that use web search, retrieval, and generation. They explain why static benchmarks such as SimpleQA and HotpotQA miss problems in real-time systems, where information changes and questions are often subjective or unexpected. Quotient AI and Tavily built a dynamic evaluation agent that creates evidence-backed question and answer pairs from current web content, using multiple search providers to improve coverage and reduce bias. In a comparison of six anonymized providers, scores and rankings changed substantially between SimpleQA and the dynamic benchmark. The talk then examines reference-free metrics for answer completeness, document relevance, and hallucination detection. These metrics reveal different failure modes and trade-offs. The speakers argue that evaluation should guide product changes, such as retrieving more documents when answers are incomplete, and eventually support agents that detect and correct their own failures.

## Key ideas
### AI search systems fail in connected ways that traditional monitoring misses
[00:36](https://www.youtube.com/watch?v=wRJD0inpmjU&t=36s)
Julia Neagu says modern AI systems change as web content, user interactions, and tool chains change. They make real-time decisions rather than following fixed software logic. Several failures can happen together, including hallucinations, retrieval failures, and reasoning errors. Quotient AI monitors live agents with expert evaluators that can identify objective failures without waiting for ground-truth data, human feedback, or a benchmark. This matters for search agents because a single bad answer can result from several linked parts of the system, and a simple pass or fail signal does not explain which part broke.

### Real-time web search requires evaluation data that changes with the web
[02:17](https://www.youtube.com/watch?v=wRJD0inpmjU&t=137s)
The speakers describe two sources of unpredictability in Tavily's agents: the web keeps changing, and users ask questions outside prepared test cases. Traditional benchmarks assume stable ground truth, but current information can make yesterday's answer wrong today. Users can also ask malformed questions or leave important context unstated. Static sets such as SimpleQA work well for short fact-seeking questions with one empirical answer, while HotpotQA tests multi-hop reasoning across documents. Neither fully addresses changing information or questions with subjective, contextual answers.

### Dynamic evaluation sets keep tests aligned with current information
[09:15](https://www.youtube.com/watch?v=wRJD0inpmjU&t=555s)
The team argues that dynamic data sets are needed for production RAG systems because they can be refreshed regularly and built for a specific domain or use case. Their open-source agent generates web search queries, gathers grounding documents from multiple real-time search providers, and creates evidence-based question and answer pairs. The use of multiple providers is intended to increase coverage and reduce bias rather than evaluating Tavily with data gathered only from Tavily. Each generated pair keeps the sources and evidence used to create it, so teams can inspect how the test was formed. The agent uses LangGraph, and the team uses LangSmith to track evaluation runs.

### Static and dynamic benchmarks can produce very different provider rankings
[10:15](https://www.youtube.com/watch?v=wRJD0inpmjU&t=615s)
Deanna Emery describes an experiment comparing six anonymized AI search providers on SimpleQA and a dynamic benchmark of about a thousand rows. The topic distributions were roughly similar, but dynamic benchmark correctness scores were substantially lower than the reported SimpleQA scores, and the relative rankings changed. Provider F performed worst on SimpleQA and best on the dynamic set. She also shows limits in the SimpleQA LLM judge. It sometimes marked an answer incorrect even when the ground-truth answer appeared in the output, while a marked-correct response could contain extra hallucinated material that the evaluation ignored.

### Reference-free metrics measure separate parts of answer quality
[13:38](https://www.youtube.com/watch?v=wRJD0inpmjU&t=818s)
When production ground truths are unavailable, the speakers use three reference-free metrics. Answer completeness checks whether every part of the question was addressed and classifies responses as fully addressed, unaddressed, or unknown. Document relevance measures the percentage of retrieved documents that help answer the question. Hallucination detection checks whether facts in the answer appear in any retrieved document. Completeness tracked the dynamic benchmark's relative performance closely, with a reported correlation of 0.94, but the speakers stress that completeness is not correctness. The other metrics use the grounding documents to provide more information about response quality.

### Retrieved documents are more useful for debugging than citations alone
[15:47](https://www.youtube.com/watch?v=wRJD0inpmjU&t=947s)
Only three of the evaluated search providers returned the documents used to generate their answers. Most returned citations, which the speakers describe as unhelpful at scale and limiting for debugging. Document relevance had a strong inverse relationship with unknown answers: when no relevant documents were available, systems were more likely to say they did not know. Hallucination results were less intuitive. One provider had the highest document relevance and the highest hallucination rate. The speakers suggest that detailed reasoning or additional interpretation may create more opportunities for unsupported claims.

### Evaluation should point to a repair strategy, not only rank systems
[18:19](https://www.youtube.com/watch?v=wRJD0inpmjU&t=1099s)
The three metrics can be read together to diagnose failures. If a response is incomplete but the retrieved documents are relevant and there are no hallucinations, the system may need more documents to cover the question. The speakers present this as an example of evaluation leading directly to a product change. Different applications may weight completeness, relevance, or hallucination risk differently because improving one dimension can hurt another. Their longer-term goal is for agents to learn which sources become outdated or unreliable, detect hallucinations during conversations, and correct their course without human intervention.

## Notable quotes
- "You can't answer today's questions with yesterday data." (05:55)
- "But the big picture idea is that your evaluation should do more than just provide relative rankings." (18:59)
- "They're measuring different dimensions of response quality and it's often a give and take." (17:58)
- "Most online and production settings this is typically the case." (13:57)

## Tools & references mentioned
- Quotient AI
- Tavily
- SimpleQA
- HotpotQA
- LangGraph
- LangSmith
- OpenAI
- GitHub Copilot

## Who should watch
- You are building a RAG or web-search agent and your offline benchmark scores do not match production behavior.
- Your system has no reliable ground-truth answers, but you still need to measure completeness, source quality, and unsupported claims.
- You need evaluation results that suggest what to change in retrieval or generation instead of only producing a provider ranking.

## Related talks

- [Shipping AI That Works: An Evaluation Framework for PMs](https://aietalks.com/talks/shipping-ai-that-works-an-evaluation-framework-for-pms) (Aman Khan, Arize, 1:26:16)
- [Mastering AI Evaluation: From Playground to Production](https://aietalks.com/talks/mastering-ai-evaluation-from-playground-to-production) (Doug & Carlos Essan, Braintrust, 1:25:08)
- [The State of AI-Powered Search and Retrieval](https://aietalks.com/talks/the-state-of-ai-powered-search-and-retrieval) (Frank Liu, MongoDB, 12:35)
- [Evals](https://aietalks.com/talks/evals) (Omar Khattab & Vitor Balogh & Rafal Wilinski, Zapier & Ido Pesal, Vercel & Randall, Bolt Foundry & Diego Rodriguez, Krea & Doug Guthrie & Ankur Goyal, Braintrust & Jeff Huber, Chroma & Jason Liu, 567 Studio & John Dickerson, Mozilla AI, 3:59:12)
- [How to Look at Your Data](https://aietalks.com/talks/how-to-look-at-your-data) (Jeff Huber, Chroma & Jason Liu, 567, 19:23)