# How to Look at Your Data

Jeff Huber, Chroma & Jason Liu, 567 | AI Engineer World's Fair 2025 | 19:23

Source: https://www.youtube.com/watch?v=jryZvCuA0Uc
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/how-to-look-at-your-data
Published: 2025-08-06
Tags: embeddings, evals, product-strategy

## TL;DR
- Fast evals built from query and document pairs let teams test retrieval changes quickly on their own data.
- Conversation histories contain feedback about frustration, retries, errors, and tool use that can guide product decisions.
- Clustering conversations and comparing performance across segments helps teams decide what to fix, build, or ignore.

## Summary
Jeff Huber and Jason Liu present a two-part method for improving AI applications through data analysis. Huber focuses on inputs, especially retrieval. He recommends building fast evals from query and document pairs rather than relying on public benchmarks, guesswork, or expensive model-based judges. Synthetic queries can fill gaps, but they should be made representative of real user questions. Liu focuses on outputs. Conversation histories contain signals about user intent, frustration, errors, retries, and tool use. Teams can extract structured metadata, embed and cluster conversations, then compare evaluation results across those groups. This makes vague scores more useful. A low factuality score becomes actionable when it is tied to a specific segment, such as queries involving time filters or contract search. The speakers argue that this process can guide tool development, prompts, routing, onboarding, and product priorities. Continuous monitoring turns those decisions into a feedback loop.

## Key ideas
### Fast evals make retrieval experiments cheap enough to repeat
[01:42](https://www.youtube.com/watch?v=jryZvCuA0Uc&t=102s)
Jeff Huber defines a fast eval as a set of query and document pairs where each document is the expected result for its query. A golden dataset holds these pairs. The team runs all queries through the retrieval system and checks whether the expected documents appear, using retrieval at 5, 10, or 20 depending on the application. Because the metrics run quickly and cost pennies, engineers can compare many experiments instead of waiting hours between runs. Huber argues that measurement makes systematic improvement easier because teams can see whether a change actually improves success on the queries they care about.

### Synthetic queries are useful only when they resemble real user questions
[03:12](https://www.youtube.com/watch?v=jryZvCuA0Uc&t=192s)
Teams that have documents and chunks but no query set can use an LLM to generate questions. Huber warns against the naive prompt, "write me a question for this document." Public benchmark examples can be unrealistically clean, such as a question whose wording closely matches the document. In work with real query data, the speakers generated synthetic queries whose specificity was aligned with real user questions. This reduces the risk of choosing an embedding model because it performs well on an artificial benchmark while failing on the questions users actually ask.

### Your own retrieval data can overturn popular embedding-model rankings
[04:24](https://www.youtube.com/watch?v=jryZvCuA0Uc&t=264s)
Huber describes an evaluation of the Weights & Biases chatbot using logged ground-truth queries and generated queries. The results compared recall at 10 across four embedding models. The synthetic and real-query results were useful when they stayed close and preserved the same ordering of models. The original text-embedding-3-small model performed worst in this application. Gemini Embeddings v3 performed very well on the English MTEB benchmark but did not perform as well on this chatbot. Voyage 3 large performed best in the application after testing on its data.

### Conversation histories contain product feedback that users do not submit directly
[07:08](https://www.youtube.com/watch?v=jryZvCuA0Uc&t=428s)
Jason Liu says teams can manually inspect a few hundred conversations, but thousands of queries and long agent traces make that impractical. Conversations still contain useful signals. Users often write messages such as "Try again" or explain that the result was not what they meant. Retry patterns and frustration can reveal problems even when there is no thumbs-up widget or explicit feedback form. Those histories can show how users are working with the product, which tools are being used, and where the agent is failing.

### Structured metadata turns conversations into data that teams can analyze
[10:15](https://www.youtube.com/watch?v=jryZvCuA0Uc&t=615s)
Liu recommends extracting fields from each conversation, including a summary, tools used, errors, satisfaction, and frustration. The resulting metadata can be embedded and clustered. Instead of treating every conversation as an isolated transcript, teams can group similar interactions and test hypotheses about those groups. He describes a pipeline with models for summarization, clustering, and aggregation. Example clusters include data visualization, SEO content requests, and authentication errors. The groups provide a clearer view of how people use the software and where additional tools or workflows might help.

### Comparing evaluation results within segments makes vague scores actionable
[10:59](https://www.youtube.com/watch?v=jryZvCuA0Uc&t=659s)
A single metric such as factuality at 0.6 does not explain what to do next. Liu shows how the result becomes more useful when compared across clusters. Factuality might be low for queries that require time filters and high for contract search. That difference points to a specific engineering problem. The fix might be a missing filter, an additional OCR extraction step, a prompt change, or a new tool. Clustering connects evaluation results to recognizable user needs instead of treating the overall average as a product diagnosis.

### Usage and performance together determine what deserves engineering time
[14:30](https://www.youtube.com/watch?v=jryZvCuA0Uc&t=870s)
Liu proposes comparing how often a category appears with how well the system performs in that category. A heavily used capability with poor evaluation results should be fixed. A heavily used capability that performs well may need no immediate work. A rarely used capability that performs well may need better user education, prefilling, or product changes. A category that is rarely used and performs badly might only need a prompt telling the user the system cannot help. This gives teams a way to weigh product decisions by both impact and performance.

### Monitoring clusters can shape the product roadmap
[15:25](https://www.youtube.com/watch?v=jryZvCuA0Uc&t=925s)
Once teams identify categories, they can build classifiers and routers, add tools, and monitor performance over time. Liu says this analysis revealed that some newly onboarded customers used the application differently from historical customers. Tracking category-level metrics can expose those changes. The speakers describe progress as the ability to form good hypotheses and test many of them. Faster evals increase the number of experiments, while segmentation makes the hypotheses more specific. Together they create a data-driven way to decide what to fix, build, or ignore.

## Notable quotes
- "Our contention is that you can really only manage what you measure." (00:49)
- "A fast eval is simply a set of query and document pairs." (02:18)
- "The idea is that the data really already exists in this conversation." (09:02)
- "The ultimate goal is to understand what to do next." (13:03)
- "Once you can do that, you can make decisions on what to fix, what to build, and what to ignore." (18:04)

## Tools & references mentioned
- Chroma
- MTEB
- Weights & Biases
- W&B Weave
- text-embedding-3-small
- Gemini Embeddings v3
- Voyage 3 large
- Cura
- Anthropic
- Claude
- Chroma Cloud
- Peter Drucker

## Who should watch
- You are choosing chunking, embedding, or retrieval settings and need a fast way to test them against the queries your application actually receives.
- Your application has enough conversations that manual review no longer scales, but you still need to understand user intent, frustration, and tool failures.
- You are deciding which agent tools or product capabilities to build and want usage and evaluation results to support those choices.

## Related talks

- [Evals](https://aietalks.com/talks/evals) (Omar Khattab & Vitor Balogh & Rafal Wilinski, Zapier & Ido Pesal, Vercel & Randall, Bolt Foundry & Diego Rodriguez, Krea & Doug Guthrie & Ankur Goyal, Braintrust & Jeff Huber, Chroma & Jason Liu, 567 Studio & John Dickerson, Mozilla AI, 3:59:12)
- [How to build world-class AI products](https://aietalks.com/talks/how-to-build-world-class-ai-products) (Sarah Sachs, Notion & Carlos Esteban, Braintrust, 1:43:46)
- [Evals Are Not Unit Tests](https://aietalks.com/talks/evals-are-not-unit-tests) (Ido Pesok, Vercel, 15:22)
- [Evals 101](https://aietalks.com/talks/evals-101) (Doug Guthrie, Braintrust, 48:31)
- [Five hard earned lessons about Evals](https://aietalks.com/talks/five-hard-earned-lessons-about-evals) (Ankur Goyal, Braintrust, 19:46)