# Building Metrics that Actually Work David Karam, Pi Labs | AI Engineer World's Fair 2025 | 40:28 Source: https://www.youtube.com/watch?v=jxrGodnopHo Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/building-metrics-that-actually-work Published: 2025-07-29 Tags: benchmarks, evals, reinforcement-learning ## TL;DR - Reliable evals start with a small set of simple signals and improve through an ongoing feedback loop. - A scoring system combines objective code checks and subjective model-based questions into one calibrated score. - Trusted metrics can drive model comparison, prompt testing, synthetic data filtering, and online selection among multiple generated responses. ## Summary David Karam presents an evaluation method based on the scoring systems used in Google Search. Instead of relying on one broad question such as whether a response is helpful, teams should break quality into smaller signals that are easier to inspect and improve. These signals can use Python, natural-language questions, or specialized scoring models. The team then calibrates the combined score against human ratings or user feedback such as thumbs up and thumbs down. The workshop applies the method to a meeting summarizer, using a co-pilot, Google Sheets, and a Colab notebook. Participants generate good and bad examples, edit dimensions, compare prompts and models, and test online sampling strategies. Karam is clear that evals are application-specific and remain difficult. His practical advice is to begin with vibe testing, add more formal checks as the system grows, and treat evaluation as the place where domain knowledge accumulates. ## Key ideas ### Evaluation is an ongoing development process, not a one-time test [05:59](https://www.youtube.com/watch?v=jxrGodnopHo&t=359s) Karam describes evals as part of how an application is developed. Teams set up benchmarks, find metrics that work, calibrate them with human ratings and user data, then repeat the process as the application changes. He compares this with Google Search, where quality work involved evaluating a stochastic system and improving it over time. The workshop does not promise a universal solution. Its goal is to teach a methodology that can be adapted to a specific application. Karam also connects offline evaluation to online feedback loops, including thumbs up and thumbs down data from users. ### Start with simple signals and add complexity when the system needs it [10:34](https://www.youtube.com/watch?v=jxrGodnopHo&t=634s) The presenters recommend starting with vibe testing because it can take an application a long way, especially early on. As systems become larger or more complex, teams can add tracing, human evaluation, code-based checks, and natural-language model judges. These techniques have different costs and returns, so they should be layered in as simpler approaches become impractical. The proposed scoring-system method starts with roughly five or ten signals that are known to correlate with goodness. Engineers then inspect failures, add signals, and measure the application again. This creates a practical feedback loop instead of requiring a complete metric design at the start. ### A scoring system decomposes a broad quality judgment into inspectable dimensions [16:26](https://www.youtube.com/watch?v=jxrGodnopHo&t=986s) Karam explains scoring through the example of Google Search. Search scores documents using many signals, such as popularity, title quality, relevance, spam, and clickbait characteristics, before combining them into a single result. The same pattern can be applied to generated content. Individual signals should be easy to understand and inspect. Some can be deterministic Python checks, while others ask a natural-language question about the output. Lower-level checks tend to be objective, while higher-level dimensions can be more subjective. Combining them gives the application one score without hiding every decision inside a single vague prompt. ### More specific metrics make errors easier to analyze [18:00](https://www.youtube.com/watch?v=jxrGodnopHo&t=1080s) A broad question such as whether an answer is helpful delegates much of the judgment to a language model or a human rater. Karam argues that decomposing quality into more objective signals lowers variance and produces a higher-fidelity score. Engineers can then slice the data by individual dimensions and see what is failing. The method also avoids an all-or-nothing choice between having evals and having none. A team can keep adding metrics as it discovers what matters for its application. Google Search is offered as an example of a system with around 300 signals, although the workshop does not suggest every application needs that many. ### The workshop turns a meeting summarizer into a metric-building exercise [19:50](https://www.youtube.com/watch?v=jxrGodnopHo&t=1190s) The practical example is a meeting summarizer that takes a multi-person transcript and produces structured JSON. The output contains a title, action items, and key insights. Participants can start from a system prompt, existing examples, or criteria. The co-pilot proposes a scoring system, then lets users generate synthetic examples, request bad outputs, and create targeted failures such as broken JSON. Users can also ask it to add or remove dimensions. One example adds a check that the title contains fewer than 20 words. The generated system is a starting point for iteration, not a finished evaluation. ### Calibration connects metric scores to real user judgments [23:38](https://www.youtube.com/watch?v=jxrGodnopHo&t=1418s) The workshop uses a spreadsheet containing examples labeled with user thumbs up or thumbs down feedback. Participants copy their scoring criteria into the sheet, run the scorer over the examples, and inspect a confusion matrix. They can change dimensions, weights, or test data and see whether alignment improves. Karam says metrics are best understood as calibrated or uncalibrated rather than simply good or bad. The important question is whether a high score corresponds to an actually good result. Correlation analysis and confusion matrices help answer that question. A scoring system with smaller dimensions also makes the numerical analysis easier to interpret. ### Trusted scoring can control online generation as well as offline evaluation [13:17](https://www.youtube.com/watch?v=jxrGodnopHo&t=797s) Once a scoring system is reliable, it can be used beyond prompt and model testing. The presenters describe generating four or five responses at a higher temperature, scoring them online, and selecting the best one. They call this a simple technique used extensively at Google and compare it with online reinforcement learning. The Colab exercise also applies the score to model comparison and prompt comparison. Engineers can test several models on the same examples, or change a system prompt and check for regressions. The same scoring specification can therefore support offline comparisons and online agent or generation workflows. ### Specialized scoring models aim for stable, fast judgments [32:26](https://www.youtube.com/watch?v=jxrGodnopHo&t=1946s) Karam distinguishes the scoring models used in the workshop from ordinary decoder models. They are designed for high precision and low variance, with bidirectional attention, a regression head, and training focused on scoring rather than token generation. The interface is simple: ask a question, provide the data, and receive a score that can be inspected. The combined metrics are calibrated with thumbs up and thumbs down data using an extension of a generalized additive model. The presenters say the scorers can evaluate around 20 dimensions in under 50 milliseconds, which makes online use practical. At the time of the workshop, multilingual support was limited and multimodal support was planned. ## Notable quotes - "The biggest takeaway I want people to have is the methodology and the stuff that we have learned." (06:24) - "Search had 300 metrics." (12:25) - "Don't think of eval as testing in the classic sense. Think of these as the primary place where domain knowledge lives." (13:52) - "Metrics that work are not necessarily good metrics or bad metrics. They're either calibrated metrics or uncalibrated metrics." (23:38) - "The best thing is to sample from your logs some number of things and evaluate or just run it online." (31:50) ## Tools & references mentioned - Google Search - Pi Labs - Google Sheets - Google Colab - Hugging Face - DSPy - Onslaught - ChatGPT ## Who should watch - You are building an AI application and have only informal vibe testing or a few brittle code checks. - Your team has human ratings or user feedback but does not know how to turn them into a metric that tracks real quality. - You need one evaluation setup for prompt and model comparison, data filtering, or online selection among multiple responses. ## Related talks - [Evals Are Broken, Use Them Anyway](https://aietalks.com/talks/evals-are-broken-use-them-anyway) (Ara Khan, Cline, 19:04) - [Engineering Better Evals: Scalable LLM Evaluation Pipelines That Work](https://aietalks.com/talks/engineering-better-evals-scalable-llm-evaluation-pipelines-that-work) (Dat Ngo & Aman Khan, Arize, 24:46) - [Practical Tactics to Build Reliable AI Apps](https://aietalks.com/talks/practical-tactics-to-build-reliable-ai-apps) (Dmitry Kuchin, Multinear, 14:55) - [How to Run Evals at Scale: Thinking Beyond Accuracy or Similarity](https://aietalks.com/talks/how-to-run-evals-at-scale-thinking-beyond-accuracy-or-similarity) (Muktesh Mishra, Adobe, 09:25) - [Building and evaluating AI Agents](https://aietalks.com/talks/building-and-evaluating-ai-agents) (Sayash Kapoor, AI Snake Oil, 20:00)