# Lessons from the Trenches: Building LLM Evals That Work IRL

Aparna Dhinakaran, Arize AI | AI Engineer World's Fair 2024 | 18:49

Source: https://www.youtube.com/watch?v=nbZzSC5A6hs
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/lessons-from-the-trenches-building-llm-evals-that-work-irl
Published: 2025-02-06
Tags: evals, observability, rag

## TL;DR
- Task evaluations test whether an LLM application performs its intended job, while model evaluations compare general model capabilities.
- Complex applications need evaluations at the router, component, trace, span, and session levels so teams can locate failures.
- Evaluations with explanations help engineers understand what went wrong and decide what to change.

## Summary
Aparna Dhinakaran distinguishes model evaluations from task evaluations. Model evaluations compare foundation models on general tests, while task evaluations check whether a particular application works. She uses a commerce application with an LLM router to show why routing and function-call arguments need their own evaluations. A wrong route can send the rest of the workflow down the wrong branch, so teams need traces, explanations, datasets, and repeated experiments to improve prompts and function descriptions. Dhinakaran also argues that evaluations should exist at several levels, from sessions and traces down to spans and individual components. She shares research showing that numeric judge scores can collapse into binary outcomes instead of providing useful detail. Her retrieval research finds that the position of information in a long context affects results. A generation test also shows that GPT-4 can improve substantially when prompted to explain itself before answering.

## Key ideas
### Task evaluations check whether an application does its job
[00:53](https://www.youtube.com/watch?v=nbZzSC5A6hs&t=53s)
Dhinakaran separates model evaluations from task evaluations. Model evaluations compare foundation models on general benchmarks, such as tests seen on the Hugging Face leaderboard, while task evaluations ask whether a specific LLM application works. For an application, the useful question is whether the system sends a user's request down the right path and produces the intended result. She describes a commerce example where a customer asks about a Kindle or a phone. The system first identifies intent, then chooses a function and executes a workflow. That application needs an evaluation designed around its task, rather than a general ranking of model capability.

### A router failure can invalidate the rest of an application workflow
[02:25](https://www.youtube.com/watch?v=nbZzSC5A6hs&t=145s)
In the commerce example, an LLM decides user intent and chooses a function call. Dhinakaran says this is where teams need to check whether the system selected the correct route and extracted the correct parameters. She shows a request about current promotions for a Samsung phone. The application chose a product-search function instead of the available promotions or discounts function. Because the first call was wrong, the rest of the execution branch was also wrong. This pattern appears in applications that combine router calls, other LLM calls, application code, search, and traditional machine-learning models.

### Evaluations should match the levels of the application
[04:08](https://www.youtube.com/watch?v=nbZzSC5A6hs&t=248s)
Dhinakaran recommends evaluating complex systems at several levels. A router evaluation checks whether the system chose the right execution branch. Component evaluations check the individual steps inside that branch. For applications with multiple turns, teams may also need session-level, trace-level, and span-level evaluations. A single pass or fail result does not tell an engineer which part needs attention. The evaluation should connect to the trace so the team can inspect what happened, identify the failing component, and decide what to change. As applications become more complex, the evaluation structure has to become more detailed as well.

### Explanations make evaluation results useful for fixing software
[10:28](https://www.youtube.com/watch?v=nbZzSC5A6hs&t=628s)
Dhinakaran says teams deploying applications in production get the most use from evaluations that include explanations. A bare incorrect result tells an engineer that something failed, but it does not say what to fix. In her example, the explanation identifies the mismatch between a promotions request and the product-search function that was selected. Teams can add these failures to a dataset, run experiments, and compare changes such as a revised prompt or a clearer function description. She describes this as a loop: inspect traces, evaluate them, understand the failure, change the application, and run the experiments again.

### Evaluation development continues from benchmarking into production
[09:07](https://www.youtube.com/watch?v=nbZzSC5A6hs&t=547s)
Dhinakaran describes evaluation work as an iterative process. Teams begin by benchmarking evaluations while they build. As they develop individual application components, they also develop and revise the evaluation templates for those components. After the application moves into production, they can monitor it and run evaluations as jobs. This process applies to more than one overall application score. The evaluation needs to provide enough detail to show which component failed and why. That information lets teams build a growing dataset of real failures and use it to test future changes.

### Numeric judge scores often fail to provide useful granularity
[11:55](https://www.youtube.com/watch?v=nbZzSC5A6hs&t=715s)
Dhinakaran shares a test of numeric outputs from LLM judges. Her team created two documents with spelling errors, one with about 80 percent corruption and another with about 11 percent corruption, then asked an LLM judge to score how bad the errors were. Both received a score of 10. Across the foundation models they tested, the outputs were often binary, with scores close to either 1 or 10 instead of a useful linear range. Dhinakaran says this makes a numeric score less informative because an 80 percent corruption level may not produce a meaningfully different result from a 10 percent level. She recommends not relying only on numeric evaluations.

### Information position in a long context affects retrieval
[13:58](https://www.youtube.com/watch?v=nbZzSC5A6hs&t=838s)
Dhinakaran discusses needle-in-a-haystack research that tests whether a model can retrieve a fact hidden in a context window. Her team varied where the fact appeared, including near the first 5 percent of the context and around 90 percent into it. The work matters for retrieval-augmented generation because retrieved information can appear at different positions in the supplied document. In the results she shows for Anthropic Claude 2.1 and GPT-4, GPT-4 retrieved the fact more successfully overall. However, the models repeatedly struggled when the context became larger and the fact appeared earlier in the document. Teams using RAG therefore need to consider information placement, not only whether retrieval found the information.

### A reasoning instruction can change generation performance
[16:14](https://www.youtube.com/watch?v=nbZzSC5A6hs&t=974s)
Dhinakaran separates retrieval from generation tasks such as rounding numbers, mapping dates, or concatenating strings from a financial document. In the research she presents, GPT-4 performed worse than Anthropic Claude 2.1 on generation, despite performing better on retrieval. The team improved GPT-4's results by changing the prompt to ask it to explain itself and then answer the question. She says the Claude model's greater verbosity helped in this setting because it worked through the task before producing the final answer. The result shows why teams should evaluate the individual task they care about, rather than assume retrieval performance predicts generation performance.

## Notable quotes
- "For most of you in the room who are probably building the application, you probably care more about task evals." (01:28)
- "Evals with explanations are by far what we see real people deploying applications finding the most useful in production." (10:48)
- "If you're using RAG, depending on how much information you're putting in the document, it's important to just balance where you place it in the document as well." (16:14)
- "We asked it to please explain yourself and then answer the question." (17:19)

## Tools & references mentioned
- Arize AI
- Phoenix
- Hugging Face
- Open LLM Leaderboard
- MMLU
- Llama
- Needle in a Haystack
- RAG
- Anthropic Claude 2.1
- GPT-4
- Mistral
- OpenAI
- Arize Observe

## Who should watch
- You are building an LLM application with routing, function calling, search, or several workflow branches and need to test more than the final answer.
- Your current evaluations return scores without telling engineers which component failed or what they should change.
- You use RAG or compare foundation models and want to test retrieval position, generation behavior, and model fit for a specific task.

## Editor's note

From the pack [Evals that survive production](https://aietalks.com/packs/production-evals):

Zapier records tool interactions so a failed run can become a test. Hetzel calls the next step rerunning production, while Feyzkhanov rebuilds the environment around a captured trace and Gupta evaluates the whole workflow. Kitaru records an agent run and replays that case after a model, prompt, tool or harness changes, so the team can compare the route as well as the final result.

Written by the AIE Talks editors (the Kitaru team), not by the speaker.

## Related talks

- [Building Blocks for LLM Systems & Products](https://aietalks.com/talks/building-blocks-for-llm-systems-products) (Eugene Yan, Amazon, 17:24)
- [How to Construct Domain Specific LLM Evaluation Systems](https://aietalks.com/talks/how-to-construct-domain-specific-llm-evaluation-systems) (Hamel Husain, Independent consultant & Emil Sedgh, Rechat, 18:45)
- [Build Evals That Actually Matter](https://aietalks.com/talks/build-evals-that-actually-matter) (Nick Ung & Akshay Sharma, Lyft, 37:45)
- [Lessons From A Year Building With LLMs](https://aietalks.com/talks/lessons-from-a-year-building-with-llms) (Eugene Yan & Shreya Shankar, UC Berkeley & Hamel Husain & Jason Liu & Bryan Bischof, Hex & Charles Frye, 35:21)
- [Strategies for LLM Evals (GuideLLM, lm-eval-harness, OpenAI Evals Workshop)](https://aietalks.com/talks/strategies-for-llm-evals-guidellm-lm-eval-harness-openai-evals-workshop) (Taylor Jordan Smith, Red Hat, 32:28)