# Strategies for LLM Evals (GuideLLM, lm-eval-harness, OpenAI Evals Workshop)

Taylor Jordan Smith, Red Hat | AI Engineer World's Fair 2025 | 32:28

Source: https://www.youtube.com/watch?v=89NuzmKokIk
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/strategies-for-llm-evals-guidellm-lm-eval-harness-openai-evals-workshop
Published: 2025-07-27
Tags: benchmarks, evals, latency, reliability

## TL;DR
- Production AI needs evaluations for reliability, safety, cost, latency, bias, and user-facing behavior, not only model accuracy.
- Teams should build evals incrementally, starting with system performance and expanding through component, integration, and end-to-end tests.
- GuideLLM, lm-eval-harness, and Promptfoo provide practical ways to measure inference performance, factual accuracy, safety, bias, and custom use cases.

## Summary
Taylor Jordan Smith explains why leaderboard scores are insufficient for deploying generative AI. A production system must also be tested for latency, throughput, cost, concurrency, formatting, factual accuracy, safety, bias, and the way its components work together. She distinguishes evaluation, which can assess a model or system end to end, from benchmarking, which uses controlled tasks and datasets to compare models. Her recommended approach is incremental. Teams can begin with inference performance, retrieval quality, or output formatting, then add integration and user-interface tests as the system grows. The workshop demonstrates GuideLLM for inference benchmarks, lm-eval-harness with MMLU Pro for factual accuracy, and Promptfoo for custom safety tests. Smith is direct about the work involved, including hardware sizing, compute costs, model configuration, and continuous updates to evaluation suites. She also connects evaluation to CI/CD so tests run alongside normal software engineering checks.

## Key ideas
### Production generative AI has failure modes that ordinary model scores do not measure
[01:38](https://www.youtube.com/watch?v=89NuzmKokIk&t=98s)
Smith describes production generative AI as complex, creative technology that must be scalable, reliable, and safe. Organizations face policy restrictions, legal exposure, bias and discrimination, cost, latency, throughput, and stale model knowledge. A model can produce a plausible answer while still being too slow or expensive for its workload. It can also repeat biased patterns from its training data or fail to guard against satire and other misleading content. Retrieval-augmented generation and agents can provide newer information, but they add more components to test. Smith's point is practical: teams need to understand their particular system and test the risks that matter before releasing it.

### Inference performance needs workload-specific measurements
[05:28](https://www.youtube.com/watch?v=89NuzmKokIk&t=328s)
The first hands-on exercise focuses on inference under enterprise workloads. Smith explains that concurrent requests can overwhelm traditional runtimes, so teams need to measure latency, throughput, time to first token, and inter-token latency with a workload that resembles the intended application. GuideLLM lets participants choose a model and dataset, sweep request rates, and inspect mean, median, and P99 performance. Smith also recommends changing input and output token levels for different use cases, such as chatbots and RAG systems. A result is only useful in context. Whether a number is good depends on the application, hardware, concurrency, and service-level objectives.

### Evaluation should grow layer by layer with the system
[12:09](https://www.youtube.com/watch?v=89NuzmKokIk&t=729s)
Smith recommends an incremental evaluation plan instead of trying to test every component on the first day. A RAG team might begin with chunk retrieval, while another team might start with latency and throughput for model output. Once those tests are in place, the suite can expand into a full system evaluation covering component integration and the end-to-end user interface. She compares this structure to the software testing pyramid, with unit-like checks at the bottom, integration tests in the middle, and UI or end-to-end tests at the top. The order should follow system priorities and the risks of the actual architecture.

### The evaluation pyramid starts with performance and adds quality and safety
[13:33](https://www.youtube.com/watch?v=89NuzmKokIk&t=813s)
The model evaluation pyramid begins with system performance because a model that cannot handle throughput or concurrent users is unsuitable for production. The next layers include formatting, such as reliably returning the JSON an application needs, and factual accuracy, including standard subject tests or checks against fine-tuning data. Higher layers can cover application-specific safety and bias. Smith uses this pyramid to show how teams can plan evaluation coverage without treating every test as equally urgent. The framework applies across different architectures, but the metrics must match the system. A RAG application, an agent, and a plain language model will need different checks.

### GuideLLM connects inference tests to deployment decisions
[14:51](https://www.youtube.com/watch?v=89NuzmKokIk&t=891s)
Smith introduces GuideLLM as a tool for system performance benchmarks associated with the vLLM inference runtime project. Participants deploy a model, select a dataset, and run a sweep of request rates. The output includes benchmark statistics and can be exported as JSON for closer inspection. GuideLLM exposes configuration choices such as the request rate and token counts, allowing teams to compare results after changing the workload. Smith also notes that the inference runtime itself has configuration knobs, including the model context window. The goal is to understand whether a particular model and hardware setup fit the intended use case before deployment.

### Open evaluation frameworks can be adapted to proprietary data
[25:13](https://www.youtube.com/watch?v=89NuzmKokIk&t=1513s)
The second activity uses MMLU Pro with lm-eval-harness, but Smith says the same multiple-choice style can be adapted to a team's own dataset. Teams can fork an open evaluation, change its data sources, or create accuracy tests based on fine-tuning data. She gives the example of evaluating fine-tuned models against proprietary information. This means a benchmark does not have to remain a fixed public leaderboard task. Its structure can become a template for a test that reflects the application's domain. The choice depends on what the team needs to learn and how much time and compute the evaluation requires.

### Custom safety evaluations are needed for application-specific risks
[29:26](https://www.youtube.com/watch?v=89NuzmKokIk&t=1766s)
The final activity uses Promptfoo for a safety evaluation, with examples that participants can modify for their own tests. Smith describes Promptfoo as a tool for building customized evaluations and points to additional examples in its repository. This allows a team to test behavior that a general accuracy benchmark will not capture, including safety and bias issues tied to its application. She connects this work to incidents such as an AI overview repeating a satirical Reddit claim and models reproducing skewed internet data. The useful test is the one that checks the failure modes a real application could expose to its users.

### Evaluation belongs in CI/CD and must keep changing
[30:48](https://www.youtube.com/watch?v=89NuzmKokIk&t=1848s)
When an attendee asks how to compare evaluation results with live customer behavior, Smith describes a CI/CD implementation of an evaluation framework. She recommends including evaluation tests alongside ordinary software engineering tests, similar to unit testing. The suite should also change over time because no evaluation catches every failure. Teams need a continuous process for adding tests and adjusting coverage as models, prompts, data, and system integrations change. This connects pre-release checks with production use cases without pretending that one benchmark can describe the whole system. The environments and workshop repository provide a place to keep experimenting with these tools.

## Notable quotes
- "Benchmarking is a subcategory of evaluation." (09:41)
- "No matter how good your model is, if it's not fast, if it's not reliable, if it's not affordable, you're screwed a little bit from the get-go." (04:54)
- "You can start with those kind of incremental approaches for specific components and then from there based on priority levels branch out into a full system eval." (12:30)
- "You should have a CI/CD framework that includes these evaluation tests just like for unit testing setups." (31:55)

## Tools & references mentioned
- Red Hat
- GuideLLM
- vLLM
- lm-eval-harness
- MMLU
- MMLU Pro
- Promptfoo
- TRT-LLM
- SGLang
- RAG
- Hugging Face
- IBM Granite
- Gemini
- Stable Diffusion
- AI Overview
- Slack

## Who should watch
- You are deploying a chatbot, copilot, RAG application, or agent and need tests that reflect its actual workload.
- Your team relies on benchmark scores but has not measured latency, throughput, safety, formatting, or integration behavior.
- You want practical starting points for adding evaluation tests to a CI/CD process or adapting open benchmarks to proprietary data.

## Related talks

- [Mastering AI Evaluation: From Playground to Production](https://aietalks.com/talks/mastering-ai-evaluation-from-playground-to-production) (Doug & Carlos Essan, Braintrust, 1:25:08)
- [Engineering Better Evals: Scalable LLM Evaluation Pipelines That Work](https://aietalks.com/talks/engineering-better-evals-scalable-llm-evaluation-pipelines-that-work) (Dat Ngo & Aman Khan, Arize, 24:46)
- [Best Practices for Evaluating Large Language Model Applications with llmeval](https://aietalks.com/talks/best-practices-for-evaluating-large-language-model-applications-with-llmeval) (Niklas Nielsen, Log10, 09:33)
- [Build Evals That Actually Matter](https://aietalks.com/talks/build-evals-that-actually-matter) (Nick Ung & Akshay Sharma, Lyft, 37:45)
- [Lessons from the Trenches: Building LLM Evals That Work IRL](https://aietalks.com/talks/lessons-from-the-trenches-building-llm-evals-that-work-irl) (Aparna Dhinakaran, Arize AI, 18:49)