# Ship Real Agents: Hands-On Evals for Agentic Applications

Laurie Voss, Arize AI | AI Engineer Europe 2026 | 2:04:18

Source: https://www.youtube.com/watch?v=Xfl50508LZM
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/ship-real-agents-hands-on-evals-for-agentic-applications
Published: 2026-05-14
Tags: debugging, evals, observability, testing

## TL;DR
- Agent evaluation replaces subjective "vibes" checks with tests that can catch regressions and run in CI.
- A useful eval suite combines deterministic code checks, LLM judges, and human-labeled examples because each method catches different failures.
- Experiments let teams compare prompt or agent changes on the same examples, so improvements can be measured instead of guessed.

## Summary
Laurie Voss builds an evaluation pipeline around a financial analysis agent. The workshop starts with tracing in Arize Phoenix, then uses the traces to find recurring failures before writing evaluations. The agent sometimes writes reports to disk instead of returning them, omits buy or sell recommendations, confuses Amazon with AWS, or produces financial claims that are hard to verify. Voss shows when to use code evaluators for deterministic properties, built-in LLM judges for semantic checks, and custom rubrics for application-specific behavior. A correctness judge gives zero passes on the example data because the judge lacks current and forward-looking financial knowledge, while a faithfulness judge gives 13 out of 13. The final stage turns failures into a data set and runs experiments against it. Voss is direct about evaluator errors, bias, overfitting, and the need for human-labeled examples.

## Key ideas
### Traces are the runtime logs that make agent tests possible
[05:17](https://www.youtube.com/watch?v=Xfl50508LZM&t=317s)
Voss asks engineers to think of evals as tests and traces as the logs that power those tests. A trace records every agent call, tool call, and LLM invocation, including inputs, outputs, timing, token counts, and other metadata. Its basic unit is a span. An LLM call or tool call can be a span, while a full agent turn can contain nested spans. This structure matters because an agent's behavior is a sequence of decisions rather than one predictable input-output exchange. Phoenix presents the nested data in a readable interface, but the underlying data can often be treated as JSON.

### Vibe checks miss regressions because agents can produce many valid answers
[10:51](https://www.youtube.com/watch?v=Xfl50508LZM&t=651s)
The common testing pattern is to run a few queries and ask whether the results look right. Voss calls this the "vibes problem." It misses edge cases, unexpected vocabulary, adversarial inputs, and changes that improve one behavior while damaging another. Exact string matching is also a poor fit because the same prompt can produce different text on different runs, with several outputs all being correct. Human review does not scale and cannot run continuously in CI. Evals give teams a repeatable way to test prompt changes, model upgrades, and the broad range of behavior affected by a small wording change.

### Code evals and LLM judges cover different kinds of failure
[10:18](https://www.youtube.com/watch?v=Xfl50508LZM&t=618s)
Code evals are deterministic functions written in Python or TypeScript. They run quickly and cheaply, and can check JSON format, length limits, required fields, forbidden phrases, or whether an output mentions the requested ticker. LLM-as-a-judge evals use a second model and a rubric to assess meaning, such as factual accuracy, faithfulness to source material, or tone. They are more flexible, but cost more, take longer, and can be wrong themselves. Voss recommends using both, with humans still involved when the team encounters failure modes it has not seen before.

### Agent evals should test outcomes without prescribing one exact path
[15:07](https://www.youtube.com/watch?v=Xfl50508LZM&t=907s)
Agents create cascading failures because each tool call depends on earlier decisions. A system may choose the wrong tool, pass bad parameters, misunderstand the result, or route work to the wrong sub-agent. Voss gives the example of a Tesla request being interpreted as a request about Nikola Tesla. At the same time, an agent can find a shorter or better path after a model upgrade. An eval that requires tool A, then tool B, then decision C may fail even when the result is correct. Tests should focus on the result and relevant behavior, rather than enforcing a fixed execution path.

### Reading traces reveals root causes that output-only checks hide
[38:12](https://www.youtube.com/watch?v=Xfl50508LZM&t=2292s)
Before writing evals, Voss reads the agent's traces and categorizes what went wrong. The financial agent researches companies and then writes a report, but one run tried to write a Markdown file to disk instead of returning the report. Another wrote about AWS when the request concerned Amazon. A Rivian report included a suspiciously precise delivery figure that needed source checking. These failures point to different fixes, so the team must distinguish bad research, tool errors, missing recommendations, reasoning gaps, and possible hallucinations. Frequency alone is not enough. Voss recommends considering severity as well, then fixing expensive frequent failures first.

### A well-chosen eval can matter more than a better-tuned judge
[59:00](https://www.youtube.com/watch?v=Xfl50508LZM&t=3540s)
The built-in correctness evaluator gives zero passes on the 13 example traces. Its explanations show why: the judge is trying to assess highly specific 2026 financial claims while relying on a model that lacks the needed current or forward-looking knowledge. Voss then uses a faithfulness evaluator, which checks whether the report follows the research produced by the first agent turn. That evaluator scores all 13 examples as faithful. The contrast shows that an evaluator can be inappropriate for the task even when its general purpose sounds right. Choosing the question the judge can actually answer comes before tuning its prompt.

### Custom rubrics work better when they use observed failures and labeled examples
[1:04:35](https://www.youtube.com/watch?v=Xfl50508LZM&t=3875s)
For the financial agent, Voss writes an actionability judge because no built-in evaluator answers whether a report gives useful investment guidance. The rubric defines observable criteria, such as specific recommendations and forward-looking analysis, along with failure cases such as merely summarizing public data. It labels the input and output clearly, gives examples of actionable and non-actionable reports, and asks for a constrained binary label. Voss says examples are especially useful because models can infer the pattern from them. She also advises separate evaluators for separate dimensions instead of one broad evaluator that mixes accuracy, tone, completeness, and policy compliance.

### Experiments turn eval results into measurable agent improvements
[1:19:14](https://www.youtube.com/watch?v=Xfl50508LZM&t=4754s)
After finding that reports lack explicit recommendations, Voss changes the research and writing prompts to request financial ratios, recent news, current price data, and a buy, sell, or hold conclusion. She adds the failing traces to a Phoenix data set and runs the revised agent against those same examples. The previously failing cases now pass the actionability check. The point is controlled comparison: the inputs and evaluator stay the same while the prompt changes. Teams can first test a small failure-focused set, then run the full corpus before shipping to catch regressions or overfitting. Capability evals guide improvement, while passed capabilities become regression tests.

## Notable quotes
- "An eval that you haven't validated is just a fancy way of being wrong at scale." (1:16:14)
- "Choosing the right eval meant can matter more than tuning your eval." (1:03:04)
- "The power of experiments is controlled comparison." (1:31:58)
- "You shouldn't just be writing rules on the basis of what you think would be a good idea. You should be writing rules based on the actual traces you've seen and the actual failures that you've observed." (1:06:58)
- "When you find that changing one thing has broken three other things without you noticing, that is when you need evals." (1:51:10)

## Tools & references mentioned
- Arize AI
- Arize Phoenix
- Arize AX
- Phoenix Cloud
- Claude Agent SDK
- Anthropic
- Claude Haiku
- Claude Sonnet
- Claude Opus
- OpenAI
- Gemini
- OpenTelemetry
- Open Inference
- CrewAI
- LangChain
- LlamaIndex
- D- Script
- Bolt
- Claude Code
- CoreBench
- Swiss cheese model
- DSPy
- Andrej Karpathy

## Who should watch
- You have an agent that looks good in manual testing but has started breaking when prompts or models change.
- Your team needs to decide which failures belong in deterministic tests and which require semantic judging.
- You are building an evaluation data set and want a practical way to validate the judge before trusting its scores.

## Related talks

- [Don't Ship Skills Without Evals](https://aietalks.com/talks/dont-ship-skills-without-evals) (Philipp Schmid, Google DeepMind, 21:46)
- [Agent Evals: Finally, With The Map](https://aietalks.com/talks/agent-evals-finally-with-the-map) (Ari Heljak, Root Signals, 13:31)
- [AI Agents, Meet Test Driven Development](https://aietalks.com/talks/ai-agents-meet-test-driven-development) (Anita, Vellum, 29:10)
- [Agents Building Agents](https://aietalks.com/talks/agents-building-agents) (Alfonso Graziano, Nearform, 30:14)
- [Build Evals That Actually Matter](https://aietalks.com/talks/build-evals-that-actually-matter) (Nick Ung & Akshay Sharma, Lyft, 37:45)