Evals are signals collected at several levels of an AI system, and they should include model judges, humans, golden datasets, and code-based checks.
2
Teams need two feedback cycles: one to improve the AI application and another to find and fix errors in the evals themselves.
3
Agent evaluation must examine paths and failure modes across many traces, rather than judging one agent run or one component in isolation.
Summary
Dat Ngo explains how to build evaluation pipelines for production AI systems. He places evals alongside observability: traces show what the system did, while evals turn large volumes of activity into useful signals. He argues that LLM-as-a-judge is only one tool. Human feedback, golden datasets, smaller encoder models, and simple code checks each have different costs and uses. Evals can target a single model call, a workflow, control flow, or an entire user session. Agent systems require broader analysis of paths across many traces, including tool use and trajectory quality. Ngo also describes two feedback loops. One improves prompts, models, and orchestration. The other checks whether the evals are accurate and tunes them with annotated failures. He discusses inline guardrails, asynchronous tracing with OpenTelemetry, confidence scores from log probabilities, and automated prompt improvement through meta-prompting.
Evals turn observability data into signals about what works
Ngo separates observability from evaluation. Observability answers what the system is doing through traces, spans, conversations, and analytics. Evals help teams avoid manually inspecting every trace by identifying what went well and what failed. He describes evals as a way to extract signal from a large amount of application data. This can happen at different levels, including a trace, a conversation, or another unit that matches how the team understands the product. Platform teams may focus on cost and latency, while application teams may care more about product quality and user outcomes.
LLM judges should be calibrated against trusted human labels
An LLM judge can evaluate parts of a retrieval-augmented generation flow, such as whether retrieved context is relevant to the user's query. Ngo distinguishes judging an output from doing the original task, using summarisation as an example. He also recommends smaller encoder-only models such as BERT-style architectures when they fit the task, since they can be much cheaper and faster. Human feedback and golden datasets provide another source of signal. Teams can run the judge on examples that people have graded, then measure and tune how closely the judge matches the trusted labels.
Simple code checks belong in the same evaluation toolbox
Ngo argues that teams should not treat LLM-as-a-judge as the whole evaluation system. Code-based checks are useful when the condition is explicit. Examples include checking whether an output contains a keyword, matching a regular expression, or verifying that a response is valid JSON. These checks are faster and cheaper than using a person or a language model. The practical design is a toolbox in which each evaluator is chosen for the signal required, rather than applying an LLM judge to every possible property.
Evaluation needs a loop for the application and a loop for the evaluators
The first feedback cycle uses traces and evals to find problems in the AI product. A team may discover hallucinations caused by poor retrieval, a weak prompt, a model choice, or agent orchestration, then change the relevant part of the system. Ngo says teams often forget the second cycle. Evals can be wrong, so people need to annotate some results and collect cases where the evaluator missed or misclassified a problem. Those failures are used to make the eval prompt more specific. Faster iterations through both cycles give teams more chances to improve the product.
Evaluation should follow the structure of the application
A complex application can be evaluated at several levels. Teams may inspect one LLM call, then zoom out to a workflow containing model calls, APIs, heuristics, and external actions. They can also evaluate control flow. Ngo recommends conditional evals when later steps depend on an earlier routing or decision: if the system took the wrong path, spending money evaluating downstream outputs may not help. At the highest level, a session can cover the back-and-forth between a user and an agent, including whether the customer became frustrated.
Agent evaluation is about failure modes across paths
For agents, looking at one trace does not answer the main operational question. Teams need to know how the agent fails across its complete set of runs. Ngo describes analysing paths through an agent graph, including how often particular tools are called and how eval results change for different sequences of components. A path can work when it goes through components one, two, and three, but fail when it goes through component four because that component depends on another step. Trajectory evaluation can compare an observed path with a reference path, use an LLM to judge the difference, or check whether expected nodes were reached directly.
Guardrails protect the system but do not replace fixing the system
Ngo describes the orchestration code as system one and the guardrails as system two. Inline evaluations can stop or redirect a request, but they add latency and operational complexity. He says teams often go to the guardrail first when the underlying prompt or orchestration is the real source of the problem. Guardrails also cover known conditions in a way similar to unit tests. Observability and evaluation are needed to learn about the wider distribution of user behaviour, including cases the team did not anticipate. Guardrails are useful protection, but they are not infallible.
Trace context must cross services for asynchronous evaluation
Large AI systems can split work across applications, model routers, containers, and Kubernetes pods. Ngo recommends OpenTelemetry propagation so a trace can connect the work across those service boundaries. For example, an application can call a model router, receive the result, call another application, and then continue processing. Without shared trace context, teams see isolated pieces instead of the full operation. This matters when an evaluation needs to cover an asynchronous or long-running process rather than a single service span.
"What is eval used for? It's actually just a really clever word for signal."04:28
Who should watch
You are building an LLM or agent product and need an evaluation plan that covers more than final-answer quality.
Your current evals are expensive, hard to trust, or disconnected from the traces and workflows where failures happen.
You need to decide when to use human labels, model judges, code checks, guardrails, or OpenTelemetry across service boundaries.
