AI agents need tracing and monitoring because a healthy system can still produce wrong or poor responses.
2
Application-specific evaluators turn user disagreement, all-caps messages, and out-of-scope requests into searchable production signals.
3
Production traces should feed maintained datasets and offline experiments so teams can check regressions before deploying changes.
Summary
Annabell Schäfer and Lotte Verheyden use a dad-focused IT support agent called Specs to explain a practical improvement loop for agents. They begin with the difference between online work, tracing and monitoring production behavior, and offline work, datasets and experiments. The workshop then instruments a TypeScript application using the OpenAI Agents SDK, examines traces containing agent observations, model calls, and tool calls, and adds monitors for user disagreement, all-caps frustration, and requests outside the system prompt. The examples show why evaluator inputs must be mapped explicitly to the relevant messages and observations. Seeded traces are later investigated with the Langfuse CLI and skill through a coding agent. That analysis finds retrieval failures, retries, latency, and gaps in evaluation coverage. The talk ends by connecting these findings to dataset maintenance, offline evaluation, regression checks, and repeated monitoring as real usage changes.
AI agents need an engineering loop because system health does not guarantee correct behavior
Lotte Verheyden says AI agents are nondeterministic, so infrastructure can look healthy while the agent gives a wrong or low-quality response. She describes AI engineering as a loop with online tracing and monitoring alongside offline datasets, experiments, and evaluation. The workshop focuses mainly on the online side. Tracing records what the agent does in production, while monitoring combines automatic evaluators, dashboards, user feedback, and human review. Teams can start with tracing and monitoring, then add structured offline testing as the product matures or the cost of mistakes increases.
A trace must contain enough detail to support later evaluation
Every trace is made of observations, which are the steps an agent takes while answering a user. These can include nested language-model calls, tool calls, and other operations. Model observations can include duration, token cost, token usage, model, and prompt version. A web-search tool might record the query it used. Schäfer argues that trace quality affects every later step: poor traces lead to missed monitoring signals, weaker datasets, and less informed evaluation decisions.
Specs makes application-specific failure signals concrete
The workshop uses Specs, a small IT support agent built from questions Schäfer's father asks about phones, printers, and computers. The sample interface accepts a question and gives the agent tools for support context and a help library. Specs uses the OpenAI API to generate replies. The example narrows the monitoring problem to practical support failures, such as an answer that is unclear, a user who says the instructions did not work, or a request that falls outside the agent's intended phone-support scope.
User feedback and human review find cases automatic checks may miss
Monitoring has several jobs. Aggregate metrics show changes in cost and quality over time. User feedback surfaces important traces when a growing application produces too many interactions to inspect manually. Automatic evaluators scale across traces, but they are less likely to detect new or nuanced failure modes. Human annotations and direct user feedback scale less easily, yet they can reveal problems the team did not anticipate. The workshop treats disagreement and frustration as useful signals for finding support interactions that need attention.
Evaluator inputs must be mapped to the question being asked
The presenters configure an evaluator model for user disagreement and target the root agent observation rather than an individual generation or tool call. They map the conversation history and the last user message into the evaluator through JSON paths. This lets the evaluator compare the answer with a follow-up such as, 'this button is actually not there'. Schäfer says that a score becomes useful when it answers a defined question about the relevant part of the trace, rather than evaluating an unspecified block of data.
Simple code evaluators can turn visible frustration into a production signal
The all-caps evaluator is a code-based check. It extracts a user message through a JSON path and returns a boolean score when the message contains all caps. The presenters use this as a proxy for frustration and mention that the same pattern could check cuss words, prohibited terms, or incorrect brand spelling. The evaluator runs on live agent observations, and the interface provides a test run before execution. The same evaluators can also be applied to offline experiments.
Scope checks compare user requests with the system prompt
The out-of-scope evaluator uses a different set of inputs. It reads the system prompt, which defines the agent as a phone IT support assistant, and the latest user message. The evaluator compares what the application is supposed to do with what the user is asking. An out-of-scope result can lead to two different actions: expand the application's scope or improve how the current scope is communicated. The presenters stress that monitoring criteria should come from the likely failure modes of the specific application.
Coding-agent analysis can expose failures that the configured monitors do not cover
After seeding production-like traces, the presenters install the Langfuse CLI and skill and ask a coding agent to investigate the traces. The report examines questions, scores, tool calls, latency, and other behavior beyond the explicit evaluators. It finds a silent retrieval failure in the search help library, retries that increase latency and cost, and an evaluation coverage gap. The coding agent can also use repository context to propose removing or fixing a tool that is not helping. Schäfer presents this as assistance for investigation, while leaving the team responsible for deciding what change to make.
Online findings must become datasets and offline checks
The final loop turns production signals into representative examples. Teams need to maintain evaluation criteria because an evaluator can stop providing useful information, and they need to add new edge cases to their datasets as usage changes. Proposed changes should run against those datasets in offline experiments, with regression checks before deployment. The presenters describe the workshop's online monitoring section as one part of a larger process that includes prompt management, datasets, experiments, and repeated review of production traces.