# LLM Observability, Evaluation, Experimentation Platform

Dat Ngo, Arize AI | AI Engineer Europe 2026 | 16:32

Source: https://www.youtube.com/watch?v=JsCCrBF7F1g
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/llm-observability-evaluation-experimentation-platform
Published: 2026-06-07
Tags: evals, observability, testing, tracing

## TL;DR
- Observability records what an agent did across traces, spans, sessions, branches, and trajectories because application code does not audit nondeterministic agent behavior.
- Evaluation signal can come from LLM judges, human feedback, golden datasets, deterministic checks, and business metrics, applied at different scopes from one span to an entire session.
- Arize is moving toward an automated loop where AI examines traces, creates evaluations, finds problems, and guides experimentation without requiring people to work manually in dashboards.

## Summary
Dat Ngo describes AI application development as software engineering with nondeterministic execution paths. Observability begins with OpenTelemetry traces and spans, which record an agent's actions, then expands to sessions, branch distributions, trajectories, and analytics. He explains five sources of evaluation signal: LLM judges, human feedback, golden datasets, deterministic checks, and business metrics. Evaluations can target one span, multiple spans, a full trajectory, or a session, but teams should use the smallest set that gives enough information because evaluations have a cost. Experiments compare changes to prompts, models, orchestration, and configuration against a dataset. Ngo also describes Arize's move toward automation through CLI tools, coding-agent integrations, and Alex, an AI system that scans traces, finds latency and errors, and creates evaluations. Phoenix is the open-source product, while AX targets large enterprises.

## Key ideas
### Agent behavior needs telemetry because application code does not audit execution
[01:50](https://www.youtube.com/watch?v=JsCCrBF7F1g&t=110s)
Ngo frames observability as the answer to what is happening inside an agent or harness. Arize uses OpenTelemetry, including auto-instrumentation that can create traces and spans with one line of code. A trace or span becomes an audit record of the agent's actions. This matters because an agent may choose different tools or branches on different runs. Ngo gives a concrete failure: the agent called tool B before tool A even though B depended on A. Looking at the trajectory exposes the ordering problem and shows where additional context or instructions are needed.

### Observability must include sessions and distributions across possible paths
[03:43](https://www.youtube.com/watch?v=JsCCrBF7F1g&t=223s)
A single trace is only one view of an agent. Ngo also discusses sessions, which capture back-and-forth conversations and state across runs. Arize AX can show a distributional view across agent instantiations, including the branches and loops traffic takes. That lets a team ask what percentage of traffic follows each branch and whether a component on one branch causes high latency. Session-level questions are different from deep tool-level questions. An enterprise user may care more about whether the user was satisfied and whether all questions were answered than about every individual tool call.

### Evaluation signal comes from five sources with different costs and strengths
[06:46](https://www.youtube.com/watch?v=JsCCrBF7F1g&t=406s)
Ngo groups evaluation signal into five flavors. LLM-as-a-judge can score outputs, while human feedback captures the experience of users and domain specialists. Golden datasets provide trusted labeled examples that can tune an LLM judge to approximate a person or dataset. Deterministic checks reduce cost when logic is enough, such as testing whether generated JSON is valid, matches a schema, and contains non-null fields. Business metrics connect the system to outcomes such as making money, saving money, or saving time. He presents these sources as complementary rather than assuming every evaluation needs another LLM call.

### Technical and domain specialists should work on different parts of evaluation
[08:42](https://www.youtube.com/watch?v=JsCCrBF7F1g&t=522s)
Ngo separates the people who build AI systems from the people who understand the intended AI experience. Engineers are suited to coding, automation, and framework work. Subject matter experts and product managers can define prompt behavior and the evaluations that matter for the domain. Arize supports both modes. A less technical user can select a model, choose an evaluation template, or customize one in the interface. A technical user can attach evaluations and run them programmatically. The division keeps implementation work with people who code and quality definitions with people who know the product and domain.

### Evaluation scope ranges from one component to an entire conversation
[10:07](https://www.youtube.com/watch?v=JsCCrBF7F1g&t=607s)
Ngo describes four evaluation scopes. A span evaluation checks the input and output of one component, such as one LLM call. A multi-span evaluation combines information from several components, which is needed to assess how agents pass data to one another. A trajectory evaluation examines all spans and asks whether the system called things in the right order to complete a business process. A session evaluation zooms out to the state of a conversation, such as whether the user became frustrated or whether every question was answered. Teams should choose both the signal type and the depth of scope.

### Teams should run the smallest evaluation set that explains whether the system works
[11:46](https://www.youtube.com/watch?v=JsCCrBF7F1g&t=706s)
Ngo cautions that the ability to evaluate something does not mean it should always be evaluated. Evaluation has a cost, especially when it uses LLM calls or human review. Teams should find the minimal set of evaluations that provides enough signal to determine whether the application behaves as intended. Observability helps locate bad signal in traces, after which teams can collect those cases into a dataset. Teams can also begin with an uploaded dataset of input-output pairs when traces are not available.

### Experiments compare changes to prompts, models, orchestration, and configuration
[12:26](https://www.youtube.com/watch?v=JsCCrBF7F1g&t=746s)
Once a dataset exists, teams can run experiments against it. Ngo defines an experiment as a comparison involving a change to the agent or harness. The changes can involve prompts, models, orchestration, or configuration. Arize supports running these comparisons in a user interface or programmatically. This gives teams a way to test an intended fix against collected examples and inspect whether the change improves the relevant signal. The earlier observability and evaluation work supplies the cases and measurements used for these comparisons.

### Arize wants AI systems to find problems and create evaluations automatically
[13:06](https://www.youtube.com/watch?v=JsCCrBF7F1g&t=786s)
Ngo says Arize expects software development to become easier to build and customize, with users interacting through coding agents as well as dashboards. Arize exposes primitives through a CLI and tools that can be called by systems such as Claude Code. Its AI layer, Alex, can inspect application data and plan tasks in response to a request such as finding problems in an application. Ngo says Alex can detect high latency and errors, then create evaluations when the traces show that new checks are needed. Arize's stated goal is to automate observability, evaluation, experimentation, and improvement out of the user's workflow.

## Notable quotes
- "It's all just engineering." (01:50)
- "It's actually the telemetry that does that." (03:23)
- "Just because you can eval something doesn't mean you always should." (12:05)
- "Our ultimate goal as a company is actually to automate you out of this process." (14:23)

## Tools & references mentioned
- Arize AI
- OpenTelemetry
- Arize Phoenix
- Arize AX
- Alex
- Claude Code
- Anthropic
- OpenAI DevDay
- Uber
- Booking
- Reddit

## Who should watch
- You are debugging an agent whose tools, branches, or loops change from run to run, and ordinary application logs do not explain the failures.
- Your team needs to choose between LLM judges, human review, golden datasets, deterministic checks, and business metrics without evaluating every possible layer.
- You are deciding whether an open-source observability setup such as Phoenix or an enterprise platform such as AX fits your deployment and automation plans.