Building Self-Learning Loops for Your Agent

Fuad Ali, Arize AI38:55 · Oct 2026 · 6,193 views
Thumbnail for Building Self-Learning Loops for Your Agent Watch on YouTube
TL;DR
  1. 1

    Agent observability should become a programmatic interface that coding agents can consume, rather than a dashboard that only people inspect.

  2. 2

    Arize Signal and Arize skills connect production traces with an agent's code so failures can be investigated and proposed fixes can be reviewed.

  3. 3

    Harness judges and agent experiments validate fixes against real failure cases before a change reaches production.

Summary

Fuad Ali presents a workflow for turning production failures into agent improvements. His Wonder Toys shopping assistant has a price-filter bug: a request for products under $30 can return items above the budget. The application is built with the OpenAI Agents SDK, runs on FastAPI, and sends traces to Arize. Ali shows how Signal can cluster recurring failures, inspect traces, consult connected sources such as a GitHub repository, write an investigation, and open a pull request. Arize skills offer a similar workflow from coding agents such as Claude Code. Ali then explains why a fixed LLM judge misses unknown failures. A harness judge can inspect tool calls, code, trace context, and external systems before deciding whether an outcome passes. Agent experiments replay failure datasets against a development endpoint and compare evaluator results with latency and cost. The proposed loop detects, investigates, fixes, verifies, and leaves code review with a human.

Key ideas
00:30

Observability must become something agents can consume

Ali describes a shift from software systems where humans read observability dashboards and write corrective code, through prompt-driven workflows, toward coding agents that consume telemetry directly. An application emits traces, an agent reads those events, and the agent can inspect the code that produced them. He argues that this change matters when teams operate thousands of agents across a stack, because people cannot inspect every event themselves. Tools such as Cursor, Claude Code, and the Datadog CLI already move in this direction, although engineers often still have to prompt them to fetch backend data. The intended interface is programmatic, so an agent can read, act on, and reflect on its own telemetry.

04:52

Production failures reveal evaluation cases that teams did not write

Ali says failures that cost a team money are usually invisible at launch because a known failure would have been fixed before release. His example is a shopping assistant that confidently returns products above a shopper's budget. A request for a birthday gift under $30 can produce products costing $30 or $50 and lead to the wrong purchase. The problem appears in production through complaints, support tickets, or an on-call page rather than in staging. The Wonder Toys Assistant provides search, product details, purchasing, order-status, and cancellation tools. It uses the OpenAI Agents SDK, a FastAPI backend, vector search with a text fallback, and Arize tracing.

09:19

Signal connects recurring trace failures to the repository

Signal is an Arize agent that runs continuously on production traces. Ali says it reads incoming traces, clusters failures by root cause, and deduplicates repeated failures. It can connect to a GitHub repository and downstream sources such as Datadog, then use those sources alongside trace data to investigate what happened. In his example, Signal identifies serialization problems that corrupt agent input, writes an investigation with a hypothesis and evidence, and opens a pull request. For the budget bug, the same approach connects a user's request for products under $30 to the code path that checks the price filter. Signal can also be configured around alerts for evaluator failures, latency spikes, or cost spikes.

10:57

Arize skills give coding agents an on-demand investigation path

Ali presents Arize skills as an alternative for teams that want to use their own coding agent instead of an always-on agent managed by Arize. The skills let agents such as Cursor, Claude Code, Codex, OpenCode, and Hermes query Arize traces while working inside an application repository. A coding agent can inspect what the deployed agent did in a trace and compare it with the code that defines the behavior. The skills include trace analysis, failure-dataset creation, and evaluator execution. Signal runs continuously in the background, while the skills support an engineer who is actively working on a product and wants an agent to investigate a failure on demand.

23:18

The improvement loop separates detection, investigation, repair, and verification

Ali frames the workflow as four steps. First, detect a failure before a user reports it. Second, investigate it with real traces and the application code. Third, propose or apply a fix, such as correcting the price-filter logic. Fourth, verify the change against actual failure cases. Trace data shows what the agent did, while the repository shows what it was supposed to do. Signal or a coding agent can connect those two sources and prepare a pull request. Ali recommends keeping code review with a person, although teams can decide how far to automate the process. Verification is separate because an agent's claim that a fix works is not enough.

26:34

A harness judge can adapt its checks to an agent's trajectory

Ali contrasts a fixed LLM judge with a harness judge. A normal judge receives an input and output and scores them against a rubric written in advance. That works for known issues, but teams cannot write every test for failures they have not seen. A harness judge can inspect the full trace tree, including tool choices, tool arguments, handoffs, and external context. It can also call outside systems, such as checking whether a user is allowed to perform an action. For the shopping example, the rubric can be stated plainly: pass when every returned product is within the requested minimum and maximum price, and fail when any result exceeds the budget. The harness can then apply that check to later runs.

30:27

Agent experiments replay real failures before a pull request ships

Agent experiments let a team deploy a development version behind an endpoint and run it against a dataset of failures collected from production traces. For the Wonder Toys example, Arize skills can create a dataset of failed searches for birthday gifts under $30. The proposed code change can then be tested against those inputs with the same price-correctness judge used in production. Engineers can compare the old and new behavior while also checking latency and cost. Ali describes a full loop in which an agent collects failures, creates a dataset, runs the proposed version against it, and puts the results in the pull request. A person can review the code and the evidence before shipping.

35:29

Tracing is the prerequisite for the rest of the workflow

Ali closes by returning to setup. Without tracing, there is no record for Signal, coding-agent skills, or failure-dataset creation to inspect. The workshop application uses environment variables for the required keys, then starts with repository cloning, dependency installation, and the development server. The skills can be installed with the command shown on screen, which provides interactive installation or installs the full set. The available skills include trace analysis, dataset curation, and evaluators. His final workflow is detect, investigate, fix, and verify, with the trace as the record of actual agent behavior and the repository as the record of intended behavior.

"Observability has to evolve from humans reading dashboards to really programmatic interfaces that agents consume, agents can operate on, and agents can self-reflect on as well."03:25
Who should watch
  • You are building an agent that behaves well in test cases but produces expensive or embarrassing edge-case failures in production.
  • Your team wants coding agents to inspect traces and repositories instead of relying on engineers to search dashboards manually.
  • You need a way to replay real failures against a proposed agent change and review quality, latency, and cost before release.