Let Your Agent Cook: Using Skills to Evaluate and Improve Your App

Ankur Duggal, Arize AI36:04 · Oct 2026 · 7,951 views
Thumbnail for Let Your Agent Cook: Using Skills to Evaluate and Improve Your App Watch on YouTube
TL;DR
  1. 1

    Agents need traces to show which tools, model calls, and decisions produced an answer.

  2. 2

    Arize CLI skills let a coding agent inspect traces and evaluations, propose code changes, build evaluations, and run experiments.

  3. 3

    Evaluations must match an application's actual failure modes and should be reviewed by a human before changes reach production.

Summary

Ankur Duggal presents a workflow for improving agents with traces, evaluations, and coding-agent skills. He starts by comparing informal checking, or "vibes," with evaluations that can find failures in production. Traces reveal the supervisor, specialist agents, model calls, retrieved documents, and tools behind an otherwise opaque agent.run call. In a financial-agent demo, the system gives investment advice without enough information, produces bad summaries, fabricates answers, repeats tool calls, and hides tool failures. Duggal connects Arize data to Claude Code through a CLI profile and skills. The coding agent downloads traces, examines evaluations and annotations, suggests new evaluators, changes the application, and tests fixes against datasets built from failed spans. An international-stock experiment shows that changing country context did not work until the underlying tool was updated. Duggal also discusses local experiments, hosted evaluations, judge-model choices, sampling, costs, and the need for human review. He recommends evaluations tailored to the application because models and user behavior change over time.

Key ideas
00:12

Vibes can start development, but production needs evidence of failure

Duggal describes "vibes" as checking an agent's input and output to see whether the answer looks correct. That approach helps teams get started, but it does not show where a production system breaks or how to fix it. Evaluations are difficult because agent behavior is not deterministic and teams may not know what to measure. He argues that teams need evaluations that identify specific failure cases and support repeated monitoring as the agent changes.

02:08

Traces expose the decisions hidden behind agent.run

Traditional software traces follow a mostly fixed path through code. An agent call is different: after agent.run receives a query, the system may call tools, retrieve documents, invoke other agents, or perform more model calls. A trace records those steps. Duggal shows a refund example where the agent calls get order status even though the user wants to know why the refund has not arrived. The trace makes the wrong action visible and points toward the need for another tool or workflow.

04:15

Improvement requires a loop from failure to evaluation to experiment

Duggal's proposed loop starts with identifying production failures, collecting enough examples for experimentation, determining whether the prompt, guardrails, tool, or application code caused the problem, and building evaluations for the affected behavior. A fix is then tested in an experiment before it is shipped. He says this process works, but is manual and tedious when engineers must find traces, inspect failures, propose changes, and repeat the work by hand.

06:03

A coding agent can combine observability data with the codebase

The workflow brings together Arize traces, evaluations, human annotations, application monitoring data, and the codebase inside a coding agent such as Claude Code, Cursor, or Codex. Duggal says the agent can pull traces, build datasets, create evaluations, run experiments, and suggest code changes. This gives it information about the exact pathway that failed, rather than only an incorrect final answer. The CLI and skills provide the connection to those platform operations.

16:03

The financial demo contains several distinct failure modes

The financial agent recommends an investment without knowing the user's financial background or asking enough questions. Duggal's analysis also finds numeric groundedness problems, fabricated answers, redundant tool calls, silent tool failures, and hallucinated responses after tools fail. One reported result is that about 44% of interactions produce a bad summary. He asks the coding agent to build an evaluator for silent tool success and to fix numeric groundedness, then test the changes against spans where the failures occurred.

20:05

A passing experiment may require changing the tool as well as the prompt

An earlier issue involved questions about stocks outside the United States, including South Korea. The coding agent created a dataset from the failed examples and ran experiments. Adding the country name did not fix the problem because the application was still calling an outdated tool. Updating the tool to the stable version produced a later iteration with about a 90% success rate. Duggal presents this as evidence that a plausible application edit must be tested against the actual failure dataset.

22:23

Evaluation infrastructure can run locally or in the cloud

Duggal says the experiment in his demo ran on his local machine, although experiments can also be hosted elsewhere. Evaluations can run in Arize, and the demo includes both a code-based evaluator and an LLM judge. Teams can choose model providers, including local or open-source models. The choice depends on the application, the desired quality, and the cost. He does not offer a universal benchmark for selecting the best judge model.

26:13

Evaluations monitor behavior after a fix instead of defining the fix alone

A summary evaluator checks whether a summary uses the relevant context and answers the question appropriately. Duggal agrees that prompts, skills, and coding workflows should be improved so the agent produces better summaries. Evaluations then monitor whether those improvements continue to work. Model regressions and changes in user questions can cause performance to degrade later, so the evaluator needs to remain active after the initial fix.

31:28

Human review keeps the coding agent focused on the real defect

Duggal warns that coding agents can spend their effort on linting or other small code issues. The skills are intended to direct attention toward evidence from traces, such as a wrong tool call, rather than generic cleanup. Even after the agent proposes changes and runs tests, a person should review whether the changes improve the agent and avoid degradation. Evaluations can also be sampled to control cost instead of running over every trace.

"You're making those fixes based on, hey, I know this is wrong. Here's where it failed. Here's why it failed. Here's how we can go ahead and fix it."24:48
Who should watch
  • You have an agent that looks good in manual checks but you cannot explain which internal step causes production failures.
  • You want a coding agent to inspect traces and evaluations before changing prompts, tools, or application code.
  • Your team needs evaluations that keep detecting regressions after models, tools, or user questions change.