Your Agent Fix Just Broke Something Else

Sofia Jakovcevic, Arize AI19:25 · Oct 2026 · 11K views
Thumbnail for Your Agent Fix Just Broke Something Else Watch on YouTube
TL;DR
  1. 1

    A one-line prompt change improved action safety by 17 points while reducing task completion by 17 points.

  2. 2

    Agent behavior changes at runtime, so production traces are a better source of truth than code diffs alone.

  3. 3

    A regression loop should turn production traces into datasets, score them with evaluators, compare experiments, and gate merges in CI.

Summary

Sofia Jakovcevic explains why agent changes are hard to review with ordinary code practices. A small instruction that asks a support agent to confirm before every action improved action safety by 17 points, but reduced task completion by 17 points because users could leave before the agent finished. The code diff was small, while the behavioral change affected runtime trajectories, tool calls, and sub-agent interactions. She proposes a five-step regression loop: build datasets from production traces, score them with evaluators, run a baseline, compare changes, and block merges that fail agreed criteria. The talk includes an Arize walkthrough covering prompt and agent playgrounds, datasets, evaluators, experiments, latency, cost, and error rates. Sofia then describes connecting coding agents to observability data through agent skills. Failed production cases can become new regression tests, and agents can eventually investigate traces and propose fixes instead of relying on humans to inspect dashboards and review every change.

Key ideas
02:26

A harmless prompt fix can improve one agent behavior while damaging another

Sofia describes a support agent whose security engineer adds one line asking it to confirm before taking any action. The change improves action safety by 17 points. It also causes a 17-point drop in task completion because the agent now waits for confirmation before every action, and a user who leaves the session leaves the task unfinished. The average of the two metrics is zero, which hides the trade-off. Sofia uses this example to argue that prompt changes need numeric evaluation across several dimensions. A single improved score does not show that the agent is better overall.

04:48

Agent failures happen in runtime behavior, not just in source code

A code reviewer can see a one-line change in a system prompt file, but that diff does not show the resulting production behavior. Sofia says the change can add steps, alter how the agent works with sub-agents, and change tool calls across thousands of possible trajectories. Those trajectories are captured in observability traces rather than static code. Her distinction is simple: the diff shows what changed, while the trace shows what happened. For agent regression testing, traces become the source of truth because the system's effective logic only appears while it runs.

06:48

Regression datasets should combine ordinary workflows with failures and protected successes

The first step is to select a dataset from real production traces rather than synthesize inputs when real user queries, tool calls, and outputs are available. The dataset should represent average user workflows, known failures, and edge cases. Sofia also says to include successful workflows that the team wants to protect. A fix must work on the cases it targets without breaking behavior that already worked. These four groups give the regression suite examples of normal usage, failures, unusual cases, and successes worth preserving.

07:50

Evaluators turn subjective agent qualities into scores

The second step is to define evaluators for the dimensions the team cares about. Sofia names hallucination, accuracy, tool trajectory, and step efficiency as examples. An evaluator can use an LLM as a judge to inspect a span, trace, or other log data against a guide, then produce a binary result such as complete or incomplete. Teams can also use a 0-to-5 scale or multiple labels. Code-based evaluators fit deterministic checks, while LLM-based evaluators fit less deterministic judgments. The result is a testing suite that can score each experiment consistently.

09:48

Baseline and comparison experiments let teams test changes before deployment

The third step runs the current agent against the production dataset and applies the evaluation suite to create a baseline. The team then makes one change, reruns the experiment, and compares the scores. Sofia says the same process can test prompt changes, handoffs, tools, and models without first deploying to production. It can also test a proposed move to a more expensive model by measuring whether the new model produces a worthwhile improvement. The comparison can include several metrics rather than relying on one overall result.

11:14

CI can block a merge when an agent regression fails agreed criteria

The fifth step moves the experiment into CI. Sofia gives GitHub Actions as an example: each pull request runs against the team's evaluation suite, and the change cannot merge until it meets the listed regression criteria. A negative result sends the team back to testing and iteration. This changes the release decision from an intuition about a natural-language prompt to a decision based on measured behavior. The gate can cover the dimensions the team has chosen, so a safety improvement cannot silently pass while task completion or another protected behavior gets worse.

16:12

Every failed production case should become a new regression test

Sofia describes a loop that starts with observability data from every production run, evaluates that data, tests a prompt, model, tool, or policy change, and ships it only if it passes. When a change fails, the failed production traces return to the observability layer and become new cases in the regression suite. This means the suite grows from actual failures rather than only from the team's original assumptions. Over time, the agent is tested against more of the behavior that previously caused problems.

17:47

Coding agents can use traces to investigate and propose fixes

Sofia says human investigation is the current bottleneck because engineers inspect dashboards, triage failures, create fixes, and review pull requests. Arize's agent skills connect coding agents with the observability backend and its UI features, giving them access to production behavior rather than static code alone. Her proposed shift is from humans using observability to understand agents toward agents using traces to understand and improve themselves. The agent would investigate what happened, evaluate a change, and propose a fix with evidence from runtime behavior.

"We don't want to just give humans the ability to debug agents. We want to give agents the ability to debug and understand themselves."18:03
Who should watch
  • You maintain an agent whose prompt or model changes can improve one metric while damaging task completion, safety, cost, or another behavior.
  • Your team reviews agent pull requests from code diffs but lacks a repeatable way to test runtime trajectories before deployment.
  • You are connecting coding agents to production traces and want failures to feed back into datasets, evaluations, and automated fixes.