# From Signal to PR: Anatomy of a Self-Improving Agent Jason Lopatecki, Arize | AI Engineer World's Fair 2026 | 20:36 Source: https://www.youtube.com/watch?v=9HbzAWnKbo4 Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/from-signal-to-pr-anatomy-of-a-self-improving-agent Published: 2026-07-24 Tags: agents, coding-agents, debugging, human-in-the-loop, observability ## TL;DR - Observability data can give coding agents the exact production path that failed, instead of forcing them to guess among many possible code paths. - Signal runs skills in a sandbox, gathers traces and logs into files beside the code, and can open an issue or prepare a pull request for human review. - The approach requires more telemetry, carefully designed skills, and human involvement for larger fixes, so the engineer's role shifts toward reviewing and guiding investigations. ## Summary Jason Lopatecki describes Arize's move from Alyx, its earlier product agent, to Signal, a system that investigates production problems before a human starts debugging. The central design choice is to pull relevant traces and logs into files in the repository, alongside the source code. Coding harnesses can work with those files and use skills to find the right evidence, inspect the failure, and suggest a fix. Signal can run periodically or in response to an event inside a chosen harness and sandbox, including an Arize sandbox in a customer's VPC. Lopatecki argues that teams will need to trace and log much more because agents can process the resulting data. He is clear about the current boundary: small fixes can be nearly automatic, while larger changes still need a human to drive the work. Evals add preprocessed assessments to production traces, giving the agent another source of evidence about recurring failures. ## Key ideas ### Observability data is becoming input for coding agents [02:03](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=123s) Lopatecki says observability was built for humans who click through UIs, graphs, and dashboards. Agents need the telemetry itself. Traces and logs show the path that the software actually took, while a developer without that evidence is guessing among a million possible paths. Skills connected to observability systems can gather the context an agent needs for debugging. In this model, telemetry is the smoke thrown off by a running system, and an agent reads that smoke to investigate and propose changes. ### The target is a loop where systems investigate and improve themselves [02:43](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=163s) Lopatecki's stated goal is to build systems that autonomously fix themselves. The current version of that process still has a human making or reviewing changes. He says teams can already build at agent speed, but they cannot yet improve their systems at the same speed. The limiting factor is confidence that a proposed fix is the right one to push. His proposed loop starts with an event or periodic trigger, gathers context through skills, investigates the issue, and gives a person evidence before that person opens the problem. ### An agent can investigate before a human picks up the issue [04:11](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=251s) Signal inverts the usual workflow. Instead of a human looking at a problem and then asking an agent to fix it, the agent examines errors first and can put up an issue. By the time someone looks at it, evidence and an initial investigation are already attached. Lopatecki says people may still drive the second and third steps, especially when the change is larger. The practical shift is from a responder starting with an empty ticket to a reviewer taking over work that already has production context. ### Files beside the source code are the main technical unlock [06:08](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=368s) The workflow starts with traces, often combined with logs, repository contents, and other context. Arize's skills pull temporary production files into the repository, next to the code. That gives the agent both the source and what actually happened in production. Lopatecki describes the combination of the right data, in a file format, with the code in the repository as the useful property of composable skills. He says some files can be 10 megabytes because coding harnesses work well with files, whereas simply pointing an agent at a data source is less effective. ### The local coding-agent workflow can run in an event-driven sandbox [07:23](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=443s) Lopatecki frames sandboxes as the next step after running an agent on a laptop. A developer first gets the workflow working locally, then runs it in an environment triggered by an error or a schedule. The sandbox can use the selected harness and skills, inspect the repository and telemetry, and continue the debugging loop. Signal can create issues in GitHub, attach evidence, and produce a starting point that a developer can resume locally. The setup lets teams choose how and where the agent runs instead of treating the product's default environment as mandatory. ### A small Alyx failure shows where automation works today [08:13](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=493s) Lopatecki shows a real issue from Alyx, Arize's in-product assistant. The agent was making repeated to-do updates and encountered a stream-canceled event. Signal gathered the relevant data and produced a one- or two-line fix. He calls this an ideal case. Larger fixes still often need a human to spearhead the work, but the agent can provide the cold start with the error, evidence, and a proposed direction. That changes the engineer's job from responding to an unexplained failure toward reviewing and completing an investigation. ### More agent-driven debugging requires much more telemetry [09:39](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=579s) Lopatecki argues that teams will trace and log about ten times more as agents become part of the debugging loop. Humans previously avoided this level of detail because searching through all the logs created too much noise. An agent can use the extra data to determine which path the software took. He imagines logging and tracing every part of a system where useful, then using continuous loops to inspect that evidence and attempt fixes. Traces and evals remain part of the loop rather than being replaced by the agent. ### Customers can choose the harness, skills, and network boundary [10:37](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=637s) Signal is presented as a way to run the local debugging experience repeatedly. Teams can choose a harness such as Claude Code, select skills, set prompts, and pick a sandbox. Arize offers its own sandbox, while customers can use cloud-managed agents or systems such as Daytona. Lopatecki says companies including Uber and Booking do not want production connections sent outside their environments, so Arize can install within a customer's VPC. AX is Arize's SaaS platform, Phoenix is open source, and Signal is currently available in AX, which can also be deployed in a VPC. ### Evals add assessed information to the raw trace [18:04](https://www.youtube.com/watch?v=9HbzAWnKbo4&t=1084s) Lopatecki explains that evals are layered onto production traces as online evals. An eval adds information about the trace, and skills let the harness retrieve both individual eval data and aggregate values across traces. Signal can use those assessments along with the raw trace data. Teams can create an evaluator after finding a recurring failure, such as prompt injection or a repeated bad response, so the system can catch it next time. He describes an eval as an AI assessment that runs repeatedly, often across a large dataset. ## Notable quotes - "The future observability actually looks a lot more like this than it does clicking around Grafana UI." (05:47) - "These harnesses are magical with files." (17:07) - "Your job kind of moves from responder to reviewer." (09:19) - "You're going to trace 10 times more. You're going to log 10 times more because that helps you know what path your software took." (09:39) ## Tools & references mentioned - Arize - Signal - Alyx - AX - Phoenix - Claude Code - Cursor - Pyroscope - Google Cloud - Daytona - GitHub - Uber - Booking ## Who should watch - You are building an AI product and need production failures to arrive with evidence instead of an empty ticket. - Your team already uses coding agents locally and wants to run the same workflows on schedules or production events. - You need to keep debugging inside a VPC while letting an agent inspect traces, logs, evals, and source code.