Automating the investigation phase makes it possible to search production systems weekly for performance work that engineers otherwise defer.
2
Runtime context, structured skills, scoring, and verification are needed before an autonomous agent can produce work that people trust.
3
Human-friendly reports and a single high-ROI recommendation are more useful than sending engineers a flood of automatically opened pull requests.
Summary
May Walter describes a workflow that uses coding agents to find and verify performance improvements in live production systems. The problem is the unpredictable investigation phase: a performance complaint can take an hour or weeks to understand, so teams defer it until the issue becomes an emergency. Hud's approach runs weekly against production traces, queries, latencies, and function-level context. It scores opportunities by impact, frequency, business importance, and risk, then has the agent propose a fix, rerun tests, and measure the affected flow. A human reviews the result. Walter explains why early versions produced plausible but unverified fixes, lazy exception handling, and inconsistent results from complex ClickHouse queries. The system improved through prod-to-code context, a query language, reusable investigation skills, and guardrails. The talk's central lesson is that autonomous engineering needs a higher trust threshold than interactive coding assistance, and that good context matters more than clever prompting.
Unpredictable investigation costs cause teams to ignore performance debt
A product complaint about a slow page often turns into an estimate ranging from an hour to a week because engineers first have to discover whether a fix exists. Older code may only be understood by one person, especially when its original contributors have left. Walter says teams repeatedly find performance and stability problems when they finally investigate, but they do not proactively search because the research cost is hard to justify. Problems then degrade until they become urgent, get fixed in emergency mode, and return to the backlog. Automating the investigation could turn that missing phase into a weekly process based on real production context.
The workflow connects production evidence to a verified code change
The proposed workflow runs regularly, analyzes production data, scores opportunities, and flags fixes that are both easy and impactful. It examines traces, queries, latencies, and endpoint behavior, then connects that evidence to functions and files. The agent is expected to identify a root cause, explain why it matters to a business flow, make a change, rerun tests, and check the effect on the specific flow. A person remains the review gate. The report should say what works, why it matters, and what changed, rather than simply offer an untested idea.
A vendor-neutral workflow needs scheduling, permissions, and maintenance
Walter says the team wanted infrastructure that was independent of the compute environment, workflow harness, and model. It also needed secure tool calls, permissions, authentication, and triggers such as scheduled runs or webhooks after an SLO breach. Maintenance mattered because agent workflows and expectations change over time. Hud chose GitHub agent workflows for its setup, with GitHub Actions running weekly, Cloud Code as the selected agent, runtime intelligence exposed through MCP, and reports sent to Slack. Walter notes that other agents and destinations could be used.
Prod-to-code gives agents function-level evidence instead of disconnected service metrics
Agents reason about code at the function and file level, while common production metrics describe services, endpoints, CPU, memory, or aggregate P90 values. Those views do not directly explain where time is spent in code. Hud's prod-to-code approach connects functions to the endpoint, event consumer, or cron job that invoked them. It can show calls to databases, LLMs, or other services and retain forensic evidence for slow requests beyond a chosen threshold. This lets an agent investigate a question such as where an endpoint's seven seconds were spent using evidence at the same level as the code.
Reusable query skills make runtime investigations more consistent
The runtime data is exposed through the HUD query language, which uses ClickHouse queries over functions, endpoints, and forensic data. Walter says direct querying created too much variance in evaluations because agents did not always reach the right query or ask it consistently. The team added skills that encode investigation procedures. For an HTTP 500, a skill can trace where the error originated. For a memory spike, it can inspect what ran on the affected pods at that time and compare it with a baseline. These skills provide a more repeatable method than relying on ad hoc queries.
Performance automation looks for concrete patterns that production data can rank
With runtime data, a query language, and investigation skills, the team can build automations for tasks such as finding dead code or improving performance. The performance workflow searches for artificial delays, timeouts, sleeps, N+1 queries, missing indexes, and sequential scans. These patterns are easier to find when the system is explicitly looking for them, especially in an old codebase with many contributors. Runtime evidence also helps distinguish a real cause from guesses based only on static code analysis. The agent can rank a specific endpoint problem by observed behavior rather than by a generic code smell.
Scoring and small reports prevent useful findings from becoming review overload
Opening a pull request for every finding created a different problem: people did not want a flood of small PRs. The team began with one recommendation at a time and scored the hot path, business importance, and risk. Payment and sign-up flows received more attention, while changes involving migrations required more human review. The report explains the observed behavior and proposed fix in a short, readable format. Walter gives an example of an endpoint that usually takes 200 milliseconds but occasionally takes 45 seconds because it uses distinct rather than a Mongo search index. Engineers can create a ticket, make the change themselves, or inspect the finding.
Autonomous agentic engineering needs a higher trust bar than interactive coding help
Walter distinguishes using an agent inside an IDE from running an unattended automation. An 80% success rate may be tolerable when an engineer is present to steer the agent, but an autonomous workflow needs high confidence that the issue is worth fixing and that the proposed fix works in runtime. The hardest part was reaching the point where a person receives a verified, worthwhile recommendation instead of spending time on false positives. Walter describes this as moving toward 80-90% trust, with context, skills, scoring, and guardrails doing more work than model cleverness alone.
"We have to pay that debt and to make sure that we spend engineering time on it in order to even know what can be done about it."03:55
Who should watch
You maintain a mature production codebase where performance work is repeatedly deferred because investigation time is unknown.
Your team is experimenting with autonomous coding workflows and needs evidence, scoring, and review gates before trusting them.
You want a concrete pattern for turning runtime traces and latency data into prioritized engineering work.
