AI coding tools are reducing the amount of human work in software development while leaving production troubleshooting largely unsolved.
2
Traditional AIOps, standalone LLM analysis, and runbook-based agents struggle with noisy telemetry, limited context, stale procedures, or slow searches.
3
Traversal combines causal machine learning, reasoning models, and swarms of parallel agents to investigate unfamiliar incidents from first principles.
Summary
Anish Agarwal divides software engineering into system design, software development, and production troubleshooting. AI tools such as Cursor, Windsurf, and GitHub Copilot are taking on more development work, but that leaves engineers with less understanding of the code and more complex systems to debug. Agarwal describes the current process as dashboard searches, code inspection, and large Slack incident channels. He argues that AIOps creates too many alerts, LLMs cannot search the scale of production telemetry, and runbook-driven agents depend on procedures that quickly become outdated. Traversal's approach combines causal machine learning to separate causes from correlated failures, reasoning models to interpret rich context, and swarms of agents to search telemetry in parallel. Matthew Schoenbauer presents a DigitalOcean case study in which Traversal investigated incidents from the initial alert and reported a likely deployment-related cause in about five minutes. He says DigitalOcean measured a 40% reduction in the time needed to find and resolve incidents.
Software engineering is being split into design, development, and troubleshooting
Agarwal describes three parts of production software work: designing the system, writing its business logic and DevOps code, and troubleshooting incidents after deployment. AI coding tools such as Cursor, Windsurf, and GitHub Copilot are increasingly handling development. His expectation is that system design and troubleshooting will remain. The intended future is for engineers to spend their time on architecture and creative decisions while AI writes code and resolves incidents. Agarwal says that future will fail if troubleshooting is left unattended.
AI-written code can leave engineers with more on-call work
Agarwal argues that incident debugging will become harder as AI writes more of the code. Engineers will have less context about how systems work because they did not write the code or hold its details in their minds. At the same time, teams will push AI-built systems toward greater complexity. The result, he says, could be engineers spending most of their time on quality assurance and on-call work. He calls this a grim outcome, since the systems will be harder to understand just as more software reaches production.
Incident response often becomes dashboard dumpster diving
The current workflow starts with observability tools such as Grafana, Datadog, Splunk, Elastic, and Sentry producing dashboards and processed data. When a system breaks, engineers search through thousands of dashboards and logs for a useful lead. They then try to connect that lead to a pull request, configuration change, or another recent system change. If the cause remains unclear, more teams join a Slack incident channel. Agarwal says this can grow to 30, 40, 50, or 100 people searching in parallel without a reliable path to the root cause.
Agarwal groups the current approaches into AIOps, LLM log analysis, and tool-using agents. Traditional anomaly detection produces too many false positives in complex, changing systems, so teams get more noise than signal. An LLM can explain a log when someone already knows which log matters, but production systems may contain terabytes of data or a trillion logs, far beyond what can fit into a context window or even memory. Runbook-based agents depend on procedures that are often outdated when they are created, while broad tool-calling searches can take days instead of the two to five minutes an incident response needs.
Traversal combines causal analysis with semantic reasoning
Traversal's approach combines three components. Causal machine learning helps distinguish a root cause from correlated failures, since many parts of a system can fail around the actual source of an incident. Reasoning models interpret semantic context in log fields, metric metadata, and code. A third component, which Agarwal calls swarms of agents, runs thousands of parallel tool calls to search telemetry exhaustively. The aim is autonomous troubleshooting for incidents the system has never seen before, using first-principles debugging rather than a memorized workflow.
The system connects an incident lead to a concrete code change
Agarwal maps Traversal's components onto the existing incident workflow. Agent swarms and causal machine learning search a large body of information for a promising lead. Code agents and vector search then connect that lead to a specific change, such as a pull request or configuration update. Because the agents build context in real time, the system can bring the right team into an incident with an explanation of why they are needed. This addresses the common experience of being added to a live incident without enough information to contribute.
Traversal investigated DigitalOcean incidents from a small initial alert
Schoenbauer describes DigitalOcean as a cloud provider serving hundreds of thousands of customers each day. Before Traversal, an on-call engineer might enter a Slack channel with 40 to 60 other engineers and search hundreds of millions of metrics across thousands of dashboards, along with tens of billions of logs from thousands of services. Traversal starts with the same small amount of context that engineers receive when an incident begins. It then sends a swarm of specialized AI SRE agents through petabytes of observability data in parallel.
The incident report gives engineers a cause, evidence, and follow-up tools
After about five minutes, Traversal reports its findings in the incident Slack channel. In the DigitalOcean example, it identified a deployment that introduced changes and caused a cascade of issues through the system. Engineers could roll back the deployment and continue their work. The Traversal interface also cites relevant observability data, assigns confidence levels to possible root causes, and explains its reasoning. An AI-generated impact map and follow-up questions let engineers examine how the incident affects the part of the stack they care about. Schoenbauer says DigitalOcean measured a 40% reduction in the time needed to find and resolve incidents.
"What we found is this idea of swarms of agent where you have these thousands of parallel agentic tool calls happening giving you this kind of exhaustive search through all of your telemetry in some sort of efficient way."08:36
Who should watch
You are responsible for production systems where incident response depends on searching large volumes of metrics and logs.
Your team is adding AI-generated code but has not decided how automated troubleshooting will work.
You are evaluating observability agents and want to understand why anomaly detection, LLM log analysis, and runbook-based agents can fall short.
