Datadog is building AI agents that investigate alerts, inspect observability data, suggest remediation, and write postmortems.
2
Task-specific agents are easier to evaluate than general-purpose agents, but teams must define measurable jobs to be done before building demos.
3
Agent products need interfaces that support human review and observability views that make complex, multi-step model workflows understandable.
Summary
Diamond Bishop describes Datadog's work on AI agents for software operations. Its AI on-call engineer starts investigations when alerts fire, reads runbooks, examines logs, metrics, and traces, tests hypotheses, suggests remediation, and can prepare a postmortem. The AI software engineer analyzes errors, proposes code fixes, creates tests, and can open a GitHub pull request or a VS Code diff. Bishop is honest about the hard parts. Demos are easy to build, while evaluating them over time is difficult. He recommends narrowly scoped agents with measurable tasks, domain experts who verify results, and test sets that include offline, online, and production feedback. He also argues that agent interfaces should support review and follow-up questions. Datadog's LLM observability and agent graph help engineers inspect complex runs that may contain hundreds of model and tool calls. He expects agents to become important users of software products, so APIs and context should be designed for them as well as for humans.
Datadog is moving from observing systems to having agents operate them
Bishop describes Datadog as an observability and security platform for cloud applications. The company has shipped AI features such as proactive alerting, root-cause analysis, impact analysis, and change tracking since around 2015. He says the current model and reasoning advances are changing what users expect from software. Datadog is responding by building agents that use its platform for customers. The first products he names are an AI software engineer, which investigates errors and recommends code, and an AI on-call engineer, which handles alert investigations.
The AI on-call engineer follows an investigation loop before asking humans to act
When an alert occurs, the AI on-call engineer starts proactively. It reads runbooks, gathers alert context, and inspects the logs, metrics, and traces that an engineer would normally review. It can investigate before the engineer reaches a computer and present a summary of what happened. The agent forms hypotheses, uses queries and other tools to test them, and validates or rejects each possibility. If it finds a likely root cause, it can suggest actions such as paging another team or scaling infrastructure. Existing Datadog workflows can also be connected to remediation.
Human review is built into the agent's operating model
Bishop says agents should act like human teammates while still allowing people to verify their work. Datadog's collaboration page shows why a hypothesis was generated, what evidence the agent found, and which runbook steps it followed. An engineer can ask follow-up questions about a decision and decide whether to accept the suggested action. This review makes the agent's work easier to inspect and helps people build trust over time. After an incident, the agent can examine what happened, what it did, and what humans did, then produce a postmortem for the team.
A software engineer agent can turn detected errors into proposed code changes
Datadog's AI software engineer watches errors and works as a proactive development assistant. Bishop shows a case where the assistant identifies a recursion issue, proposes a fix, and creates a recursion test that could catch the problem later. The engineer can create a pull request in GitHub or open the diff in VS Code for editing. Bishop presents this agent and the on-call engineer as complementary: one helps reduce the number of incidents by addressing code problems, while the other investigates incidents that still occur.
Agents should be scoped around measurable jobs to be done
Bishop's first lesson is to define the work clearly before building. Teams should describe the task step by step from a human point of view, then decide how another human would evaluate it. This leads Datadog toward vertical, task-specific agents rather than generalized agents. Each step should be measurable and verifiable where possible. He says teams can build a convincing demo quickly, yet struggle to prove that it continues to work and improves over time. Domain experts should act as design partners or task verifiers, rather than writing fixed rules for stochastic models.
Evaluation must continue from offline tests into production use
Bishop says evaluation was one of Datadog's largest sources of mistakes when it was not considered early enough. Agent teams need offline tests, online evaluation, and end-to-end task measurements. They also need instrumentation that shows whether people use the product and provide feedback. That feedback should update a living test set. His point is practical: an agent can appear successful in a demo while its behavior remains difficult to verify across changing models, tools, and real tasks.
Agent teams need a small amount of specialist knowledge and many fast-moving builders
Bishop does not think an agent team needs to be made up of machine learning specialists. He recommends starting with one or two ML experts and adding optimistic generalists who write code well and are willing to try ideas quickly. Front-end and user experience work matter more than he initially expected, especially when people collaborate with agents. He also wants teammates who actively use AI, explore new capabilities, and are comfortable with a field that changes quickly. Without that attitude, he says, teams can get stuck.
Agent observability needs workflow views instead of a flat call list
Bishop argues that observability should be designed into agent systems from the start. An agent may make hundreds of model and tool calls, loop through decisions, and interact with hosted or API-based models. A flat list of those calls does not make the workflow easy to understand. Datadog's LLM observability brings these interactions into one view, while its agent graph shows the workflow structure. In the example he describes, a red node identifies an error, and engineers can zoom in to inspect what happened in a readable form.
Products should be designed for agents that use them as customers
Bishop expects agents to become users of software products alongside humans. He estimates that agents may surpass humans as users within about five years, while acknowledging that estimates range from one to ten years. He gives third-party agents such as Claude using Datadog through MCP as an example. Product teams should consider the context and API information those agents need. He also imagines teams hiring groups of DevOps, security, and coding agents that integrate with platforms and handle operational work.
"You can quickly build something that looks like it works, but then it's very hard to actually verify that over time and improve it."08:47
Who should watch
You are building an incident-response or DevOps agent and need a practical way to scope its tasks and evaluate its output.
Your team has a promising agent demo but lacks production feedback, human review, or a lasting test set.
You build an observability or SaaS product and need to prepare its APIs and context for agents as users.
