PostHog is turning product signals such as errors, Slack messages, and session replays into grouped reports that can produce pull requests.
2
Embedding raw signals groups them by source format, so PostHog generates LLM queries from signals and embeds those queries instead.
3
Agents need enough evidence and specificity to act safely, because a generic report such as "Onboarding is broken" can lead to a meaningless fix.
Summary
Joshua Snyder describes PostHog's attempt to turn observability data into pull requests. The system ingests signals from product analytics, error tracking, logs, Slack, and session replay, filters unsafe inputs, normalizes them, groups related signals into reports, and sends worthwhile reports to research and coding agents. The research agent gathers more evidence from PostHog data, the codebase, Linear, and Notion before deciding whether a problem is actionable. A coding agent then works in a sandbox, opens a PR, and reruns after CI failures or review comments. Snyder is direct about the hard parts. Raw embeddings group errors with errors and Slack messages with Slack messages because they capture structural similarity. Generic reports cause agents to invent fixes. He also argues that early experiments should use agents heavily, even when the cost looks unreasonable, because repeated runs reveal patterns that can later be replaced with cheaper calls or models. The system is in alpha.
PostHog wants product signals to produce pull requests instead of dashboards
Snyder describes the current workflow as a signal changing a metric, an engineer noticing it in a dashboard hours or days later, investigating it, filing an issue, writing a PR, reviewing it, and shipping it. PostHog is building a background agent that investigates the signal and creates a PR automatically. Engineers would review ready PRs in GitHub rather than inspect analytics dashboards, errors, or logs. Low-risk changes could potentially ship behind a feature flag without manual approval.
The pipeline groups noisy signals before sending them to agents
The system ingests a large stream of events, including errors, session recordings, logs, experiments, and customer messages. It normalizes each item into a common signal structure with a source product, type, content, and weight. Related signals are grouped into a report. When a report's weight crosses a threshold, the system promotes it and starts research. An error tracking issue and a customer message saying checkout is broken may be different signal types that describe the same product problem.
Raw embeddings cluster source formats instead of shared problems
Snyder says PostHog first embedded the signals themselves and used those embeddings to find related issues. The result was poor because off-the-shelf models noticed structural similarity. Errors clustered with other errors, Slack messages with Slack messages, and session replays with session replays. A checkout error therefore did not reliably group with an onboarding error-related message from another source. PostHog changed the process by asking an LLM what each signal is about, generating several queries, and matching those queries in embedding space.
The research agent gathers evidence and decides who should act
PostHog runs the research agent with the Claude agent SDK inside a Modal sandbox. The agent can pull additional product data through PostHog's MCP server, inspect the codebase, and use external MCPs. Snyder says Linear and Notion help connect the problem to useful context. The output includes a problem summary, a priority, and a reviewer suggestion based on Git blame. This research gives the later coding step more context than the original signal alone.
A report can be returned to the pool when it lacks enough data, placed in an inbox when it needs a product decision, or sent directly to a coding agent when it is actionable. Error tracking tends to provide specific problems that coding agents can address. Slack messages and session replays often describe generic problems with several possible solutions. Snyder warns that a report such as "Onboarding is broken" gives an agent permission to fix something at random, producing noisy PRs.
The coding agent keeps working until the PR is green
The execution step clones the repository into a sandbox and runs the Claude agent SDK to implement the fix. It pushes a PR, then reruns when CI fails or someone comments on the PR. PostHog snapshots the sandbox and rehydrates it when more work is needed. The intended result is that an engineer can start the morning with green PRs instead of CI failures and review comments that still require local investigation.
Representative production evals are needed for this pipeline
Snyder says local testing and informal vibe checks were inadequate because the pipeline handles varied customer data. The team needed representative production data to understand whether the system worked. Without that, he describes iteration as fumbling in the dark. He also repeats that teams must think carefully about what they embed when their data contains several formats, since structural similarity can dominate the grouping behavior.
Repeated agent runs reveal expensive steps that can be simplified
PostHog initially tried to avoid agents or delay them because the pipeline looked too expensive. Snyder says this was a mistake during experimentation. Running the same problem through an agent many times exposed repeated patterns in the agent's behavior. Those patterns allowed the team to replace an expensive agent step with a one-shot LLM call or a faster trained model. His practical advice is to accept high experimental cost long enough to learn what can later be simplified.
The long-term system will learn from every PR outcome
The alpha system currently turns incoming product data into grouped reports and PRs. Snyder wants it to ship experiments, measure their effects, approve simple changes with an agent, and deploy them behind feature flags. He also wants later PRs to use feedback from rejected PRs, deployment problems, and whether an error actually disappeared after release. The team planned to keep iterating on that feedback loop as the system rolled out.
"We want to turn observability data instead of something that you read and that you interpret based on dashboards, we're trying to turn that into something that submits pull requests for you."00:14
Who should watch
You operate a product that produces errors, logs, replays, analytics events, or customer messages, and want those signals to create engineering work automatically.
You are designing an agent pipeline that must connect evidence from several data formats before it changes code.
You are deciding whether agent costs are too high during prototyping and need a way to find steps that can later become cheaper calls.
