Fresh, task-specific documentation helps agents avoid inventing keys, patterns, and APIs for fast-moving software projects.
2
Small reference applications, called model airplanes, give agents repeatable integration patterns across frameworks and languages.
3
PostHog improves agent runs by breadcrumbing tasks, interrogating failures, and restricting access to sensitive files such as .env files.
Summary
Danilo Campos describes how PostHog makes its Wizard more reliable for automated integrations. The system serves around 15,000 people each month, and its main defense against model rot is to supply current Markdown documentation from posthog.com at runtime. PostHog also maintains lightweight reference projects, called model airplanes, across frameworks and languages. These show the shape of a correct integration without carrying the complexity of a production application. The Wizard guides agents through a sequence of small steps: find business-relevant files, propose events, record them, then load the relevant documentation and edit the code. Campos also describes failures caused by human-maintained instructions, including contradictory tool guidance, missing tools, and language mismatches. A stop-hook asks the agent what would have helped it succeed. Security requires tighter controls. The Wizard stopped reading full .env contents and instead checks whether a key exists or writes a new value. Campos argues that most of the system's value is now in Markdown and the order in which information reaches the agent.
Model rot makes static training knowledge unreliable for fast-moving projects
Campos says models are snapshots of the web from perhaps 6, 8, 12, or 18 months earlier. That becomes a problem when an agent works on a software project whose APIs and patterns change quickly. PostHog handles this by giving the agent fresh Markdown documentation from posthog.com. The agent first identifies what integration is needed, then uses tools to choose the relevant documents and place them into its context. This approach addresses the failures Campos saw when primitive agents invented keys, patterns, and APIs that did not exist.
Model airplanes show agents the shape of a correct integration
PostHog maintains lightweight example projects across several frameworks and languages. Campos calls these projects model airplanes because they resemble real applications without implementing all of their production behavior. Authentication, for example, may accept any password, while retaining the structure of an authentication flow. That gives the agent a concrete pattern for placing PostHog event tracking around logins and identity. The examples also use fewer tokens than full production applications, so the agent can inspect them efficiently and reproduce the integration more consistently.
Breadcrumbs keep agents from improvising thousands of integration patterns
With roughly 15,000 integrations each month, an unconstrained agent could produce 15,000 different setups and create a difficult support problem. PostHog therefore leads the agent through intermediate tasks instead of announcing the final integration immediately. It first finds files with business value, such as login, Stripe, or churn-related code. It then identifies interesting events without writing code, records event names and descriptions, and only later loads the framework-specific documentation and edits the files. Campos says business concepts cast a large shadow in code, which makes this staged discovery reliable.
A stop-hook exposes failures caused by human-maintained instructions
Campos says the team itself is a major source of agent failures. PostHog found contradictory MCP instructions, references to a tool that did not exist, and JavaScript guidance sent to an agent working in a Python project. These problems affected hundreds of runs in the missing-tool case. At the end of each run, the team asks the agent what would have better prepared it for success. This inexpensive inference-time interrogation revealed missing permissions and conflicting directives that the team would not have found by simply observing successful or failed outputs.
Sensitive file access should be reduced to narrow operations
An early Wizard version read .env files because the agent needed to write configuration. Campos says sending complete environment contents to a cloud service could leave secrets in logs. PostHog restricted which reads the agent could perform and replaced full-file access with a narrow tool. The tool can check whether a named key exists and write a new value to a key. It does not send the environment file contents through inference. This limits what the agent can learn while still supporting the setup task.
Markdown and information sequence can matter more than generated code
Campos says the Wizard is about 90% Markdown files, 8% tools that deliver and process Markdown, and the remainder agent-harness code. He argues that prose can keep gaining value as models improve, because newer models can do more with the same instructions. The goal is therefore to give the agent enough information and arrange it in a useful sequence, while avoiding unnecessary constraints on how it solves the problem. PostHog's examples, documentation, and staged prompts provide direction without encoding every implementation detail.
The context service packages examples into skill files
In the question period, Campos explains that PostHog uses skill files generated by a context service. The service flattens the model airplanes into a Markdown file and includes that file as a reference in the skill. The agent can search through the complete example when working. The skill also includes ordinary documentation. PostHog found that both kinds of material are useful: prose explains the integration, while the model airplane shows what a successful implementation looks like.
"The wizard that makes everybody so happy is 90% markdown files, 8% tools for delivering and processing markdown files, And then the rest is like Agent Harness stuff, right?"15:01
Who should watch
You are building an autonomous coding agent that must work across many frameworks and need examples of how to guide its decisions.
Your agent produces plausible but inconsistent integrations, and you need a staged workflow that reduces improvisation.
You run code-generating tools on users' machines and need to limit access to secrets while finding failures in your own instructions.
