# LLM codegen fails and how to stop 'em

Danilo Campos, PostHog | AI Engineer Europe 2026 | 19:18

Source: https://www.youtube.com/watch?v=juoNbJiZUi0
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/llm-codegen-fails-and-how-to-stop-em
Published: 2026-04-30
Tags: agents, coding-agents, context-engineering, reliability, security

## TL;DR
- Fresh, task-specific documentation helps agents avoid inventing keys, patterns, and APIs for fast-moving software projects.
- Small reference applications, called model airplanes, give agents repeatable integration patterns across frameworks and languages.
- PostHog improves agent runs by breadcrumbing tasks, interrogating failures, and restricting access to sensitive files such as .env files.

## Summary
Danilo Campos describes how PostHog makes its Wizard more reliable for automated integrations. The system serves around 15,000 people each month, and its main defense against model rot is to supply current Markdown documentation from posthog.com at runtime. PostHog also maintains lightweight reference projects, called model airplanes, across frameworks and languages. These show the shape of a correct integration without carrying the complexity of a production application. The Wizard guides agents through a sequence of small steps: find business-relevant files, propose events, record them, then load the relevant documentation and edit the code. Campos also describes failures caused by human-maintained instructions, including contradictory tool guidance, missing tools, and language mismatches. A stop-hook asks the agent what would have helped it succeed. Security requires tighter controls. The Wizard stopped reading full .env contents and instead checks whether a key exists or writes a new value. Campos argues that most of the system's value is now in Markdown and the order in which information reaches the agent.

## Key ideas
### Model rot makes static training knowledge unreliable for fast-moving projects
[02:09](https://www.youtube.com/watch?v=juoNbJiZUi0&t=129s)
Campos says models are snapshots of the web from perhaps 6, 8, 12, or 18 months earlier. That becomes a problem when an agent works on a software project whose APIs and patterns change quickly. PostHog handles this by giving the agent fresh Markdown documentation from posthog.com. The agent first identifies what integration is needed, then uses tools to choose the relevant documents and place them into its context. This approach addresses the failures Campos saw when primitive agents invented keys, patterns, and APIs that did not exist.

### Model airplanes show agents the shape of a correct integration
[04:51](https://www.youtube.com/watch?v=juoNbJiZUi0&t=291s)
PostHog maintains lightweight example projects across several frameworks and languages. Campos calls these projects model airplanes because they resemble real applications without implementing all of their production behavior. Authentication, for example, may accept any password, while retaining the structure of an authentication flow. That gives the agent a concrete pattern for placing PostHog event tracking around logins and identity. The examples also use fewer tokens than full production applications, so the agent can inspect them efficiently and reproduce the integration more consistently.

### Breadcrumbs keep agents from improvising thousands of integration patterns
[06:44](https://www.youtube.com/watch?v=juoNbJiZUi0&t=404s)
With roughly 15,000 integrations each month, an unconstrained agent could produce 15,000 different setups and create a difficult support problem. PostHog therefore leads the agent through intermediate tasks instead of announcing the final integration immediately. It first finds files with business value, such as login, Stripe, or churn-related code. It then identifies interesting events without writing code, records event names and descriptions, and only later loads the framework-specific documentation and edits the files. Campos says business concepts cast a large shadow in code, which makes this staged discovery reliable.

### A stop-hook exposes failures caused by human-maintained instructions
[09:25](https://www.youtube.com/watch?v=juoNbJiZUi0&t=565s)
Campos says the team itself is a major source of agent failures. PostHog found contradictory MCP instructions, references to a tool that did not exist, and JavaScript guidance sent to an agent working in a Python project. These problems affected hundreds of runs in the missing-tool case. At the end of each run, the team asks the agent what would have better prepared it for success. This inexpensive inference-time interrogation revealed missing permissions and conflicting directives that the team would not have found by simply observing successful or failed outputs.

### Sensitive file access should be reduced to narrow operations
[12:02](https://www.youtube.com/watch?v=juoNbJiZUi0&t=722s)
An early Wizard version read .env files because the agent needed to write configuration. Campos says sending complete environment contents to a cloud service could leave secrets in logs. PostHog restricted which reads the agent could perform and replaced full-file access with a narrow tool. The tool can check whether a named key exists and write a new value to a key. It does not send the environment file contents through inference. This limits what the agent can learn while still supporting the setup task.

### Markdown and information sequence can matter more than generated code
[13:50](https://www.youtube.com/watch?v=juoNbJiZUi0&t=830s)
Campos says the Wizard is about 90% Markdown files, 8% tools that deliver and process Markdown, and the remainder agent-harness code. He argues that prose can keep gaining value as models improve, because newer models can do more with the same instructions. The goal is therefore to give the agent enough information and arrange it in a useful sequence, while avoiding unnecessary constraints on how it solves the problem. PostHog's examples, documentation, and staged prompts provide direction without encoding every implementation detail.

### The context service packages examples into skill files
[16:19](https://www.youtube.com/watch?v=juoNbJiZUi0&t=979s)
In the question period, Campos explains that PostHog uses skill files generated by a context service. The service flattens the model airplanes into a Markdown file and includes that file as a reference in the skill. The agent can search through the complete example when working. The skill also includes ordinary documentation. PostHog found that both kinds of material are useful: prose explains the integration, while the model airplane shows what a successful implementation looks like.

## Notable quotes
- "The wizard that makes everybody so happy is 90% markdown files, 8% tools for delivering and processing markdown files, And then the rest is like Agent Harness stuff, right?" (15:01)
- "And so, one of the things that we do is we start off barely even telling the agent that this is what we're doing." (07:54)
- "You have to ask to find out." (11:59)
- "Plain text prose is where so much of our value now lives." (15:27)

## Tools & references mentioned
- PostHog Wizard
- PostHog
- Anthropic
- Cursor
- Claude Code
- Claude agent SDK
- MCP
- BlueSky
- Twitter

## Who should watch
- You are building an autonomous coding agent that must work across many frameworks and need examples of how to guide its decisions.
- Your agent produces plausible but inconsistent integrations, and you need a staged workflow that reduces improvisation.
- You run code-generating tools on users' machines and need to limit access to secrets while finding failures in your own instructions.

## Related talks

- [MCPs are Boring (or: Why We Are Losing the Sparkle of LLMs)](https://aietalks.com/talks/mcps-are-boring-or-why-we-are-losing-the-sparkle-of-llms) (Manuel Odendahl, 28:32)
- [Building Enterprise LLM Agents That Work](https://aietalks.com/talks/building-enterprise-llm-agents-that-work) (Shaan Desai, Cohere, 18:29)
- [12-Factor Agents: Patterns of Reliable LLM Applications](https://aietalks.com/talks/12-factor-agents-patterns-of-reliable-llm-applications) (Dex Horthy, HumanLayer, 17:06)
- [Can LLMs Generate Enterprise Quality Code?](https://aietalks.com/talks/can-llms-generate-enterprise-quality-code) (Prasenjit Sarkar, Sonar, 15:12)
- [Code Mode: Let the Code do the Talking](https://aietalks.com/talks/code-mode-let-the-code-do-the-talking) (Sunil Pai, Cloudflare, 19:40)