Coding agents need current documentation and guided discovery because pre-training context produces outdated Langfuse integrations.
2
Tracing the agent's work reveals most of the problems, while documentation sitemaps, Markdown formats, and a natural-language search endpoint reduce wasted exploration.
3
Auto-research can improve a skill, but its target function can remove the documentation steps that keep the skill current.
Summary
Marc Klingen explains how Langfuse built a skill that helps coding agents add observability and evaluation to applications. The starting problem was a large, changing documentation set. An agent using pre-training knowledge could write outdated instrumentation, discover the failure, then fetch current documentation and repair it. Klingen's team used traces to inspect this process, added ways for agents to find and request documentation, exposed a natural-language search endpoint, and created basic evaluations for different application types. They learned that dynamic material should be referenced rather than copied into the skill, since copied content becomes stale. An auto-research loop proposed six improvements to a prompt-migration task, and the team accepted three. The experiment also exposed a failure in the target function: optimizing for fewer turns caused the agent to remove documentation-fetching steps. Klingen is candid about unresolved problems around skill distribution, versioning, trust, and deciding whether a skill should aim for a quick first result or a complete setup.
Skills give agents a procedure for work they could not infer from tools alone
Klingen compares an agent with a Bash tool to a person holding a Rubik's Cube without knowing how to solve it. A manual changes the situation because the agent can follow a procedure rather than guess from the available actions. He places skills between fixed workflows and fully open-ended agents. A workflow is reliable for a narrow case, such as routing a password-reset request to a dedicated agent. It becomes awkward when a request crosses domains, such as resetting a password and changing an email address together. A skill lets the agent progressively obtain the context needed for the combined task.
Runtime traces reveal missing use cases and weaknesses in existing skills
Klingen says teams can trace agent execution to learn what users actually ask for and how the agent handles those requests. A trace may reveal an unexpected use case, such as users wanting to change a password even though the team did not plan for it. That evidence can lead to a new skill. Traces also show when an existing skill is outdated or takes an inefficient route. Langfuse was used to inspect the coding agent's actions, rather than relying only on the final code. This gave the team a way to see where the agent wandered and adjust the skill toward a more direct implementation.
Pre-training context can make an integration look easy while producing obsolete code
Langfuse had grown to 478 documentation pages across several feature areas, with substantial flexibility in how users could implement tracing and evaluation. An agent could produce Langfuse SDK code immediately from its pre-training context, but interfaces had changed over time. That led to hallucinated methods and old instructions. In Klingen's example, Claude Code first added instrumentation based on stale knowledge, tried to verify it, discovered that it failed, and only then fetched current documentation to fix the code. The process was slow and initially wrong. A trace of two LLM calls also failed to show what the agent was actually doing.
Documentation discovery needs explicit routes that agents can use efficiently
The team added guidance for finding information across a large documentation set. Its skill exposes an agent sitemap so the coding agent can see what material exists before fetching pages one by one. It also tells agents how to request Markdown, either with a request header or by adding .md to a documentation URL. Without that hint, an agent might fetch and process HTML, which uses additional tokens. The team also surfaced its existing documentation question-answering system through a natural-language search endpoint. An agent can ask a question and receive relevant documentation chunks instead of crawling several pages.
A documentation search endpoint creates useful signals about agent problems
The search endpoint improves the immediate interaction because the agent can ask a natural-language question and receive material related to that question. It also gives Langfuse visibility into what agents are trying to learn. Ordinary documentation fetching makes it difficult to tell what Claude Code and the user's laptop were doing. Search requests can be tracked, so the team can see recurring problems and decide where more documentation is needed. Klingen describes this production signal as one of the most useful additions. The endpoint turns agent information requests into data about gaps in the product's guidance.
Small evaluations can catch regressions before a skill is changed further
The team struggled to define one evaluation approach because Langfuse users build chat applications, real-time voice systems, video-generation tools, and background invoice-processing systems. It started with five basic evaluations instead. These checked the filesystem and the diff before and after the skill ran, using an LLM judge to assess natural-language conditions. One check looked for OpenAI instrumentation. Another expected retrieval spans in a trace for a retrieval-augmented application, since seeing only LLM calls would indicate incomplete instrumentation. The evaluations let the team change the skill and check that an earlier behavior had not been broken.
Skills should point to changing documentation instead of copying it
Klingen describes the temptation to put more material directly into a skill because it feels like a local documentation cache. That makes information immediately available, but it creates another copy of the product's explanation. The copied content can become stale in the same way that pre-training context becomes stale. His team therefore prefers references that point the agent to the current documentation. This reduces duplication and gives the skill a better chance of remaining useful as Langfuse changes. The tradeoff is that the agent must fetch information when it needs it, rather than relying on a static block of embedded instructions.
Auto-research expands experimentation, but the target function can reward the wrong behavior
Langfuse used an auto-research loop to improve a skill for moving prompts from a local Git repository into Langfuse prompt management. The system proposed six improvements, and the team accepted three after human review. The experiment exposed a problem with the target function. The team measured speed by the number of turns, so an optimizer removed documentation-fetching notes because it already knew how prompt management worked. That made the run shorter while defeating the reason for using a skill: obtaining current context after the skill has been installed for a while. Missing goals had similar effects. The optimizer removed steps about linking prompt versions to production traces because those steps were outside the measured task.
Skill distribution and versioning are still practical obstacles
Once a skill is installed into an agent environment, its copied instructions can become old. Klingen considers adding a fetch timestamp and asking the agent to refresh a skill after it has passed a certain age. Distribution is harder because installation and upgrades depend on the coding agent, and users may need to perform the install themselves. He is not enthusiastic about plugin marketplaces because a small team would have to maintain proprietary integrations and coordinate updates with multiple agent vendors. His preferred direction is a well-known skill that an agent can discover, with a timestamp that lets it warn the user or fetch a newer version.
"Dynamic content should be referenced because there's a huge, I'd say, incentive for developers on the team, but also for users in the community to just contribute a lot of context to the skill."14:01
Who should watch
You maintain an SDK or developer platform whose APIs and documentation change faster than model pre-training can keep up.
You are building coding-agent skills and need ways to inspect agent behavior, test changes, and guide documentation retrieval.
You are experimenting with automated skill improvement and need to choose an objective that rewards correct long-term behavior rather than only fewer turns.
