Skills improve agent performance, but without evals it is hard to tell whether a failure comes from the skill or from the model struggling with the task.
2
Model-invoked skills need short, directive descriptions with clear positive and negative cases, while fixed workflows should usually become scripts.
3
A lightweight harness using JSON test cases, isolated runs, regex checks, and occasional LLM judges can catch regressions before a skill change is merged.
Summary
Philipp Schmid explains why agent skills need tests before they reach users. A skill usually contains a short description, a fuller instruction file, and deeper reference files loaded through progressive disclosure. Schmid distinguishes temporary capability skills from durable preference skills, then covers how to make model-triggered skills work reliably. Descriptions should state why, when, and how to use a skill. They should be concise and directive, with negative cases that prevent over-triggering. Fixed workflows should become scripts instead of rigid skill instructions. His testing approach starts with small sets of positive and negative prompts, adds production traces, runs isolated trials several times, and tests across different agent harnesses. A practical Gemini Interactions API example uses 117 cases, a Python runner, and inexpensive regex assertions. Schmid also recommends ablation tests with and without the skill, so teams can retire skills when models no longer need them.
Skills fail in production because teams rarely test them
Schmid opens with a gap between widespread skill use and scarce evaluation. He cites SkillsBench, which indexed more than 50,000 skills and found that almost none had evals. Many were AI-generated and had not been tested carefully. Since agents are nondeterministic, a failed task does not immediately reveal whether the skill is poor or the task is too difficult for the model. That uncertainty makes manual, occasional checks weak evidence. Testing gives teams a way to separate skill failures from model limitations and to check whether a change actually improves behavior.
Customer-facing agents cannot depend on users knowing how to invoke skills
Schmid distinguishes coding agents that engineers use from agents built into applications for customers. An engineer notices quickly when a skill fails to trigger, then can reprompt the agent or use a slash command. A customer does not know the available skill names and will not write a request such as "use refund skill." Customer-facing systems therefore depend much more on model-triggered skills. Their descriptions must help the model infer when a skill applies from ordinary user language.
Skills use progressive disclosure to control context
A skill is a folder containing a skills.md file and supporting assets. Its first layer is a title and description, usually included in the model's context so the model can decide whether to use it. The second layer contains fuller instructions and references. The third layer contains detailed files that the model can inspect for a specific task. Schmid recommends keeping descriptions and main skill files concise because the description is paid for on every model call, while deeper references can hold details such as separate AWS, Google Cloud, and Azure deployment instructions.
Capability skills may expire while preference skills usually persist
Capability skills teach a model to perform something it cannot yet do consistently, such as tracing logs or creating a React app. They are temporary because model improvements may remove the need for them. Preference skills encode company-specific workflows, language styles, or other domain conventions that foundation models may not know. Those skills tend to last longer and need protection against regressions. Evals can show when a capability skill is safe to retire and whether a preference skill has begun to degrade an agent's performance.
Descriptions should give direct triggers and include cases where the skill must not run
For model-invoked skills, the description tells the model why and how to use the skill, along with when it applies. Schmid prefers a directive such as "use the Interactions API if you're working on a chat application" over a passive explanation. Descriptions should stay short because they appear in every model context. Teams also need negative cases. A broad instruction for web development could trigger on Angular work when it was intended only for React or Tailwind CSS. Specific exclusions help prevent that behavior.
A skill should describe goals and constraints, while fixed procedures belong in scripts
Schmid criticizes skills that prescribe every step of a stable workflow. If a process always follows the same sequence, a script can run it without spending model tokens on each step. Skills should instead define the goal and the constraints. For a configuration change, the instruction can identify the file and ask the model to make the change, rather than dictating that it read the config, update the port, and deploy again. This leaves the model room to choose an appropriate path.
Small positive and negative test sets expose trigger failures early
Schmid recommends testing a new skill with roughly 10 to 20 prompts. His initial split is five happy-path prompts for cases where the skill should run and five prompts where it should not run. Production traces should be added when available because they reflect real requests. He also advises removing no-ops, meaning instructions that do not change agent behavior, such as generic reminders to write clear code. Removing them can reduce token cost even when eval scores stay the same.
A lightweight harness can test generated code with cheap assertions
For a Gemini Interactions API skill, Schmid's team created 117 test cases from synthetic examples, real-user patterns, and observed feedback. Each case included a prompt, a language such as TypeScript or Python, a should-trigger value, and expected checks. A Python script ran the Gemini CLI and inspected the result. Regex assertions checked whether the output used the correct SDK, model, and methods, and whether it avoided old patterns. The team reported improving valid Interactions API code generation to almost 90% with this setup.
Regression evals should run in isolated environments and include ablation tests
At Google DeepMind, skills have evals alongside them. Cases run in clean workspaces with optional files, startup commands, script checks, and LLM judges for more complex expectations. Evals run whenever a skill changes, and a change cannot be merged unless it improves existing cases or adds new ones. Schmid recommends several trials per case because model behavior varies, plus testing across harnesses such as Gemini, Claude, Cursor, and Codex. Running with and without the skill shows whether the model still needs it and preserves an eval for future regressions.
"You wouldn't merge code without tests, so why are we shipping skills without evals?"00:22
Who should watch
You maintain skills for a coding agent and currently judge changes by trying a few prompts by hand.
Your team is building a customer-facing agent where users will not know skill names or slash commands.
You need a small testing setup for generated code, trigger behavior, or deciding whether an older skill can be removed.
