Skills should be evaluated by comparing model answers with and without the skill loaded.
2
A line-by-line novelty test can show which parts of a skill add knowledge, have no effect, or reduce performance.
3
Models often understand a skill's general ideas while missing its exact details, so removing familiar material should still be validated with task evaluations.
Summary
Nikolay Petrov presents a way to measure what a skill adds to an agent. He separates skills that teach new knowledge from skills that mainly steer a process, then focuses on the first type. The method breaks a skill into checkable claims, asks questions about each claim with and without the skill, and uses a separate judge or panel to assess the answers. The results form a four-part matrix: knowledge the model already had, knowledge the skill adds, cases where the skill hurts performance, and cases neither version can answer. Petrov says this first test often finds that models already know much of a skill, while exact parameters, values, and steps are less well understood. Novelty measurements can guide what to remove, but task evaluations still decide whether a change works. He describes this functionality in the docume framework, which also evaluates other agent context such as agents.md files and MCP tools.
Petrov separates skills into two broad types. Some add knowledge or capabilities that the model did not have. Others steer the model through a repeatable sequence of actions without adding information. Both can be useful, but this talk focuses on knowledge skills. The starting point is often informal: people repeat a successful prompt, package it into a skill, and check whether it seems to work. Petrov argues that this feeling can be replaced with a more deterministic comparison. A skill should be treated as a piece of context whose effect can be measured against the same task without that context.
The basic evaluation compares answers with and without the skill
Petrov describes two evaluation arms. In one, the agent receives the task with no skill loaded. In the other, it receives the same task plus the skill. The evaluator then compares the results, including how many tasks were solved, token use, elapsed time, and other factors. His example has five tasks: the model solves three without the skill and all five with it. This comparison turns a skill from a promising document into an instruction that has demonstrated effects. It also gives a basis for judging later edits instead of relying on whether a new model version appears to handle the skill better.
Breaking a skill into claims makes its contribution visible
The proposed method splits the skill into individual items, with roughly 25 or 30 items per skill in the examples. Each item gets a question and an expected answer. An item can express a general proposition or a more definite contract about an action. The questions are asked with and without the skill, then a judge or panel evaluates the answers. Petrov says the judge should ideally come from a different model family. Questions should avoid leaking their answers. Depending on the use case, the test can measure the model's internal knowledge alone or include tools such as web access.
The four outcomes distinguish useful context from harmful context
The item-level results form a simple matrix. If the model answers correctly both with and without the skill, that part was already known and the skill had no effect. If the model answers without the skill but performs worse with it, the skill degrades performance. The desired result is a question the model cannot answer without the skill but can answer with it. A fourth case occurs when neither version succeeds, which can mean that the item is relevant but the skill is not written clearly enough. Petrov says harmful instructions are the easiest problems to fix because they can be removed without reducing the model's existing ability.
General understanding can hide missing exact details
Petrov compares lenient and strict judging. Lenient judging checks whether the model understands the general idea. Strict judging checks exact keys, parameters, values, or steps. In one result, Opus shows 70 percent understanding under the general test but only 22 percent under the specific-detail test. The gap reflects a common pattern in newer models: they can generalize from concepts while still needing the exact operational information that a skill provides. A skill may therefore contain many familiar explanations and only a smaller set of details that materially changes the model's answer.
Novelty measurements guide editing, but task evaluations decide
Once familiar items have been identified, Petrov suggests removing some or all of them and rerunning the task evaluation. If the scores stay the same, the skill can become smaller and use fewer tokens and less context. The novelty map provides guidance about what to try removing, rather than forcing an editor to guess line by line. It is still only an initial direction. Every change must be tested against the actual tasks the skill is meant to solve, because an item that looks familiar in isolation may still affect the complete workflow.
The docume framework applies the same approach beyond skills
Petrov says the measurements were made with docume, a framework for comparing agent context. Its scope includes skills, agents.md files, and MCP tools. Novelty is one part of the framework, alongside evaluations of context impact, routing, descriptions, and naming. He recommends running the measurements across a broader set of models because the amount of prior knowledge differs by model family and skill. The result is a way to make deliberate decisions about agent context, while retaining end-to-end evaluations for the final check.
"What we see across the models is the difference is large especially with their larger frontier models where we see Opus dropping from 70% understanding of their skill in case of the general understanding to just 22% of the specific details."09:32
Who should watch
You maintain skills that keep growing because removing a line feels risky, and you need evidence about which parts still matter.
You want to compare a model's existing knowledge with what a skill actually teaches, including exact parameters, values, or steps.
You are building agent evaluations and need a way to test skills, agents.md files, or MCP tools across different models.