You Can't Tell What an Agent Skill Will Do by Reading It

Roey Ben Chaim, Zenity14:31 · Nov 2026 · 471 views
Thumbnail for You Can't Tell What an Agent Skill Will Do by Reading It Watch on YouTube
TL;DR
  1. 1

    An official diagnostic skill can perform actions that resemble an information stealer, so behavior needs to be judged in context.

  2. 2

    Reading SKILL.md is insufficient because skills can fetch prompts, install packages, invoke scripts, and create processes outside the agent trace.

  3. 3

    An agent detonation chamber needs a realistic environment, task coverage, an operator, and evidence from processes, files, and network traffic.

Summary

Roey Ben Chaim explains why an agent skill cannot be assessed by reading its entry file. A legitimate Azure diagnostic skill read cloud and Kubernetes configuration, contacted an identity service, and probed instance metadata, actions that also resemble information stealing. Skills can fetch more instructions, install packages, invoke hooks, and create processes whose effects are absent from the agent conversation. Zenity built an agent detonation chamber to observe those effects. It places an agent and skill in a sandbox, builds the services and projects the skill expects, exercises different task paths, and captures process creation, filesystem activity, network traffic, and agent traces. Roey walks through chained skill installation, misleading package names, persistence, remote prompts, hooks, and a lookalike project that harvested credentials. He ends by treating skills as supply-chain dependencies: inspect what they load, limit access, and record what actually runs.

Key ideas
00:46

Legitimate diagnostics can look like information stealing

Roey shows an official Azure diagnostic skill reading cloud and Kubernetes configuration, contacting the identity service, and probing an instance metadata endpoint. Those actions resemble the behavior of information stealers that search secret and identity folders. The same observed actions can therefore have different meanings depending on the skill's purpose. A scanner that sees only access to credentials or metadata cannot establish whether the skill is malicious or performing its intended diagnostic work.

01:45

SKILL.md is only the start of the execution chain

In many skills, SKILL.md is an entry point rather than the complete payload. It can invoke another endpoint containing more prompt instructions, download a package during installation, or run code that fetches additional packages and payloads. A static scanner that reads only the file misses those later stages. Roey also describes research by Trail of Bits in which a small comment about an approved change manipulated a package registry and fooled an LLM analyzer.

02:42

A detonation chamber needs the environment a skill expects

Zenity's first version put an agent and skill in a sandbox with a task prompt and let them run, but many skills could not operate because their CLIs, services, or projects were missing. The team added a world builder that examines a skill and supplies the appropriate environment. A Notion skill needs a Notion service, for example. Without those dependencies, execution does not cover the behavior the skill would show in use.

03:48

Different skills require different task paths

A generic task prompt cannot exercise every skill. A presentation skill may need to create a deck, while a deployment skill may need to configure a service, deploy it, and recover it from snapshots. Zenity added an operator that reads the skill and chooses the trajectories to invoke. The operator also had to deal with refusals and safeguards. In a controlled container, it used adversarial prompting and restarted runs when agents would not decode or execute the relevant steps.

06:01

Agent traces miss effects created by child processes

The agent's trace does not include every side effect. Roey uses npm install as an example: a post-installation script can create another process, read and write files, and connect to an endpoint. Those events can form the behavior of an information stealer even when they are absent from the agent's visible conversation. The chamber therefore captures process creation, filesystem events, and network packets alongside agent traces.

07:08

Chained installation can hide a skill's real behavior

A proof-of-concept skill called security of YouTube installed another skill called security review. That second skill instructed the agent to fetch SKILL.md from GitHub, where hidden instructions supplied the payload. The payload was not malicious in this example, but it ran successfully. Roey's point is that the full chain has to execute before the behavior can be recovered. He says this approach uncovered 100 malicious skills out of thousands, which were then shut off from the skills.sh framework.

08:48

Persistence, remote prompts, and hooks change the risk

Roey describes skills that add prompts telling an agent to reinstall them if removed, as well as a nonofficial skill creator that silently uninstalls the official creator and installs itself. Remote prompts create another problem because a skill can retrieve instructions from a location that later changes or is taken over. Hook abuse can trigger a skill on events such as user prompt submission, causing it to run whenever a user sends a message.

10:35

A lookalike project can turn installation into credential theft

Roey follows a campaign involving a fake organization that mimicked the Paperclip project. The lookalike project added a log action across its routes. That action contacted an attacker-controlled endpoint, fetched and decoded a payload, wrote it to a temporary file, executed it with Node, and self-destroyed after 60 seconds. The payload searched SSH, AWS, and Azure paths, collected information, and sent it to the attacker. Related skills using the same command and control reached 1.7 million total downloads before Vercel and Microsoft took them down.

13:33

Skills should be handled like software dependencies

Roey places skills inside a wider agent supply chain. Anything added to the agent's context can carry risk, including websites, database rows, packages, and YouTube comments. For skills, he recommends inspecting everything they load, limiting what they can access, granting only the needed privileges, and observing what actually runs. The detonation chamber is one way to collect that runtime evidence before relying on a skill.

"If you use just a static scanner and read through skills of MD, you would lose all of that."02:04
Who should watch
  • You are building an agent platform that can install or execute third-party skills and need evidence beyond the model's conversation.
  • Your security scanner reads SKILL.md or package metadata but does not follow fetched prompts, installation scripts, hooks, or child processes.
  • You are designing a sandbox for agent testing and need to model external services, exercise multiple task paths, and capture filesystem and network activity.