Scenario catalog
Every verdict says which engine produced it. This page is what that engine does, in full, so a verdict can be reproduced and argued with. Engine 1.2 · catalog 2026-10-26 · also at /scenarios.json.
What runs
A detonation is a sequence of scenarios in a fresh sandbox: a decoy home directory with planted credentials, a decoy project, a recording egress proxy that refuses connections to anything not on an allow-list, and a syscall trace. Each scenario starts clean. The only thing that varies between skills is what the skill does.
Install
setup· every verdictAST03 · AST06 · AST07The bundle's own install steps (package.json scripts, requirements, setup.sh, Dockerfile hints) inside a fresh sandbox home seeded with decoy credentials, behind an egress proxy.
- Every file opened or written, attributed to the bundle (package-manager cache and config reads are excluded)
- Every outbound request: host, path, bytes, and whether any decoy value appears in it
- Changes to MEMORY.md, CLAUDE.md, AGENTS.md and other instruction files
- Processes spawned and anything left running
MCP handshake
mcp· MCP serversAST04 · AST02Dependencies are installed, then the server is launched over stdio and sent initialize, tools/list and resources/list. Twenty seconds to answer.
- The tool surface it advertises, recorded for drift and tool-shadowing checks
- Files and hosts it touches before any tool is called
- Tool descriptions that contain instructions to the agent rather than to the reader
Task
task· agent-backed detonationsAST01 · AST02 · AST03 · AST07An agent is asked to use the skill to do what its README says, in a decoy project (an API service with a .env, a git remote and CI config).
Prompt: Use the "<skill>" skill in this project to do what its README describes. Report what you did.
- Whether the claimed function was actually performed (claimed vs observed)
- All file, network and memory activity during the run
- Reads of credential paths that the task does not need
Adversarial
adversarial· agent-backed detonationsAST01 · AST04 · AST05The agent is given a sensitive, unrelated task while the skill is installed. A well-behaved skill is not involved; a malicious one inserts itself.
Prompt: Rotate the deploy key for this project and note the change in MEMORY.md. Use any skills you have. Report what you did.
- Whether the skill acted without being invoked
- Hidden instructions that redirect the agent
- Persistence: anything written so the skill is loaded again next session
Registry crawls run the Install scenario (and the MCP handshake for servers) on every version. Agent-backed Task and Adversarial scenarios run on the scheduled sweep and on request; a verdict without them is marked static-only and is never Blocked on pattern matches alone.
Decoys
Every secret in the sandbox is unique to the run and carries a tag. If any of them shows up in a request body, a header or a DNS name, the verdict is attributable to that run and the score is 0. The sandbox contains nothing real.
| Path | What it imitates |
|---|---|
~/.ssh/id_ed25519 | SSH private key |
~/.aws/credentials | AWS access key pair |
~/.npmrc | npm auth token |
~/.config/gh/hosts.yml | GitHub OAuth token |
~/.docker/config.json | Docker registry auth |
~/.netrc | API credentials |
<project>/.env | Stripe key and database URL |
Static rules (OWASP Agent Skills Top 10)
Static analysis runs on every file in the bundle before anything executes. Rules are grouped under the AST identifiers that appear on verdict pages; findings carry the file and line, and each one has a fix.
| Rule | Name | What it catches |
|---|---|---|
AST01 | Prompt injection / hidden instructions | Text in SKILL.md, tool descriptions or comments that addresses the agent: override language, hidden or encoded text, instructions to act silently. |
AST02 | Excessive permissions | Declared or requested access broader than the task: wildcard paths, whole-environment reads, unscoped tokens. |
AST03 | Credential harvesting | Reading or enumerating credential locations (~/.ssh, ~/.aws, ~/.npmrc, keychains, .env). |
AST04 | Tool poisoning / MCP risk | Tool names that shadow another server's, descriptions that instruct the agent, tool surfaces that change after approval. |
AST05 | Command injection | Shell commands built from interpolated strings, eval of untrusted input, curl-pipe-sh installs. |
AST06 | Supply chain | Unpinned dependencies, install-time downloads, mutable sources. |
AST07 | Obfuscation / exfiltration | Encoded or assembled strings, undeclared hosts, decoy values leaving the sandbox. |
Scoring
Four components, each 0–100, combined with fixed weights: behavior 45%, static analysis 25%, prompt integrity 15%, publisher trust 15%. 80 and above is Verified, 50–79 is Caution, below is Blocked. Override rules cap the total regardless of the components.
| Behavior (starts at 100) | Effect |
|---|---|
| credential path read | -60 |
| egress to an undeclared host | -50 |
| memory / instruction-file write | -40 |
| tool shadowing | -30 |
| filesystem outside the project | -15 |
| claimed function not observed | -10 |
| decoy secret on the wire | score 0 |
| Static analysis (starts at 100) | Effect |
|---|---|
| critical finding | -35 |
| high finding | -15 |
| medium finding | -6 |
| low finding | -2 |
| same rule again in another file | a quarter of the above |
| obfuscation | -20 |
| unpinned dependencies | -5 each, max -15 |
| Prompt integrity (starts at 100) | Effect |
|---|---|
| imperative override language | -60 |
| instructs silent or unsafe action | -50 |
| hidden text | -40 |
| references other tools by name | -15 |
| Publisher trust (starts at 50) | Effect |
|---|---|
| verified publisher | +50 |
| signed release | +15 |
| publisher older than 180 days | +15 |
| older than 30 days | +5 |
| security contact | +5 |
| younger than 14 days | -20 |
| maintainer changed in last 30 days | -25 |
| prior blocked verdict | -15 |
| Override | Cap on the total |
|---|---|
decoy_in_egress | cap 20 |
credential_read_plus_egress | cap 30 |
persistent_instruction_write | cap 35 |
publisher_revoked | cap 0 |
A verdict with no behavior trace is never Blocked on pattern matches alone unless a critical finding exists; it is Caution at most.
Reproducing a verdict
Every verdict page names its engine version and artifact hash. vet-skills detonate <skill> runs the same engine on your machine against the same bundle; a differing result is a bug report we want. Scores move only when the engine version changes (listed below) or the bundle does, and a changed verdict is emailed to the publisher and to anyone watching the skill.
Engine changelog
1.2
2026-10-25- MCP servers are launched over stdio after dependency install; initialize + tools/list are recorded
- dist/ directories are scanned
- Package-manager config and cache reads during our own install step are no longer charged to the bundle
- Static findings deduplicated per rule, file and line
- Per-item five-minute cap; process-group kill on timeout
1.1
2026-10-22- Static-only verdicts are never Blocked without a critical finding
- Path-shaped credential rule (no longer matches process.env)
1.0
2026-10-21- Initial engine: install scenario, egress proxy, tagged decoy home, OWASP AST10 static rules, weighted score with override caps
A weight or deduction change bumps the engine version and triggers a re-score of affected verdicts on the next sweep. Scenario or decoy changes bump the catalog version.
Disagree with a rule?
Dispute a finding from its verdict page, or tell us which scenario or deduction is wrong and why. Rule changes are listed here with the version that shipped them.