A user asked what a clean trace of a server that never started was worth

Our crawler writes a short description stub for every registry package so the sandbox knows what it is looking at. The scorer treated that sentence, "MCP server for interacting with Supabase", as a claim, and when the server exited at startup for want of a token, it charged −10 for not doing what its description said. try uploads the raw package, which has no stub, so no claim and no deduction. Same engine, same bytes, four points apart.

We were pricing our own coverage gap as the package's fault. A claim can only be unmet by a run that got far enough to meet it; the deduction now applies only to agent-task runs, where a task either happens or does not. An MCP server's description is not a testable claim at all.

Both "outbound HTTP to a raw IP" findings pointed at dist/cli.js:2 and dist/cli.cjs:2: the ESM and CommonJS builds of the same source line, which assembles http://127.0.0.1:{port}/mcp, the URL of the server's own --http mode listener, bound to loopback eight lines earlier. A loopback listener is not outbound anything.

Raw addresses are now classified before they become findings: loopback and 0.0.0.0 are a local listener and no finding; the cloud metadata address 169.254.169.254 is critical; private ranges are low; public addresses keep the original meaning. And the evidence shown for a finding is now the text around the match, not the first two hundred characters of a minified line, so you can see what tripped it.

Reproducing his run showed the sandbox had been far blinder on MCP servers than our pages implied, in four ways:

• Dependency installs were refused. Our own egress proxy blocked the registries, so every npm and PyPI server with dependencies died on a missing module in under a second and was recorded as "no tools/list reply". The production run for this package took 958 milliseconds, total.
• stdin closed too early. We sent the three handshake messages and closed the pipe. A stdio server treats end-of-input as the client leaving, so many quit before processing tools/list.
• Node's fetch ignores the proxy. Servers built on it reached the internet straight past our recording proxy, and the hosts they contacted were never written down.
• A server that asks for a credential stopped there. We recorded the exit and moved on.

All four are fixed in engine 1.8: registries pass through for the install step (the traffic is still recorded, and a decoy value in it still scores zero), stdin stays open for the window, the run sets the variable that makes Node honor the proxy and reads direct connections from the syscall trace besides, and a server that exits asking for a named credential is launched again with a tagged decoy in that variable. Most servers validate lazily, so the tool list comes out, and a decoy that then travels anywhere is attributable. No real credential is ever involved.

The honest consequence: our "with a honeypot trace" figure was true but thinner than it sounded for MCP servers. Every server scored before 1.8 is being re-run, most installed first, and we are not quoting MCP coverage numbers until that has run for a while.

The structural fix is a field, not a rule. A run now records its coverage: full (the server answered tools/list, or an agent task ran), startup only (it exited before answering, usually asking for a credential), not started (its dependencies failed to install), install only, or static only (nothing has run yet). The label is never shown without it when the run was partial:

VERIFIED  85/100 · startup only   [email protected] · supabase · 588,685 installs
 
Coverage  Started with a decoy SUPABASE_ACCESS_TOKEN and answered initialize; the tool list did not come back. Authenticated tool behavior not tested.
  tested      server startup with a tagged decoy in SUPABASE_ACCESS_TOKEN · MCP initialize · every file read, file written and host contacted by what ran
  not tested  the tool list · what the tools do when called with a real credential
  needs       SUPABASE_ACCESS_TOKEN — give it to the guard with --secrets, never to a chat or a report

The same field is on the verdict page, in the API as trace_summary.coverage, and in policy: a team that wants Verified to mean full coverage sets partial_coverage: caution (or approval, or block) and a startup-only Verified is handled accordingly. The report ends with a recommendation that follows from the evidence rather than from the label, and it never asks anyone to paste a token.

Before anyone posted about the fix, an independent tester ran the shipped release and the live site against the claims above. The first pass confirmed the engine fixes on a fresh run of the same package (Verified, dependencies installed, decoy relaunch, 36 tools listed, the loopback findings gone) and found four more problems: the ordinary check could time out and fail open, because the verdict API was loading a package's entire history to answer for one version; the report link try printed was a 404; publisher age still showed how long we had known the publisher; and the trace labelled opaque tunnels as HTTPS. A second pass showed two of our fixes had not gone far enough (the report link still 404ed for a reason the first fix did not reach; publisher age had moved from 16 days to 79 when npm's history says 589). Everything is fixed; the second pass is what caught the last two.

We are saying all of this because the alternative is a changelog line. A verdict system that cannot show you its own mistakes is asking you to trust the label, which is the one thing this whole product exists to replace.

The developer's workflow, generalised, is now the front door. You are about to connect an agent to GitHub, a database, a deployment tool; the server will want a credential. killerskills.ai/connect takes the package name and answers, in this order: what it will ask for (environment variables and flags, read from the code), where it talks to (its own vendor hosts, with any third party called out), what the honeypot saw it do with a decoy credential, and what could not be tested. Then the one command that keeps the credential out of the server:

vet-skills run --secrets SUPABASE_ACCESS_TOKEN=supabase.com -- npx -y @supabase/mcp-server-supabase

The server gets a placeholder; the guard holds the value and puts it only into requests to that host. A placeholder sent anywhere else is a random string, and the guard logs the attempt. Free, no account, nothing of yours uploaded. Bring the MCP you want to use. Not your credentials.

The package that started this is at its verdict page: Verified, full coverage, 36 tools, reached only its own host, no high findings. It is a normal, well-built server that needs a token. Our verdict on it moved up because the errors were ours. Thanks to the developer who asked, and to the tester who kept asking. If you can get a decoy credential out of the trap and come out Verified, we publish the technique with your name and fix it.