They graded every key by what it could reach

Blog · November 17, 2026 · Karim Hijazi

Amazon published a report on November 13 that every team running AI agents should read. Attackers pointed AI agents at internet-exposed AI apps for about a month. The agents pulled out API keys, checked that they worked and pooled them. Then the stolen keys paid for the model calls that ran the next round of hunting.

One line in that report stuck with me. The operators told their agents to grade every stolen credential by whether it worked and what it could reach.

That is the right question. Attackers are asking it at scale. Most teams have not asked it about their own agents yet.

1. What Amazon saw

Amazon runs MadPot, a network of decoy sensors. Some of them posed as popular LLM APIs. From October 3 to November 5, 2026, agents running on ARTEX, an open-source agentic penetration testing framework, sent requests to those decoys. Agent frameworks send their history along with each request. So the requests carried the agents' own operating memory: instructions, tool history and findings.

The full write-up is on the AWS Security Blog. Cybernews has a short summary.

Targets named in those records included LLM gateways and proxies such as LiteLLM, One API and Veloera, and app builders such as FastGPT and Dify. The agents went after apps that hold keys for many providers. One weak admin page there can expose many accounts.

Every key got a grade. Amazon describes three levels. A lead. A confirmed working API call. Then quantified impact, such as readable data or a balance that could be spent. Keys that had been public for more than 30 days were treated as low value. Someone else had probably found them first.

Two caveats matter. Amazon says it could not verify the outcomes the agents reported, and not every target named in the records was breached. Also, none of the techniques were new. Default credentials. Keys left in code. Endpoints with no login. Unpatched software.

Amazon's advice is plain: “organizations should assume exposed secrets and misconfigurations will be found quickly and remediate accordingly.”

2. The keys were the target, not the model

Nothing in this campaign needed a clever trick against a model. The agents wanted credentials. A working key is money, compute and access in one string.

Look at what the operators valued. Not how a key was found. What it worked for, and what it reached. A key to a premium model with money behind it ranked above a stale one in a public repo.

That way of thinking carries straight over to the machines where your own agents run. A coding agent holds a model provider key. An MCP server holds a GitHub token or a database URL. A skill reads an env file because its script told it to. Those are the same kind of credential these agents were hunting.

3. Same class of risk, different surface

I want to be precise here. The exposed gateways and app builders in Amazon's report are a server-side problem. An admin interface on the internet. A default key. A missing patch. You fix those on the server. Put the gateway behind a login. Change the default master key. Watch for spend you did not expect. Amazon's post covers that well.

KillerSkills works on a different surface. We look at the agent and tool side, on developer machines, and it works in GitHub Actions. Which agents, MCP servers and skills are installed. Which credentials they sit next to. What a tool actually does when it runs.

We did not detect this campaign, and I am not going to claim we would have stopped it. The overlap is the question, not the surface. If a bad tool or an attacker got a foothold on a developer machine, what would the keys there reach?

Most teams I talk to do not have that list. Keys live in agent configs, MCP config files, env files and shell history. Some end up in agent transcripts and memory files because the agent read them during a task. A credential store helps. It is not isolation. A tool that runs as you can often read what you can read.

4. Grade your own keys first

Run the attackers' exercise on your own setup before anyone else does. Start with two free commands. No account.

npx vet-skills inventory

It lists the agents, MCP servers and skills on the machine, and it shows credentials sitting in their configs. That is your list of leads.

npx vet-skills prove

It checks transcripts and memory files for six CLI agents: Claude Code, Codex CLI, Gemini CLI, Qwen Code, Grok CLI and Copilot CLI. It names the file and the kind of credential. It does not print the value. It does not cover agents that log to a database, like Cursor. And a clean result is not proof that a key is off the machine. It tells you where keys did show up.

Then grade each one the way the attackers did. Does it work? What can it reach? Is it scoped to the job, or is it an org-wide token sitting in a config for a weekend project? Rotate what is stale. Scope down what is broad. Remove what nothing uses.

Their agents asked what every key could reach. Ask it about yours first.

Start at killerskills.ai.