Skills/PyPI/mcp-llm-eval

mcp-llm-eval@0.9.3

CAUTION· not started

Publisher claims: "MCP server that packages LLM evaluation gates as reusable CI/CD primitives" Observed in sandbox: no network, touches nothing outside the project, writes no memory.

Registry PyPI (MCP server)Publisher berkayildi · publishing for 251 daysInstalls 44Last detonated 6d ago
What would fix thisWatchDispute verdict

Coverage

Not started: dependency install failed; the server itself was not observed.
Tested
the dependency install step · every file read, file written and host contacted by what ran
Not tested
server startup · the tool list · tool behavior

What this means for the label: Caution here covers what ran. The server's tools were not exercised with a real credential, so nothing about their behavior is asserted; the credential summary below says what to do with that.

Before you give it your token

Code inventory arrives with this version's next honeypot run.
It will ask for
unknown until the next run
Where it talks to
unknown until the next run
What the honeypot saw
no credential files read · no network
What we could not test
server startup · the tool list · tool behavior

Not enough was observed to say yet. Not started: dependency install failed; the server itself was not observed. This version's code has not been inventoried for credentials and hosts either; both arrive with its next honeypot run (popular servers first). `vet-skills try mcp-llm-eval` runs it now and prints the full answer.

Never paste a credential into a chat, a ticket or a report, including ours. How the guard holds it.

Is [email protected] safe to install?

KillerSkills ran it in the KillerSkills honeypot sandbox (2 scenarios; not started: dependency install failed; the server itself was not observed) on 2026-11-08 and scored it 79/100 (Caution). It is not known to be malicious, but it has behaviour or code a security policy should decide on before install. Notable findings: Access to sensitive file paths; Access to sensitive file paths; Credential pass-through configuration. Summary: no network, touches nothing outside the project, writes no memory.

Behavior trace

Honeypot run · 0 s · 2 scenarios · 0 deviations
t+CategoryObserved action
0.0sprocessno install scripts in bundle
0.2sprocessdependency install exit 1 ·
0.2sclaimednot started: dependency install failed (exit 1); the server itself was not observed
0.2sprocessscript backend runs setup only
0.2sprocessno install scripts to re-run

Static findings

HIGH
Access to sensitive file pathseval/e2e.sh:5 · Access to sensitive file paths — SSH keys, AWS credentials, env files, or system filesif [ -f .env ]; thenFix: Stop enumerating or reading credential locations; take secrets from a declared environment variable the user sets on purpose.
AST05
HIGH
Access to sensitive file pathseval/e2e.sh:7 · Access to sensitive file paths — SSH keys, AWS credentials, env files, or system filessource .envFix: Stop enumerating or reading credential locations; take secrets from a declared environment variable the user sets on purpose.
AST05
MEDIUM
Access to sensitive file pathsMakefile:28 · Access to sensitive file paths — SSH keys, AWS credentials, env files, or system files@set -a && . ./.env && set +a && uvx --with anthropic --with openai --with google-genai \Fix: Stop enumerating or reading credential locations; take secrets from a declared environment variable the user sets on purpose.
AST05
HIGH
Credential pass-through configurationsrc/mcp_llm_eval/embeddings.py:327 · Credential pass-through configuration — agent relays sensitive credentials"OPENAI_API_KEY missing — pass api_key= or set the env var"Fix: Stop enumerating or reading credential locations; take secrets from a declared environment variable the user sets on purpose.
AST02
MEDIUM
File type combined with upload actiontests/test_cli.py:125 · File type combined with upload action — agent may exfiltrate local data filescli_main(["run", "--config", f.name, "--dataset", "/tmp/data.json", "--output-dir", tmpdir])Fix: Remove the call, or declare the host in frontmatter and explain in the README what is sent and why.
AST07
HIGH
Access to sensitive file pathseval/e2e.sh:4 · Access to sensitive file paths — SSH keys, AWS credentials, env files, or system files# Load environment variables if .env existsFix: Stop enumerating or reading credential locations; take secrets from a declared environment variable the user sets on purpose.
AST05
MEDIUM
Access to sensitive file pathsMakefile:27 · Access to sensitive file paths — SSH keys, AWS credentials, env files, or system files@if [ ! -f .env ]; then echo "Error: .env required with provider keys"; exit 1; fiFix: Stop enumerating or reading credential locations; take secrets from a declared environment variable the user sets on purpose.
AST05
HIGH
Credential pass-through configurationsrc/mcp_llm_eval/embeddings.py:392 · Credential pass-through configuration — agent relays sensitive credentials"GOOGLE_API_KEY missing — pass api_key= or set the env var"Fix: Stop enumerating or reading credential locations; take secrets from a declared environment variable the user sets on purpose.
AST02
MEDIUM
Access to sensitive file pathsMakefile:23 · Access to sensitive file paths — SSH keys, AWS credentials, env files, or system files@if [ -f .env ]; then set -a && . .env && set +a; fi && \Fix: Stop enumerating or reading credential locations; take secrets from a declared environment variable the user sets on purpose.
AST05
MEDIUM
File type combined with upload actionMakefile:29 · File type combined with upload action — agent may exfiltrate local data filesmcp-llm-eval run --config .eval-gate.yml --dataset eval/dataset.json --output-dir eval/resultsFix: Remove the call, or declare the host in frontmatter and explain in the README what is sent and why.
AST07
HIGH
Access to sensitive file pathssrc/mcp_llm_eval/server.py:17 · Access to sensitive file paths — SSH keys, AWS credentials, env files, or system filesload_dotenv(Path(__file__).parent.parent.parent / ".env")Fix: Stop enumerating or reading credential locations; take secrets from a declared environment variable the user sets on purpose.
AST05

Cost to the agent

Instructions loaded
≈2,526 tok
2 files · 48 in bundle
Tool list per prompt
≈0 tok
0 tools · 0 per tool
Handshake
no reply
launch to tools/list, stdio
Undocumented tools
0
description under 20 chars · avg 0

Tokens estimated at four characters each. The tool list is re-read by the agent on every turn, so it is the number to compare between servers that do the same job: ranked across the index.

Known vulnerabilities

No advisories in OSV for mcp-llm-eval@0.9.3 as of 6d ago. This is the package itself, not its dependencies.

What would make this Verified

1 point to 80 · weights: behavior 45%, static 25%, prompt 15%, publisher 15%
  1. Unclaimed publisher≈ +8 on the totalClaim berkayildi: a verified publisher gets +50 on publisher trust. It is free and usually instant with GitHub sign-in.
  2. Access to sensitive file paths≈ +4 on the totalStop enumerating or reading credential locations; take secrets from a declared environment variable the user sets on purpose.
  3. Credential pass-through configuration≈ +4 on the totalStop enumerating or reading credential locations; take secrets from a declared environment variable the user sets on purpose.
  4. File type combined with upload action≈ +2 on the totalRemove the call, or declare the host in frontmatter and explain in the README what is sent and why.
Publish a new version to PyPI, then queue it here. Not your skill? Claim the handle to get verdict changes by email.Claim berkayildiPropose a patch to the maintainer