evalsmith-mcp@0.2.0
VERIFIED· install onlyPublisher claims: "Enables MCP clients to score retrieval quality with recall, precision, nDCG, hit rate and MRR, compare 2-3 configurations with seeded paired-bootstrap confidence intervals, audit agent tool calls against expected traces, and detect prompt regressions against golden sets. It also generates Markdown, JSON or JUnit reports with pass/fail gates, all deterministically and offline without paid API keys." Observed in sandbox: no network, touches nothing outside the project, writes no memory.
Coverage
Nothing executable in the bundle ran; the trace covers the install step only.- Tested
- every file read, file written and host contacted by what ran
- Not tested
- use of the skill by an agent
What this means for the label: Verified here covers what ran. The server's tools were not exercised with a real credential, so nothing about their behavior is asserted; the credential summary below says what to do with that.
Before you give it your token
- It will ask for
EVALSMITH_JUDGE_API_KEYConfiguration, not credentials: EVALSMITH_JUDGE_BASE_URL, EVALSMITH_JUDGE_MODEL, EVALSMITH_TRANSPORT, GITHUB_OUTPUT, GITHUB_STEP_SUMMARY- Where it talks to
- no vendor host named in the code
- What the honeypot saw
- no credential files read · no network
- What we could not test
- use of the skill by an agent
Connect it with a scoped credential, under the guard. It asks for EVALSMITH_JUDGE_API_KEY and talks to its vendor. Give the credential to the guard, not the server: the value never enters the server's environment and only travels to the host you bind. What the tools do with a real credential was not observed.
Never paste a credential into a chat, a ticket or a report, including ours. How the guard holds it.
Is [email protected] safe to install?
KillerSkills ran it in the KillerSkills honeypot sandbox (1 scenario; nothing executable in the bundle ran; the trace covers the install step only) on 2026-11-15 and scored it 83/100 (Verified). In what ran, nothing we could detect went wrong: no undeclared network egress, no credential reads, no persistent instruction writes. What it does with a real credential was not observed; the static findings and the publisher carry the rest of the score. Summary: no network, touches nothing outside the project, writes no memory.
Behavior trace
Honeypot run · 0 s · 1 scenarios · 0 deviationsStatic findings
- run: npx playwright install --with-deps chromiumFix: Pin versions and sources; vendor anything fetched at install time.// Treat empty strings from .env templates as unset.Fix: Stop enumerating or reading credential locations; take secrets from a declared environment variable the user sets on purpose.- run: npx playwright install --with-deps chromiumFix: Pin versions and sources; vendor anything fetched at install time."inspect": "npx -y @modelcontextprotocol/inspector node dist/mcp/server.js",Fix: Pin versions and sources; vendor anything fetched at install time.Cost to the agent
Tokens estimated at four characters each. Skills load their instruction text when triggered; the figure is what that costs.
Known vulnerabilities
Vulnerability advisories (OSV) cover npm and PyPI packages; this registry has no advisory database.