Claude Code Plugin Eval: Test a Plugin Against a Baseline

September 12, 2026 claude-codeplugins

claude plugin eval runs your plugin against a set of test prompts, scores what Claude produced, and then runs the same prompts again with no plugin loaded so you can see what the plugin actually changed. It shipped in Claude Code 2.1.269 on September 12, 2026, and it answers the question claude plugin validate cannot: not "does this plugin load" but "does this plugin help".

This article is what we learned running it the day it landed, on a one-skill plugin, twice. The first run scored zero with and without the plugin even though the skill fired. The second, after a rewrite of the skill body, scored 1.00 with and 0.00 without. Both runs together cost $0.42 in list-price model calls. Everything below was checked against the plugin-evals page at code.claude.com and the --help output on 2.1.269.

What one run does

For each case, Claude Code starts a fresh, isolated, non-interactive session with only your plugin loaded, sends the case's prompt, and lets Claude work until it finishes or hits the case's turn or time limit (10 turns and 300 seconds by default). Then each grader checks the final reply, the transcript, or a file Claude created, and passes or fails.

Three things about that session matter when you write cases:

• Nothing of yours loads. No user settings, CLAUDE.md, MCP servers, other plugins, or memory. Each run gets a throwaway home and an empty working directory, so put what the task needs in the prompt or seed the workspace.
• Read-only tools only, unless you grant more. Bash, Write, Edit, WebFetch, and WebSearch are removed unless you pass --allow-tools. Runs never stop to ask permission.
• Two arms by default. Every case runs with the plugin and again without it, and the summary prints both scores and their difference, Δ. A case that scores 1.00 in both arms passed because Claude is good at the task, not because your plugin did anything.

Each case runs three times per arm by default, so one case is six sessions before you touch any flags.

The smallest suite that runs

We started from the plugin in our plugin tutorial: a release-notes skill that turns commits into release notes. From the plugin root:

claude plugin eval init --bare draft-notes

That wrote two placeholder files and ran nothing:

evals/draft-notes/
├── prompt.md
└── graders/
    └── criteria.md

The full init (without --bare) opens an interactive session in which Claude reads your plugin, proposes prompts that should and should not trigger it, drafts graders, pilots them once, and writes one case per prompt. That path costs model usage. We wrote the case by hand to see what the files hold.

prompt.md is the message Claude receives, with run limits in frontmatter. Ours pastes the commits in, because the run's workspace is empty and there is no git history to read:

---
max_turns: 6
allowed_tools: [Skill]
---

Write release notes for these commits since v1.2.0:

- feat: add CSV export to the reports page
- fix: dashboard chart no longer clips on narrow screens
- chore: bump eslint to 9.3
- feat: keyboard shortcut (Ctrl+K) for search
- fix: login redirect loop on expired sessions

Then three graders, one file each under graders/. A judged rubric on the result:

---
type: llm
---

PASS if the reply groups the changes under Added and Fixed headings, describes each change as something a user can do or no longer suffers, and does NOT mention eslint or any dependency bump.
FAIL if the eslint bump appears, if the entries still carry commit prefixes like "feat:" or "fix:", or if there are no Added/Fixed groupings.

A free regex check for the one line the skill is supposed to always append:

---
type: regex
pattern: 'Full changelog: v1\.2\.0\.\.\.HEAD'
---

And a check that the skill is what produced the answer, using the pattern the docs give (the (?:[\w-]+:)? part accepts the namespaced plugin-name:skill-name form):

---
type: tool_used
tool: Skill
input_match: '"skill"\s*:\s*"(?:[\w-]+:)?release-notes"'
---

What the first run showed

We ran it once per arm to keep the pilot cheap:

claude plugin eval . --trust-plugin --runs 1 --no-publish --max-cost-usd 2

--trust-plugin is there because we launched it from a script rather than a terminal. The first run against a directory otherwise stops at a Trust this plugin directory? prompt, and when there is no terminal to ask, the run is refused with exit 1 instead.

Here is the summary, verbatim:

CASE         WITH  W/OUT Δ      RUNS COST    NOTES
draft-notes  0.00  0.00  0.00   2    $0.24   changelog-link: pattern not found in last_message

1 case(s) · mean Δ 0.00 · 27s · $0.24

The per-run lines above the table are where the explanation lives:

draft-notes run 1/1 [with]: score 0.00  $0.15
  ✗ changelog-link (weight 1): pattern not found in last_message
  ✗ criteria (weight 1): judge votes: FAIL FAIL FAIL
  ✓ skill-fired [with-only, not scored]: Skill called 1x (expected 1..∞)

The skill fired. Claude invoked it once, exactly as the tutorial version of the plugin intended. It still scored zero, and the judge's evidence in the JSON result said why: the reply listed "Bumped ESLint to 9.3" under a ### Changed heading and never wrote the changelog line. The skill body at that point was one sentence ("group commits into Added, Fixed, Changed"), and Claude did precisely that. Nothing in the skill said to drop chores or to end with a link, so the model did what a model does without instructions, and the with-arm looked identical to the without-arm.

This is the failure the tool exists to catch. claude plugin validate passed the plugin, the skill triggered on natural phrasing, and a person testing by hand would have read a tidy draft and moved on. Only the baseline comparison shows the plugin contributed nothing the graders could detect.

Note the [with-only, not scored] tag on the skill-fired grader. A tool_used grader on Skill can never pass without the plugin, so counting it would inflate Δ. In a two-arm run Claude Code excludes it from the score and reports it as a "plugin-fired indicator"; under --ablation none nothing is excluded, so the same suite can print a different absolute score in the two modes.

What the second run showed

We rewrote the skill body to say what the graders check: three headings in a fixed order, drop chore: and ci: commits, plain sentences without prefixes, and a closing Full changelog: <tag>...HEAD line. Same command, same case:

CASE         WITH  W/OUT Δ      RUNS COST    NOTES
draft-notes  1.00  0.00  +1.00  2    $0.18

1 case(s) · mean Δ +1.00 · 22s · $0.18

With the plugin, both scored graders passed (the regex matched, the judge voted PASS three times). Without it, both failed, as before. Δ of +1.00 means every point of the score came from the plugin. Exit code went from 1 to 0, because the default --threshold is 1.0 and the case now meets it.

Two caveats on our own numbers. One run per arm is noisy; the docs say to confirm any change at the default three runs before trusting it, and we agree. And a Δ this clean is partly an artifact of graders that check for things only the skill would produce. That is fine for a regression suite (it tells you the moment the skill stops doing its job), but a suite meant to show the plugin improves quality needs rubrics a plugin-less Claude could plausibly pass.

The six grader types

Four are computed from the transcript and files and cost nothing. Two call a judge model and add to the run's cost.

regex graders take a target and llm graders take a focus, with the same values: last_message (the default), trace (the session as JSON, so quotes appear as \"), files (created paths, not contents), { source: file, path: <path> } (one file's contents), or mock_calls. Targeting files when you meant a file's contents scores zero even when the right file was produced; the docs list it as the first troubleshooting item. The judge defaults to a small fast model; pass --judge-model sonnet when it marks correct answers wrong for formatting. There are no custom-code graders.

Fixtures and mocked MCP servers

When the prompt alone is not enough, a case.yaml beside it can name a scaffold_script that builds fixture files or a git repository in the empty workspace (runs as you, only with --scaffold), a history_file transcript to resume, or add_dirs Claude may read. For a plugin whose skills call MCP tools, one Markdown file per tool under evals/mocks/<server>/<tool>.md stands in for the real server: the body is the tool result, and an expect: block aborts the run with score 0 if the plugin calls the tool with the wrong shape. Real MCP servers never start unless you pass --allow-real-servers or --mocks off.

What it costs, and how to keep it down

Every eval run and every judge vote is a real model call on your account, against your plan's usage or your API bill. The arithmetic is roughly cases × runs agent sessions with the plugin, as many again for the baseline, plus three short judge calls per llm or baseline grader per run. Our one-case, one-run-per-arm pilot cost $0.24 and then $0.18 at list price; the default three runs per arm would have been about three times that.

The levers, in the order we would pull them:

1. --runs 1 and --case <name> while iterating on a single case.
2. --ablation none while iterating on graders, which halves the sessions because the baseline arm is skipped.
3. Free graders for quick every-change suites; save llm graders for a nightly or pre-release run.
4. --max-cost-usd <n> as a ceiling on the total. It is checked before each run starts, so spend can overshoot by the runs in flight, and hitting it exits 2 with partial: true in the JSON.

Gate CI on it

The docs' CI shape, which we would copy as written:

claude plugin eval . \
  --trust-plugin \
  --json results.json \
  --threshold 0.8 \
  --model claude-sonnet-5 \
  --judge-model claude-haiku-4-5 \
  --no-publish \
  --max-cost-usd 20

Pin both models so a model rollout is not mistaken for a plugin regression. --json writes a schemaVersion: 1 document (camelCase, new fields added without renaming) and silences the progress lines, so when a case fails in CI, rerun it locally without --json to see the grader explanations. Exit 0 means every case met the threshold and every case file loaded; 1 is a failing case, a load error, no cases, an untrusted directory, or a bad option; 2 is a partial run (cost ceiling or rejected credential); 130 and 143 are interrupted and terminated. Report problems never change the exit code. The runner needs a Claude Code install and credentials such as ANTHROPIC_API_KEY.

What this is not

It is not claude plugin validate, which checks manifests and frontmatter for schema errors and says nothing about behavior. It is not the skill-creator plugin's eval loop, which stores cases in evals/evals.json inside a skill directory and runs them as subagents in your session; the docs say the two formats are separate. And it is not a security review: a passing suite says nothing about whether the plugin is safe, and the plugin's hooks and any real MCP servers you start run as you, outside the sandbox. Only evaluate plugins you would run anyway.

One platform note: when you grant Bash in any form, commands run under Claude Code's OS-level sandbox, and native Windows has no sandbox backend, so Claude Code refuses each such run rather than running it unconfined. Our runs above granted nothing beyond Skill, which is why they worked on a Windows machine. Suites that need a shell run under WSL2 there.

FAQ

Do I need a marketplace plugin, or does a local one work?

A local directory works. claude plugin eval . from a plugin root loads the plugin you are standing in. You can also name an installed plugin (name or name@marketplace), or a skills-directory plugin as name@skills-dir; those targets skip the trust prompt because you already installed them. A named plugin's results are written under ./evals/results/ in your current directory rather than inside the plugin.

Why does my tool_used: Skill grader show passed: false under a run that scored 1.0, or scored: false?

Because in a two-arm run it is excluded from the score by design and reported as a plugin-fired indicator. Mark a grader arm: both to force it into the score in both arms, which is what you want for a "must not invoke the skill" case with min: 0 and max: 0.

The summary has no W/OUT column. What happened?

No plugin resolved for the case, so there was nothing to ablate. Add plugins: ["../.."] to the case's prompt.md frontmatter, giving the path from the case directory to the plugin root, and rerun.

Where does the HTML report go, and is it published anywhere?

Every run writes evals/results/<timestamp>/report.html, a self-contained file with no external requests; add results/ to .gitignore. On a claude.ai subscription with artifacts available, Claude Code also publishes it as a private artifact and prints a Published: URL unless you pass --no-publish. With API-key authentication it stays local.

Can I use it on a skill that is not inside a plugin?

Not directly. The target has to be a directory with a plugin.json or .claude-plugin/plugin.json, or a skills-directory plugin. Wrapping a skill in the three-file plugin from our skills tutorial and plugin tutorial takes a minute, and then the same suite tells you whether the skill's description triggers on the phrasings your users actually type, which is the first thing most suites find out.