Regression Test: Golden Dataset Management

Manage golden test datasets for LLM prompt regression testing using Promptfoo. Ensures prompt changes don't degrade output quality.

One eval tells you today's score. A kept set tells you what your change did. There is a recorded

three-run walkthrough of exactly that, runnable with no API key, at

https://github.com/7alexhale5-rgb/alexhale-skills/tree/main/example

**Prerequisite:** `npm install -g promptfoo` (check with `which promptfoo`)

---

Commands

`/regression-test init`

Scaffold regression testing infrastructure in the current project.

1. Create directory structure:

```bash

mkdir -p .promptfoo/golden

```

2. Generate `.promptfoo/promptfooconfig.yaml` (the path matters: `tests: "golden/*.yaml"` below resolves relative to this file, and `/regression-test run` looks for it here):

```yaml

description: "Golden dataset regression tests"

# Without this, promptfoo has no prompt to send and every case scores an empty string.

# The vars come from the golden files that `/regression-test add` writes.

prompts:

- |

[{"role":"system","content":"{{system_prompt}}"},{"role":"user","content":"{{user_input}}"}]

providers:

- id: anthropic:messages:claude-sonnet-5

config:

temperature: 0

tests: "golden/*.yaml"

defaultTest:

assert:

- type: llm-rubric

value: "Output should match the expected behavior described in the test case"

```

3. Add `.promptfoo/results/` to `.gitignore` if not already present. Ignore the results, commit the goldens: a golden dataset nobody else can see is not a shared baseline, and the whole point is that your teammates and your CI run the same cases you do. If a prompt in a golden case is genuinely sensitive, keep that one case out of the repo rather than hiding the dataset.

4. Report: "Initialized `.promptfoo/golden/` — run `/regression-test add <name>` to capture test cases."

---

`/regression-test add <name>`

Capture a prompt+response pair as a golden test case.

1. Parse `<name>` from user input (slug format, e.g., `auth-flow`, `error-handling`)

2. Prompt user for:

- **System prompt** (or detect from current project's CLAUDE.md/skill)

- **User input** (the test prompt)

- **Expected behavior** (natural language description of good output)

- **Vars** (optional template variables)

3. Write golden test case:

```yaml

# .promptfoo/golden/{name}.yaml

- description: "{name}"

vars:

system_prompt: |

{system prompt content}

user_input: |

{user input}

assert:

- type: llm-rubric

value: |

{expected behavior description}

- type: not-contains

value: "error"

```

4. Report: "Added golden test `{name}` — {N} total test cases."

---

`/regression-test run`

Execute Promptfoo evaluation against all golden datasets.

1. Check prerequisites:

```bash

which promptfoo 2>&1 # must return path

ls .promptfoo/golden/*.yaml 2>/dev/null | wc -l # must be > 0

```

2. Run evaluation:

```bash

cd {project_root}

promptfoo eval --config .promptfoo/promptfooconfig.yaml \

--output .promptfoo/results/eval-$(date +%Y%m%d-%H%M%S).json \

--no-cache 2>&1

```

3. Parse results JSON:

- Count PASS/FAIL per test case

- Extract failure details (expected vs actual)

4. Report:

```

Regression Test Results

├─ Total: {N} test cases

├─ Pass: {N} ({%})

├─ Fail: {N} ({%})

└─ Duration: {N}s

Failures:

- {test_name}: {assertion_type} — expected "{expected}", got "{actual}"

```

5. If all pass: "All golden tests pass — safe to ship."

If failures: "Regressions detected — review before shipping."

---

`/regression-test report`

Show regression trends over the last 5 evaluations.

1. Read result files:

```bash

ls -t .promptfoo/results/eval-*.json | head -5

```

2. For each result file, extract:

- Date (from filename)

- Pass/fail counts

- Any new failures vs previous run

3. Report:

```

Regression Trends (last 5 evals)

├─ 2026-03-31 14:30: 12/12 PASS ✓

├─ 2026-03-30 09:15: 11/12 PASS (1 fail: auth-flow)

├─ 2026-03-29 16:45: 12/12 PASS ✓

├─ 2026-03-28 11:00: 10/12 PASS (2 fail)

└─ 2026-03-27 08:30: 12/12 PASS ✓

Trend: STABLE (4/5 clean)

```

---

Where to wire it in

The value shows up when the run is automatic, not when you remember to type it.

| Moment | What to do |

|--------|------------|

| After changing any prompt | `/regression-test run` before you commit. This is the one that catches most regressions. |

| Before a release | Run it in CI and fail the build on a new failure. `promptfoo eval` returns non-zero when an assertion fails. |

| After a model or version change | Re-run the whole set. A provider upgrade is a silent prompt change. |

| Weekly, on a schedule | `/regression-test report` to see whether quality is drifting rather than breaking. |

If your agent harness supports hooks or a pre-commit step, put the run there. A check that depends on discipline is a check you will lose.

---

Graceful Degradation

| Component | If Missing | Fallback |

|-----------|-----------|----------|

| promptfoo CLI | Not installed | Prompt user: `npm install -g promptfoo` |

| .promptfoo/ directory | Not initialized | Prompt: `/regression-test init` |

| Golden datasets | No .yaml files | Skip with note: "No golden tests — add with `/regression-test add`" |

| Results directory | Missing | Create on first `run` |