Skip to content

Reference

Eval Format

Use this page for the exact formats bmad-eval reads and writes. For when to run which mode, see Eval Runner.

<repository or project root>/
├── evals/
│ ├── suites/
│ │ └── <suite-name>/
│ │ └── trigger-matrix.json # queries across a whole install
│ └── <module>/
│ └── <skill-name>/
│ ├── cases.json # baseline, variant, and quality cases
│ ├── triggers.json # trigger queries
│ └── files/ # fixtures the cases name
└── skills/
└── <module>/
└── <skill-name>/

Evals sit outside the skill, so installing the skill never copies them. bmad-eval looks for cases in the file you name, then the skill’s folder under evals/ in the nearest folder above the skill holding one, mirroring its path below skills/ (evals/core-tools/bmad-brainstorming/), then <skill>/evals/ for a skill built elsewhere, then anywhere under <project-root>/evals/. A run never stages the skill’s own evals/ folder, so the model under test cannot read its rubric.

cases.json is a JSON array of cases, or an object with a cases array. One case looks like this:

{
"id": "create-1",
"input": "I want a brief for InsuLens, a claims-triage tool for mid-market insurers. Notes are in evals/insulens/files/memo.md",
"rubric": [
"brief.md exists and its word count is between 250 and 1500",
"brief.md names InsuLens and the mid-market insurer segment"
],
"state_prefix": null,
"files": ["evals/insulens/files/memo.md"]
}
FieldMeaning
idStable identifier, used as the case’s folder name in the run
inputThe realistic user request, sent as given after any state_prefix
rubricNamed expectations, each graded to {text, passed, evidence}
state_prefixOptional bracketed text that places the skill mid-workflow. Null or absent starts cold
filesOptional fixtures staged into the clean working directory

Each entry in files is a path relative to --project-root or the cases file’s folder, or a GitHub file URL pinned to a commit or tag, https://github.com/<owner>/<repo>/blob/<ref>/<path>, for data kept in another repo. A relative path keeps its folders, so the input can name it as written; a GitHub file lands at the workspace root under its file name. Absolute and ~ paths are refused so the cases run on anyone’s machine, and a missing fixture stops the run before any case starts.

The runner prepends the prefix to input and sends the result as one message, so one case can test any mid-workflow moment.

[the skill has already worked through discovery; on turn 4 the user was asked about stakeholders and responded:] User said: "just me and a PM"

Only a case where the conversation itself is the deliverable needs real multiple turns.

triggers.json is a JSON array of {query, should_trigger} records.

[
{ "query": "help me turn my deploy script into a reusable skill", "should_trigger": true },
{ "query": "my deploy script keeps failing on the rollback step", "should_trigger": false }
]

Aim for about twenty queries, roughly half of each kind, varied in form. Make the should-not queries near misses that share keywords and domain with the real ones.

A suite that spans skills lives in evals/suites/<suite-name>/. Its trigger-matrix.json is a JSON array of {query, expect, also_ok} records: expect is the skill that should load, or null for none, and also_ok names skills that may load instead.

[
{ "query": "help me come up with names for my bakery", "expect": "bmad-brainstorming" },
{ "query": "do some market research on AI note-taking apps", "expect": "bmad-deep-recon", "also_ok": ["bmad-agent-analyst"] },
{ "query": "rename this variable to camelCase", "expect": null }
]

Every skill of the install is staged at once, each with its real description, so a query fails when another skill loads in its place. Write queries where two descriptions could both claim the request, and keep also_ok to skills that would serve it as well.

A wrong output must not be able to pass an expectation. A weak one is worse than none, because a passing check reads as proof.

WeakWhy
“brief.md exists”Passes for an empty file. Pair it with a content check
“the brief is high quality”Cannot be graded. State the property concretely
Anything that follows from understanding the promptProves nothing
StrongExample
Specific facts“incorporates at least two findings from section X”
Structure“word count between 250 and 1500”
Negative assertions“does not introduce content from unrelated sections”
Frontmatter“frontmatter contains title, status, created, updated”
Bounded output“the final message contains a JSON object with intent=‘create’”
Side artifacts“the Decisions section records the pricing decision with its rejected alternative”
Tool calls“the transcript contains a call invoking bmad-editorial-review-prose”
Phase order“the polish call occurs after the brief Write and before the final JSON block”
Read-only inputs“the input is byte-identical to the fixture”

Single-shot modes need the skill to finish without stopping to ask. Start the input with the literal Run headless., use a keyword from the skill’s own headless section, or give enough context that no question is needed. A state_prefix can also supply the answer the skill would ask for. A skill with no headless path needs a headless mode, or a human in the loop for that case.

Agents run these for you. They are listed for reading a run or running one by hand.

Baseline, variant, and quality:

Terminal window
uv run {skill-root}/scripts/run_evals.py \
--cases <cases-file> --skill-path <skill> --output-dir <dir> \
--mode quality|baseline|variant [--variant-path <skill>] \
--label <skill-name>-<mode> [--runs N]

Trigger:

Terminal window
uv run {skill-root}/scripts/run_triggers.py \
--skill-path <skill> --queries <queries-file> --output-dir <dir> \
[--runs-per-query N]

Trigger matrix, given the folders that hold the install’s skills:

Terminal window
uv run {skill-root}/scripts/run_matrix.py \
--skills <folder> [--skills <folder> ...] --queries <trigger-matrix.json> --output-dir <dir> \
[--runs-per-query N]

Aggregate the time, tokens, and tool calls of repeated runs into mean, sample standard deviation, min, max, and delta:

Terminal window
uv run {skill-root}/scripts/aggregate_benchmark.py --baseline <run-dir>/<config-a> --variant <run-dir>/<config-b>

Convert a skill-creator file, or add --to skill-creator to reverse it:

Terminal window
uv run {skill-root}/scripts/convert_cases.py <evals.json> --output <cases.json>
Exit codeMeaning
0The run completed. A trigger or matrix query failing its threshold is a result, not an error
1A case errored or timed out, or the command was not found
2Usage error
3No harness is recorded

--runs defaults to 1, which counts as a quick look. A result is settled at 3 or more.

The harness is the coding tool that evals run through. These keys are recorded under [workflow.harness] in bmad-eval’s customization. In a project without BMad, write the same keys to a JSON file and pass its path with --harness.

KeyHolds
commandThe argv for one non-interactive run, with {prompt} standing for the input and the flag that turns off permission prompts
skill_dirThe folder under the workspace that the tool reads skills from
envEnvironment variables a run needs. A value is set as given, ~ means the fresh home folder, and an empty value forwards the host’s
home_filesFiles or folders under your home that keep you logged in, placed at the same path in the fresh home folder
[workflow.harness]
command = ["claude", "-p", "{prompt}", "--output-format", "stream-json", "--verbose", "--dangerously-skip-permissions"]
skill_dir = ".claude/skills"
env = { USER = "" }
home_files = ["Library/Keychains"]

Further harnesses, one per tool or model to compare, are named tables with the same keys. Pass --harness-name <name> to any runner to run through one; repeat it to run once per name. Each name gets its own run folder, and the output is one JSON object keyed by name.

[workflow.harnesses.codex]
command = ["codex", "exec", "--json", "--ephemeral", "--skip-git-repo-check", "--dangerously-bypass-approvals-and-sandbox", "{prompt}"]
skill_dir = ".agents/skills"
env = { CODEX_HOME = "~/.codex" }
home_files = [".codex/auth.json"]

Each run is a dated folder under the output folder, which is the configured reports folder when BMad is set up and ~/bmad-evals/ otherwise. run.json names the harness command. Each case folder, and each trigger attempt, holds:

FileHolds
The promptWhat was sent
The transcriptWhat the harness printed
stderr.txtThe error stream
cwd/The workspace after the run
timing.jsonTiming, written when the invocation ends
grading.jsonGrades, when quality mode ran

Run folders are never deleted, overwritten, or rotated.