# Eval loop

The shared loop that `approaches/eval-first.md` and `approaches/skill-creator-loop.md` load after their own opening. Its outcome: a skill whose cases pass because of what the skill adds, with the evidence on record. Every run here is "invoke the `bmad-eval` skill" on `{target}`; this file never runs a skill itself.

## Cases

Cases live at `{target}/evals/cases.json` as `{id, input, rubric[], state_prefix, files[]}`. Seed them from real requests and transcripts before inventing any, because invented cases come out homogeneous and easy. Two or three realistic, messy inputs to start, with no rubric yet. A `state_prefix` places a case mid-flow without a multi-turn harness.

## One round

1. Run with and without. Invoke the `bmad-eval` skill in baseline mode, so the skill and the bare model see the same inputs in parallel. Runs land in bmad-eval's reports folder, by default `{output_folder}/eval-reports`; bmad-eval writes them.
2. Write the rubric after the first outputs. Each entry is binary and names one failure mode you saw; prefer graders computed from the transcript and files, and keep a model judge for short outputs only, per the `bmad-eval` skill's eval-format reference. An assertion that passes in both arms tests nothing the skill adds: sharpen it or drop it.
3. Grade. Three runs per case, because output varies run to run. Invoke the `bmad-eval` skill in quality mode for the rubric and read the delta between arms, never the with-skill score alone.
4. Read the transcripts, not just the grades. Several approaches tried before one works: an instruction is too vague. An instruction followed where it does not apply: too broad. Stalling between alternatives: no default named. The same helper rewritten every run: bundle a script.
5. Change one thing, with its why, and cut any line whose removal did not change results.
6. Rerun. A variant run (invoke the `bmad-eval` skill in variant mode against the prior snapshot) says whether the change, not chance, moved the score.

## When to stop

- Every grader passes three runs: done.
- The user says it is good enough: done, and tell them which cases still fail.
- Two rounds move nothing: the cases are the problem, not the words. Re-examine them for weak assertions, inputs too tidy to fail, or a rubric grading what the bare model already does.
- The with-skill arm loses to the bare model and stays behind: the model has caught up. Say so plainly and propose retiring the skill, not patching it.

## The honest check

Before leaving the loop, say in one line what the cases do not cover and whether any assertion passed for a trivial reason. A passing run on easy cases is not proof; the user decides whether to add harder ones.