basic BMAD setup

.agents/skills/bmad-eval/references/description-optimization.md

@@ -0,0 +1,64 @@
+# Description optimization: the trigger-eval loop
+
+A skill's description is its only trigger. The router reads it, decides whether the user's request belongs to this skill, and either loads it or moves on. A description that is too narrow stays quiet when it should fire; one that is too broad fires on requests it cannot serve. This loop measures real firing against a held-out test set and improves the description until it triggers on what it should and stays silent on what it should not, without the improver ever overfitting to the cases it is being graded on.
+
+The whole loop runs through the harness recorded in this skill's customization, so "did the skill fire" means the staged skill's canary token appeared in the output (`references/harness.md`).
+
+## Step 1: generate the query set
+
+Generate about twenty near-miss queries, roughly half that should trigger the skill and half that should not. The signal lives in the near misses, so make the should-not queries share keywords, domain, and phrasing with the should queries. A should-not query that obviously belongs to another skill teaches the description nothing, because any wording already handles it. The pairs that matter are the ones a careless reader would lump together: a request to build a workflow versus a request to debug an existing one, a request to write a brief versus a request to critique a brief someone already wrote.
+
+Each query is a `{query, should_trigger}` record:
+
+```json
+{ "query": "help me turn my deploy script into a reusable skill", "should_trigger": true }
+{ "query": "my deploy script keeps failing on the rollback step", "should_trigger": false }
+```
+
+Aim for variety in surface form (casual speech, a pasted error, a one-line ask, a paragraph of context) so the description is tested against the shapes real requests arrive in, not one tidy template.
+
+## Step 2: stratified 60/40 split
+
+Split the queries into a train set and a test set, 60 percent train and 40 percent test, stratified so the should and should-not ratio is preserved in both halves. Stratifying matters because an unstratified split can land most of the should-not queries in one half and leave the improver blind to the false-positive problem on train, or leave the test set unable to detect it.
+
+The split is fixed once at the start of the loop and never reshuffled between rounds, because reshuffling would let a query that exposed a weakness in one round hide in the train set the next. The improver works only from the train set. It never sees the test queries, their labels, or the test score, which is what keeps the loop honest.
+
+## Step 3: measure real triggering
+
+Run every query through the harness with the current description in place, several times per query because firing is probabilistic. The trigger rate for a query is the fraction of completed runs that loaded the skill; a query with a failed attempt is unmeasured and rerun, never scored. Turn each rate into a verdict against a threshold (a query "triggers" when its rate clears the bar, for example more than half its runs loaded the skill), then score against the labels:
+
+- a should-trigger query that triggered is a true positive,
+- a should-trigger query that stayed quiet is a false negative (the description is too narrow here),
+- a should-not query that triggered is a false positive (the description is too broad here),
+- a should-not query that stayed quiet is a true negative.
+
+Score train and test separately. The train score and its per-query verdicts are what the improver sees; the test score is recorded but withheld.
+
+## Step 4: improve from train failures, test blinded
+
+Hand the improver the current description, the train queries with their labels, and the train verdicts, and ask for a rewritten description that fixes the train failures. False negatives mean the description needs to claim ground it is leaving uncovered; false positives mean it needs to draw a sharper boundary against the near misses it is wrongly catching. The improver works the train failures only and never sees a test query or the test score, so it cannot tune to the held-out set.
+
+Also hand the improver the descriptions it already tried and why each fell short, so it tries something structurally different rather than nudging the same wording round after round. Without this, the loop tends to oscillate between two phrasings that each fix one failure and reintroduce the other. Feeding the history back pushes the improver toward a different cut of the boundary: reframing around intent instead of keywords, naming the adjacent skill the near misses belong to, or moving a qualifier from the trigger clause into the body.
+
+Keep the description within whatever length and format bounds the runtime enforces (character cap, no angle brackets, and so on); a rewrite that triggers well but violates the bound is not a candidate.
+
+## Step 5: re-measure and iterate
+
+Apply the new description, re-measure train and test, and record both scores plus the description text for this round. Continue for up to five rounds. Stop early if train reaches a clean separation (all should fire, all should-not stay quiet) and the test score agrees, because more rounds past a clean split only invite overfitting.
+
+## Step 6: pick the winner by test score
+
+After the rounds finish, pick the description with the best test score, not the best train score. Train measures how well the improver fixed the failures it could see; test measures whether that fix generalizes to queries it never saw, which is the only thing that matters in production. When two rounds tie on test, prefer the one with the better train score as the tiebreaker, and failing that the shorter, sharper description.
+
+Report the winning description, its test score, and the round-by-round trail (each description, its train score, its test score) so the choice is auditable and a human can override it. Each round's description and scores are written to the run folder as the loop runs.
+
+## Why each guard is here
+
+| Guard | What it prevents |
+|---|---|
+| near-miss should-not queries | a test set so easy the description never has to draw a real boundary |
+| 60/40 stratified split | a split that hides the false-positive or false-negative problem in one half |
+| fixed split across rounds | a weakness escaping into the train set on a later round |
+| test score blinded from improver | the improver tuning its wording to the held-out queries |
+| pick by test score, not train | shipping a description that fixed the visible failures but does not generalize |
+| prior attempts fed back | the loop oscillating between two phrasings instead of finding a new boundary |

.agents/skills/bmad-eval/references/eval-format.md

@@ -0,0 +1,91 @@
+# Eval format and the four modes
+
+A case is the unit of evaluation. Every case is `input + rubric + optional state_prefix`. The same case shape feeds all four modes; what changes is which invocations the runner sets up and how the result is judged.
+
+## The case
+
+```json
+{
+  "id": "create-1",
+  "input": "I want a brief for InsuLens, a claims-triage tool for mid-market insurers. Notes are in evals/insulens/files/memo.md",
+  "rubric": [
+    "brief.md exists and its word count is between 250 and 1500",
+    "brief.md names InsuLens and the mid-market insurer segment",
+    "brief.md incorporates at least two specific points from memo.md without inventing claims absent from it"
+  ],
+  "state_prefix": null,
+  "files": ["evals/insulens/files/memo.md"]
+}
+```
+
+Field semantics:
+
+- `id`: stable identifier; used as the case's folder name in the run.
+- `input`: the realistic, messy user request. Use real file paths, company names, typos, and casual speech, because a polished input tests a situation the skill rarely meets. The runner sends this verbatim to the invocation, after prepending any `state_prefix`.
+- `rubric`: a list of named expectations, each gradeable to `{text, passed, evidence}` by the grader. The strong-versus-weak taxonomy below decides whether each one is worth keeping.
+- `state_prefix`: optional bracketed prime that places the skill mid-workflow (see below). Null or absent means the skill starts cold.
+- `files`: optional fixture paths staged into the case's clean working directory before the run. A bare filename lands at the workspace root; a nested path keeps its directory structure, so the input can reference it verbatim. Sources resolve against `--project-root`, then the cases file's directory, then as absolute paths.
+
+For trigger cases the shape is lighter: a `query` and a `should_trigger` boolean, because there is no artifact to grade, only whether the skill fired.
+
+## state_prefix: turn simulation in one shot
+
+Most multi-turn skills can be evaluated single-shot if the case is designed right. The `state_prefix` is the trick that makes mid-workflow points reachable without a multi-turn simulator. It is a bracketed prime prepended to the input that tells the skill where in its own flow this turn lands and what the user already said:
+
+```
+[the skill has already worked through discovery; on turn 4 the user was asked about stakeholders and responded:] User said: "just me and a PM"
+```
+
+The runner prepends the `state_prefix` to `input` and sends the combined text as a single message. One input then exercises any mid-workflow moment: a clarifying turn, a correction, a resume after an interruption. Only a case where the conversation arc itself is the deliverable needs real multi-turn.
+
+Subjective skills (coaching, brainstorming, design facilitation) skip the rubric and rely on human judgment. The `state_prefix` still earns its place there, because it lets a human see the exact mid-run moment they want to judge.
+
+## Strong versus weak expectations
+
+The grader's job is easier and the result is more honest when an expectation is discriminating, meaning a wrong output cannot pass it. A weak expectation is worse than no expectation, because a green check on it reads as proof when it is noise. The grader flags weak expectations when it sees them; write them out of the rubric before they ship.
+
+Weak patterns to avoid:
+
+- Filename-only checks. "brief.md exists" passes for an empty file. Pair existence with a content check.
+- Wholly subjective phrasing. "the brief is high quality" cannot be graded. State the property concretely.
+- Tautologies. Anything that follows automatically from the prompt being understood proves nothing.
+
+Strong patterns for artifact correctness:
+
+- Specific facts that must appear, such as "incorporates at least two findings from section X."
+- Structural claims a wrong output would fail, such as "word count between 250 and 1500."
+- Negative assertions, such as "does not introduce content from unrelated sections."
+- Frontmatter checks, such as "frontmatter contains title, status, created (ISO 8601), updated."
+- Bounded output blocks, such as "the final message contains a JSON object with intent='create'."
+
+Strong patterns for process discipline:
+
+- Side-artifact existence paired with content, such as ".memlog.md captures the pricing decision with its rejected alternative and rationale."
+- Transcript tool-call patterns, such as "the transcript contains a call invoking bmad-editorial-review-prose."
+- Phase ordering, such as "the polish call occurs after the brief Write and before the final JSON block."
+- Read-only enforcement, such as "the input brief.md is byte-identical to the fixture and no Write or Edit targeted it."
+- Bidirectional fidelity, such as "every decision in the memlog is reflected in the brief, and no claim in the brief is absent from the input or the memlog."
+
+Most process-discipline checks are deterministic reads of the transcript and filesystem, so the grader confirms them by quoting evidence rather than judging.
+
+## The four modes in detail
+
+### Baseline: skill versus bare model
+
+Run the case input twice in parallel in the same turn, once wrapped by the skill and once against the bare model with nothing around it. The bare-model run is the long-term floor. The skill earns its existence only by producing something the bare model cannot, so when the skill stops beating the bare model the right call is retirement, not another patch. Use baseline when the user asks whether the skill is worth keeping, or as the release check.
+
+### Variant: full versus stripped smallest-version
+
+Run the full skill against a stripped smallest-version of the same skill (passed as `--variant-path`), or against a snapshot of the prior version for an edit, on the same input. It settles whether a section suspected of being ceremony does real work. If the two outputs tie on the dimension the section was supposed to protect, the section is decoration and gets cut. If the small version is materially and durably worse, the section earned its keep.
+
+### Quality: output versus rubric
+
+Grade a single config's output against the named rubric with the read-only grader. The grader gives no partial credit, puts the burden of proof on a passing grade, and flags any non-discriminating assertion. Use quality when a rubric exists and the user wants to know whether the output meets it, independent of any comparison.
+
+### Trigger and description
+
+Generate near-miss should-trigger and should-not-trigger queries that share keywords, split them, measure real firing through the harness, and improve the description across bounded rounds with the held-out scores blinded from the improver. Trigger detection is a canary: the staged skill asks for a token at the start of the reply, and the token in the output is the load (`references/harness.md`).
+
+## Getting a skill to behave non-interactively
+
+Single-shot modes need the skill to produce its deliverable without stopping to ask. Most multi-turn skills expose a headless flag or keyword that suppresses clarifying questions and ends with a structured status block. Trigger it from the input: the literal `Run headless.` at the start, a skill-specific keyword from the skill's own headless section, or enough context that no clarification is genuinely needed. The `state_prefix` also helps here, because a turn that already supplies the answer the skill would ask for keeps the run moving. If a skill has no headless path and the input cannot satisfy its questions, either add a headless mode to the skill or accept that this case needs a human in the loop.

.agents/skills/bmad-eval/references/grader.md

@@ -0,0 +1,83 @@
+# Grader: LLM-as-judge contract
+
+The grader inspects one case's captured transcript and artifacts and answers, per expectation, whether it held. It writes its verdict to `grading_path` so the grade lives in the case folder, not just in a subagent's reply. It is otherwise read-only against the run folder: it does not execute the skill, fix an artifact, or rerun anything; its only job is to judge what was produced and cite the evidence.
+
+The grader has a second job that matters as much as the first: it critiques the rubric. A passing grade on a weak assertion is worse than useless, because it reads as proof while measuring nothing, so the grader flags assertions that a wrong output would also pass and names important outcomes that no assertion covers.
+
+## Inputs
+
+The grader receives:
+
+- `case_id`: identifier for this case.
+- `input`: the message that was sent to the skill, including any prepended `state_prefix`.
+- `rubric`: the list of expectation strings it grades, each independently.
+- `transcript_path`: absolute path to the run's transcript: what the harness printed, line-delimited JSON events when it emits them, plain text otherwise.
+- `artifacts_dir`: absolute path to the directory of files the skill wrote.
+- `grading_path`: absolute path where the grader writes `grading.json`.
+
+## Process
+
+1. Read the transcript. When it is line-ordered JSON events, note the input that was sent, every tool call the skill made (with its name and arguments), the order those calls happened in, the final message (often a JSON status block for headless runs), and any errors. When it is plain text, it is the final output and the tool calls are not visible: grade tool-call and ordering expectations from the artifacts where they leave evidence, and otherwise mark them failed with "not observable in this transcript" as the evidence.
+
+2. List and read the artifacts. Walk `artifacts_dir` and open the files each expectation implicates. Read their contents rather than trusting filenames, and note modification times when ordering or read-only behavior is in scope.
+
+3. Grade each expectation independently. Identify what kind of check it is and gather the matching evidence:
+
+   - Artifact existence + content ("brief.md exists AND names X") → open the file, read it, check the content matches; existence alone never passes a content claim.
+   - Transcript tool-call patterns ("transcript contains a Skill call to X") → scan for `tool_use` events with the matching `name` and `input`; quote the matching event.
+   - Phase ordering ("the polish call occurs after the Write and before the final JSON block") → find each landmark's line number or event index and verify the order.
+   - Read-only enforcement ("input file is byte-identical; no Write/Edit targeted it") → compare content against the fixture AND scan the transcript for any Write/Edit whose `input.file_path` falls in the protected path.
+   - Frontmatter checks → parse the frontmatter, verify each named field and its format.
+   - Output-block checks ("final message contains a JSON object with intent='create'") → take the last assistant message's text, extract the object, check the field.
+   - Bidirectional fidelity ("every decision in the log appears in the artifact AND nothing in the artifact lacks a source") → list claims on each side and trace both directions.
+
+4. Decide pass or fail with specific evidence. Pass only when there is clear evidence the expectation holds and the evidence reflects substance rather than surface compliance, so a file that exists but holds only placeholders fails a content expectation. Fail when no evidence is found, the evidence contradicts the expectation, or the assertion is technically satisfied while the underlying outcome is wrong. Cite the evidence every time by quoting a line, naming a file with its path, or pointing to a tool call by its index and arguments.
+
+5. Critique the rubric. After grading, surface assertions that look weak, meaning ones that passed but would also pass for a clearly wrong output, and name important outcomes you observed, good or bad, that no assertion checks. Keep the bar at what a rubric author would call a good catch rather than a nit.
+
+6. Write the verdict to `grading_path` as `grading.json`, then summarize it in your reply.
+
+## Output
+
+`grading.json` holds one record per expectation plus a summary and rubric feedback:
+
+```json
+{
+  "case_id": "create-1",
+  "expectations": [
+    {
+      "text": "brief.md exists and word count is between 250 and 1500",
+      "passed": true,
+      "evidence": "artifacts/insulens/brief.md, 487 words"
+    },
+    {
+      "text": "the memlog references having ingested the memo as source material",
+      "passed": false,
+      "evidence": ".memlog.md exists but contains only the init entry; no mention of memo.md"
+    }
+  ],
+  "summary": { "passed": 1, "failed": 1, "total": 2, "pass_rate": 0.5 },
+  "rubric_feedback": {
+    "weak": [
+      {
+        "assertion": "brief.md exists",
+        "reason": "Existence alone passes for an empty file; pair with a content or word-count check."
+      }
+    ],
+    "uncovered": [
+      "The brief invented a competitor not present in the input or the memlog; no assertion would have caught this."
+    ],
+    "overall": "Assertions check structure but not content fidelity in two places."
+  }
+}
+```
+
+When `weak` and `uncovered` would both be empty, set them to `[]` and `overall` to `"No suggestions; the rubric looks discriminating."`
+
+## Rules
+
+- Verdicts come from evidence, not impressions, so quote, name files, and point to event indices.
+- No partial credit. Each expectation is pass or fail.
+- The burden of proof is on a passing grade, so when the evidence is uncertain the expectation fails.
+- Read-only against the run folder except `grading.json`. The grader never edits an artifact.
+- No silent defaults. If a file or the transcript genuinely cannot be read, mark the affected expectations failed with that as the evidence rather than guessing.

.agents/skills/bmad-eval/references/harness.md

@@ -0,0 +1,30 @@
+# The harness
+
+The harness is the agent CLI the evals run through: yours, the one you are running in. The runner starts it once per case from an empty workspace in a fresh HOME, with the skill under test copied where the CLI reads skills, stdin closed and output captured. It needs four facts, recorded once per project in this skill's customization under `[workflow.harness]` through the `bmad-customize` skill:
+
+| Key | What to work out |
+|---|---|
+| `command` | argv for one non-interactive run, `{prompt}` standing for the input, with your flag that turns off permission prompts. `--help` has it. |
+| `skill_dir` | the folder under the workspace you read skills from. |
+| `env` | env vars the run needs: a value is set as given, `~` meaning the fresh HOME; an empty value forwards the host's. Your config-folder variable, if you have one, and whatever the login needs. |
+| `home_files` | files or folders under your home that must come along for you to stay logged in, placed at the same path in the fresh HOME. Look in your config folder; on a Mac it may be the keychain. |
+
+Two shapes that have run, as evidence of the form rather than a catalog:
+
+```toml
+command = ["codex", "exec", "--json", "--ephemeral", "--skip-git-repo-check", "--dangerously-bypass-approvals-and-sandbox", "{prompt}"]
+skill_dir = ".agents/skills"
+env = { CODEX_HOME = "~/.codex" }
+home_files = [".codex/auth.json"]
+```
+
+```toml
+command = ["claude", "-p", "{prompt}", "--output-format", "stream-json", "--verbose", "--dangerously-skip-permissions"]
+skill_dir = ".claude/skills"
+env = { USER = "" }
+home_files = ["Library/Keychains"]
+```
+
+Prove the facts on one case before the set; a wrong flag, folder or login shows there. A run bypasses your permission prompts and is not contained unless the command wraps it in a sandbox, so say that when confirming a run. When nobody is at the keyboard and nothing is recorded, stop and show the table to record. In a project without BMad, write the same keys as JSON and pass `--harness <file>`.
+
+Trigger mode detects a load with a canary: the staged skill's body asks for a token at the start of the reply, and the token in the output is the load, on any CLI. Token counts are read when the output is line-delimited JSON with usage blocks; otherwise elapsed time is the measure.

.agents/skills/bmad-eval/references/self-improvement.md

@@ -0,0 +1,49 @@
+# Self-improvement: the bounded auto-iterate loop
+
+This is the loop that reviews a skill, evaluates it, proposes a fix, applies it, and re-evaluates, repeating until the skill passes or a round bound is hit. It is the most autonomous mode the runner offers, so it carries the most guardrails: it is opt-in, calibrated to what is at stake, fully logged, and bounded.
+
+The benchmark is a guardrail, never the judge. The human stays the judge. A green run means the change cleared the bar the loop was given, not that the change is correct, and the loop's job is to do the mechanical iteration a human would otherwise do by hand and then hand back a fix plus the evidence for it.
+
+## When to run it, and how hard
+
+The loop is opt-in. It never starts on its own, because applying changes to a skill in a loop is a stronger action than reporting findings, and the user decides when that is warranted.
+
+Calibrate the aggressiveness to the stakes. A throwaway skill the user is still shaping can take a longer loop and a looser bar, because a wrong iteration costs little and is easy to throw away. A skill that other skills already depend on, or one that is shipped and in use, takes a short loop, a strict pass bar, and a close human read of every applied change, because a regression there propagates. Agree the round bound and the pass condition with the user before the first round, and write both into the run report so the run is auditable against the terms it was given.
+
+## The loop
+
+Each round runs four beats:
+
+1. Review. Invoke the `bmad-toolsmith` skill to review the skill and collect its findings. On rounds after the first, review again rather than trusting the prior findings, because the last fix may have moved something.
+
+2. Eval. Run the modes that apply to this skill: quality against its rubric where one exists, variant to settle a finding that a section does no work, baseline to confirm the skill still beats the bare model. The review says what looks wrong; the eval says whether it measurably is. A finding the eval cannot confirm is a candidate to note for a human, not to auto-fix.
+
+3. Propose a fix. From the confirmed findings, propose one concrete change. Address the cause the finding names rather than the single case that exposed it (see generalizing, below). Keep the change small enough that the next eval can attribute the delta to it; a round that rewrites five things at once cannot tell you which one moved the score.
+
+4. Apply and re-eval. Apply the proposed change, then re-run the eval from beat 2 and compare. A round that improves the score and breaks nothing else is kept; a round that regresses any mode is reverted before the next round, because an applied change that made things worse is not a base to build on.
+
+Stop when the pass condition is met or the round bound is reached, whichever comes first. The bound is a hard stop: hitting it without passing ends the loop and reports the best state reached, it does not earn extra rounds.
+
+## The trail is the run folder
+
+Each round's proposed fix, scores and transcripts are already on disk in the run folder, so a reviewer reads the rounds back in order and sees what was tried, what each attempt did to the numbers, and why the loop stopped; note a reverted round and its reason in the round's report.
+
+## Generalize to intent, do not overfit to the case
+
+The failure that ends most auto-iterate loops is fixing the example instead of the cause. A case fails because the skill mishandled a class of input; patching the skill to special-case that one input passes the case and leaves the class broken, and often the patch is a hardcoded branch that makes the skill worse. Read each finding as a representative of an intent category and fix the category. A case where the skill invented a fact absent from the source is not "handle this memo," it is "the skill does not ground its output in the provided source," and the fix belongs at that level.
+
+When a proposed fix reaches for ALL-CAPS ALWAYS or NEVER or a stack of MUSTs, treat that as a warning sign. Shouting at the model is usually a sign the fix is patching a symptom; a sharper outcome statement or a small worked example generalizes where a louder rule does not. Prefer the version that explains the reasoning over the version that issues the command.
+
+## Why each guard is here
+
+| Guard | What it prevents |
+|---|---|
+| opt-in | a loop applying changes the user never authorized |
+| stakes calibration | the same aggressiveness on a throwaway and a depended-on skill |
+| eval confirms the review | auto-fixing a finding the evidence does not support |
+| one change per round | a round whose delta cannot be attributed to a specific fix |
+| revert on regression | building the next round on a change that made things worse |
+| round bound | a loop that runs away instead of handing back to a human |
+| trail on disk | reasoning that lives only in the model and cannot be audited |
+| benchmark as guardrail, human as judge | treating a green run as proof the change is correct |
+| generalize to intent | a hardcoded patch that passes the case and leaves the class broken |