basic BMAD setup

This commit is contained in:
Ebbe Baß
2026-10-15 08:57:23 +02:00
parent 44029127dc
commit 5e747654b9
438 changed files with 41818 additions and 0 deletions
@@ -0,0 +1,19 @@
# Eval-first
For an output that can be checked objectively and a user who wants proof. Its outcome: the cases exist before the skill does, so the skill is built to pass them and the passing run is on record.
## Cases before code
From the discovery, write `{target}/evals/cases.json` (`{id, input, rubric[], state_prefix, files[]}`). Inputs are the user's real requests where they exist, as messy as they were typed. Each rubric entry is a PASS or FAIL condition over the output, the transcript or the files, never a quality adjective. A `state_prefix` reaches a mid-flow turn in one shot. Show the file and confirm it with the user, because cases they do not believe in prove nothing to them.
## Smallest skill
Scaffold with `uv run {skill-root}/scripts/init_skill.py --name <name> --dest <parent folder> --shape <shape> --description "<approved description>"`, adding `--bmod <record> --source <update_source>` when the read-back registers it as a member, and load `shapes/<shape>/shape.md`, then the module shape the registration line names, if any. Write `SKILL.md` to `canon.md`, aimed at the cases, never to the rubric word for word, because a skill that quotes its grader passes the test and fails the user.
## Loop
Load `approaches/eval-loop.md` and run it to a stop condition.
## Then
Tell the user which run folder holds the passing run, and which cases still fail if they stopped early. Load `ship.md`.
@@ -0,0 +1,27 @@
# Eval loop
The shared loop that `approaches/eval-first.md` and `approaches/skill-creator-loop.md` load after their own opening. Its outcome: a skill whose cases pass because of what the skill adds, with the evidence on record. Every run here is "invoke the `bmad-eval` skill" on `{target}`; this file never runs a skill itself.
## Cases
Cases live at `{target}/evals/cases.json` as `{id, input, rubric[], state_prefix, files[]}`. Seed them from real requests and transcripts before inventing any, because invented cases come out homogeneous and easy. Two or three realistic, messy inputs to start, with no rubric yet. A `state_prefix` places a case mid-flow without a multi-turn harness.
## One round
1. Run with and without. Invoke the `bmad-eval` skill in baseline mode, so the skill and the bare model see the same inputs in parallel. Runs land in bmad-eval's reports folder, by default `{output_folder}/eval-reports`; bmad-eval writes them.
2. Write the rubric after the first outputs. Each entry is binary and names one failure mode you saw; prefer graders computed from the transcript and files, and keep a model judge for short outputs only, per the `bmad-eval` skill's eval-format reference. An assertion that passes in both arms tests nothing the skill adds: sharpen it or drop it.
3. Grade. Three runs per case, because output varies run to run. Invoke the `bmad-eval` skill in quality mode for the rubric and read the delta between arms, never the with-skill score alone.
4. Read the transcripts, not just the grades. Several approaches tried before one works: an instruction is too vague. An instruction followed where it does not apply: too broad. Stalling between alternatives: no default named. The same helper rewritten every run: bundle a script.
5. Change one thing, with its why, and cut any line whose removal did not change results.
6. Rerun. A variant run (invoke the `bmad-eval` skill in variant mode against the prior snapshot) says whether the change, not chance, moved the score.
## When to stop
- Every grader passes three runs: done.
- The user says it is good enough: done, and tell them which cases still fail.
- Two rounds move nothing: the cases are the problem, not the words. Re-examine them for weak assertions, inputs too tidy to fail, or a rubric grading what the bare model already does.
- The with-skill arm loses to the bare model and stays behind: the model has caught up. Say so plainly and propose retiring the skill, not patching it.
## The honest check
Before leaving the loop, say in one line what the cases do not cover and whether any assertion passed for a trivial reason. A passing run on easy cases is not proof; the user decides whether to add harder ones.
@@ -0,0 +1,19 @@
# From logs
The one approach that starts before `discover.md`. Its outcome: up to three skill candidates mined from what the user actually did, one chosen and handed to discovery as mined input.
## Which logs
Ask once, with the default offered: this project's Claude Code sessions (`--project {project-root}`), their memlogs, or files they name. The formats read today are Claude Code transcripts and memlogs. Say that the digest is computed locally and nothing leaves the machine, and wait for a yes before reading.
## Digest
`uv run {skill-root}/scripts/read_session_log.py --project {project-root}` or `--format claude-code|memlog|auto <paths...>`, with `--max-items N` to cap the lists. The JSON holds `user_requests` with counts, `tool_sequences`, `corrections`, `files_touched` and `skills_invoked`. Read it whole before proposing anything.
## Candidates
A candidate is a request made repeatedly, the tool sequence that answered it, and the corrections the user made along the way. The corrections are the know-how a skill exists to encode; a request with none is probably something the model already does well and needs no skill. Propose at most three, each with its counts: how often asked, in which words, which tools in which order, and the corrections verbatim. Leave out what an installed skill already covers (`skills_invoked` says). The user picks one or none; none ends here with the digest path noted.
## Hand-off
Load `discover.md` with the candidate as the mined input: discovery confirms the purpose, consumer, trigger phrases and know-how from the digest rather than asking for them, and asks only what the logs cannot say. Name the declined candidates so discovery does not re-propose them. From the read-back on, the approach is lean unless the user named another.
@@ -0,0 +1,27 @@
# Lean
The default approach. Its outcome: the smallest skill that does the job, tried once on the user's real input before it is polished, at `{target}`.
## Scaffold
`uv run {skill-root}/scripts/init_skill.py --name <name> --dest <parent folder> --shape <shape> --description "<approved description>"`, with `--dirs` for only the folders the read-back listed and `--bmod <record> --source <update_source>` when the read-back registers it as a member; a single-skill module's record and help come from `shapes/single-skill-module/shape.md` at the write step. Then load `shapes/<shape>/shape.md` for the files the shape emits and the rules they follow.
## Write the smallest version
Write `SKILL.md` to `canon.md`. Show the draft and the description in chat before writing any further file, because a user who moves the bar now saves every file written after it.
## Try it
When the user has a real input (a request they made by hand, a file, a transcript), run the skill on it once in a fresh context, a subagent that sees only `{target}` and the input, and read the transcript, not just the output. What the trace means:
- Several approaches tried before one worked: an instruction is too vague. Name the default.
- An instruction followed where it did not apply: too broad. Narrow its condition.
- Stalling between alternatives: no default was named. Name one.
- The same helper rewritten each run: bundle it as a script.
- A line the run never needed: cut it.
Fix what the trace shows, once, and record the decision. Where discovery ran the trigger phrase bare, compare against it: what the skill changed is what it is for, and a line that changed nothing goes. Otherwise judge the trace against the read-back's outcome and bar. Without a real input, ask for one; if none exists, apply the canon's tests line by line and tell the user the skill has not run.
## Then
Load `ship.md`. The lenses run there and the skill improves from them; evals are offered there, not run here.
@@ -0,0 +1,9 @@
# Scaffold
For a skill the user will write themselves. Its outcome: a valid folder at `{target}` with frontmatter, manifest and starting files in place, and the user knowing what to fill in.
Load `shapes/<shape>/shape.md` to learn which `assets/` templates the shape starts from. Then `uv run {skill-root}/scripts/init_skill.py --name <name> --dest <parent folder> --shape <shape> --dirs <the folders they asked for> --description "<approved description>"`, adding `--bmod <record> --source <update_source>` when the read-back registers it as a member; a single-skill module's record comes from `shapes/single-skill-module/shape.md`. When no description was approved, leave `--description` off: the scaffold carries a `[TODO:` marker that the checks flag until they write one. Copy the shape's template files into place as starting points, placeholders left visible.
Say, in about five lines in chat: that the description is the whole trigger and where it lives; where the know-how goes; what each extra folder is for; that references stand alone, one level deep; and the canon's core test, so they cut what a capable model would do unasked. Point at `canon.md` for the rest.
Then load `ship.md`. `init_skill.py --check` fails on any `[TODO:` left, so the checks stay red until they finish, and the trigger eval waits for a real description.
@@ -0,0 +1,23 @@
# Skill-creator loop
Anthropic's skill-creator loop with BMad's shapes, packaging and harness. Its outcome: a skill iterated on measured results, its description tuned by a trigger eval.
## Draft
Scaffold (`uv run {skill-root}/scripts/init_skill.py --name <name> --dest <parent folder> --shape <shape> --description "..."`, with `--bmod <record> --source <update_source>` for a member), load `shapes/<shape>/shape.md` and the module shape the read-back's registration line names, write `SKILL.md` to `canon.md`. Make the description deliberately specific, because widening one that stays quiet is easier than narrowing one that fires everywhere.
## Measure
Write 2 to 3 realistic prompts into `{target}/evals/cases.json` with empty rubrics, from the user's real requests. Load `approaches/eval-loop.md` and run it to a stop condition.
## Description round
If `{target}/evals/triggers.json` is absent, write it to the spec in `ship.md` under Trigger eval. Then invoke the `bmad-eval` skill in trigger mode with its description-optimization loop on `{target}`; it splits, measures, revises and picks. Apply the winning description.
## BMad's step
The customization decision: does a team need to swap a rule or step without forking? Then `customize.toml` and the rendered shape, else nothing. The canon pass is the lens round in `ship.md`.
## Then
Load `ship.md`.