diff --git a/docs/calibration.md b/docs/calibration.md new file mode 100644 index 0000000..4d1011c --- /dev/null +++ b/docs/calibration.md @@ -0,0 +1,88 @@ +# Calibrating the accept rule + +`weksa loop calibrate` measures how the live accept rule behaves on a run's own +Validation corpus: how often it accepts, rejects as no gain, or regresses a +candidate that should be none of those. It does not reimplement the rule. It +reads the answered draws through `RunStores::read_draws` (the scoring +`weksa loop relower` uses), decides every trial with `decide_panel` over +`paired_floor` (`crates/weksa/src/run.rs`), and prints counts. The numbers in +`docs/targets/interlingua-lowering-map.md` ("Recalibration on the grown +corpus") come from it. It is a diagnostic: it records nothing, writes no +prompts, and never touches Test or Train. + +## What it counts + +The draws are lowerings of one round's Validation packets: the round's own +(recorded) lowering `R`, plus draws you label. `P` is `R` and the `--prompted` +draws; the floor size is `FLOOR_ARMS_MIN` (3), the live rule's. + +- Prompted trials: every ordered pair (base, candidate) in `P`, against every + 3-subset of the rest of `P` as the floor. A repeat of the same lowering + should read NoGain; Accepted is a false accept, Regressed a false reject. + (`R` comes from the round, not from `relower`, so it may not be + exchangeable with a repeat; read the pairs that involve it with that in mind.) +- Degraded trials, per `--degraded` arm: every base in `P` against every + 3-subset of the rest of `P`, the degraded draw as the candidate. These should + read Regressed. +- Candidate trials, pooled over the `--candidate` arms: the same, tallied + Accepted, NoGain, Regressed. This is the rule's power against that arm. +- Noise: the mean `panel_noise` over every (base, floor) choice. + +With six prompted draws including `R` that is 120 prompted trials, 60 per +degraded arm and 240 candidate trials. Trials share draws and are not +independent. Each count is printed for `all` Validation items and for each +`--class NAME=PREFIX[,PREFIX...]` (items whose key starts with any prefix), from +the same per-item scores. A class with fewer than two read items in a language +has no floor and reads Regressed in every trial; that is printed. + +## Steps + +1. `weksa loop open ROOT ...` and `weksa loop round ROOT` record round N and + leave its answers in `ROOT/round-N/`. +2. For each draw label: `weksa loop relower ROOT --round N --arm LABEL + --lowerer NAME` writes the byte-identical lowering prompts to + `ROOT/relower-N-LABEL/.prompt` and exits 3. Answer each prompt into + `ROOT/relower-N-LABEL/.answer`. +3. `weksa loop calibrate ROOT --round N --prompted p1 --prompted p2 ... + [--degraded LABEL]... [--candidate LABEL]... [--class NAME=PREFIX]...`. + At least four `--prompted` labels are needed (a pair and three floors with + `R` make five draws). It exits 1 and lists `unanswered LABEL COUNT` for any + draw with prompts still unanswered. Labels match `[a-z0-9-]{1,32}` and differ + across the three kinds. It needs the Ollama embedder of the run; a full + 60-item, 13-draw calibration is on the order of 1500 embedded texts, so + launch it detached with a log. + +The scratch packet store is `ROOT/calibrate-N/base.cc`; delete it at will. + +## Answer file format + +`.answer` is one JSON object and nothing else, no code fence: +`{"text": "..."}`, a JSON-escaped string. The answerer reads only the prompt +(its first line says what to write) and writes the answer. Every prompt of the +draw must be answered. + +## Briefs per arm + +Independence rule: a fresh agent per batch of answers; no batch sees another +draw's answers. Only `mild` reads `round-N/*.answer`. + +- Plain prompted: answer each prompt as its own first line asks; read nothing + else in the run root. +- Mild: for each prompt, read the recorded lowering in + `ROOT/round-N/.answer` and write the same-format answer whose text is + that utterance with exactly one meaning detail blurred: change or drop one of + tense, polarity, modality or hedge, a referent, a quantity or a predicate's + concept (replace it with a near but different word). Keep language, register, + length and fluency; the result must read as a natural utterance and must not + mention the change. A second mild draw blurs a different detail than the first + and is written without reading the first's answers. The wording of the + distil-base original is lost, and the dispatch used for growth-base is not on + disk, so this is the wording from now on. +- Train-guide: before answering, read the Train sentences of the corpus in the + batch's language as a register guide and let it steer word choice; the guide + gives phrasing only and each answer must still say exactly what its own packet + says. The guide section is the live Train items of that language, built from + `weksa corpus dump` filtered to Train, and is given to the answerer in the + brief, never pasted into this document, and never includes Validation or Test + text. Read nothing else in the run root. +- Control: a script writes `{"text":"Yes."}` as the answer to every prompt. diff --git a/docs/targets/interlingua-lowering-map.md b/docs/targets/interlingua-lowering-map.md index 3fc873c..e8d4313 100644 --- a/docs/targets/interlingua-lowering-map.md +++ b/docs/targets/interlingua-lowering-map.md @@ -505,11 +505,19 @@ findings shaped the wording. `validation-growth-writer-claude-checked`). The new scenes use a different `in_group` convention from the old ones (follow-up `vgrow-scene-in-group-convention`), so the new-item scores are for scenes that - will be re-annotated; the corpus digest changes again then. + will be re-annotated; the speaker-state store changes then (a run pins its + digest), the corpus does not. -Evidence lives in the Self workspace (`cal2/`, `runs/growth-base`), not in git, -and uses a DUMP patch kept out of the repo; no committed harness reruns these -numbers (Soul finding `evidence-outside-git`, follow-up owed). +These numbers were first computed by scripts outside git (Soul finding +`evidence-outside-git`). `weksa loop calibrate` (`docs/calibration.md`) +computes them from `run.rs`'s own scoring and accept rule, and its rerun on +`runs/growth-base` reproduces them: prompted 116 NoGain, 4 Accepted, 0 +Regressed of 120; control-yes 60 of 60 Regressed, mild-1 19 of 60, mild-2 0 of +60; the Train-guide arm 51 Accepted, 189 NoGain, 0 Regressed of 240; noise term +0.0451; items en-US n=28 skipped=7, ja-JP n=32 skipped=3. By class the harness +adds: old items (`item:original*`) prompted 111/9/0, noise 0.1069, Train-guide +71 Accepted of 240; new items (`item:vgrow*`) prompted 116/4/0, noise 0.0334, +Train-guide 0 Accepted and 74 Regressed of 240. ### Smallest measurable distil slice (`distil-incontext`, not built)