diff --git a/docs/calibration.md b/docs/calibration.md index 4d1011c..45a1c5a 100644 --- a/docs/calibration.md +++ b/docs/calibration.md @@ -5,10 +5,16 @@ Validation corpus: how often it accepts, rejects as no gain, or regresses a candidate that should be none of those. It does not reimplement the rule. It reads the answered draws through `RunStores::read_draws` (the scoring `weksa loop relower` uses), decides every trial with `decide_panel` over -`paired_floor` (`crates/weksa/src/run.rs`), and prints counts. The numbers in -`docs/targets/interlingua-lowering-map.md` ("Recalibration on the grown -corpus") come from it. It is a diagnostic: it records nothing, writes no -prompts, and never touches Test or Train. +`paired_floor` (`crates/weksa/src/run.rs`), and prints, per class, the items +read and skipped per language, the trial totals with their Accepted, NoGain and +Regressed counts, and the mean panel noise. That is all it produces: the +Accepted/NoGain/Regressed counts and the noise mean in the map's "Recalibration +on the grown corpus" block are reproduced by it; the other figures there (the +35% with R removed, the noise range, the 0.0185 gain and its 0.41 ratio, the +bootstrap intervals, the per-item SDs, R's +0.024 and the mild-arm duplicate +counts) were computed by scratch scripts that are not in the repo. It is a +diagnostic: it records nothing, writes no prompts, and never touches Test or +Train. ## What it counts @@ -74,10 +80,14 @@ draw's answers. Only `mild` reads `round-N/*.answer`. tense, polarity, modality or hedge, a referent, a quantity or a predicate's concept (replace it with a near but different word). Keep language, register, length and fluency; the result must read as a natural utterance and must not - mention the change. A second mild draw blurs a different detail than the first - and is written without reading the first's answers. The wording of the - distil-base original is lost, and the dispatch used for growth-base is not on - disk, so this is the wording from now on. + mention the change. Pick the detail that is not the most salient one. A + second mild draw blurs a different detail than the first and is written + without reading the first's answers. The wording of the distil-base original + is lost, and the dispatch used for growth-base is not on disk, so this is the + wording from now on. Warning: on growth-base two mild arms briefed this way + came out identical in 27 of 60 items, so independently briefed mild arms + are not independent degradations; read a mild draw's regress count with that + in mind. - Train-guide: before answering, read the Train sentences of the corpus in the batch's language as a register guide and let it steer word choice; the guide gives phrasing only and each answer must still say exactly what its own packet diff --git a/docs/targets/interlingua-lowering-map.md b/docs/targets/interlingua-lowering-map.md index e8d4313..a4e85ce 100644 --- a/docs/targets/interlingua-lowering-map.md +++ b/docs/targets/interlingua-lowering-map.md @@ -508,16 +508,19 @@ findings shaped the wording. will be re-annotated; the speaker-state store changes then (a run pins its digest), the corpus does not. -These numbers were first computed by scripts outside git (Soul finding -`evidence-outside-git`). `weksa loop calibrate` (`docs/calibration.md`) -computes them from `run.rs`'s own scoring and accept rule, and its rerun on -`runs/growth-base` reproduces them: prompted 116 NoGain, 4 Accepted, 0 -Regressed of 120; control-yes 60 of 60 Regressed, mild-1 19 of 60, mild-2 0 of -60; the Train-guide arm 51 Accepted, 189 NoGain, 0 Regressed of 240; noise term -0.0451; items en-US n=28 skipped=7, ja-JP n=32 skipped=3. By class the harness -adds: old items (`item:original*`) prompted 111/9/0, noise 0.1069, Train-guide -71 Accepted of 240; new items (`item:vgrow*`) prompted 116/4/0, noise 0.0334, -Train-guide 0 Accepted and 74 Regressed of 240. +`weksa loop calibrate` (`docs/calibration.md`) reproduces from `run.rs`'s own +scoring and accept rule the counts per class, the trial totals and the mean +noise of this block: prompted 116 NoGain, 4 Accepted, 0 Regressed of 120; +control-yes 60 of 60 Regressed, mild-1 19 of 60, mild-2 0 of 60; the +Train-guide arm 51 Accepted, 189 NoGain, 0 Regressed of 240; noise 0.0451; +items en-US n=28 skipped=7, ja-JP n=32 skipped=3. By class it adds: old items +(`item:original*`) prompted 111/9/0, noise 0.1069, Train-guide 71 Accepted of +240; new items (`item:vgrow*`) prompted 116/4/0, noise 0.0334, Train-guide 0 +Accepted and 74 Regressed of 240. The other figures of this block (35% with R +removed, the noise range, the 0.0185 gain and 0.41 ratio, the bootstrap +intervals, the per-item SDs, R's +0.024, the mild-arm duplicate counts) were +computed by scratch scripts outside git (Soul finding `evidence-outside-git`) +and are not reproduced by the harness. ### Smallest measurable distil slice (`distil-incontext`, not built)