docs: calibration procedure; the map's evidence paragraph points at weksa loop calibrate and states its reproduction

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Weksa 2026-11-13 08:08:15 +00:00
commit 451e263db1
2 changed files with 101 additions and 5 deletions

88
docs/calibration.md Normal file
View file

@ -0,0 +1,88 @@
# Calibrating the accept rule
`weksa loop calibrate` measures how the live accept rule behaves on a run's own
Validation corpus: how often it accepts, rejects as no gain, or regresses a
candidate that should be none of those. It does not reimplement the rule. It
reads the answered draws through `RunStores::read_draws` (the scoring
`weksa loop relower` uses), decides every trial with `decide_panel` over
`paired_floor` (`crates/weksa/src/run.rs`), and prints counts. The numbers in
`docs/targets/interlingua-lowering-map.md` ("Recalibration on the grown
corpus") come from it. It is a diagnostic: it records nothing, writes no
prompts, and never touches Test or Train.
## What it counts
The draws are lowerings of one round's Validation packets: the round's own
(recorded) lowering `R`, plus draws you label. `P` is `R` and the `--prompted`
draws; the floor size is `FLOOR_ARMS_MIN` (3), the live rule's.
- Prompted trials: every ordered pair (base, candidate) in `P`, against every
3-subset of the rest of `P` as the floor. A repeat of the same lowering
should read NoGain; Accepted is a false accept, Regressed a false reject.
(`R` comes from the round, not from `relower`, so it may not be
exchangeable with a repeat; read the pairs that involve it with that in mind.)
- Degraded trials, per `--degraded` arm: every base in `P` against every
3-subset of the rest of `P`, the degraded draw as the candidate. These should
read Regressed.
- Candidate trials, pooled over the `--candidate` arms: the same, tallied
Accepted, NoGain, Regressed. This is the rule's power against that arm.
- Noise: the mean `panel_noise` over every (base, floor) choice.
With six prompted draws including `R` that is 120 prompted trials, 60 per
degraded arm and 240 candidate trials. Trials share draws and are not
independent. Each count is printed for `all` Validation items and for each
`--class NAME=PREFIX[,PREFIX...]` (items whose key starts with any prefix), from
the same per-item scores. A class with fewer than two read items in a language
has no floor and reads Regressed in every trial; that is printed.
## Steps
1. `weksa loop open ROOT ...` and `weksa loop round ROOT` record round N and
leave its answers in `ROOT/round-N/`.
2. For each draw label: `weksa loop relower ROOT --round N --arm LABEL
--lowerer NAME` writes the byte-identical lowering prompts to
`ROOT/relower-N-LABEL/<digest>.prompt` and exits 3. Answer each prompt into
`ROOT/relower-N-LABEL/<digest>.answer`.
3. `weksa loop calibrate ROOT --round N --prompted p1 --prompted p2 ...
[--degraded LABEL]... [--candidate LABEL]... [--class NAME=PREFIX]...`.
At least four `--prompted` labels are needed (a pair and three floors with
`R` make five draws). It exits 1 and lists `unanswered LABEL COUNT` for any
draw with prompts still unanswered. Labels match `[a-z0-9-]{1,32}` and differ
across the three kinds. It needs the Ollama embedder of the run; a full
60-item, 13-draw calibration is on the order of 1500 embedded texts, so
launch it detached with a log.
The scratch packet store is `ROOT/calibrate-N/base.cc`; delete it at will.
## Answer file format
`<digest>.answer` is one JSON object and nothing else, no code fence:
`{"text": "..."}`, a JSON-escaped string. The answerer reads only the prompt
(its first line says what to write) and writes the answer. Every prompt of the
draw must be answered.
## Briefs per arm
Independence rule: a fresh agent per batch of answers; no batch sees another
draw's answers. Only `mild` reads `round-N/*.answer`.
- Plain prompted: answer each prompt as its own first line asks; read nothing
else in the run root.
- Mild: for each prompt, read the recorded lowering in
`ROOT/round-N/<digest>.answer` and write the same-format answer whose text is
that utterance with exactly one meaning detail blurred: change or drop one of
tense, polarity, modality or hedge, a referent, a quantity or a predicate's
concept (replace it with a near but different word). Keep language, register,
length and fluency; the result must read as a natural utterance and must not
mention the change. A second mild draw blurs a different detail than the first
and is written without reading the first's answers. The wording of the
distil-base original is lost, and the dispatch used for growth-base is not on
disk, so this is the wording from now on.
- Train-guide: before answering, read the Train sentences of the corpus in the
batch's language as a register guide and let it steer word choice; the guide
gives phrasing only and each answer must still say exactly what its own packet
says. The guide section is the live Train items of that language, built from
`weksa corpus dump` filtered to Train, and is given to the answerer in the
brief, never pasted into this document, and never includes Validation or Test
text. Read nothing else in the run root.
- Control: a script writes `{"text":"Yes."}` as the answer to every prompt.

View file

@ -505,11 +505,19 @@ findings shaped the wording.
`validation-growth-writer-claude-checked`). The new scenes use a different
`in_group` convention from the old ones (follow-up
`vgrow-scene-in-group-convention`), so the new-item scores are for scenes that
will be re-annotated; the corpus digest changes again then.
will be re-annotated; the speaker-state store changes then (a run pins its
digest), the corpus does not.
Evidence lives in the Self workspace (`cal2/`, `runs/growth-base`), not in git,
and uses a DUMP patch kept out of the repo; no committed harness reruns these
numbers (Soul finding `evidence-outside-git`, follow-up owed).
These numbers were first computed by scripts outside git (Soul finding
`evidence-outside-git`). `weksa loop calibrate` (`docs/calibration.md`)
computes them from `run.rs`'s own scoring and accept rule, and its rerun on
`runs/growth-base` reproduces them: prompted 116 NoGain, 4 Accepted, 0
Regressed of 120; control-yes 60 of 60 Regressed, mild-1 19 of 60, mild-2 0 of
60; the Train-guide arm 51 Accepted, 189 NoGain, 0 Regressed of 240; noise term
0.0451; items en-US n=28 skipped=7, ja-JP n=32 skipped=3. By class the harness
adds: old items (`item:original*`) prompted 111/9/0, noise 0.1069, Train-guide
71 Accepted of 240; new items (`item:vgrow*`) prompted 116/4/0, noise 0.0334,
Train-guide 0 Accepted and 74 Regressed of 240.
### Smallest measurable distil slice (`distil-incontext`, not built)