From 915863b5b301987fa8f69daa3d058e9fef125565 Mon Sep 17 00:00:00 2001 From: Weksa Date: Fri, 13 Nov 2026 07:24:51 +0000 Subject: [PATCH 01/22] map: recalibration on the 35-item Validation corpus; the Train-guide gain was in-sample Co-Authored-By: Claude Sonnet 5.5 --- docs/targets/interlingua-lowering-map.md | 63 ++++++++++++++++++++++-- 1 file changed, 59 insertions(+), 4 deletions(-) diff --git a/docs/targets/interlingua-lowering-map.md b/docs/targets/interlingua-lowering-map.md index 0182cc0..046773f 100644 --- a/docs/targets/interlingua-lowering-map.md +++ b/docs/targets/interlingua-lowering-map.md @@ -406,7 +406,7 @@ of 1120 trials, NoGain 71%, Regressed 0%: a net gain of 0.079 against a mean noise term of 0.079. At 14 items per language the rule is therefore a low-power filter for a gain of this size; by 1/sqrt(n), about 35 or more items per language would bring the noise term near 0.05 (an extrapolation, not measured). -### Validation growth (`validation-growth`, built; recalibration not run) +### Validation growth (`validation-growth`, built and recalibrated) Power needs about 35 or more Validation items per language. `data/original` now holds 35 per language (it held 14; Train 14 and Test 8 are unchanged): @@ -441,9 +441,64 @@ packets will be refused at encode (inventory coverage) and score 0 as the means do; they stay in the corpus. That is a result about inventory generalization, not a defect. -Every calibration number above (92/0/8 prompted rates, the 29% power, the 0.079 -noise term) is for the 14-item corpus; the corpus digest changed, and the -numbers are redone on the grown corpus in a separate recalibration step. +Every calibration number in the Distil measure section above (92/0/8 prompted +rates, the 29% power, the 0.079 noise term) is for the 14-item corpus and stays +the stated baseline of that corpus. + +Recalibration on the grown corpus (`runs/growth-base`, main 44b66f9, panel +en-US and ja-JP, round 1 recorded by briefed subagents; 6 prompted draws +including the recorded one, 2 mild degradations, an all-"Yes." control and 4 +Train-guide draws by Claude sonnet-5-5; the same decision rule at FLOOR_Z 3 and +3-arm floors; 120 prompted trials from the 30 ordered pairs of 6 draws with the +other 4 as floor candidates, so trials share draws and are not independent): + +- Live Validation items: 28 en-US and 32 ja-JP of 35 each; 10 are refused at + parse (en-US original:18, 23, 25, 27 and vgrow:4, 5, 6; ja-JP original-ja:17, + 22 and vgrow-ja:2) and score 0 in every arm. Old 14: 10 en-US and 12 ja-JP + live; new 21: 18 and 20. +- Prompted repeat: NoGain 96.7%, Accepted 3.3%, Regressed 0% (14 items: 92 / 0 / + 8). The false-accept bound is no better than before: 3.3% rests on 120 trials + over 6 draws. +- Degraded arms read Regressed: all-"Yes." 100% (as before); mild-1 31.7%; + mild-2 0% (14 items: 100% and 75% on a subtler arm). Not comparable: these + mild arms were rewritten from the recorded lowering by blurring one detail + and move the mean composite by only 0.003 to 0.015, against 0.044 for the old + subtler arm; the original mild wording is not on disk, so this is a weaker + degradation, and the rule's sensitivity to a real small regression is + unchanged in kind but not re-measured at the old strength. +- Noise: the panel noise term of the sum rule is 0.045 on average over the + base and floor choices (0.031 to 0.064), against 0.079 at 14 items; by + 1/sqrt(n) the extrapolation said near 0.05. That part of the extrapolation + held. +- Power: the Train-guide arm (14 Train sentences as a register guide) is + Accepted in 21.2% of 240 trials, NoGain 78.8%, against 29% at 14 items, and + its mean gain over the prompted draws fell from 0.079 to 0.018 (gain over + noise 0.37). Split by item class (paired bootstrap over items, 95%): the + gain is on the old Validation items, composite +0.029 [+0.005, +0.052] (en-US + +0.049, ja-JP +0.013 with an interval through zero; 22 items), and nil on the + new ones, -0.002 [-0.011, +0.007] (en-US -0.017 [-0.027, -0.007], ja-JP + +0.011; 38 items). The old items share a writer, a vocabulary and a register + with Train; the new ones were written free of both. So the 14-item power + figure measured a gain that was in-sample register fit on text shaped like + the guide, and the "better arm" premise does not hold on independent + Validation. This is a measured difference; its cause is inferred, not + isolated. The power of the rule against a gain that exists is therefore not + re-measured: no better arm is known on the new items. +- The two classes are different strata (Soul s2 `register-stratum-shift`): the + new items score lower (composite 0.41 against 0.64 pooled) and vary less per + item between draws (SD of the difference between two prompted draws 0.034 + against 0.085), so a pooled 60-item mean mixes them; report old and new + separately. +- The self-preference effect is not measured (ruling + `validation-growth-writer-claude-checked`). The new scenes use a different + `in_group` convention from the old ones (follow-up + `vgrow-scene-in-group-convention`), so the new-item scores are for scenes that + will be re-annotated; the corpus digest changes again then. + +Measured: everything listed above. Extrapolated: nothing new; the 1/sqrt(n) +noise forecast is now a measurement, the old power figure is withdrawn as +in-sample. Evidence: `cal2/` and `runs/growth-base` in the Weksa Self +workspace, not in git. ### Smallest measurable distil slice (`distil-incontext`, not built) From 2b29efbc2d9828ae2b782c472f6f5bb022e50dbf Mon Sep 17 00:00:00 2001 From: Weksa Date: Fri, 13 Nov 2026 07:33:08 +0000 Subject: [PATCH 02/22] map: recalibration wording after Soul s3 (ja effect, R, mild arms, noise stratum) Co-Authored-By: Claude Sonnet 5.5 --- docs/targets/interlingua-lowering-map.md | 81 ++++++++++++++---------- 1 file changed, 46 insertions(+), 35 deletions(-) diff --git a/docs/targets/interlingua-lowering-map.md b/docs/targets/interlingua-lowering-map.md index 046773f..3fc873c 100644 --- a/docs/targets/interlingua-lowering-map.md +++ b/docs/targets/interlingua-lowering-map.md @@ -450,55 +450,66 @@ en-US and ja-JP, round 1 recorded by briefed subagents; 6 prompted draws including the recorded one, 2 mild degradations, an all-"Yes." control and 4 Train-guide draws by Claude sonnet-5-5; the same decision rule at FLOOR_Z 3 and 3-arm floors; 120 prompted trials from the 30 ordered pairs of 6 draws with the -other 4 as floor candidates, so trials share draws and are not independent): +other 4 as floor candidates, so trials share draws and are not independent). +Soul pass `cut-validation-growth.s3` re-derived every number below; its +findings shaped the wording. - Live Validation items: 28 en-US and 32 ja-JP of 35 each; 10 are refused at parse (en-US original:18, 23, 25, 27 and vgrow:4, 5, 6; ja-JP original-ja:17, - 22 and vgrow-ja:2) and score 0 in every arm. Old 14: 10 en-US and 12 ja-JP - live; new 21: 18 and 20. + 22 and vgrow-ja:2); `relower` skips an item with no packet, so every reading + here is over live items only. Old 14: 10 en-US and 12 ja-JP live; new 21: 18 + and 20. - Prompted repeat: NoGain 96.7%, Accepted 3.3%, Regressed 0% (14 items: 92 / 0 / - 8). The false-accept bound is no better than before: 3.3% rests on 120 trials - over 6 draws. + 8). All four false accepts are one pair, base p6 and candidate the recorded + lowering R: R comes from the round, the other five draws from `relower`, and + in en-US R scores +0.024 per item above them [+0.002, +0.052], so R is not + exchangeable with a repeat (a Plausible Soul finding, cause not isolated). No + false-accept bound can be drawn from 6 draws. - Degraded arms read Regressed: all-"Yes." 100% (as before); mild-1 31.7%; - mild-2 0% (14 items: 100% and 75% on a subtler arm). Not comparable: these - mild arms were rewritten from the recorded lowering by blurring one detail - and move the mean composite by only 0.003 to 0.015, against 0.044 for the old - subtler arm; the original mild wording is not on disk, so this is a weaker - degradation, and the rule's sensitivity to a real small regression is - unchanged in kind but not re-measured at the old strength. -- Noise: the panel noise term of the sum rule is 0.045 on average over the - base and floor choices (0.031 to 0.064), against 0.079 at 14 items; by - 1/sqrt(n) the extrapolation said near 0.05. That part of the extrapolation - held. + mild-2 0% (14 items: 100% and 75% on a subtler arm). The mild arms are weak + and not independent: they were rewritten from R with one detail blurred, the + original wording is not on disk, mild-2 equals R word for word in 5 of 60 + items and mild-1 in 2, and mild-1 equals mild-2 in 27 of 60 although they + were asked to blur different details. Their mean composite is 0.003 to 0.015 + below the prompted mean (the old subtler arm: 0.037 and 0.052). The + sensitivity to a small regression is not re-measured. +- Noise: the panel noise term of the sum rule is 0.045 on average over the base + and floor choices (0.031 to 0.064), against 0.079 at 14 items. The 1/sqrt(n) + forecast (about 0.05) assumed per-item variance stayed the same; it did not: + the per-item SD of the difference between two prompted draws is about 0.085 + on the old items and 0.034 on the new ones, so the pooled term landed near + the forecast because less variable items were added. Not a confirmation of + the forecast. - Power: the Train-guide arm (14 Train sentences as a register guide) is Accepted in 21.2% of 240 trials, NoGain 78.8%, against 29% at 14 items, and - its mean gain over the prompted draws fell from 0.079 to 0.018 (gain over - noise 0.37). Split by item class (paired bootstrap over items, 95%): the - gain is on the old Validation items, composite +0.029 [+0.005, +0.052] (en-US - +0.049, ja-JP +0.013 with an interval through zero; 22 items), and nil on the - new ones, -0.002 [-0.011, +0.007] (en-US -0.017 [-0.027, -0.007], ja-JP - +0.011; 38 items). The old items share a writer, a vocabulary and a register - with Train; the new ones were written free of both. So the 14-item power - figure measured a gain that was in-sample register fit on text shaped like - the guide, and the "better arm" premise does not hold on independent - Validation. This is a measured difference; its cause is inferred, not - isolated. The power of the rule against a gain that exists is therefore not - re-measured: no better arm is known on the new items. + its mean composite gain over the prompted draws is 0.0185, 0.41 of the mean + noise term (0.37 on the first floor choice per base); with `relower`-only + bases (R removed) it is Accepted in 35% (28 of 80). Split by item class + (paired bootstrap over items, 95%): old items +0.029 [+0.005, +0.052] (en-US + +0.049 [+0.023, +0.084] on 10 items, ja-JP +0.013 [-0.021, +0.037] on 12); + new items -0.002 [-0.011, +0.007] pooled, but two opposite effects: en-US + -0.017 [-0.027, -0.007] on 18 items and ja-JP +0.011 [+0.001, +0.022] on 20. + So the old-versus-new contrast exists in en-US; in ja-JP old (+0.013) and new + (+0.011) do not differ. Measured: the 14-item gain does not carry to the new + en-US items. Inferred, not isolated: that it was in-sample register fit. The + new items differ from the old in several ways at once (written free of the + inventory, longer, a different scene convention, lower scores, a smaller + vocabulary overlap with Train in en-US: 0.29 against 0.52) and were briefed to + the same register as Train, by Claude as the old were. The 29% power figure + stays the stated baseline of the 14-item corpus; no better arm is known on + the new items, so the rule's power against a real gain is not re-measured. - The two classes are different strata (Soul s2 `register-stratum-shift`): the - new items score lower (composite 0.41 against 0.64 pooled) and vary less per - item between draws (SD of the difference between two prompted draws 0.034 - against 0.085), so a pooled 60-item mean mixes them; report old and new - separately. + new items score lower (composite 0.41 against 0.64 pooled) so a pooled + 60-item mean mixes them; report old and new separately. - The self-preference effect is not measured (ruling `validation-growth-writer-claude-checked`). The new scenes use a different `in_group` convention from the old ones (follow-up `vgrow-scene-in-group-convention`), so the new-item scores are for scenes that will be re-annotated; the corpus digest changes again then. -Measured: everything listed above. Extrapolated: nothing new; the 1/sqrt(n) -noise forecast is now a measurement, the old power figure is withdrawn as -in-sample. Evidence: `cal2/` and `runs/growth-base` in the Weksa Self -workspace, not in git. +Evidence lives in the Self workspace (`cal2/`, `runs/growth-base`), not in git, +and uses a DUMP patch kept out of the repo; no committed harness reruns these +numbers (Soul finding `evidence-outside-git`, follow-up owed). ### Smallest measurable distil slice (`distil-incontext`, not built) From 76fa6856b2d939a99c0890a6137e58fd4cfb0916 Mon Sep 17 00:00:00 2001 From: Weksa Date: Fri, 13 Nov 2026 07:33:17 +0000 Subject: [PATCH 03/22] map: recalibration on the 35-item Validation corpus (#76) Co-Authored-By: Claude Sonnet 5.5 --- docs/targets/interlingua-lowering-map.md | 74 ++++++++++++++++++++++-- 1 file changed, 70 insertions(+), 4 deletions(-) diff --git a/docs/targets/interlingua-lowering-map.md b/docs/targets/interlingua-lowering-map.md index 0182cc0..3fc873c 100644 --- a/docs/targets/interlingua-lowering-map.md +++ b/docs/targets/interlingua-lowering-map.md @@ -406,7 +406,7 @@ of 1120 trials, NoGain 71%, Regressed 0%: a net gain of 0.079 against a mean noise term of 0.079. At 14 items per language the rule is therefore a low-power filter for a gain of this size; by 1/sqrt(n), about 35 or more items per language would bring the noise term near 0.05 (an extrapolation, not measured). -### Validation growth (`validation-growth`, built; recalibration not run) +### Validation growth (`validation-growth`, built and recalibrated) Power needs about 35 or more Validation items per language. `data/original` now holds 35 per language (it held 14; Train 14 and Test 8 are unchanged): @@ -441,9 +441,75 @@ packets will be refused at encode (inventory coverage) and score 0 as the means do; they stay in the corpus. That is a result about inventory generalization, not a defect. -Every calibration number above (92/0/8 prompted rates, the 29% power, the 0.079 -noise term) is for the 14-item corpus; the corpus digest changed, and the -numbers are redone on the grown corpus in a separate recalibration step. +Every calibration number in the Distil measure section above (92/0/8 prompted +rates, the 29% power, the 0.079 noise term) is for the 14-item corpus and stays +the stated baseline of that corpus. + +Recalibration on the grown corpus (`runs/growth-base`, main 44b66f9, panel +en-US and ja-JP, round 1 recorded by briefed subagents; 6 prompted draws +including the recorded one, 2 mild degradations, an all-"Yes." control and 4 +Train-guide draws by Claude sonnet-5-5; the same decision rule at FLOOR_Z 3 and +3-arm floors; 120 prompted trials from the 30 ordered pairs of 6 draws with the +other 4 as floor candidates, so trials share draws and are not independent). +Soul pass `cut-validation-growth.s3` re-derived every number below; its +findings shaped the wording. + +- Live Validation items: 28 en-US and 32 ja-JP of 35 each; 10 are refused at + parse (en-US original:18, 23, 25, 27 and vgrow:4, 5, 6; ja-JP original-ja:17, + 22 and vgrow-ja:2); `relower` skips an item with no packet, so every reading + here is over live items only. Old 14: 10 en-US and 12 ja-JP live; new 21: 18 + and 20. +- Prompted repeat: NoGain 96.7%, Accepted 3.3%, Regressed 0% (14 items: 92 / 0 / + 8). All four false accepts are one pair, base p6 and candidate the recorded + lowering R: R comes from the round, the other five draws from `relower`, and + in en-US R scores +0.024 per item above them [+0.002, +0.052], so R is not + exchangeable with a repeat (a Plausible Soul finding, cause not isolated). No + false-accept bound can be drawn from 6 draws. +- Degraded arms read Regressed: all-"Yes." 100% (as before); mild-1 31.7%; + mild-2 0% (14 items: 100% and 75% on a subtler arm). The mild arms are weak + and not independent: they were rewritten from R with one detail blurred, the + original wording is not on disk, mild-2 equals R word for word in 5 of 60 + items and mild-1 in 2, and mild-1 equals mild-2 in 27 of 60 although they + were asked to blur different details. Their mean composite is 0.003 to 0.015 + below the prompted mean (the old subtler arm: 0.037 and 0.052). The + sensitivity to a small regression is not re-measured. +- Noise: the panel noise term of the sum rule is 0.045 on average over the base + and floor choices (0.031 to 0.064), against 0.079 at 14 items. The 1/sqrt(n) + forecast (about 0.05) assumed per-item variance stayed the same; it did not: + the per-item SD of the difference between two prompted draws is about 0.085 + on the old items and 0.034 on the new ones, so the pooled term landed near + the forecast because less variable items were added. Not a confirmation of + the forecast. +- Power: the Train-guide arm (14 Train sentences as a register guide) is + Accepted in 21.2% of 240 trials, NoGain 78.8%, against 29% at 14 items, and + its mean composite gain over the prompted draws is 0.0185, 0.41 of the mean + noise term (0.37 on the first floor choice per base); with `relower`-only + bases (R removed) it is Accepted in 35% (28 of 80). Split by item class + (paired bootstrap over items, 95%): old items +0.029 [+0.005, +0.052] (en-US + +0.049 [+0.023, +0.084] on 10 items, ja-JP +0.013 [-0.021, +0.037] on 12); + new items -0.002 [-0.011, +0.007] pooled, but two opposite effects: en-US + -0.017 [-0.027, -0.007] on 18 items and ja-JP +0.011 [+0.001, +0.022] on 20. + So the old-versus-new contrast exists in en-US; in ja-JP old (+0.013) and new + (+0.011) do not differ. Measured: the 14-item gain does not carry to the new + en-US items. Inferred, not isolated: that it was in-sample register fit. The + new items differ from the old in several ways at once (written free of the + inventory, longer, a different scene convention, lower scores, a smaller + vocabulary overlap with Train in en-US: 0.29 against 0.52) and were briefed to + the same register as Train, by Claude as the old were. The 29% power figure + stays the stated baseline of the 14-item corpus; no better arm is known on + the new items, so the rule's power against a real gain is not re-measured. +- The two classes are different strata (Soul s2 `register-stratum-shift`): the + new items score lower (composite 0.41 against 0.64 pooled) so a pooled + 60-item mean mixes them; report old and new separately. +- The self-preference effect is not measured (ruling + `validation-growth-writer-claude-checked`). The new scenes use a different + `in_group` convention from the old ones (follow-up + `vgrow-scene-in-group-convention`), so the new-item scores are for scenes that + will be re-annotated; the corpus digest changes again then. + +Evidence lives in the Self workspace (`cal2/`, `runs/growth-base`), not in git, +and uses a DUMP patch kept out of the repo; no committed harness reruns these +numbers (Soul finding `evidence-outside-git`, follow-up owed). ### Smallest measurable distil slice (`distil-incontext`, not built) From ba42639c3bba281226a8694b58d0d77c1ee2561c Mon Sep 17 00:00:00 2001 From: Weksa Date: Fri, 13 Nov 2026 08:24:45 +0000 Subject: [PATCH 04/22] scene-in-group: rubric r1 and 98 labelled scenes (#77) Co-Authored-By: Claude Sonnet 5.5 --- README.md | 4 +- crates/weksa/tests/original_corpus.rs | 83 +++++++++++++++- data/original/speaker-state.cc | Bin 32508 -> 60158 bytes docs/scene-annotation.md | 116 +++++++++++++++++++++++ scripts/scene-annotation/agreement.py | 99 +++++++++++++++++++ scripts/scene-annotation/build_briefs.py | 37 ++++++++ scripts/scene-annotation/put_labels.py | 15 +++ 7 files changed, 349 insertions(+), 5 deletions(-) create mode 100644 docs/scene-annotation.md create mode 100644 scripts/scene-annotation/agreement.py create mode 100644 scripts/scene-annotation/build_briefs.py create mode 100644 scripts/scene-annotation/put_labels.py diff --git a/README.md b/README.md index ff6eadd..2b5d0d3 100644 --- a/README.md +++ b/README.md @@ -5,8 +5,8 @@ culture, and constraint, and the planned lowerer that turns an agent's meaning packet into the reader's display language. **What is valuable here.** The interlingua standard and the lowering design: a -meaning packet owns what is meant, the scene state (live: `weksa scene put|dump`; speaker -documents are planned) owns the speaker's relation to the addressee, which selects register, +meaning packet owns what is meant, the scene state (live: `weksa scene put|dump`, labelled by the rule in +`docs/scene-annotation.md`; speaker documents are planned) owns the speaker's relation to the addressee, which selects register, address and politeness, and a lowerer owns surface text and leaves a trace. **The problem it solves.** An agent's outward speech today is one English diff --git a/crates/weksa/tests/original_corpus.rs b/crates/weksa/tests/original_corpus.rs index a3eae63..44af0cc 100644 --- a/crates/weksa/tests/original_corpus.rs +++ b/crates/weksa/tests/original_corpus.rs @@ -1039,7 +1039,7 @@ fn committed_scenes_cover_original_groups() { let held: BTreeSet = scenes.iter().map(|s| s.group.clone()).collect(); assert_eq!(held, live, "a scene for each live Train or Validation group, none other"); // Revisions append: a group's later revisions re-derived its relation. - assert!(scenes.iter().all(|s| s.n <= 2)); + assert!(scenes.iter().all(|s| s.n <= 3)); const S7: &str = "claude-sonnet-5-5 (Weksa Hands, model-annotated original scenes)"; const GROWTH: &str = "claude-sonnet-5-5 (Weksa Hands, model-annotated validation-growth scenes; annotator is not the writer)"; let grown: BTreeSet = CorpusStore::open(&data.join("corpus.cc")) @@ -1051,16 +1051,31 @@ fn committed_scenes_cover_original_groups() { .map(|i| i.group) .collect(); assert_eq!(grown.len(), 42); + const RUBRIC: &str = + "claude-sonnet-5-5 (Weksa Hands, scene rubric r1, docs/scene-annotation.md)"; + assert!( + Path::new(env!("CARGO_MANIFEST_DIR")) + .join("../../docs/scene-annotation.md") + .is_file(), + "the rubric r1 names a doc, and the doc is in the repo" + ); let mut by_growth = BTreeSet::new(); for s in &scenes { match &s.annotator { Annotator::Model { family, generator } => { assert_eq!(*family, ModelFamily::Anthropic, "{}", s.group); - assert!(generator == S7 || generator == GROWTH, "{}", s.group); + assert!( + generator == S7 || generator == GROWTH || generator == RUBRIC, + "{}", + s.group + ); if generator == GROWTH { by_growth.insert(s.group.clone()); - // One relation, keeper to unnamed, the three Scene features. + // Written once, never revised by its own generator. assert_eq!(s.n, 1, "{}", s.group); + } + if generator == GROWTH || generator == RUBRIC { + // One relation, keeper to unnamed, the three Scene features. assert_eq!(s.relations.len(), 1, "{}", s.group); let rel = &s.relations[0]; assert_eq!(rel.from.as_str(), "entity:keeper", "{}", s.group); @@ -1075,8 +1090,70 @@ fn committed_scenes_cover_original_groups() { by_growth, grown, "the growth scenes are exactly the vgrow groups" ); + // The label a run reads is each group's latest revision: all 98 are rubric r1's. + let mut latest = std::collections::BTreeMap::new(); + for s in &scenes { + if latest.get(&s.group).is_none_or(|l: &&weksa::speaker::Scene| l.n < s.n) { + latest.insert(s.group.clone(), s); + } + } + let mut lines = Vec::new(); + for (group, s) in &latest { + let Annotator::Model { generator, .. } = &s.annotator else { + panic!("{group} is not model-annotated"); + }; + assert_eq!(generator, RUBRIC, "{group}: the latest revision follows rubric r1"); + lines.push(format!("{group}\n{}\n{:?}", s.n, s.relations[0].features)); + } + let parts: Vec<&[u8]> = lines.iter().map(|l| l.as_bytes()).collect(); + assert_eq!( + weksa::split::fnv1a64_of(&parts), + DIGEST_OF_THE_SCENES, + "a latest scene changed without a rubric revision" + ); + // The revisions before rubric r1 are provenance and never change: 113 of them. + let mut earlier: Vec<&weksa::speaker::Scene> = scenes + .iter() + .filter(|s| !matches!(&s.annotator, Annotator::Model { generator, .. } if generator == RUBRIC)) + .collect(); + earlier.sort_by(|a, b| (&a.group, a.n).cmp(&(&b.group, b.n))); + assert_eq!(earlier.len(), 113); + let lines: Vec = earlier + .iter() + .map(|s| format!("{}\n{}\n{:?}\n{:?}", s.group, s.n, s.annotator, s.relations)) + .collect(); + let parts: Vec<&[u8]> = lines.iter().map(|l| l.as_bytes()).collect(); + assert_eq!( + weksa::split::fnv1a64_of(&parts), + DIGEST_OF_THE_EARLIER_SCENES, + "an earlier scene revision changed" + ); + // Rubric r1 is text the labels cite: the section, as committed, is pinned. + let doc = std::fs::read_to_string( + Path::new(env!("CARGO_MANIFEST_DIR")).join("../../docs/scene-annotation.md"), + ) + .unwrap(); + let from = doc.find("## Rubric r1").expect("the doc holds Rubric r1"); + let rest = &doc[from + "## Rubric r1".len()..]; + let section = &doc[from..from + "## Rubric r1".len() + rest.find("\n## ").expect("a section follows r1")]; + assert_eq!( + weksa::split::fnv1a64_of(&[section.trim().as_bytes()]), + DIGEST_OF_THE_RUBRIC, + "rubric r1 was edited; a new rubric is a new section and a new generator" + ); } +/// Taken over the 113 scene revisions that precede rubric r1, 2026-11-13. +const DIGEST_OF_THE_EARLIER_SCENES: u64 = 2598828882325200913; + +/// Taken over the text of the `## Rubric r1` section of docs/scene-annotation.md +/// (trimmed), 2026-11-13. +const DIGEST_OF_THE_RUBRIC: u64 = 11823581212652524671; + +/// Taken over each live group's latest revision (group, n, features) at rubric r1, +/// 2026-11-13. +const DIGEST_OF_THE_SCENES: u64 = 2505219080875817063; + /// The sweep both seed-pair tests share: over a grid of grades per feature (each /// also ABSENT) and the production `Condition::holds`, no listed pair of one /// language's entries co-holds; and each witness entry fires where it should and diff --git a/data/original/speaker-state.cc b/data/original/speaker-state.cc index 2503674ddbe09cd0598caeb70e5130c618d4ccd8..07bd001fcc60dc9db82fb63563f121474be7943f 100644 GIT binary patch literal 60158 zcmd^|OK&7s6~|+T57L@NM5OUB_Pi3a2^&C>011VMj6kaHu5nlTQC3y8tp>3ei5061 ziJC7!KtLuym?S&`NWtug5)xh$!kcwgh(LJWTUFh2`qVky$3Atd%f&9SeLU{Je&>B} z)#J}}#)G8eb$b4VI0+;F!r}JKFMj>V@5kPSxZ{s}_}@>1ubUr)-@5bgXU@guw+y`x zdPD!*?m6(I=g#Nf`luhpK{#6cuK7jg2ZsKzZN7VM_Y`nNKAA94T*PeJj{Zn?s zDA*51-eAicb$Vg+(86DEd*|ZL)vfLAt?f&XT)eWo{owBQ1AE7RpMC>bZM@^=Eq{~* z$z-?h`v-n>^YO+1|H?;3Bl8)$kALB7*L^P;N51)GH@^I#gT(ypPY(P;e{lCZI~Q*} z`M${O`Y&xf2EMv|<4f-yc>6Ejc=7{b*Zllf-!n>jQFst^?p*H-ym8mxio?;!PqwaX zU1`1ZV^;6h`@K;&KHnNz3-b|2Vd5ozw{_?Zg05%(@vZ$RJW6`4RE_xEt^8-gs1?L5 z^W&|g=eLfcAn~L7In8#UxC@$H+r46ByS8`yMnSVpSDO7*rrE=*G)om}MdNl9bXw8& z`BpdV#P3SKI#bfNQoxVfwOGei-VB_513_B zBWHXF>otAsptTSH0y9ghMWYGv2ZPCp39&}b2&pW)6bMPRSVf3wZx#r#Mol855Ld&a z)rGn#J{B^Ju{ryEkV**fSjaF&i6+re@R`W>x+jhw)=0@m+aH8Spi7zeaT}ikUtKa0 zaFAhAEf_$!KCW$zl!WkFytrO1R`qc>mRc>8eH^caFt3zZtrxu|42PVz%m-*@81xqC zZrhPDQhE#9!))GHB!rrJR5d@N6=QsA-t!eWGKwjEidRfHCn}Sp@1m}+;|=^#*IPY{ zH7LD3{notmo7uUdiJFzHlO|R-0Z_jqbaTZNr5iPGE0ZZa-S3z}Rz?(@?q^m?p7`0D zFqc;qofypoG{Nb9rdg9%`VNjyL2R}mqRGQ+({J7lPnnsLCN^?Qy-Z&ja(W0r`E5~; zCXs^kdnX+4v$JikB1(;vl&~>p^M!g>M44g&lHimgHFCw#=9@Cb%FJH4H}8i0W*V*& z;grJKC@HEztRa;U-~p!;RtZ;hNTY6ijfWY{XYSCyy zqA*Ul&Kw31xGA}Ajg*A&d5GxDsh&VztrQ4JwO|P0=2)o~>n6mN7AKrzrCKQGSS%Oo zh^M%o?Ril<9JAvG=r6Fo&&`w`!{^+}c8i=I0-quDPu@?+ZyydqxRZ`@p`X!YR-~sz* zt3;FNDELe`^4U{DY?QG^Rq|1*D3dBCz-S@LSfeI!g!gZcdRyMm>xPnq1^1RSJ0(pg zs+8AB6RVp5D7d$r>DDBs@D9#?#t~Zr&0A43+Qo>Ghs=IN-sKHPIM|{|4Voym4uLjYBx~cdWH3Y*7 z>@%^aoK^5@2)B@BZcToGw)i$G9C_vvq~8xX&w<6?f2PmQSNYIZ=|60g-4M>VsyZFA zR6oYo;OOE{jDAXw0Kk zj{i0NHbf@3bj3{Fj&3i}PJq@QNbQ=$7G7$Y66yLlVrL;R7Qv-qZlh{wGplV%un;L#1Ei#1bH!shF`AKl15{zC$qP!_YQDS6@}^%^emkvv7B zBv#-l3Y0vZ$kV8Lr>Y#rsdgHkqCm;hiBxl)!Col#s&=-~_O`A{<{Q77-iFQU%(P0b zD24RdY?&?sYW_&;(j;<-PCe-R`<`sfkER^Cr7LdEz?5UAod7O0<;X3Y#1_$W1d$0e zTu6dt6YAAcJ7-)8`}(T#9IKmvD%7i`x+$6BE2lLqCW(b@kg&}fBnn*=|CAZPc8B2r@QZ+%DT^-?PeXlKeN}f4D^mFh?2t!k9r{sz4>o(38j}TGX z=<6j`5r!2wh@}*9MT>OZ?@*CdqgXQ%L9sYxL^+G)86o!46eF(v#}E2!G=^&oOpCQs z`j5IJb3;q22;)EVDJwyXwNv_!T9GnoA&$iK!=Wh$#|e8F59V)lq-bvIikiB08%y;Q z@P&>P&Gl;%V^sc{j75iu79NgeGrG0=e_@N&O~4h-a#lAbQ*33uj_nXJb(G#)K(ktF z=8Pz!ey-W9HdRdk6q@~8GbK;8%7Z^F&T5Cjr%e(Zda}{Tf2YsEY3*HOS8P#Uw@$W} zD<~lB&zyoy;*5%0W1cWroTD=}Lt|DYTU6hougy6<1iavgWb|kfDa3pz?6U{3At>ld zh_O2RoG@w#FhSEYqejyynXsMPGD;@eYgFodu8$@ZB}R=?P%_b8qf)C;W|vRw6Qf|X zANuSQKw*!M_KCTrGN!OIv$}QoOSBVkh4zWLcFG9EYDd+CRNgL=E#hpN?94Perx!Wv z!lbI`;}5Nc02BC-0W;=^8r547bBx#vu8V3JG?7m2l#xdrnf08oiU?3albuu%Wjs+U zQf53M?mxv|-y4aF8gwTtGgR`z&b{vaTyXMnlcug2!(n2qPDJ7|ik{Di-53gSn z!4eSNlgl_!ZUJlWMSj08EBOAoc#AMj4V7NRsv~{KLRIH7C|}EzV|)#+3o%!zq0-k_ z<)n2@nbDazyBYVq{@7d(mmDDb_u|>jnW;0jYBqmgrrrv=b}sH6qm!>Q^_s*NadY0F zH4G|jAAqM9fuP@%52i8i;*qkSQe2%^jSfLJs5G)3Gb(9qG>Ii+%xjHM#0_$Mt zsSALeCe;SSL?iBnj|MXT`sLz1@Z3`ATk0IAr)P;Oj9<+Xzah z_spm4kE2P*A1;GqM6@BjYwV1jnmvAKxq<@7(1!G`f=%L#*a*g9TXcL8ZAh(|Gq!{$ zJN0pi(?h@u+K^g3noi1C7K_jspxBsINeVS3s;a-Gc(SUUf+u6vc%FuAUkjU#SHY_Q z%=|tc54C4AxE;?=-!2Z=s1=W!z+*la2MX*ma3D5jN>DgCUaIaI2I)9P4?$mn1F_LV z*-)tUD6=#q))|R+x+@2%U6rBOjqk9E2%;=n@T7_;y{A@<1TkPuyneWKSu)*0%Z$uW z$qPHWsO~}&uL%PTjJBjEN@93TDl2?>O#~4a4dOFSlrdMWNtrR1xOieeq#0Ss#yG!T ze3tFpR_ROXNTaV?mMX)z6CCBBD>-vzlpdv4rc8Q>S|;(vqKEXO+RmDteP%d41h}Bu z&g#)5Qm9JCMj1)6Mu&G2Yj^fpVHFX;f@?dgNRwzGhUnu_H|%>NUeIbGw>8m<9np941pkc}AdC z85+mnfj3J0NR*`EmNz$3dJSLIR(3&%(?j4kgdR%2;q|z$=r>eO(nk?a4}s&rY?Je$ z^c-G~HM|N4S9F6g97@_ASle0~rO&9NiJlQs3C3aOr)W%Mg|)3!Lg_7PCCWTKgCFx^ zo)H=MBzV9_-lRrnpAT9K0U+>^H>nm)A_P|?!yq#6Sm6+X%@2GbMrP%Tk-E*|7|jGI z!747(%oR)Pn7|c0Y&__&(FUUQ*7WK5pk-){lq9hknmQ*ktYS)z zc*Tef3Z+nJ!B4ED^930Bjok&+~}hv~a5 zS_=UpFmt6^G>H)0gGOPPaJBS&(_{dF-MF<;@==3HC#eL32b}t1C6sjVN(gh2-cv{= z1cNI$m#|DIr&j8i;W8`7ok`LYt=wRRmsu%&h1aF#6<)3x<5O_L1uMMFO6gU+W2^}4ayw^Cy;qxvaQ9;X)KF4dMATW1=KU9l7=O z5T%0v4|qP!=%6G+twWg^3|EM8=pArqz)15qa6Zl2xMHJjFS;Q0T7}Tg;-=5))g*@S z{4YMRZ}8wKngL{~v#Zh%P+8j4ag3sfJCW1W#?^Bpk7A*BlZw!AgzJJ|nai z0z}{hE7hXmgoFbwjLegfS-rp=o>3(nxFV!(kKrdcGK!s?qi~?)h@Evw_vdBSG&lk$ zzIVXSLLo?S!joAkN#Z+8q6?_g4lmbCfD+7FnU#_ywPt0qgzLSe=N)k&2_`K#OSU#j zj?_JazIP&(5byzK$yNy^9cm@Yq@xgNdXevSIX>7(lUgYW;k`yU!>Vdd!)PWTX(7_2 znkiXgHIsG=WwKQ8ub?X`j~611u~|1q4kC?Jg5e{Lx%oT1uoAB5plTXwg6BwyRr@fx#w delta 882 zcmXX^X-Je&6y<%+rjyM#y-{;gMjexeGR=}mD8~vSwZ(-ZjLLSSL}@RiC`$=thHhpm zrDQ5iO4_xE$QeZ0NTY-{@kbUKsWW5md!PRv_r7!QIp-elfREcl>~$o+r9Pg3 zt4#!*LWRCQf&dn^0peBT5Ys3^ie?Jz$s*?X`4cFaGtG-lbLC>UXcTaoh7qv!ikK9* zh(OPvh-LF52@Iu*wZlfj!P?LW0uygU$S}#k{BSm{CQ^wNX11^7s)!v?W&ynxGpB+n zOJ)JER0g!fvVyc^GH!OsrucaTPp{Mn&;}G83DE?sYfNZd9Yw%>mV(WyhRAX=G*;=L zHL(k}R8nk8vPi@z`mXmEpeBprx(y3u@ioaRsNDFMaiQ=lRf*1&d79t=ri{sy1i&V%ZLp0R20?oStVFqbGM1xHp^I zL-FxGCU;vQMdiU5&I^tm_P~Wz6w8n32sn>U@r9;54UXorh0>9}XSwB?iR?v|kYk zj1?=;StQB5=Eq%uwWTv`FvUDufMjndv|smdXU1=MB-$tEv=%!+CL`BD`j zd{yi1AcCXS0^!ob?lS2k4FPbcPK~Aa^aR`M*%he60H*s=#sVPW!357hMiaXoGZI`a zYIHnQ%E@^DNECSL^iOyeAm*u?nIvM`Gp54T7KpXaL*xT|bFLt`tb>y8D$$x4EenJ2M diff --git a/docs/scene-annotation.md b/docs/scene-annotation.md new file mode 100644 index 0000000..5fca125 --- /dev/null +++ b/docs/scene-annotation.md @@ -0,0 +1,116 @@ +# Scene annotation + +How the keeper scenes of `data/original/speaker-state.cc` are labelled. A scene +label has a committed rule that produced it; this file is that rule. + +## Why + +Two findings forced a written rule. + +- `annotation-rank-from-register`: an annotator who reads the line's register + (politeness forms, honorifics, endings) to decide the speaker's rank hands the + lowering its own answer back, because register is what the lowering must + produce from the scene. +- The addressee gate (`with_scene` in `crates/weksa/src/validate.rs`): a scene is + read only when the packet names a discourse addressee. A line that addresses no + one reads `NoAddressee` and no Scene entry selects, so labels on narration are + inert and the only labels that bite are those on lines said to someone. + +The earlier labels followed two briefs that were never committed and that +disagreed with each other (the relation to the reader, and a situation read with +the register ignored). The rubric below replaces both for every live group. + +## Provenance + +Cut `weksa-lowering:cut_spec:cut-scene-in-group.r1`, ruling +`weksa-lowering:ruling:scene-rubric-whole-bed`: all 98 live Train and Validation +groups of `data/original` receive a new scene revision from annotator A under +rubric r1, generator `claude-sonnet-5-5 (Weksa Hands, scene rubric r1, +docs/scene-annotation.md)`. Earlier revisions stay as provenance. The rubric uses +no corpus line as an example on purpose: a labelled example would make the blind +check against the old labels circular. + +A later rubric revision gets a new section and a new generator string; r1 is +never edited. + +## Rubric r1 + +The speaker of every line is the keeper, a quiet caretaker of a house and garden. First decide whether the line addresses a listener: it asks them something, asks them to do something, tells them about themselves, wishes something for them, or speaks to them as you. A line that addresses no one (narration, description, a remark about the world or about other people, a thought or wish with no one addressed) is told to the reader, a stranger: answer addressed false, relative_status level, intimacy 500, in_group false. People named inside a line are not its listener. + +For a line that addresses a listener, answer addressed true and judge the keeper's relation to that listener from the situation the line states or implies, never from its register: politeness forms, honorifics, sentence endings, contractions, please and casual particles are what the lowering must produce from the scene, so they cannot decide it. relative_status: above when the situation gives the keeper standing over the listener (instructing, advising or permitting as elder, host, employer or teacher); below when it gives the listener standing over the keeper (the keeper serves, thanks, or asks a favour of a guest, employer, elder or official); level when it shows neither. + +intimacy, 0 to 1000: 500 when the line shows nothing; up to 800 when it shows shared daily life (the listener's body, sleep or habits, the household's routine); above 800 only for a stated close bond; down to 200 for a stated first meeting or a service relation. in_group: true only when the line shows the listener belongs to the keeper's household, family or working circle, stated or plainly implied; a request, a question or a kind wish alone is not enough; otherwise false. Answer one JSON line per group, in the order given and nothing else: {"group": G, "addressed": true|false, "relative_status": S, "intimacy": N, "in_group": true|false}. + +## Procedure + +- Annotator A (committed) and annotator B (measurement only) are separate fresh + model agents, one pair per language (49 live groups each). Each sees only the + text of Rubric r1 above and that language's sentences as `group"text"`, + shuffled by a fixed seed. Neither sees an existing scene, the other annotator's + answer, the ontology, which items are old or new, or any Test text. +- An output that does not parse or misses a group is re-drawn once and logged. +- A's labels are written with `weksa scene put` (one revision per group, + `--to unnamed --inventory-rev 4`). `addressed` is not stored: no Scene feature + carries it. +- Disagreements are reported, never adjudicated. + +## Agreement at r1 (this cut, 98 live groups, 49 per language) + +Four fresh agents drew the labels (A-en, A-ja, B-en, B-ja); the briefs were the +rubric section plus one language's shuffled sentences (SHA-256 of the brief +bodies: en-US `db087476...dbc04e`, ja-JP `c5b81b27...97488c`; no Test text in +either). No draw was repeated. Annotator model: `claude-sonnet-5-5`. + +A against B (B measures only): pooled `addressed` equal 96/98 (kappa 0.96), +`in_group` equal 93/98 (kappa 0.79), `relative_status` equal 89/98, intimacy +within 150 on 98/98. en-US: addressed 47/49, in_group 47/49 (0.86), +relative_status 45/49. ja-JP: addressed 49/49, in_group 46/49 (0.64), +relative_status 44/49. + +Blind check, A against the 56 older labels: `relative_status` equal 49/56, +`in_group` equal 53/56 (kappa 0.64), intimacy within 150 on 55/56. Three +in_group differences: `original-ja-shimoyado-2` and `original-ja-sakana` (both +narrations, true to false) and `original-ja-asa` (an instruction to a household +member, false to true). + +In_group true across the 98: 42 before, 15 after; every unaddressed line is now +false (it was true on 17 of the 21 unaddressed vgrow groups). Disagreements are +reported, not adjudicated. + +## Reproducing the cut + +`scripts/scene-annotation/` holds the standard-library scripts of this cut, run +from the repo root with `WEKSA` (the binary) and `SCENE_SCRATCH` (a directory +that is never committed): `build_briefs.py` writes the two annotator briefs from +the Rubric r1 section and prints each brief's Test-leak count (must be 0) and +sha256 (it reproduces the briefs of this cut byte for byte); `put_labels.py` +puts A's `A-en.jsonl` and `A-ja.jsonl`; `agreement.py` prints the tables above and +must run before the puts, against the store at 76fa685, because the older labels +are then its latest revisions (it also reads the round-1 encoder answers of +`runs/growth-base` through `ROUND1`). `DIGEST_OF_THE_RUBRIC` and +`DIGEST_OF_THE_EARLIER_SCENES` in `crates/weksa/tests/original_corpus.rs` pin the +rubric text and the 113 earlier revisions. + +## Known limits of r1 labels + +Found by Soul's pass on this cut (`weksa-lowering:verdict:cut-scene-in-group.s1`). +They are not fixed in r1; a reviewer's pass over the labels is follow-up +`weksa-lowering:follow_up:scene-labels-reader-review`. + +- The new labels change which ontology entries are selected for 37 groups. Eight + addressed lines move onto the casual in-group entries (en-US:12: lantern, + broom, mirror, seedling; ja-JP:12: furo, asa, irori, yurikago), while their + references are polite requests. The reader default (level, 500, false) is + exactly the condition of ja-JP:11. +- `relative_status` tracks request form for some lines: imperatives read above, + favour questions read level or below, so the same household request gets + different standing in two forms (hane above/650/true, kabin below/500/false; + B gave both level/false). Form is what signals the speech act, which is what + part 2 asks for, and register is what part 2 forbids. +- `in_group` is true on some requests alone (lantern, mulch, stool, hane) and on + a line whose only family member is named inside it (original-ja-ane), though r1 + says a request or a person named inside the line is not enough. A and B + agreeing does not clear this. +- A run opened on the old speaker-state refuses to continue after merge: + `runs/growth-base` pins the store of `/var/lib/mind-weksa-self/Weksa-growth` + (main), and that checkout must not be pulled while the run is wanted. diff --git a/scripts/scene-annotation/agreement.py b/scripts/scene-annotation/agreement.py new file mode 100644 index 0000000..0a200e3 --- /dev/null +++ b/scripts/scene-annotation/agreement.py @@ -0,0 +1,99 @@ +"""Agreement of annotator A, B, the older labels and the round-1 encoder. Run it BEFORE the +rubric r1 revisions are put: the older labels are the store's latest revision then. +Standard library only. Usage: as build_briefs.py, plus ROUND1=.""" +import re,sys,subprocess,json,glob,os +import os +W=os.environ.get("WEKSA","target/debug/weksa") # the weksa binary +R=os.getcwd() # run from the repo root +S=os.environ.get("SCENE_SCRATCH","scene-scratch")+"/" # A-*.jsonl, B-*.jsonl, briefs; never committed +A=["--corpus","data/original/corpus.cc","--entities","data/original/entity.cc","--inventory","data/original/inventory.cc","--speaker-state","data/original/speaker-state.cc"] +def run(*a): return subprocess.run([W]+A+list(a),cwd=R,capture_output=True,text=True).stdout.split("\n") +out=run("corpus","dump") +items=[] +for i,l in enumerate(out): + m=re.match(r'^item:(\S+) \[(\S+) \S+ group=(\S+) (\w+)\]$',l) + if m and m.group(4)!="Test": + t=out[i+1].strip(); t=json.loads(t) if '\\' in t else t[1:-1] + items.append(dict(id=m.group(1),lang=m.group(2),group=m.group(3),text=t,new=m.group(1).startswith("vgrow"))) +G={x['group']:x for x in items} +assert len(G)==98 +# scene dump: latest and @1 per group +sd=run("scene","dump"); lab={} +for i,l in enumerate(sd): + m=re.match(r'^scene:(\S+)@(\d+) rev:\d+ Model \S+ (.*) \[entity:keeper\]$',l) + if m: + f=re.search(r'relative_status=(\w+) intimacy=(\d+) in_group=(\w+)',sd[i+1]) + lab.setdefault(m.group(1),{})[int(m.group(2))]=dict(rs=f.group(1),im=int(f.group(2)),ig=f.group(3)=="true",gen=m.group(3)) +old={g:v[max(v)] for g,v in lab.items()} +def load(f): return {json.loads(l)['group']:json.loads(l) for l in open(S+f+".jsonl")} +Aa={**load("A-en"),**load("A-ja")}; Bb={**load("B-en"),**load("B-ja")} +assert set(Aa)==set(G)==set(Bb) +def kappa(x,y): + n=len(x); + if n==0: return float('nan') + po=sum(a==b for a,b in zip(x,y))/n + pa=sum(x)/n; pb=sum(y)/n + pe=pa*pb+(1-pa)*(1-pb) + return float('nan') if pe==1 else (po-pe)/(1-pe) +def tab(x,y): + return "[TT=%d TF=%d FT=%d FF=%d]"%(sum(a and b for a,b in zip(x,y)),sum(a and not b for a,b in zip(x,y)),sum((not a) and b for a,b in zip(x,y)),sum((not a) and (not b) for a,b in zip(x,y))) +def cmp(name,P,Q,gs,bkey=None): + # P,Q: dict group-> record with keys addressed(optional) relative_status intimacy in_group + def rs(r): return r.get('relative_status',r.get('rs')) + def im(r): return r.get('intimacy',r.get('im')) + def ig(r): return r.get('in_group',r.get('ig')) + n=len(gs) + if n==0: print(f"{name}: n=0"); return + e=lambda f: sum(f(P[g])==f(Q[g]) for g in gs) + s=f"{name}: n={n} rs_eq={e(rs)}/{n} ig_eq={e(ig)}/{n} im<=150={sum(abs(im(P[g])-im(Q[g]))<=150 for g in gs)}/{n}" + s+=f" ig_kappa={kappa([ig(P[g]) for g in gs],[ig(Q[g]) for g in gs]):.2f} ig_tab(A.row,B.col)={tab([ig(P[g]) for g in gs],[ig(Q[g]) for g in gs])}" + if all('addressed' in P[g] and 'addressed' in Q[g] for g in gs): + s+=f" addr_eq={e(lambda r:r['addressed'])}/{n} addr_kappa={kappa([P[g]['addressed'] for g in gs],[Q[g]['addressed'] for g in gs]):.2f} addr_tab={tab([P[g]['addressed'] for g in gs],[Q[g]['addressed'] for g in gs])}" + print(s) +L={"en-US":[g for g in G if G[g]['lang']=="en-US"],"ja-JP":[g for g in G if G[g]['lang']=="ja-JP"],"pooled":list(G)} +print("== (1) A vs B") +for k,gs in L.items(): cmp(k,Aa,Bb,gs) +oldg=[g for g in G if not G[g]['new']]; newg=[g for g in G if G[g]['new']] +print("== (2) blind check: A vs old latest, the 56") +for k,gs in L.items(): + gg=[g for g in gs if g in oldg] + cmp(k+" all",Aa,old,gg) + cmp(k+" A.addr=true",Aa,old,[g for g in gg if Aa[g]['addressed']]) + cmp(k+" A.addr=false",Aa,old,[g for g in gg if not Aa[g]['addressed']]) +print("differing in_group (old->A):") +for g in oldg: + if old[g]['ig']!=Aa[g]['in_group']: print(" ",g,old[g]['ig'],"->",Aa[g]['in_group'],"addr",Aa[g]['addressed']) +print("differing any label, old vs A:") +for g in oldg: + o=old[g];a=Aa[g] + if (o['rs'],o['im'],o['ig'])!=(a['relative_status'],a['intimacy'],a['in_group']): print(" %s old %s/%d/%s A %s/%d/%s addr=%s"%(g,o['rs'],o['im'],o['ig'],a['relative_status'],a['intimacy'],a['in_group'],a['addressed'])) +print("== (3) A vs vgrow @1, the 42") +for k,gs in L.items(): cmp(k,Aa,{g:lab[g][1] for g in newg},[g for g in gs if g in newg]) +print("== (4) A addressed vs round-1 encoder addressee") +enc={} +RD=os.environ.get("ROUND1","runs/growth-base/round-1")+"/" +for p in glob.glob(RD+"*.prompt"): + pt=open(p).read() + if not pt.startswith("Encode the line"): continue + ap=p[:-7]+".answer" + if not os.path.exists(ap): continue + # the line text; take the item whose text appears in the prompt + hit=[g for g in G if G[g]['text'] in pt] + if len(hit)!=1: continue + try: a=json.load(open(ap)) + except Exception: continue + enc[hit[0]]=a.get('discourse',{}).get('addressee') is not None +print("encoder answers matched:",len(enc),"of 98") +for k,gs in list(L.items()): + gg=[g for g in gs if g in enc] + print(k,"n=",len(gg),"A.row,enc.col",tab([Aa[g]['addressed'] for g in gg],[enc[g] for g in gg]),"kappa=%.2f"%kappa([Aa[g]['addressed'] for g in gg],[enc[g] for g in gg])) +for nm,grp in (("old",oldg),("new",newg)): + gg=[g for g in grp if g in enc]; print(nm,"enc-addressee true:",sum(enc[g] for g in gg),"of",len(gg),"; A addressed true among old/new:",sum(Aa[g]['addressed'] for g in grp),"of",len(grp)) +print("== (5) in_group true counts, old/new x addressed, before -> after") +for nm,grp in (("old56",oldg),("new42",newg)): + for ad in (True,False): + gg=[g for g in grp if Aa[g]['addressed']==ad] + before=sum(old[g]['ig'] for g in gg); after=sum(Aa[g]['in_group'] for g in gg) + print(f"{nm} A.addressed={ad}: n={len(gg)} in_group true before={before} after={after}") +print("totals before",sum(old[g]['ig'] for g in G),"after",sum(Aa[g]['in_group'] for g in G)) +print("A addressed true:",sum(Aa[g]['addressed'] for g in G),"B:",sum(Bb[g]['addressed'] for g in G)) diff --git a/scripts/scene-annotation/build_briefs.py b/scripts/scene-annotation/build_briefs.py new file mode 100644 index 0000000..472f475 --- /dev/null +++ b/scripts/scene-annotation/build_briefs.py @@ -0,0 +1,37 @@ +"""Build the two annotator briefs (rubric r1 section + one language's shuffled live +sentences), print the Test-leak count of each (must be 0) and its sha256. +Standard library only. Usage: from the repo root, WEKSA=... SCENE_SCRATCH=dir python3 -I scripts/scene-annotation/build_briefs.py""" +import re,sys,subprocess,random,json,hashlib,os +import os +W=os.environ.get("WEKSA","target/debug/weksa") # the weksa binary +R=os.getcwd() # run from the repo root +S=os.environ.get("SCENE_SCRATCH","scene-scratch")+"/" # A-*.jsonl, B-*.jsonl, briefs; never committed +A=["--corpus","data/original/corpus.cc","--entities","data/original/entity.cc","--inventory","data/original/inventory.cc","--speaker-state","data/original/speaker-state.cc"] +out=subprocess.run([W]+A+["corpus","dump"],cwd=R,capture_output=True,text=True).stdout.split("\n") +live=[];test=[] +for i,l in enumerate(out): + m=re.match(r'^item:(\S+) \[(\S+) \S+ group=(\S+) (\w+)\]$',l) + if not m: continue + t=out[i+1].strip() + assert t.startswith('"') and t.endswith('"') + t=json.loads(t) if '\\' in t else t[1:-1] + if m.group(4)=="Test": test.append(t) + else: live.append((m.group(2),m.group(3),t)) +assert len(live)==98,len(live) +rub=open(R+"/docs/scene-annotation.md").read() +sec=rub[rub.index("## Rubric r1"):] +# rubric section ends at next "## " heading +rest=sec[len("## Rubric r1"):] +nx=rest.find("\n## ") +sec=sec[:len("## Rubric r1")+(nx if nx>=0 else len(rest))].strip()+"\n" +def leaks(brief,text): + w=text.split(); c=list(text) + return text in brief or any(" ".join(w[k:k+5]) in brief for k in range(len(w)-4)) or (len(w)==1 and any("".join(c[k:k+12]) in brief for k in range(len(c)-11))) +for lang in ("en-US","ja-JP"): + rows=[(g,t) for (l,g,t) in live if l==lang] + assert len(rows)==49 + random.Random(f"scene-r1-{lang}").shuffle(rows) + body=sec+"\nThe groups, one per line as group\"text\":\n\n"+"\n".join(f'{g}\t"{t}"' for g,t in rows)+"\n" + n=sum(leaks(body,t) for t in test) + print(lang,"lines",len(rows),"test_leaks",n,hashlib.sha256(body.encode()).hexdigest()) + open(S+f"brief-{lang}.txt","w").write(body) diff --git a/scripts/scene-annotation/put_labels.py b/scripts/scene-annotation/put_labels.py new file mode 100644 index 0000000..9de31a1 --- /dev/null +++ b/scripts/scene-annotation/put_labels.py @@ -0,0 +1,15 @@ +"""Put annotator A's labels through weksa scene put, one revision per group. +Standard library only. Usage: as build_briefs.py.""" +import json,subprocess +import os +W=os.environ.get("WEKSA","target/debug/weksa") # the weksa binary +R=os.getcwd() # run from the repo root +S=os.environ.get("SCENE_SCRATCH","scene-scratch")+"/" # A-*.jsonl, B-*.jsonl, briefs; never committed +A=["--corpus","data/original/corpus.cc","--entities","data/original/entity.cc","--inventory","data/original/inventory.cc","--speaker-state","data/original/speaker-state.cc"] +rows=[json.loads(l) for f in ("A-en","A-ja") for l in open(S+f"{f}.jsonl")] +for r in sorted(rows,key=lambda r:r['group']): + c=[W]+A+["scene","put","--group",r['group'],"--to","unnamed","--inventory-rev","4", + "--set",f"relative_status={r['relative_status']}","--set",f"intimacy={r['intimacy']}","--set",f"in_group={'true' if r['in_group'] else 'false'}", + "--model-family","anthropic","--generator","claude-sonnet-5-5 (Weksa Hands, scene rubric r1, docs/scene-annotation.md)"] + p=subprocess.run(c,cwd=R,capture_output=True,text=True) + print(r['group'],p.returncode,p.stdout.strip()[:80],p.stderr.strip()[:200]) From d8afc4ab85c9462fc36ea802e55eeba18923afa3 Mon Sep 17 00:00:00 2001 From: Weksa Date: Fri, 13 Nov 2026 09:18:54 +0000 Subject: [PATCH 05/22] calibrate-harness: weksa loop calibrate over answered draws (#78) Co-Authored-By: Claude Sonnet 5.5 --- crates/weksa/src/main.rs | 130 +++++- crates/weksa/src/run.rs | 524 ++++++++++++++++++----- crates/weksa/tests/cli.rs | 93 ++++ crates/weksa/tests/golf_loop.rs | 254 +++++++++++ docs/calibration.md | 99 +++++ docs/targets/interlingua-lowering-map.md | 21 +- 6 files changed, 1001 insertions(+), 120 deletions(-) create mode 100644 docs/calibration.md diff --git a/crates/weksa/src/main.rs b/crates/weksa/src/main.rs index f70887a..3cd2235 100644 --- a/crates/weksa/src/main.rs +++ b/crates/weksa/src/main.rs @@ -15,7 +15,7 @@ use weksa::packet::{FeatValue, FeatureValue, Grade}; use weksa::packet::proof_packet; use weksa::model::{ModelPort, RecordedAnswers}; use weksa::run::{ - AblationOutcome, CycleOutcome, LowererReach, OntologyMintOutcome, RecallOutcome, RelowerOutcome, RoundOutcome, RunPaths, RunStores, cycle, round, + AblationOutcome, CalibrateOutcome, CycleOutcome, FLOOR_ARMS_MIN, LowererReach, OntologyMintOutcome, RecallOutcome, RelowerOutcome, RoundOutcome, RunPaths, RunStores, cycle, round, }; use weksa::score::ScoreInput; use weksa::{ @@ -174,6 +174,31 @@ enum LoopCommand { #[arg(long, default_value = "http://127.0.0.1:11434")] ollama_url: String, }, + /// Replay the accept rule over answered relower draws of round ROUND, to + /// measure how often it accepts, rejects or regresses what it should. Reads + /// the recorded lowering and each draw (--prompted: repeats of it, at least + /// four; --degraded: arms that should regress; --candidate: arms to rate), + /// answered through `weksa loop relower ROOT --round N --arm LABEL`, and + /// prints counts over all Validation items and over each --class + /// NAME=PREFIX[,PREFIX...] (items whose key starts with a prefix). Writes + /// no prompts; exits 1 naming the draws still unanswered. Procedure: + /// docs/calibration.md. A diagnostic, never a gate; records nothing; Test + /// and Train are never touched. Labels match [a-z0-9-]{1,32} and differ. + Calibrate { + root: PathBuf, + #[arg(long)] + round: u32, + #[arg(long)] + prompted: Vec, + #[arg(long)] + degraded: Vec, + #[arg(long)] + candidate: Vec, + #[arg(long)] + class: Vec, + #[arg(long, default_value = "http://127.0.0.1:11434")] + ollama_url: String, + }, /// Run one edit-accept cycle of the run at ROOT. The stores are read only /// from the run's record. The editor's prompt, then the candidate round's /// encode prompts, then its lowering prompts are written in turn; the @@ -660,6 +685,12 @@ impl AddArgs { } } +/// A draw or arm label: [a-z0-9-]{1,32}. +fn is_label(l: &str) -> bool { + (1..=32).contains(&l.len()) + && l.bytes().all(|b| b.is_ascii_lowercase() || b.is_ascii_digit() || b == b'-') +} + /// The split of an item as `Train`, `Validation`, `Test` or `excluded`. fn split_text(item: &CorpusItem) -> String { SplitScheme::V1 @@ -1146,12 +1177,8 @@ fn main() -> Result<()> { floor, ollama_url, }) => { - let label = |l: &str| { - (1..=32).contains(&l.len()) - && l.bytes().all(|b| b.is_ascii_lowercase() || b.is_ascii_digit() || b == b'-') - }; - if !label(&arm) - || floor.iter().any(|f| !label(f) || *f == arm) + if !is_label(&arm) + || floor.iter().any(|f| !is_label(f) || *f == arm) || floor.iter().enumerate().any(|(i, f)| floor[..i].contains(f)) { anyhow::bail!("arm and floor labels must match [a-z0-9-]{{1,32}} and differ"); @@ -1197,6 +1224,95 @@ fn main() -> Result<()> { } } } + Command::Loop(LoopCommand::Calibrate { + root, + round, + prompted, + degraded, + candidate, + class, + ollama_url, + }) => { + let labels: Vec<&String> = prompted.iter().chain(°raded).chain(&candidate).collect(); + if labels.iter().any(|l| !is_label(l)) + || labels.iter().enumerate().any(|(i, l)| labels[..i].contains(l)) + { + anyhow::bail!("draw labels must match [a-z0-9-]{{1,32}} and differ"); + } + if prompted.len() < FLOOR_ARMS_MIN + 1 { + anyhow::bail!("at least {} --prompted labels are needed", FLOOR_ARMS_MIN + 1); + } + let mut classes: Vec<(String, Vec)> = Vec::new(); + for c in &class { + let Some((name, prefixes)) = c.split_once('=') else { + anyhow::bail!("--class is NAME=PREFIX[,PREFIX...]"); + }; + if !is_label(name) + || name == "all" + || classes.iter().any(|(n, _)| n == name) + || prefixes.split(',').any(str::is_empty) + { + anyhow::bail!("--class is NAME=PREFIX[,PREFIX...] with a new label NAME other than all"); + } + classes.push((name.to_string(), prefixes.split(',').map(String::from).collect())); + } + let run = RunStores::open(&root)?; + let scorer = Scorer::new(OllamaEmbedder::new(&ollama_url, &run.record().embedder)); + match run.calibrate(round, &prompted, °raded, &candidate, &classes, &scorer)? { + CalibrateOutcome::Unanswered(missing) => { + for (label, n) in &missing { + println!("unanswered {label} {n}"); + } + eprintln!( + "answer it through weksa loop relower ROOT --round N --arm LABEL first" + ); + std::process::exit(1); + } + CalibrateOutcome::Read { unusable, classes } => { + println!( + "{}", + run.labelled(&format!( + "calibrate draws prompted={} degraded={} candidate={} floor-arms={FLOOR_ARMS_MIN} unusable={unusable}", + prompted.len(), + degraded.len(), + candidate.len(), + )) + ); + for c in classes { + let items: String = c + .languages + .iter() + .map(|l| format!(" {} n={} skipped={}", l.language, l.n, l.skipped)) + .collect(); + let t = &c.trials; + println!("{}", run.labelled(&format!("calibrate class={} items{items}", c.name))); + println!( + "{}", + run.labelled(&format!( + "calibrate class={} prompted trials={} nogain={} accepted={} regressed={} noise={:.4}", + c.name, t.prompted.trials, t.prompted.nogain, t.prompted.accepted, t.prompted.regressed, t.noise + )) + ); + for (label, d) in degraded.iter().zip(&t.degraded) { + println!( + "{}", + run.labelled(&format!( + "calibrate class={} degraded={label} trials={} regressed={}", + c.name, d.trials, d.regressed + )) + ); + } + println!( + "{}", + run.labelled(&format!( + "calibrate class={} candidate trials={} accepted={} nogain={} regressed={}", + c.name, t.candidate.trials, t.candidate.accepted, t.candidate.nogain, t.candidate.regressed + )) + ); + } + } + } + } Command::Loop(LoopCommand::Recall { root }) => { let run = RunStores::open(&root)?; let mut files = run.answer_files(run.recall_dir())?; diff --git a/crates/weksa/src/run.rs b/crates/weksa/src/run.rs index 463db61..b15fd07 100644 --- a/crates/weksa/src/run.rs +++ b/crates/weksa/src/run.rs @@ -44,7 +44,7 @@ use crate::encode::{ EditDraft, LoweringDraft, MAX_EDIT_CHANGES, OntologyDraft, PacketDraft, encoder_id, }; use crate::form::{is_attestation, is_language_tag, is_slug}; -use crate::model::{AnswerFiles, ModelPort, PortError}; +use crate::model::{AnswerFiles, ModelPort, PortError, RecordedAnswers}; use crate::packet::{Packet, Stance}; use crate::score::{CHRF_VERSION, chrf, Embedder, ItemScore, ScoreInput, Scorer, SplitMean, aggregate, validation_items}; use crate::split::{Split, SplitScheme, fnv1a64, fnv1a64_of, hex16}; @@ -1675,27 +1675,90 @@ impl RunStores { floors: &mut [&mut dyn ModelPort], scorer: &Scorer, ) -> Result { - let store = |_| RunError::Store; check_embedder(self, scorer)?; - let held = self.held_splits(n, &[Split::Validation])?; - let inventory = self.inventory.inventory().map_err(store)?; - let ontology = self.ontology.ontology().map_err(store)?; - - let dir = self.relower_dir(n, arm_label); - std::fs::create_dir_all(&dir).map_err(|e| RunError::Io(e.kind()))?; - for suffix in ["cc", "cc.lock"] { - let _ = std::fs::remove_file(dir.join(format!("base.{suffix}"))); - } - let mut scratch = PacketStore::open(&dir.join("base.cc")).map_err(store)?; - if (1..FLOOR_ARMS_MIN).contains(&floors.len()) { return Err(RunError::FloorArms); } - // Per language, per item: the item text, then the recorded and arm - // lowerings and one per floor arm (`None` when unusable). - type Row = (String, Option, Option, Vec>); - let mut waiting: Vec = Vec::new(); - let mut floor_waiting = false; + let dir = self.relower_dir(n, arm_label); + // Ports in the order the prompts are asked: recorded, floors, arm. + let floor_count = floors.len(); + let mut ports: Vec<&mut dyn ModelPort> = vec![recorded]; + for f in floors.iter_mut() { + let f: &mut dyn ModelPort = &mut **f; + ports.push(f); + } + ports.push(arm); + let drawn = self.read_draws(n, &dir.join("base.cc"), &mut ports, scorer)?; + let arm_at = floor_count + 1; + if drawn.waiting[1..arm_at].iter().any(|w| !w.is_empty()) { + return Err(RunError::FloorUnanswered); + } + if !drawn.waiting[arm_at].is_empty() { + return Ok(RelowerOutcome::Unanswered { prompts: drawn.waiting[arm_at].clone() }); + } + let mut languages = Vec::new(); + for l in &drawn.languages { + let of_port = |p: usize| -> Vec { l.items.iter().map(|(_, s)| s[p].clone()).collect() }; + let rec = of_port(0); + let floor_scores: Vec> = (1..arm_at).map(of_port).collect(); + languages.push(RelowerReading { + language: l.language.clone(), + n: l.items.len() as u32, + skipped: l.skipped, + unusable: l.unusable[arm_at], + floor_unusable: l.unusable[1..arm_at].iter().sum(), + recorded: reading_of(&rec)?, + arm: reading_of(&of_port(arm_at))?, + floor: if floor_count == 0 { None } else { Some(paired_floor(&rec, &floor_scores)) }, + }); + } + let verdict = if floor_count > 0 { + let panel: Vec<(Reading, Reading, Reading)> = languages + .iter() + .filter_map(|l| Some((l.recorded, l.arm, l.floor?))) + .collect(); + Some(decide_panel(&panel)) + } else { + None + }; + Ok(RelowerOutcome::Read { languages, verdict }) + } + + /// Asks every port, in the order given, each live Validation item's + /// byte-identical lowering prompt of round `n` and scores the answers against + /// the items in one [`Scorer::score`] call: the one owner of per-item + /// scoring for [`relower`](Self::relower) and + /// [`calibrate`](Self::calibrate). An answer that is not a usable lowering + /// scores 0. Port 0 is the recorded lowering, which must be answered + /// (`Unrecorded` at once); any other port that is not is listed in + /// `waiting`, and nothing is scored. `scratch` is the packet store the + /// prompts are built in, removed and reopened here; its directory is + /// created here, after the round is found. + fn read_draws( + &self, + n: u32, + scratch: &Path, + ports: &mut [&mut dyn ModelPort], + scorer: &Scorer, + ) -> Result { + let store = |_| RunError::Store; + let held = self.held_splits(n, &[Split::Validation])?; + let inventory = self.inventory.inventory().map_err(store)?; + let ontology = self.ontology.ontology().map_err(store)?; + if let Some(dir) = scratch.parent() { + std::fs::create_dir_all(dir).map_err(|e| RunError::Io(e.kind()))?; + } + for suffix in ["", ".lock"] { + let mut name = scratch.as_os_str().to_owned(); + name.push(suffix); + let _ = std::fs::remove_file(PathBuf::from(name)); + } + let mut scratch = PacketStore::open(scratch).map_err(store)?; + + // Per language, per item: the item id and text, then each port's lowering + // (`None` when unusable or unanswered). + let mut waiting: Vec> = vec![Vec::new(); ports.len()]; + type Row = (String, String, Vec>); let mut rows: Vec> = Vec::new(); for (_, _, items) in &held { let mut language_rows = Vec::new(); @@ -1707,49 +1770,33 @@ impl RunStores { else { continue; }; - let text = |a: Asked| match a { - Asked::Text(t) => Some(t), - _ => None, - }; - let recorded_text = match asked(recorded, &prompt)? { - Asked::Waiting => return Err(RunError::Unrecorded), - other => text(other), - }; - let mut floor_texts = Vec::new(); - for f in floors.iter_mut() { - floor_texts.push(match asked(&mut **f, &prompt)? { + let mut texts = Vec::new(); + for (p, port) in ports.iter_mut().enumerate() { + texts.push(match asked(&mut **port, &prompt)? { + Asked::Text(t) => Some(t), + Asked::Unusable => None, + Asked::Waiting if p == 0 => return Err(RunError::Unrecorded), Asked::Waiting => { - floor_waiting = true; + let d = prompt.digest(); + if !waiting[p].contains(&d) { + waiting[p].push(d); + } None } - other => text(other), }); } - let arm_text = match asked(arm, &prompt)? { - Asked::Waiting => { - let d = prompt.digest(); - if !waiting.contains(&d) { - waiting.push(d); - } - None - } - other => text(other), - }; - language_rows.push((item.text.clone(), recorded_text, arm_text, floor_texts)); + language_rows.push((item.id.as_str().to_string(), item.text.clone(), texts)); } rows.push(language_rows); } - if floor_waiting { - return Err(RunError::FloorUnanswered); - } - if !waiting.is_empty() { - return Ok(RelowerOutcome::Unanswered { prompts: waiting }); + if waiting.iter().any(|w| !w.is_empty()) { + return Ok(Drawn { languages: Vec::new(), waiting }); } // One score call: each usable lowering against its item, in row order. let mut inputs: Vec = Vec::new(); - for (reference, recorded, arm, floor_texts) in rows.iter().flatten() { - for output in [recorded, arm].into_iter().chain(floor_texts).flatten() { + for (_, reference, texts) in rows.iter().flatten() { + for output in texts.iter().flatten() { inputs.push(ScoreInput { reference, output }); } } @@ -1759,67 +1806,272 @@ impl RunStores { scorer.score(&inputs).map_err(|_| RunError::Embed)? } .into_iter(); - let mut next = |t: &Option| -> Result { - match t { - Some(_) => scores.next().ok_or(RunError::Store), - None => Ok(ItemScore { chrf: 0.0, meaning: Some(0.0) }), - } - }; - let reading_of = |scored: &[ItemScore]| { - Reading::of(&SplitMean { - count: scored.len(), - chrf: mean(scored.iter().map(|s| s.chrf)), - meaning: mean(scored.iter().filter_map(|s| s.meaning)), - }) - .ok_or(RunError::NoValidation) - }; - let mut languages = Vec::new(); let live = self.corpus.all().map_err(store)?; - for ((language, _, _), language_rows) in held.iter().zip(&rows) { - let (mut rec, mut arm_scores) = (Vec::new(), Vec::new()); - let mut floor_scores: Vec> = vec![Vec::new(); floors.len()]; - let (mut unusable, mut floor_unusable) = (0u32, 0u32); - for (_, r, a, f) in language_rows { - rec.push(next(r)?); - arm_scores.push(next(a)?); - if a.is_none() { - unusable += 1; - } - for (scores, t) in floor_scores.iter_mut().zip(f) { - scores.push(next(t)?); - if t.is_none() { - floor_unusable += 1; - } + let mut languages = Vec::new(); + for ((language, _, _), language_rows) in held.iter().zip(rows) { + let mut unusable = vec![0u32; ports.len()]; + let mut items = Vec::new(); + for (id, _, texts) in language_rows { + let mut item_scores = Vec::new(); + for (p, t) in texts.iter().enumerate() { + item_scores.push(match t { + Some(_) => scores.next().ok_or(RunError::Store)?, + None => { + unusable[p] += 1; + ItemScore { chrf: 0.0, meaning: Some(0.0) } + } + }); } + items.push((id, item_scores)); } - let recorded_reading = reading_of(&rec)?; - languages.push(RelowerReading { + let live_validation = live + .iter() + .filter(|i| &i.language == language && SplitScheme::V1.live(i) == Some(Split::Validation)) + .count() as u32; + languages.push(DrawnLanguage { language: language.clone(), - n: language_rows.len() as u32, - skipped: live - .iter() - .filter(|i| { - &i.language == language && SplitScheme::V1.live(i) == Some(Split::Validation) - }) - .count() as u32 - - language_rows.len() as u32, + skipped: live_validation - items.len() as u32, + items, unusable, - floor_unusable, - recorded: recorded_reading, - arm: reading_of(&arm_scores)?, - floor: if floors.is_empty() { None } else { Some(paired_floor(&rec, &floor_scores)) }, }); } - let verdict = if !floors.is_empty() { - let panel: Vec<(Reading, Reading, Reading)> = languages - .iter() - .filter_map(|l| Some((l.recorded, l.arm, l.floor?))) - .collect(); - Some(decide_panel(&panel)) - } else { - None + Ok(Drawn { languages, waiting }) + } +} + +/// One panel language of [`RunStores::read_draws`]. +struct DrawnLanguage { + language: String, + /// Live Validation items of the language with no lowering prompt. + skipped: u32, + /// Per read item in key order: its id and one score per port. + items: Vec<(String, Vec)>, + /// Per port, answers that were not a usable lowering. + unusable: Vec, +} + +struct Drawn { + /// Per panel language in panel order; empty while any port is waiting. + languages: Vec, + /// Per port, the digests of the prompts it holds no answer to. + waiting: Vec>, +} + +/// A reading over per-item scores; `NoValidation` without items. +fn reading_of(scored: &[ItemScore]) -> Result { + Reading::of(&SplitMean { + count: scored.len(), + chrf: mean(scored.iter().map(|s| s.chrf)), + meaning: mean(scored.iter().filter_map(|s| s.meaning)), + }) + .ok_or(RunError::NoValidation) +} + +/// How trials of one kind read: counts, not rates. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct Counts { + pub trials: u32, + pub accepted: u32, + pub nogain: u32, + pub regressed: u32, +} + +impl Counts { + fn tally(&mut self, v: Verdict) { + self.trials += 1; + match v { + Verdict::Accepted => self.accepted += 1, + Verdict::Rejected(Why::Regressed(_)) => self.regressed += 1, + Verdict::Rejected(_) => self.nogain += 1, + } + } +} + +/// What [`calibration_trials`] counted over one set of items. +#[derive(Clone, Debug, PartialEq)] +pub struct Trials { + /// Ordered (base, candidate) pairs of prompted draws, each over every floor. + pub prompted: Counts, + /// Per degraded arm in the order given: the arm as the candidate. + pub degraded: Vec, + /// Every candidate arm as the candidate, pooled. + pub candidate: Counts, + /// The mean of [`panel_noise`] over every (base, floor) choice. + pub noise: f64, +} + +/// Every `k`-subset of `pool`, in order. +fn subsets(pool: &[usize], k: usize) -> Vec> { + if k == 0 { + return vec![Vec::new()]; + } + let mut out = Vec::new(); + for (i, &first) in pool.iter().enumerate() { + for mut rest in subsets(&pool[i + 1..], k - 1) { + rest.insert(0, first); + out.push(rest); + } + } + out +} + +/// The accept rule replayed over draws that are all lowerings of one round, as +/// the live loop would meet them: [`decide_panel`] over (base, candidate, +/// [`paired_floor`] of the base against `FLOOR_ARMS_MIN` other prompted draws), +/// one row per language. `draws` is `[port][language][item]`; ports +/// `0..prompted` are prompted draws (the recorded lowering first), then +/// `degraded` and `candidate` ports follow. A language with no items reads +/// NaN, which fails closed. +pub fn calibration_trials( + draws: &[Vec>], + prompted: usize, + degraded: usize, + candidate: usize, +) -> Trials { + let languages = draws.first().map_or(0, Vec::len); + let rows = |b: usize, c: usize, floors: &[usize]| -> Vec<(Reading, Reading, Reading)> { + let read = |scores: &[ItemScore]| { + reading_of(scores).unwrap_or(Reading { chrf: f64::NAN, meaning: f64::NAN, composite: f64::NAN }) }; - Ok(RelowerOutcome::Read { languages, verdict }) + (0..languages) + .map(|l| { + let arms: Vec> = floors.iter().map(|&f| draws[f][l].clone()).collect(); + (read(&draws[b][l]), read(&draws[c][l]), paired_floor(&draws[b][l], &arms)) + }) + .collect() + }; + let pool: Vec = (0..prompted).collect(); + let without = |out: &[usize]| -> Vec { pool.iter().copied().filter(|p| !out.contains(p)).collect() }; + let (mut prompted_counts, mut candidate_counts) = (Counts::default(), Counts::default()); + let mut degraded_counts = vec![Counts::default(); degraded]; + let (mut noise, mut noises) = (0.0, 0u32); + for &b in &pool { + for floors in subsets(&without(&[b]), FLOOR_ARMS_MIN) { + for (d, counts) in degraded_counts.iter_mut().enumerate() { + counts.tally(decide_panel(&rows(b, prompted + d, &floors))); + } + for c in prompted + degraded..prompted + degraded + candidate { + candidate_counts.tally(decide_panel(&rows(b, c, &floors))); + } + noise += panel_noise(&rows(b, b, &floors)); + noises += 1; + } + for &c in pool.iter().filter(|&&c| c != b) { + for floors in subsets(&without(&[b, c]), FLOOR_ARMS_MIN) { + prompted_counts.tally(decide_panel(&rows(b, c, &floors))); + } + } + } + Trials { + prompted: prompted_counts, + degraded: degraded_counts, + candidate: candidate_counts, + noise: noise / f64::from(noises), + } +} + +/// One panel language of a class: items read and live Validation items of the +/// class not read. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct ClassLanguage { + pub language: String, + pub n: u32, + pub skipped: u32, +} + +/// The trials over one class of items. +#[derive(Clone, Debug, PartialEq)] +pub struct ClassTrials { + pub name: String, + pub languages: Vec, + pub trials: Trials, +} + +#[derive(Debug)] +pub enum CalibrateOutcome { + /// Draw labels with the number of prompts each has no answer to. + Unanswered(Vec<(String, u32)>), + /// `unusable` over every prompted, degraded and candidate draw; `all` first, + /// then the classes in the order given. + Read { unusable: u32, classes: Vec }, +} + +impl RunStores { + /// Replays the accept rule over answered relower draws: reads every draw of + /// round `n` (the recorded lowering and the `relower--