map: recalibration on the 35-item Validation corpus (#76)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
44b66f94dc
commit
76fa6856b2
1 changed files with 70 additions and 4 deletions
|
|
@ -406,7 +406,7 @@ of 1120 trials, NoGain 71%, Regressed 0%: a net gain of 0.079 against a mean
|
|||
noise term of 0.079. At 14 items per language the rule is therefore a low-power
|
||||
filter for a gain of this size; by 1/sqrt(n), about 35 or more items per
|
||||
language would bring the noise term near 0.05 (an extrapolation, not measured).
|
||||
### Validation growth (`validation-growth`, built; recalibration not run)
|
||||
### Validation growth (`validation-growth`, built and recalibrated)
|
||||
|
||||
Power needs about 35 or more Validation items per language. `data/original`
|
||||
now holds 35 per language (it held 14; Train 14 and Test 8 are unchanged):
|
||||
|
|
@ -441,9 +441,75 @@ packets will be refused at encode (inventory coverage) and score 0 as the means
|
|||
do; they stay in the corpus. That is a result about inventory generalization,
|
||||
not a defect.
|
||||
|
||||
Every calibration number above (92/0/8 prompted rates, the 29% power, the 0.079
|
||||
noise term) is for the 14-item corpus; the corpus digest changed, and the
|
||||
numbers are redone on the grown corpus in a separate recalibration step.
|
||||
Every calibration number in the Distil measure section above (92/0/8 prompted
|
||||
rates, the 29% power, the 0.079 noise term) is for the 14-item corpus and stays
|
||||
the stated baseline of that corpus.
|
||||
|
||||
Recalibration on the grown corpus (`runs/growth-base`, main 44b66f9, panel
|
||||
en-US and ja-JP, round 1 recorded by briefed subagents; 6 prompted draws
|
||||
including the recorded one, 2 mild degradations, an all-"Yes." control and 4
|
||||
Train-guide draws by Claude sonnet-5-5; the same decision rule at FLOOR_Z 3 and
|
||||
3-arm floors; 120 prompted trials from the 30 ordered pairs of 6 draws with the
|
||||
other 4 as floor candidates, so trials share draws and are not independent).
|
||||
Soul pass `cut-validation-growth.s3` re-derived every number below; its
|
||||
findings shaped the wording.
|
||||
|
||||
- Live Validation items: 28 en-US and 32 ja-JP of 35 each; 10 are refused at
|
||||
parse (en-US original:18, 23, 25, 27 and vgrow:4, 5, 6; ja-JP original-ja:17,
|
||||
22 and vgrow-ja:2); `relower` skips an item with no packet, so every reading
|
||||
here is over live items only. Old 14: 10 en-US and 12 ja-JP live; new 21: 18
|
||||
and 20.
|
||||
- Prompted repeat: NoGain 96.7%, Accepted 3.3%, Regressed 0% (14 items: 92 / 0 /
|
||||
8). All four false accepts are one pair, base p6 and candidate the recorded
|
||||
lowering R: R comes from the round, the other five draws from `relower`, and
|
||||
in en-US R scores +0.024 per item above them [+0.002, +0.052], so R is not
|
||||
exchangeable with a repeat (a Plausible Soul finding, cause not isolated). No
|
||||
false-accept bound can be drawn from 6 draws.
|
||||
- Degraded arms read Regressed: all-"Yes." 100% (as before); mild-1 31.7%;
|
||||
mild-2 0% (14 items: 100% and 75% on a subtler arm). The mild arms are weak
|
||||
and not independent: they were rewritten from R with one detail blurred, the
|
||||
original wording is not on disk, mild-2 equals R word for word in 5 of 60
|
||||
items and mild-1 in 2, and mild-1 equals mild-2 in 27 of 60 although they
|
||||
were asked to blur different details. Their mean composite is 0.003 to 0.015
|
||||
below the prompted mean (the old subtler arm: 0.037 and 0.052). The
|
||||
sensitivity to a small regression is not re-measured.
|
||||
- Noise: the panel noise term of the sum rule is 0.045 on average over the base
|
||||
and floor choices (0.031 to 0.064), against 0.079 at 14 items. The 1/sqrt(n)
|
||||
forecast (about 0.05) assumed per-item variance stayed the same; it did not:
|
||||
the per-item SD of the difference between two prompted draws is about 0.085
|
||||
on the old items and 0.034 on the new ones, so the pooled term landed near
|
||||
the forecast because less variable items were added. Not a confirmation of
|
||||
the forecast.
|
||||
- Power: the Train-guide arm (14 Train sentences as a register guide) is
|
||||
Accepted in 21.2% of 240 trials, NoGain 78.8%, against 29% at 14 items, and
|
||||
its mean composite gain over the prompted draws is 0.0185, 0.41 of the mean
|
||||
noise term (0.37 on the first floor choice per base); with `relower`-only
|
||||
bases (R removed) it is Accepted in 35% (28 of 80). Split by item class
|
||||
(paired bootstrap over items, 95%): old items +0.029 [+0.005, +0.052] (en-US
|
||||
+0.049 [+0.023, +0.084] on 10 items, ja-JP +0.013 [-0.021, +0.037] on 12);
|
||||
new items -0.002 [-0.011, +0.007] pooled, but two opposite effects: en-US
|
||||
-0.017 [-0.027, -0.007] on 18 items and ja-JP +0.011 [+0.001, +0.022] on 20.
|
||||
So the old-versus-new contrast exists in en-US; in ja-JP old (+0.013) and new
|
||||
(+0.011) do not differ. Measured: the 14-item gain does not carry to the new
|
||||
en-US items. Inferred, not isolated: that it was in-sample register fit. The
|
||||
new items differ from the old in several ways at once (written free of the
|
||||
inventory, longer, a different scene convention, lower scores, a smaller
|
||||
vocabulary overlap with Train in en-US: 0.29 against 0.52) and were briefed to
|
||||
the same register as Train, by Claude as the old were. The 29% power figure
|
||||
stays the stated baseline of the 14-item corpus; no better arm is known on
|
||||
the new items, so the rule's power against a real gain is not re-measured.
|
||||
- The two classes are different strata (Soul s2 `register-stratum-shift`): the
|
||||
new items score lower (composite 0.41 against 0.64 pooled) so a pooled
|
||||
60-item mean mixes them; report old and new separately.
|
||||
- The self-preference effect is not measured (ruling
|
||||
`validation-growth-writer-claude-checked`). The new scenes use a different
|
||||
`in_group` convention from the old ones (follow-up
|
||||
`vgrow-scene-in-group-convention`), so the new-item scores are for scenes that
|
||||
will be re-annotated; the corpus digest changes again then.
|
||||
|
||||
Evidence lives in the Self workspace (`cal2/`, `runs/growth-base`), not in git,
|
||||
and uses a DUMP patch kept out of the repo; no committed harness reruns these
|
||||
numbers (Soul finding `evidence-outside-git`, follow-up owed).
|
||||
|
||||
### Smallest measurable distil slice (`distil-incontext`, not built)
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue