map: recalibration wording after Soul s3 (ja effect, R, mild arms, noise stratum)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
915863b5b3
commit
2b29efbc2d
1 changed files with 46 additions and 35 deletions
|
|
@ -450,55 +450,66 @@ en-US and ja-JP, round 1 recorded by briefed subagents; 6 prompted draws
|
|||
including the recorded one, 2 mild degradations, an all-"Yes." control and 4
|
||||
Train-guide draws by Claude sonnet-5-5; the same decision rule at FLOOR_Z 3 and
|
||||
3-arm floors; 120 prompted trials from the 30 ordered pairs of 6 draws with the
|
||||
other 4 as floor candidates, so trials share draws and are not independent):
|
||||
other 4 as floor candidates, so trials share draws and are not independent).
|
||||
Soul pass `cut-validation-growth.s3` re-derived every number below; its
|
||||
findings shaped the wording.
|
||||
|
||||
- Live Validation items: 28 en-US and 32 ja-JP of 35 each; 10 are refused at
|
||||
parse (en-US original:18, 23, 25, 27 and vgrow:4, 5, 6; ja-JP original-ja:17,
|
||||
22 and vgrow-ja:2) and score 0 in every arm. Old 14: 10 en-US and 12 ja-JP
|
||||
live; new 21: 18 and 20.
|
||||
22 and vgrow-ja:2); `relower` skips an item with no packet, so every reading
|
||||
here is over live items only. Old 14: 10 en-US and 12 ja-JP live; new 21: 18
|
||||
and 20.
|
||||
- Prompted repeat: NoGain 96.7%, Accepted 3.3%, Regressed 0% (14 items: 92 / 0 /
|
||||
8). The false-accept bound is no better than before: 3.3% rests on 120 trials
|
||||
over 6 draws.
|
||||
8). All four false accepts are one pair, base p6 and candidate the recorded
|
||||
lowering R: R comes from the round, the other five draws from `relower`, and
|
||||
in en-US R scores +0.024 per item above them [+0.002, +0.052], so R is not
|
||||
exchangeable with a repeat (a Plausible Soul finding, cause not isolated). No
|
||||
false-accept bound can be drawn from 6 draws.
|
||||
- Degraded arms read Regressed: all-"Yes." 100% (as before); mild-1 31.7%;
|
||||
mild-2 0% (14 items: 100% and 75% on a subtler arm). Not comparable: these
|
||||
mild arms were rewritten from the recorded lowering by blurring one detail
|
||||
and move the mean composite by only 0.003 to 0.015, against 0.044 for the old
|
||||
subtler arm; the original mild wording is not on disk, so this is a weaker
|
||||
degradation, and the rule's sensitivity to a real small regression is
|
||||
unchanged in kind but not re-measured at the old strength.
|
||||
- Noise: the panel noise term of the sum rule is 0.045 on average over the
|
||||
base and floor choices (0.031 to 0.064), against 0.079 at 14 items; by
|
||||
1/sqrt(n) the extrapolation said near 0.05. That part of the extrapolation
|
||||
held.
|
||||
mild-2 0% (14 items: 100% and 75% on a subtler arm). The mild arms are weak
|
||||
and not independent: they were rewritten from R with one detail blurred, the
|
||||
original wording is not on disk, mild-2 equals R word for word in 5 of 60
|
||||
items and mild-1 in 2, and mild-1 equals mild-2 in 27 of 60 although they
|
||||
were asked to blur different details. Their mean composite is 0.003 to 0.015
|
||||
below the prompted mean (the old subtler arm: 0.037 and 0.052). The
|
||||
sensitivity to a small regression is not re-measured.
|
||||
- Noise: the panel noise term of the sum rule is 0.045 on average over the base
|
||||
and floor choices (0.031 to 0.064), against 0.079 at 14 items. The 1/sqrt(n)
|
||||
forecast (about 0.05) assumed per-item variance stayed the same; it did not:
|
||||
the per-item SD of the difference between two prompted draws is about 0.085
|
||||
on the old items and 0.034 on the new ones, so the pooled term landed near
|
||||
the forecast because less variable items were added. Not a confirmation of
|
||||
the forecast.
|
||||
- Power: the Train-guide arm (14 Train sentences as a register guide) is
|
||||
Accepted in 21.2% of 240 trials, NoGain 78.8%, against 29% at 14 items, and
|
||||
its mean gain over the prompted draws fell from 0.079 to 0.018 (gain over
|
||||
noise 0.37). Split by item class (paired bootstrap over items, 95%): the
|
||||
gain is on the old Validation items, composite +0.029 [+0.005, +0.052] (en-US
|
||||
+0.049, ja-JP +0.013 with an interval through zero; 22 items), and nil on the
|
||||
new ones, -0.002 [-0.011, +0.007] (en-US -0.017 [-0.027, -0.007], ja-JP
|
||||
+0.011; 38 items). The old items share a writer, a vocabulary and a register
|
||||
with Train; the new ones were written free of both. So the 14-item power
|
||||
figure measured a gain that was in-sample register fit on text shaped like
|
||||
the guide, and the "better arm" premise does not hold on independent
|
||||
Validation. This is a measured difference; its cause is inferred, not
|
||||
isolated. The power of the rule against a gain that exists is therefore not
|
||||
re-measured: no better arm is known on the new items.
|
||||
its mean composite gain over the prompted draws is 0.0185, 0.41 of the mean
|
||||
noise term (0.37 on the first floor choice per base); with `relower`-only
|
||||
bases (R removed) it is Accepted in 35% (28 of 80). Split by item class
|
||||
(paired bootstrap over items, 95%): old items +0.029 [+0.005, +0.052] (en-US
|
||||
+0.049 [+0.023, +0.084] on 10 items, ja-JP +0.013 [-0.021, +0.037] on 12);
|
||||
new items -0.002 [-0.011, +0.007] pooled, but two opposite effects: en-US
|
||||
-0.017 [-0.027, -0.007] on 18 items and ja-JP +0.011 [+0.001, +0.022] on 20.
|
||||
So the old-versus-new contrast exists in en-US; in ja-JP old (+0.013) and new
|
||||
(+0.011) do not differ. Measured: the 14-item gain does not carry to the new
|
||||
en-US items. Inferred, not isolated: that it was in-sample register fit. The
|
||||
new items differ from the old in several ways at once (written free of the
|
||||
inventory, longer, a different scene convention, lower scores, a smaller
|
||||
vocabulary overlap with Train in en-US: 0.29 against 0.52) and were briefed to
|
||||
the same register as Train, by Claude as the old were. The 29% power figure
|
||||
stays the stated baseline of the 14-item corpus; no better arm is known on
|
||||
the new items, so the rule's power against a real gain is not re-measured.
|
||||
- The two classes are different strata (Soul s2 `register-stratum-shift`): the
|
||||
new items score lower (composite 0.41 against 0.64 pooled) and vary less per
|
||||
item between draws (SD of the difference between two prompted draws 0.034
|
||||
against 0.085), so a pooled 60-item mean mixes them; report old and new
|
||||
separately.
|
||||
new items score lower (composite 0.41 against 0.64 pooled) so a pooled
|
||||
60-item mean mixes them; report old and new separately.
|
||||
- The self-preference effect is not measured (ruling
|
||||
`validation-growth-writer-claude-checked`). The new scenes use a different
|
||||
`in_group` convention from the old ones (follow-up
|
||||
`vgrow-scene-in-group-convention`), so the new-item scores are for scenes that
|
||||
will be re-annotated; the corpus digest changes again then.
|
||||
|
||||
Measured: everything listed above. Extrapolated: nothing new; the 1/sqrt(n)
|
||||
noise forecast is now a measurement, the old power figure is withdrawn as
|
||||
in-sample. Evidence: `cal2/` and `runs/growth-base` in the Weksa Self
|
||||
workspace, not in git.
|
||||
Evidence lives in the Self workspace (`cal2/`, `runs/growth-base`), not in git,
|
||||
and uses a DUMP patch kept out of the repo; no committed harness reruns these
|
||||
numbers (Soul finding `evidence-outside-git`, follow-up owed).
|
||||
|
||||
### Smallest measurable distil slice (`distil-incontext`, not built)
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue