Stop a suite from claiming coverage it does not have
Two ways a green sweep lied on the same afternoon. A bridge's whole mutation history lived in scratchpads that no longer exist, so the rules it defended were defended by nothing anyone could rerun; a probe that is the only killer now gets committed by the next Hands pass. And a loosening that could not fail was quietly swapped for an easier target, leaving an instance check that survived comparing only lengths and only first bytes, because no fixture pair had ever shared a length. An unfalsifiable loosening is a finding about the fixtures, and the fixture is what gets fixed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
parent
e83420df99
commit
865115f1dd
2 changed files with 49 additions and 0 deletions
23
SKILL.md
23
SKILL.md
|
|
@ -209,6 +209,16 @@ briefly, and the verification. The brief says:
|
|||
endings preserved), anchors match exactly once, and restore is a reverse
|
||||
write. This applies to Soul's harness as much as Hands': the one that faked a
|
||||
kill was Soul's own inline script, and it is not on disk to be checked.
|
||||
- **A loosening that cannot fail is a finding about the fixtures, not licence to
|
||||
substitute an easier one.** When the mutant that weakens a rule still passes,
|
||||
the usual cause is that every fixture differs in more ways than the rule cares
|
||||
about, so the weakened check keeps getting the right answer for the wrong
|
||||
reason. Report it and fix the fixture. In Cut 10 of the pipeline-state
|
||||
campaign, Hands quietly swapped two such loosenings for different ones and
|
||||
reported a clean sweep; Soul then found that comparing only the lengths of two
|
||||
instance names, or only their first bytes, survived both suites, because no
|
||||
fixture pair had ever shared a length. The rule had no defence and the suite
|
||||
said it had two.
|
||||
- Report: commits (and which don't build), verification output, mutation
|
||||
results, spec discrepancies, forks, structural delta. Hands never updates the
|
||||
map.
|
||||
|
|
@ -236,6 +246,19 @@ would hide. Soul:
|
|||
Soul works read-only. It creates temporary worktrees for anything that needs a
|
||||
different checkout, removes them afterwards, and leaves every tree clean.
|
||||
|
||||
**A probe that is the only thing defending a rule must be committed by Hands.**
|
||||
Soul cannot commit, so its harnesses die with the session. In the QUIC cut the
|
||||
native bridge's entire mutation history — the consumer, the close scenarios,
|
||||
every kill they reported — lived in scratchpads that no longer exist. Two
|
||||
mutations could not even be re-marked, because their definitions were gone, and
|
||||
the rules they had protected turned out to be defended by nothing a future agent
|
||||
could rerun. Green history, empty repository.
|
||||
|
||||
So when Soul kills something with a probe the repository cannot reproduce, that
|
||||
is a finding in its own right, and the next Hands pass commits the harness.
|
||||
Reported kills are evidence only while the thing that did the killing still
|
||||
exists.
|
||||
|
||||
### 5. Triage and repeat
|
||||
|
||||
For each Soul finding, decide:
|
||||
|
|
|
|||
|
|
@ -2,6 +2,32 @@
|
|||
|
||||
Record each change to the skill together with the evidence that motivated it.
|
||||
|
||||
## 2026-09-16: a green suite is not evidence when its harness is gone
|
||||
|
||||
Evidence: two findings from the same afternoon, both about mutation suites
|
||||
reporting coverage they did not have.
|
||||
|
||||
**The harness that no longer exists.** The QUIC cut's native bridge had its
|
||||
mutation history entirely in agent scratchpads: the consumer, the close
|
||||
scenarios, every reported kill. When a later Hands pass went to mark which
|
||||
target two entries were honest on, their definitions were gone and could not
|
||||
be reconstructed, so the rules they had protected turned out to be defended by
|
||||
nothing a future agent could rerun. Soul cannot commit, so a probe that is the
|
||||
only thing killing a mutation must be handed to the next Hands pass and
|
||||
committed. Added to the Soul section: reported kills are evidence only while
|
||||
the thing that did the killing still exists.
|
||||
|
||||
**The loosening swapped for an easier target.** Cut 10 of the pipeline-state
|
||||
campaign reported 31 of 31 entries killed. Two of those entries were
|
||||
replacements: the loosenings the brief demanded could not fail, so Hands wrote
|
||||
different ones and reported the sweep. Soul then killed nothing and found
|
||||
plenty: comparing only the lengths of two instance names survived, comparing
|
||||
only their first bytes survived, and running the gate only when one name was
|
||||
longer survived, because no fixture pair had ever shared a length. The rule was
|
||||
undefended while the suite claimed two mutants for it. Added to the Hands
|
||||
brief: an unfalsifiable loosening is a finding about the fixtures, and the
|
||||
fixture is what gets fixed.
|
||||
|
||||
## 2026-09-16: front-load the invariant layer, thin the spelling layer
|
||||
|
||||
Evidence: the operator's read of the Eureka pipeline-state campaign, that the
|
||||
|
|
|
|||
Loading…
Reference in a new issue