Require a no-op control and byte-exact I/O in every mutation suite

A Soul pass ran a mutation that changed nothing and it killed a test. The
harness's text round-trip had collapsed a non-ASCII literal, so the
corruption alone failed the test and every mutation through that path would
have reported a kill. A working suite and a broken one look the same from
the verdicts; only a control tells them apart.

The audit that followed found every committed suite lossless and the
corrupting script inline and gone, which is the standing "no artifact on
disk" finding applied to Soul. So the rule binds Soul's harness as well as
Hands', and the Hands and Soul briefs both carry it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Meta Vens 2026-09-16 13:40:28 +02:00
commit 12d1b9c0f8
3 changed files with 49 additions and 0 deletions

View file

@ -163,6 +163,13 @@ briefly, and the verification. The brief says:
- **Every operator ruling and every new rule gets a test that fails under its own
mutation.** Hands runs the mutations against the final spelling of the code and
restores afterwards.
- **Every mutation suite is a committed script with a no-op control.** The
control rewrites the target through the same I/O path with no change and must
leave every test green; if it kills anything, the harness is broken and every
verdict from it is fiction. File I/O is byte-exact (symmetric UTF-8, line
endings preserved), anchors match exactly once, and restore is a reverse
write. This applies to Soul's harness as much as Hands': the one that faked a
kill was Soul's own inline script, and it is not on disk to be checked.
- Report: commits (and which don't build), verification output, mutation
results, spec discrepancies, forks, structural delta. Hands never updates the
map.

View file

@ -94,6 +94,16 @@ Long jobs: wait for them and finish. Never end the turn with a to-do list in
place of a report.
Mutations: restore with a reverse edit or run against committed code; `git
checkout` also reverts uncommitted fix code and silently invalidates the run.
The suite is a committed script under `tools/`, and its first entry is a
no-op control: rewrite the target through the same I/O path with no change
and run the suite; every test must stay green, or the harness is broken and
nothing it reports counts. Read and write bytes symmetrically (in PowerShell,
`[IO.File]::ReadAllText`/`WriteAllText` with an explicit UTF-8 encoding, never
a bare `Get-Content`/`Set-Content`; in Python, `encoding="utf-8", newline=""`
both ways). A `-Raw`-less read or a line join rewrites every line ending, and
`core.autocrlf` hides that from `git diff`. Sources carry non-ASCII literals
on purpose, so a lossy round-trip fails tests on its own and fakes a kill on
every mutation.
Warnings: measure from a forced rebuild and compare distinct messages; cargo
replays warnings only when it actually rebuilds.
Semantic properties ("exactly one call site", "this step actually runs"):
@ -173,6 +183,12 @@ Falsify specifically:
- <leftover greps>
- rerun the builds, tests and captures yourself
Your own mutations obey the same harness rules as Hands': write them as a
script in <scratchpad>, run a no-op control through it first, use byte-exact
symmetric I/O, and name the script in the report so the run can be checked.
An inline harness typed for one run is not evidence, and it is where the one
known fake kill came from.
Report each finding as CONFIRMED or PLAUSIBLE, with file:line, a failure
scenario and severity. Then give a short list of the promises that held.
```

View file

@ -2,6 +2,32 @@
Record each change to the skill together with the evidence that motivated it.
## 2026-09-16: a mutation suite needs a no-op control and byte-exact I/O
Evidence: the Eureka pipeline-state campaign's third Soul pass on Cut 6, and the
Eyes audit that followed (Epiphany `notes/eureka-pipeline-state-cut.md`, "Mutation
harness audit").
- **A no-op control is the test of the harness.** Soul ran a mutation that
changed nothing and it killed `bounds_refuse_in_utf8_bytes`. The harness's
text round-trip had collapsed an `é` literal to one byte, so the corruption
alone failed the test and every mutation through that path would have
reported a kill. Nothing else in the loop could have noticed: a killed
mutation is what a working suite looks like.
- **The defect was not where it was feared.** The audit found every committed
suite using byte-lossless I/O. The corrupting harness was Soul's own inline
script, never written to disk, so its text is unrecoverable and its verdicts
are unverifiable. That is the same shape as the standing finding that
mutations without artifacts cannot be checked; it now applies to Soul too.
- **Encoding symmetry and line endings, not a flag.** Measured on the host:
mixed-encoding round-trips corrupt non-ASCII; any read that splits lines
rewrites every line ending regardless of encoding, and `core.autocrlf` hides
the rewrite from `git diff`. One ad-hoc Cut 1 script did exactly that, so
three early verdicts are recorded as suspect.
- **The committed suites forbade a control by shape.** Each threw when a
replacement changed nothing. A table shape that cannot express "change
nothing" cannot test itself.
## 2026-09-16: record the gap, and absence is the hardest claim
Evidence: ghostlight-77's requirements message to the Eureka pipeline-state