Require a no-op control and byte-exact I/O in every mutation suite
A Soul pass ran a mutation that changed nothing and it killed a test. The harness's text round-trip had collapsed a non-ASCII literal, so the corruption alone failed the test and every mutation through that path would have reported a kill. A working suite and a broken one look the same from the verdicts; only a control tells them apart. The audit that followed found every committed suite lossless and the corrupting script inline and gone, which is the standing "no artifact on disk" finding applied to Soul. So the rule binds Soul's harness as well as Hands', and the Hands and Soul briefs both carry it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
parent
e1fd2a1d66
commit
12d1b9c0f8
3 changed files with 49 additions and 0 deletions
7
SKILL.md
7
SKILL.md
|
|
@ -163,6 +163,13 @@ briefly, and the verification. The brief says:
|
|||
- **Every operator ruling and every new rule gets a test that fails under its own
|
||||
mutation.** Hands runs the mutations against the final spelling of the code and
|
||||
restores afterwards.
|
||||
- **Every mutation suite is a committed script with a no-op control.** The
|
||||
control rewrites the target through the same I/O path with no change and must
|
||||
leave every test green; if it kills anything, the harness is broken and every
|
||||
verdict from it is fiction. File I/O is byte-exact (symmetric UTF-8, line
|
||||
endings preserved), anchors match exactly once, and restore is a reverse
|
||||
write. This applies to Soul's harness as much as Hands': the one that faked a
|
||||
kill was Soul's own inline script, and it is not on disk to be checked.
|
||||
- Report: commits (and which don't build), verification output, mutation
|
||||
results, spec discrepancies, forks, structural delta. Hands never updates the
|
||||
map.
|
||||
|
|
|
|||
|
|
@ -94,6 +94,16 @@ Long jobs: wait for them and finish. Never end the turn with a to-do list in
|
|||
place of a report.
|
||||
Mutations: restore with a reverse edit or run against committed code; `git
|
||||
checkout` also reverts uncommitted fix code and silently invalidates the run.
|
||||
The suite is a committed script under `tools/`, and its first entry is a
|
||||
no-op control: rewrite the target through the same I/O path with no change
|
||||
and run the suite; every test must stay green, or the harness is broken and
|
||||
nothing it reports counts. Read and write bytes symmetrically (in PowerShell,
|
||||
`[IO.File]::ReadAllText`/`WriteAllText` with an explicit UTF-8 encoding, never
|
||||
a bare `Get-Content`/`Set-Content`; in Python, `encoding="utf-8", newline=""`
|
||||
both ways). A `-Raw`-less read or a line join rewrites every line ending, and
|
||||
`core.autocrlf` hides that from `git diff`. Sources carry non-ASCII literals
|
||||
on purpose, so a lossy round-trip fails tests on its own and fakes a kill on
|
||||
every mutation.
|
||||
Warnings: measure from a forced rebuild and compare distinct messages; cargo
|
||||
replays warnings only when it actually rebuilds.
|
||||
Semantic properties ("exactly one call site", "this step actually runs"):
|
||||
|
|
@ -173,6 +183,12 @@ Falsify specifically:
|
|||
- <leftover greps>
|
||||
- rerun the builds, tests and captures yourself
|
||||
|
||||
Your own mutations obey the same harness rules as Hands': write them as a
|
||||
script in <scratchpad>, run a no-op control through it first, use byte-exact
|
||||
symmetric I/O, and name the script in the report so the run can be checked.
|
||||
An inline harness typed for one run is not evidence, and it is where the one
|
||||
known fake kill came from.
|
||||
|
||||
Report each finding as CONFIRMED or PLAUSIBLE, with file:line, a failure
|
||||
scenario and severity. Then give a short list of the promises that held.
|
||||
```
|
||||
|
|
|
|||
|
|
@ -2,6 +2,32 @@
|
|||
|
||||
Record each change to the skill together with the evidence that motivated it.
|
||||
|
||||
## 2026-09-16: a mutation suite needs a no-op control and byte-exact I/O
|
||||
|
||||
Evidence: the Eureka pipeline-state campaign's third Soul pass on Cut 6, and the
|
||||
Eyes audit that followed (Epiphany `notes/eureka-pipeline-state-cut.md`, "Mutation
|
||||
harness audit").
|
||||
|
||||
- **A no-op control is the test of the harness.** Soul ran a mutation that
|
||||
changed nothing and it killed `bounds_refuse_in_utf8_bytes`. The harness's
|
||||
text round-trip had collapsed an `é` literal to one byte, so the corruption
|
||||
alone failed the test and every mutation through that path would have
|
||||
reported a kill. Nothing else in the loop could have noticed: a killed
|
||||
mutation is what a working suite looks like.
|
||||
- **The defect was not where it was feared.** The audit found every committed
|
||||
suite using byte-lossless I/O. The corrupting harness was Soul's own inline
|
||||
script, never written to disk, so its text is unrecoverable and its verdicts
|
||||
are unverifiable. That is the same shape as the standing finding that
|
||||
mutations without artifacts cannot be checked; it now applies to Soul too.
|
||||
- **Encoding symmetry and line endings, not a flag.** Measured on the host:
|
||||
mixed-encoding round-trips corrupt non-ASCII; any read that splits lines
|
||||
rewrites every line ending regardless of encoding, and `core.autocrlf` hides
|
||||
the rewrite from `git diff`. One ad-hoc Cut 1 script did exactly that, so
|
||||
three early verdicts are recorded as suspect.
|
||||
- **The committed suites forbade a control by shape.** Each threw when a
|
||||
replacement changed nothing. A table shape that cannot express "change
|
||||
nothing" cannot test itself.
|
||||
|
||||
## 2026-09-16: record the gap, and absence is the hardest claim
|
||||
|
||||
Evidence: ghostlight-77's requirements message to the Eureka pipeline-state
|
||||
|
|
|
|||
Loading…
Reference in a new issue