diff --git a/SKILL.md b/SKILL.md index 0d1b9cf..f265239 100644 --- a/SKILL.md +++ b/SKILL.md @@ -163,6 +163,13 @@ briefly, and the verification. The brief says: - **Every operator ruling and every new rule gets a test that fails under its own mutation.** Hands runs the mutations against the final spelling of the code and restores afterwards. +- **Every mutation suite is a committed script with a no-op control.** The + control rewrites the target through the same I/O path with no change and must + leave every test green; if it kills anything, the harness is broken and every + verdict from it is fiction. File I/O is byte-exact (symmetric UTF-8, line + endings preserved), anchors match exactly once, and restore is a reverse + write. This applies to Soul's harness as much as Hands': the one that faked a + kill was Soul's own inline script, and it is not on disk to be checked. - Report: commits (and which don't build), verification output, mutation results, spec discrepancies, forks, structural delta. Hands never updates the map. diff --git a/references/briefs.md b/references/briefs.md index 241ab02..3e73ca1 100644 --- a/references/briefs.md +++ b/references/briefs.md @@ -94,6 +94,16 @@ Long jobs: wait for them and finish. Never end the turn with a to-do list in place of a report. Mutations: restore with a reverse edit or run against committed code; `git checkout` also reverts uncommitted fix code and silently invalidates the run. +The suite is a committed script under `tools/`, and its first entry is a +no-op control: rewrite the target through the same I/O path with no change +and run the suite; every test must stay green, or the harness is broken and +nothing it reports counts. Read and write bytes symmetrically (in PowerShell, +`[IO.File]::ReadAllText`/`WriteAllText` with an explicit UTF-8 encoding, never +a bare `Get-Content`/`Set-Content`; in Python, `encoding="utf-8", newline=""` +both ways). A `-Raw`-less read or a line join rewrites every line ending, and +`core.autocrlf` hides that from `git diff`. Sources carry non-ASCII literals +on purpose, so a lossy round-trip fails tests on its own and fakes a kill on +every mutation. Warnings: measure from a forced rebuild and compare distinct messages; cargo replays warnings only when it actually rebuilds. Semantic properties ("exactly one call site", "this step actually runs"): @@ -173,6 +183,12 @@ Falsify specifically: - - rerun the builds, tests and captures yourself +Your own mutations obey the same harness rules as Hands': write them as a +script in , run a no-op control through it first, use byte-exact +symmetric I/O, and name the script in the report so the run can be checked. +An inline harness typed for one run is not evidence, and it is where the one +known fake kill came from. + Report each finding as CONFIRMED or PLAUSIBLE, with file:line, a failure scenario and severity. Then give a short list of the promises that held. ``` diff --git a/references/changelog.md b/references/changelog.md index 0cd9222..720c6b7 100644 --- a/references/changelog.md +++ b/references/changelog.md @@ -2,6 +2,32 @@ Record each change to the skill together with the evidence that motivated it. +## 2026-09-16: a mutation suite needs a no-op control and byte-exact I/O + +Evidence: the Eureka pipeline-state campaign's third Soul pass on Cut 6, and the +Eyes audit that followed (Epiphany `notes/eureka-pipeline-state-cut.md`, "Mutation +harness audit"). + +- **A no-op control is the test of the harness.** Soul ran a mutation that + changed nothing and it killed `bounds_refuse_in_utf8_bytes`. The harness's + text round-trip had collapsed an `é` literal to one byte, so the corruption + alone failed the test and every mutation through that path would have + reported a kill. Nothing else in the loop could have noticed: a killed + mutation is what a working suite looks like. +- **The defect was not where it was feared.** The audit found every committed + suite using byte-lossless I/O. The corrupting harness was Soul's own inline + script, never written to disk, so its text is unrecoverable and its verdicts + are unverifiable. That is the same shape as the standing finding that + mutations without artifacts cannot be checked; it now applies to Soul too. +- **Encoding symmetry and line endings, not a flag.** Measured on the host: + mixed-encoding round-trips corrupt non-ASCII; any read that splits lines + rewrites every line ending regardless of encoding, and `core.autocrlf` hides + the rewrite from `git diff`. One ad-hoc Cut 1 script did exactly that, so + three early verdicts are recorded as suspect. +- **The committed suites forbade a control by shape.** Each threw when a + replacement changed nothing. A table shape that cannot express "change + nothing" cannot test itself. + ## 2026-09-16: record the gap, and absence is the hardest claim Evidence: ghostlight-77's requirements message to the Eureka pipeline-state