diff --git a/tools/INDEX.md b/tools/INDEX.md index deb8c50..5a5caf1 100644 --- a/tools/INDEX.md +++ b/tools/INDEX.md @@ -8,7 +8,7 @@ gets a Soul pass before merge. | script | purpose | consumer | named by | |---|---|---|---| -| `tools/stopgap/ygg-verify.sh` | Pushes an exact revision to a mirror on Yggdrasil and runs one command in a capped, niced container. `SUMMARY=` keeps the full log and prints a bounded summary (verdict, counts, ZERO TESTS RAN, failing tests); `wait ` blocks at most 270 s and returns the summary once; call it with a Bash tool `timeout` of at least 280000 ms. Dies with the stopgap when Idunn's verify transaction lands. | Hands, Soul, Imagination probes | `references/briefs.md` (Hands long jobs and verification, Soul long jobs), `SKILL.md` (Yggdrasil) | -| `tools/mutation-table.sh` | One verify job that runs a list of hand mutants (file, before, after, test) against a revision and returns a kill table: CAUGHT with the killing test, SURVIVED, UNVIABLE, TIMEOUT, NOAPPLY, ERROR. Refuses a red or zero-test baseline and a red BUILD; rows print live; the whole tree is restored after each mutant. Replaces per-agent runmut/mutrun/strykrun scripts. | Soul, Hands (scoped run) | `references/briefs.md` (Hands verification, Soul reruns) | -| `tools/testparse.sh` | Sourced library: failing test names and pass/fail/skip counts from cargo, dotnet, vitest and pytest logs. Shared by the summary and the kill table. | the two scripts above | not named by a brief; it lives and dies with them | -| `tools/test-tools.sh` | Local fixture check of the parsers and the summary over `tools/fixtures/logs` (real cargo and mutation logs, plus hand-written dotnet, vitest, pytest, exit-0-but-failed and no-verdict logs). Run it before merging a change to the scripts above. | whoever changes those scripts | `SKILL.md` (Shared tools) | +| `tools/stopgap/ygg-verify.sh` | Pushes an exact revision to a mirror on Yggdrasil and runs one command in a capped, niced container (`RUN_ON_HOST=1`: run a host-side command in the same slot, clone and watchdog instead; the watchdog kills every container carrying the job label). `SUMMARY=` keeps the full log and prints a bounded summary (verdict, counts, ZERO TESTS RAN, failing tests); `wait ` blocks at most 270 s and returns the summary once; call it with a Bash tool `timeout` of at least 280000 ms. Dies with the stopgap when Idunn's verify transaction lands. | Hands, Soul, Imagination probes | `references/briefs.md` (Hands long jobs and verification, Soul long jobs), `SKILL.md` (Yggdrasil) | +| `tools/mutation-table.sh` | One verify job that runs a list of hand mutants (file, before, after, test) against a revision and returns a kill table: CAUGHT with the killing test, SURVIVED, UNVIABLE, TIMEOUT, NOAPPLY, ERROR. Refuses a red or zero-test baseline and a red BUILD; rows print live. Isolation is Docker's: the baselines run once in a container that holds the revision, that container is committed to a throwaway image, and every mutant runs in a fresh `--rm` container from it, so nothing a run writes (tracked, untracked or ignored, such as `target/`) reaches the next mutant; the image is removed when the job ends, killed or not. The shared registry and sccache volumes and the network are not isolated. A file-level load failure (vitest `FAIL [ ]`, jest `FAIL ` plus `Test suite failed to run`) reads UNVIABLE. Replaces per-agent runmut/mutrun/strykrun scripts. | Soul, Hands (scoped run) | `references/briefs.md` (Hands verification, Soul reruns) | +| `tools/testparse.sh` | Sourced library: failing test names and pass/fail/skip counts from cargo, dotnet, vitest, jest and pytest logs, and test files that failed to load (`LOADFAIL`). Shared by the summary and the kill table. | the two scripts above | not named by a brief; it lives and dies with them | +| `tools/test-tools.sh` | Local fixture check of the parsers and the summary over `tools/fixtures/logs` (real cargo and mutation logs, plus hand-written dotnet, vitest, jest, pytest, exit-0-but-failed and no-verdict logs), and of the mutation runner's host driver and container half under a mock docker. Run it before merging a change to the scripts above. | whoever changes those scripts | `SKILL.md` (Shared tools) | diff --git a/tools/fixtures/logs/mutation-table-nonce.log b/tools/fixtures/logs/mutation-table-nonce.log index 4e18302..92a1506 100644 --- a/tools/fixtures/logs/mutation-table-nonce.log +++ b/tools/fixtures/logs/mutation-table-nonce.log @@ -1,9 +1,9 @@ warning: You appear to have cloned an empty repository. -ygg-verify: probe@7d3a1f5284 slot 2, image eureka-verify-rust:b99d30ec441e, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/probe-7d3a1f5284-Elfg -YGG-SUMMARY/c0ffee12: MUTATION 5 mutants: caught=2 survived=1 unviable=1 timeout=0 noapply=1 error=0 (baseline green, tests ran) -YGG-SUMMARY/c0ffee12: clamp-positive | SURVIVED | tests passed with the mutation applied: cargo test -YGG-SUMMARY/c0ffee12: add-typo | UNVIABLE | build exit 101 -YGG-SUMMARY/c0ffee12: ghost | NOAPPLY | before text not found -YGG-SUMMARY/c0ffee12: add-minus | CAUGHT | tests::add_adds -YGG-SUMMARY/c0ffee12: even-flip | CAUGHT | tests::even_detects +ygg-verify: tool-calling-s5@675669150b slot 1, image eureka-verify-rust:b99d30ec441e, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/tool-calling-s5-675669150b-sepz +YGG-SUMMARY/474d380d662e71a5: START 3 mutants; per-run timeout 300s; whole-job timeout 900s; one row per mutant follows as it finishes, so a timeout keeps the finished rows +YGG-SUMMARY/474d380d662e71a5: SNAPSHOT committed in 3s; each mutant runs in a fresh container from it +YGG-SUMMARY/474d380d662e71a5: [1/3] stamp2-bad | CAUGHT | tests::d_cache +YGG-SUMMARY/474d380d662e71a5: [2/3] untested-equiv | SURVIVED | tests passed with the mutation applied: cargo test --lib -- --test-threads=1 +YGG-SUMMARY/474d380d662e71a5: [3/3] add-minus | CAUGHT | tests::c_add +YGG-SUMMARY/474d380d662e71a5: MUTATION 3 mutants: caught=2 survived=1 unviable=0 timeout=0 noapply=0 error=0 (baseline green, tests ran) ygg-verify: exit 1 diff --git a/tools/fixtures/mutation-probe/README.txt b/tools/fixtures/mutation-probe/README.txt index a76b4ae..e084a0c 100644 --- a/tools/fixtures/mutation-probe/README.txt +++ b/tools/fixtures/mutation-probe/README.txt @@ -6,3 +6,7 @@ Run each on Yggdrasil through the stopgap; expected rows: js/mutants-d.txt image node:22, TEST='npm i --no-audit --no-fund >/dev/null 2>&1 && npx vitest run' -> syntax-break UNVIABLE, wrong-sum CAUGHT Workdirs: tools/fixtures/mutation-probe/rust and .../js. + soul2/mutants.txt TEST='cargo test --lib -- --test-threads=1', workdir .../soul2 -> a test keeps state in an IGNORED file + (target/soul-cache.txt): stamp2-bad CAUGHT tests::d_cache, untested-equiv SURVIVED, add-minus CAUGHT + tests::c_add; exit 1. Before the per-mutant snapshot, untested-equiv read CAUGHT d_cache and the job + exited 0 (the cache outlived the mutant that wrote it). diff --git a/tools/test-tools.sh b/tools/test-tools.sh index 721c0c9..3149243 100644 --- a/tools/test-tools.sh +++ b/tools/test-tools.sh @@ -1,8 +1,8 @@ #!/usr/bin/env bash # Local, cheap checks of the verdict-bearing text parsers: tools/testparse.sh and # the summary and wait modes of tools/stopgap/ygg-verify.sh, over -# tools/fixtures/logs. cargo-*.log and mutation-table-nonce.log are real -# Yggdrasil runs (the tiny-crate fixture, 2026-10-01); soul-s5/*.log are real +# tools/fixtures/logs. cargo-*.log (the tiny-crate fixture) and mutation-table-nonce.log (the soul2 +# reproducer) are real Yggdrasil runs, 2026-10-01; soul-s5/*.log are real # dotnet, pytest, vitest and cargo runs of Soul's probe repo (several runs per # log, split by "=== MARK " lines); the rest are hand-written. # Run: bash tools/test-tools.sh (no network, no builds) @@ -29,11 +29,11 @@ check cargo-red.log 101 "exit 101 RED" check cargo-red.log 101 "FAILED tests::add_adds" check cargo-red.log 101 "left: 5" check cargo-zero-tests.log 0 "ZERO TESTS RAN" -NONCE=c0ffee12 -check mutation-table-nonce.log 1 "clamp-positive | SURVIVED" -check mutation-table-nonce.log 1 "add-typo | UNVIABLE" -check mutation-table-nonce.log 1 "add-minus | CAUGHT | tests::add_adds" -check mutation-table-nonce.log 1 "caught=2 survived=1 unviable=1 timeout=0 noapply=1 error=0" "no test counts recognised" +NONCE=474d380d662e71a5 # a real run: the soul2 reproducer through the snapshot runner, 2026-10-01 +check mutation-table-nonce.log 1 "untested-equiv | SURVIVED" +check mutation-table-nonce.log 1 "stamp2-bad | CAUGHT | tests::d_cache" +check mutation-table-nonce.log 1 "add-minus | CAUGHT | tests::c_add" +check mutation-table-nonce.log 1 "caught=2 survived=1 unviable=0 timeout=0 noapply=0 error=0" "no test counts recognised" NONCE= check dotnet-red.log 1 "FRAMEWORK dotnet passed=11 failed=1 skipped=2" check dotnet-red.log 1 "FAILED MyNs.AddTests.Subtracts"