GameCult/Eureka
Watch
9
0
Fork
You've already forked Eureka
 0
Code Issues Pull requests 3 Projects Releases Packages Wiki Activity Actions

Fixtures and docs: real nonce log from the soul2 reproducer, INDEX describes the snapshot isolation

Browse source 
mutation-table-nonce.log is now a real run of the snapshot runner (untested-equiv
SURVIVED, exit 1). The probe README names the soul2 case. ygg-verify.sh no longer says
a reused log path is guarded by a run id (fresh paths are the rule).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Meta Vens 2026-10-01 17:46:11 +02:00
commit 2fbf696f79
4 changed files with 22 additions and 18 deletions
Download patch file Download diff file Expand all files Collapse all files

8
tools/INDEX.md
View file

@ -8,7 +8,7 @@ gets a Soul pass before merge.
| script | purpose | consumer | named by |
|---|---|---|---|
| `tools/stopgap/ygg-verify.sh` | Pushes an exact revision to a mirror on Yggdrasil and runs one command in a capped, niced container. `SUMMARY=<log>` keeps the full log and prints a bounded summary (verdict, counts, ZERO TESTS RAN, failing tests); `wait <log>` blocks at most 270 s and returns the summary once; call it with a Bash tool `timeout` of at least 280000 ms. Dies with the stopgap when Idunn's verify transaction lands. | Hands, Soul, Imagination probes | `references/briefs.md` (Hands long jobs and verification, Soul long jobs), `SKILL.md` (Yggdrasil) |
| `tools/mutation-table.sh` | One verify job that runs a list of hand mutants (file, before, after, test) against a revision and returns a kill table: CAUGHT with the killing test, SURVIVED, UNVIABLE, TIMEOUT, NOAPPLY, ERROR. Refuses a red or zero-test baseline and a red BUILD; rows print live; the whole tree is restored after each mutant. Replaces per-agent runmut/mutrun/strykrun scripts. | Soul, Hands (scoped run) | `references/briefs.md` (Hands verification, Soul reruns) |
| `tools/testparse.sh` | Sourced library: failing test names and pass/fail/skip counts from cargo, dotnet, vitest and pytest logs. Shared by the summary and the kill table. | the two scripts above | not named by a brief; it lives and dies with them |
| `tools/test-tools.sh` | Local fixture check of the parsers and the summary over `tools/fixtures/logs` (real cargo and mutation logs, plus hand-written dotnet, vitest, pytest, exit-0-but-failed and no-verdict logs). Run it before merging a change to the scripts above. | whoever changes those scripts | `SKILL.md` (Shared tools) |
| `tools/stopgap/ygg-verify.sh` | Pushes an exact revision to a mirror on Yggdrasil and runs one command in a capped, niced container (`RUN_ON_HOST=1`: run a host-side command in the same slot, clone and watchdog instead; the watchdog kills every container carrying the job label). `SUMMARY=<log>` keeps the full log and prints a bounded summary (verdict, counts, ZERO TESTS RAN, failing tests); `wait <log>` blocks at most 270 s and returns the summary once; call it with a Bash tool `timeout` of at least 280000 ms. Dies with the stopgap when Idunn's verify transaction lands. | Hands, Soul, Imagination probes | `references/briefs.md` (Hands long jobs and verification, Soul long jobs), `SKILL.md` (Yggdrasil) |
| `tools/mutation-table.sh` | One verify job that runs a list of hand mutants (file, before, after, test) against a revision and returns a kill table: CAUGHT with the killing test, SURVIVED, UNVIABLE, TIMEOUT, NOAPPLY, ERROR. Refuses a red or zero-test baseline and a red BUILD; rows print live. Isolation is Docker's: the baselines run once in a container that holds the revision, that container is committed to a throwaway image, and every mutant runs in a fresh `--rm` container from it, so nothing a run writes (tracked, untracked or ignored, such as `target/`) reaches the next mutant; the image is removed when the job ends, killed or not. The shared registry and sccache volumes and the network are not isolated. A file-level load failure (vitest `FAIL <file> [ <file> ]`, jest `FAIL <file>` plus `Test suite failed to run`) reads UNVIABLE. Replaces per-agent runmut/mutrun/strykrun scripts. | Soul, Hands (scoped run) | `references/briefs.md` (Hands verification, Soul reruns) |
| `tools/testparse.sh` | Sourced library: failing test names and pass/fail/skip counts from cargo, dotnet, vitest, jest and pytest logs, and test files that failed to load (`LOADFAIL`). Shared by the summary and the kill table. | the two scripts above | not named by a brief; it lives and dies with them |
| `tools/test-tools.sh` | Local fixture check of the parsers and the summary over `tools/fixtures/logs` (real cargo and mutation logs, plus hand-written dotnet, vitest, jest, pytest, exit-0-but-failed and no-verdict logs), and of the mutation runner's host driver and container half under a mock docker. Run it before merging a change to the scripts above. | whoever changes those scripts | `SKILL.md` (Shared tools) |

14
tools/fixtures/logs/mutation-table-nonce.log
View file

@ -1,9 +1,9 @@
warning: You appear to have cloned an empty repository.
ygg-verify: probe@7d3a1f5284 slot 2, image eureka-verify-rust:b99d30ec441e, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/probe-7d3a1f5284-Elfg
YGG-SUMMARY/c0ffee12: MUTATION 5 mutants: caught=2 survived=1 unviable=1 timeout=0 noapply=1 error=0 (baseline green, tests ran)
YGG-SUMMARY/c0ffee12: clamp-positive | SURVIVED | tests passed with the mutation applied: cargo test
YGG-SUMMARY/c0ffee12: add-typo | UNVIABLE | build exit 101
YGG-SUMMARY/c0ffee12: ghost | NOAPPLY | before text not found
YGG-SUMMARY/c0ffee12: add-minus | CAUGHT | tests::add_adds
YGG-SUMMARY/c0ffee12: even-flip | CAUGHT | tests::even_detects
ygg-verify: tool-calling-s5@675669150b slot 1, image eureka-verify-rust:b99d30ec441e, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/tool-calling-s5-675669150b-sepz
YGG-SUMMARY/474d380d662e71a5: START 3 mutants; per-run timeout 300s; whole-job timeout 900s; one row per mutant follows as it finishes, so a timeout keeps the finished rows
YGG-SUMMARY/474d380d662e71a5: SNAPSHOT committed in 3s; each mutant runs in a fresh container from it
YGG-SUMMARY/474d380d662e71a5: [1/3] stamp2-bad | CAUGHT | tests::d_cache
YGG-SUMMARY/474d380d662e71a5: [2/3] untested-equiv | SURVIVED | tests passed with the mutation applied: cargo test --lib -- --test-threads=1
YGG-SUMMARY/474d380d662e71a5: [3/3] add-minus | CAUGHT | tests::c_add
YGG-SUMMARY/474d380d662e71a5: MUTATION 3 mutants: caught=2 survived=1 unviable=0 timeout=0 noapply=0 error=0 (baseline green, tests ran)
ygg-verify: exit 1

4
tools/fixtures/mutation-probe/README.txt
View file

@ -6,3 +6,7 @@ Run each on Yggdrasil through the stopgap; expected rows:
js/mutants-d.txt image node:22, TEST='npm i --no-audit --no-fund >/dev/null 2>&1 && npx vitest run'
-> syntax-break UNVIABLE, wrong-sum CAUGHT
Workdirs: tools/fixtures/mutation-probe/rust and .../js.
soul2/mutants.txt TEST='cargo test --lib -- --test-threads=1', workdir .../soul2 -> a test keeps state in an IGNORED file
(target/soul-cache.txt): stamp2-bad CAUGHT tests::d_cache, untested-equiv SURVIVED, add-minus CAUGHT
tests::c_add; exit 1. Before the per-mutant snapshot, untested-equiv read CAUGHT d_cache and the job
exited 0 (the cache outlived the mutant that wrote it).

14
tools/test-tools.sh
View file

@ -1,8 +1,8 @@
#!/usr/bin/env bash
# Local, cheap checks of the verdict-bearing text parsers: tools/testparse.sh and
# the summary and wait modes of tools/stopgap/ygg-verify.sh, over
# tools/fixtures/logs. cargo-*.log and mutation-table-nonce.log are real
# Yggdrasil runs (the tiny-crate fixture, 2026-10-01); soul-s5/*.log are real
# tools/fixtures/logs. cargo-*.log (the tiny-crate fixture) and mutation-table-nonce.log (the soul2
# reproducer) are real Yggdrasil runs, 2026-10-01; soul-s5/*.log are real
# dotnet, pytest, vitest and cargo runs of Soul's probe repo (several runs per
# log, split by "=== MARK <name>" lines); the rest are hand-written.
# Run: bash tools/test-tools.sh (no network, no builds)
@ -29,11 +29,11 @@ check cargo-red.log 101 "exit 101 RED"
check cargo-red.log 101 "FAILED tests::add_adds"
check cargo-red.log 101 "left: 5"
check cargo-zero-tests.log 0 "ZERO TESTS RAN"
NONCE=c0ffee12
check mutation-table-nonce.log 1 "clamp-positive | SURVIVED"
check mutation-table-nonce.log 1 "add-typo | UNVIABLE"
check mutation-table-nonce.log 1 "add-minus | CAUGHT | tests::add_adds"
check mutation-table-nonce.log 1 "caught=2 survived=1 unviable=1 timeout=0 noapply=1 error=0" "no test counts recognised"
NONCE=474d380d662e71a5 # a real run: the soul2 reproducer through the snapshot runner, 2026-10-01
check mutation-table-nonce.log 1 "untested-equiv | SURVIVED"
check mutation-table-nonce.log 1 "stamp2-bad | CAUGHT | tests::d_cache"
check mutation-table-nonce.log 1 "add-minus | CAUGHT | tests::c_add"
check mutation-table-nonce.log 1 "caught=2 survived=1 unviable=0 timeout=0 noapply=0 error=0" "no test counts recognised"
NONCE=
check dotnet-red.log 1 "FRAMEWORK dotnet passed=11 failed=1 skipped=2"
check dotnet-red.log 1 "FAILED MyNs.AddTests.Subtracts"