From 60a09dc17ca564d5c2b19ed1d5f57d19882dd436 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 1 Oct 2026 15:09:07 +0200 Subject: [PATCH 01/16] Tools: ygg-verify summary and wait modes, mutation-table runner, shared test parser SUMMARY= keeps the full log and prints a bounded verdict summary; wait blocks at most 270 s and returns it once. mutation-table.sh runs hand mutants in one verify job and returns a kill table. testparse.sh is the one parser for both. Fixtures: real cargo and mutation logs from Yggdrasil (tiny-crate), plus hand-written dotnet, vitest, pytest, exit-0-but-failed and no-verdict logs. Every commit here builds; verified live on Yggdrasil. Co-Authored-By: Claude Sonnet 5.5 --- tools/INDEX.md | 10 +- tools/fixtures/logs/cargo-green.log | 20 +++ tools/fixtures/logs/cargo-red.log | 29 ++++ tools/fixtures/logs/cargo-zero-tests.log | 11 ++ tools/fixtures/logs/dotnet-red.log | 6 + tools/fixtures/logs/exit0-but-failed.log | 3 + tools/fixtures/logs/mutation-table.log | 9 ++ tools/fixtures/logs/no-verdict.log | 2 + tools/fixtures/logs/pytest-red.log | 3 + tools/fixtures/logs/vitest-red.log | 5 + tools/fixtures/tiny-crate/Cargo.toml | 4 + tools/fixtures/tiny-crate/mutants.txt | 40 ++++++ tools/fixtures/tiny-crate/src/lib.rs | 32 +++++ tools/mutation-table.sh | 167 +++++++++++++++++++++++ tools/stopgap/ygg-verify.sh | 116 +++++++++++++++- tools/test-tools.sh | 38 ++++++ tools/testparse.sh | 90 ++++++++++++ 17 files changed, 581 insertions(+), 4 deletions(-) create mode 100644 tools/fixtures/logs/cargo-green.log create mode 100644 tools/fixtures/logs/cargo-red.log create mode 100644 tools/fixtures/logs/cargo-zero-tests.log create mode 100644 tools/fixtures/logs/dotnet-red.log create mode 100644 tools/fixtures/logs/exit0-but-failed.log create mode 100644 tools/fixtures/logs/mutation-table.log create mode 100644 tools/fixtures/logs/no-verdict.log create mode 100644 tools/fixtures/logs/pytest-red.log create mode 100644 tools/fixtures/logs/vitest-red.log create mode 100644 tools/fixtures/tiny-crate/Cargo.toml create mode 100644 tools/fixtures/tiny-crate/mutants.txt create mode 100644 tools/fixtures/tiny-crate/src/lib.rs create mode 100644 tools/mutation-table.sh create mode 100644 tools/test-tools.sh create mode 100644 tools/testparse.sh diff --git a/tools/INDEX.md b/tools/INDEX.md index 84b0647..25efd0c 100644 --- a/tools/INDEX.md +++ b/tools/INDEX.md @@ -2,9 +2,13 @@ One line per script. Briefs name the exact script they want; agents do not browse. A line whose consuming brief no longer names the script is dead and is -deleted. Contributions come by branch and pull request. A script that verdicts -depend on gets a Soul pass before merge. +deleted. Contributions come by branch and Forgejo pull request (this repo lives on +Yggdrasil's Forgejo), never a push to main. A script that verdicts depend on +gets a Soul pass before merge. | script | purpose | consumer | named by | |---|---|---|---| -| `tools/stopgap/ygg-verify.sh` | Pushes an exact revision to a mirror on Yggdrasil and runs one command in a capped, niced container. Dies with the stopgap when Idunn's verify transaction lands. | Hands, Soul, Imagination probes | `references/briefs.md` (Hands and Soul verification), `SKILL.md` (Yggdrasil) | +| `tools/stopgap/ygg-verify.sh` | Pushes an exact revision to a mirror on Yggdrasil and runs one command in a capped, niced container. `SUMMARY=` keeps the full log and prints a bounded summary (verdict, counts, ZERO TESTS RAN, failing tests); `wait ` blocks at most 270 s and returns the summary once. Dies with the stopgap when Idunn's verify transaction lands. | Hands, Soul, Imagination probes | `references/briefs.md` (Hands long jobs and verification, Soul long jobs), `SKILL.md` (Yggdrasil) | +| `tools/mutation-table.sh` | One verify job that runs a list of hand mutants (file, before, after, test) against a revision and returns a kill table: CAUGHT with the killing test, SURVIVED, UNVIABLE, TIMEOUT, NOAPPLY, ERROR. Refuses a red or zero-test baseline. Replaces per-agent runmut/mutrun/strykrun scripts. | Soul, Hands (scoped run) | `references/briefs.md` (Hands verification, Soul reruns) | +| `tools/testparse.sh` | Sourced library: failing test names and pass/fail/skip counts from cargo, dotnet, vitest and pytest logs. Shared by the summary and the kill table. | the two scripts above | not named by a brief; it lives and dies with them | +| `tools/test-tools.sh` | Local fixture check of the parsers and the summary over `tools/fixtures/logs` (real cargo and mutation logs, plus hand-written dotnet, vitest, pytest, exit-0-but-failed and no-verdict logs). Run it before merging a change to the scripts above. | whoever changes those scripts | `SKILL.md` (Shared tools) | diff --git a/tools/fixtures/logs/cargo-green.log b/tools/fixtures/logs/cargo-green.log new file mode 100644 index 0000000..06afc5c --- /dev/null +++ b/tools/fixtures/logs/cargo-green.log @@ -0,0 +1,20 @@ +warning: You appear to have cloned an empty repository. +ygg-verify: probe@7d3a1f5284 slot 1, image eureka-verify-rust:b99d30ec441e, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/probe-7d3a1f5284-Pvbq + Compiling tiny-crate v0.0.0 (/src) + Finished `test` profile [unoptimized + debuginfo] target(s) in 0.35s + Running unittests src/lib.rs (target/debug/deps/tiny_crate-7b09195d0b4b7f00) + Doc-tests tiny_crate + +running 3 tests +test tests::add_adds ... ok +test tests::even_detects ... ok +test tests::clamp_negative ... ok + +test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + + +running 0 tests + +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + +ygg-verify: exit 0 diff --git a/tools/fixtures/logs/cargo-red.log b/tools/fixtures/logs/cargo-red.log new file mode 100644 index 0000000..c98c3e0 --- /dev/null +++ b/tools/fixtures/logs/cargo-red.log @@ -0,0 +1,29 @@ +warning: You appear to have cloned an empty repository. +ygg-verify: probe@7d3a1f5284 slot 2, image eureka-verify-rust:b99d30ec441e, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/probe-7d3a1f5284-axtF + Compiling tiny-crate v0.0.0 (/src) + Finished `test` profile [unoptimized + debuginfo] target(s) in 0.34s + Running unittests src/lib.rs (target/debug/deps/tiny_crate-7b09195d0b4b7f00) + +running 3 tests +test tests::add_adds ... FAILED +error: test failed, to rerun pass `--lib` +test tests::clamp_negative ... ok +test tests::even_detects ... ok + +failures: + +---- tests::add_adds stdout ---- + +thread 'tests::add_adds' (113) panicked at src/lib.rs:19:9: +assertion `left == right` failed + left: 5 + right: 6 +note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace + + +failures: + tests::add_adds + +test result: FAILED. 2 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + +ygg-verify: exit 101 diff --git a/tools/fixtures/logs/cargo-zero-tests.log b/tools/fixtures/logs/cargo-zero-tests.log new file mode 100644 index 0000000..cbd303f --- /dev/null +++ b/tools/fixtures/logs/cargo-zero-tests.log @@ -0,0 +1,11 @@ +warning: You appear to have cloned an empty repository. +ygg-verify: probe@7d3a1f5284 slot 1, image eureka-verify-rust:b99d30ec441e, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/probe-7d3a1f5284-OoDx + Compiling tiny-crate v0.0.0 (/src) + Finished `test` profile [unoptimized + debuginfo] target(s) in 0.32s + Running unittests src/lib.rs (target/debug/deps/tiny_crate-7b09195d0b4b7f00) + +running 0 tests + +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.00s + +ygg-verify: exit 0 diff --git a/tools/fixtures/logs/dotnet-red.log b/tools/fixtures/logs/dotnet-red.log new file mode 100644 index 0000000..183a623 --- /dev/null +++ b/tools/fixtures/logs/dotnet-red.log @@ -0,0 +1,6 @@ +SYNTHETIC: hand-written in the format dotnet test prints; not captured from a run. + Passed MyNs.AddTests.Adds [2 ms] + Failed MyNs.AddTests.Subtracts [12 ms] + Error Message: + Assert.Equal() Failure: Expected 1, Actual 2 +Failed! - Failed: 1, Passed: 11, Skipped: 2, Total: 14, Duration: 1 s - MyNs.Tests.dll (net10.0) diff --git a/tools/fixtures/logs/exit0-but-failed.log b/tools/fixtures/logs/exit0-but-failed.log new file mode 100644 index 0000000..f6ca9b5 --- /dev/null +++ b/tools/fixtures/logs/exit0-but-failed.log @@ -0,0 +1,3 @@ +SYNTHETIC: a command whose last statement hid a failing test run (the exit status is 0 in the test). +test tests::add_adds ... FAILED +test result: FAILED. 2 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s diff --git a/tools/fixtures/logs/mutation-table.log b/tools/fixtures/logs/mutation-table.log new file mode 100644 index 0000000..71b01b3 --- /dev/null +++ b/tools/fixtures/logs/mutation-table.log @@ -0,0 +1,9 @@ +warning: You appear to have cloned an empty repository. +ygg-verify: probe@7d3a1f5284 slot 2, image eureka-verify-rust:b99d30ec441e, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/probe-7d3a1f5284-Elfg +YGG-SUMMARY: MUTATION 5 mutants: caught=2 survived=1 unviable=1 timeout=0 noapply=1 error=0 (baseline green, tests ran) +YGG-SUMMARY: clamp-positive | SURVIVED | tests passed with the mutation applied: cargo test +YGG-SUMMARY: add-typo | UNVIABLE | build exit 101 +YGG-SUMMARY: ghost | NOAPPLY | before text not found +YGG-SUMMARY: add-minus | CAUGHT | tests::add_adds +YGG-SUMMARY: even-flip | CAUGHT | tests::even_detects +ygg-verify: exit 1 diff --git a/tools/fixtures/logs/no-verdict.log b/tools/fixtures/logs/no-verdict.log new file mode 100644 index 0000000..9bcb259 --- /dev/null +++ b/tools/fixtures/logs/no-verdict.log @@ -0,0 +1,2 @@ +SYNTHETIC: the remote script died before printing its verdict line. +ygg-verify: NO VERDICT from the remote job - treating as failure diff --git a/tools/fixtures/logs/pytest-red.log b/tools/fixtures/logs/pytest-red.log new file mode 100644 index 0000000..2e8603b --- /dev/null +++ b/tools/fixtures/logs/pytest-red.log @@ -0,0 +1,3 @@ +SYNTHETIC: hand-written in the format pytest prints; not captured from a run. +FAILED tests/test_add.py::test_carries - assert 4 == 5 +========================= 1 failed, 5 passed, 1 skipped in 0.12s ========================= diff --git a/tools/fixtures/logs/vitest-red.log b/tools/fixtures/logs/vitest-red.log new file mode 100644 index 0000000..fabf42d --- /dev/null +++ b/tools/fixtures/logs/vitest-red.log @@ -0,0 +1,5 @@ +SYNTHETIC: hand-written in the format vitest prints; not captured from a run. + FAIL src/add.test.ts > add > carries +AssertionError: expected 4 to be 5 + Test Files 1 failed | 2 passed (3) + Tests 1 failed | 10 passed | 1 skipped (12) diff --git a/tools/fixtures/tiny-crate/Cargo.toml b/tools/fixtures/tiny-crate/Cargo.toml new file mode 100644 index 0000000..1e7a91d --- /dev/null +++ b/tools/fixtures/tiny-crate/Cargo.toml @@ -0,0 +1,4 @@ +[package] +name = "tiny-crate" +version = "0.0.0" +edition = "2021" diff --git a/tools/fixtures/tiny-crate/mutants.txt b/tools/fixtures/tiny-crate/mutants.txt new file mode 100644 index 0000000..c611989 --- /dev/null +++ b/tools/fixtures/tiny-crate/mutants.txt @@ -0,0 +1,40 @@ +Fixture for tools/mutation-table.sh. Expected: add-minus CAUGHT by add_adds, +even-flip CAUGHT by even_detects, clamp-positive SURVIVED (no test reads the +positive branch), add-typo UNVIABLE (with BUILD='cargo test --no-run'), +ghost NOAPPLY. Run with TEST='cargo test' BUILD='cargo test --no-run'. + +=== add-minus +file: src/lib.rs +--- before + a + b +--- after + a - b + +=== even-flip +file: src/lib.rs +test: cargo test even_ +--- before +n % 2 == 0 +--- after +n % 2 == 1 + +=== clamp-positive +file: src/lib.rs +--- before +else { n } +--- after +else { n + 1 } + +=== add-typo +file: src/lib.rs +--- before + a + b +--- after + a + c + +=== ghost +file: src/lib.rs +--- before +this text is nowhere +--- after +x diff --git a/tools/fixtures/tiny-crate/src/lib.rs b/tools/fixtures/tiny-crate/src/lib.rs new file mode 100644 index 0000000..1bc490c --- /dev/null +++ b/tools/fixtures/tiny-crate/src/lib.rs @@ -0,0 +1,32 @@ +pub fn add(a: i32, b: i32) -> i32 { + a + b +} + +pub fn is_even(n: i32) -> bool { + n % 2 == 0 +} + +pub fn clamp_pos(n: i32) -> i32 { + if n < 0 { 0 } else { n } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn add_adds() { + assert_eq!(add(2, 3), 5); + } + + #[test] + fn even_detects() { + assert!(is_even(4)); + assert!(!is_even(3)); + } + + #[test] + fn clamp_negative() { + assert_eq!(clamp_pos(-5), 0); + } +} diff --git a/tools/mutation-table.sh b/tools/mutation-table.sh new file mode 100644 index 0000000..99bb764 --- /dev/null +++ b/tools/mutation-table.sh @@ -0,0 +1,167 @@ +#!/usr/bin/env bash +# Hand-mutant kill table: run a list of mutants against a revision on +# Yggdrasil, in ONE verify job, and get one table back. +# +# tools/mutation-table.sh [workdir] +# workdir directory inside the repo where the commands run (e.g. +# packages/cultcache-rs; CultLib has no Cargo.toml at its root). +# env TEST default test command for mutants with no `test:` line +# BUILD optional compile-only command; when it fails the mutant +# is UNVIABLE. Without it a mutant that does not compile +# has no failing test and reads ERROR, never CAUGHT. +# MUT_TIMEOUT seconds per build or test run (default 600) +# everything ygg-verify.sh reads (SUMMARY=, TIMEOUT, MEM, ...) +# Typical, from an agent: launch as ONE background shell with +# SUMMARY=/mut1.log TEST='cargo test' tools/mutation-table.sh ... +# then `tools/stopgap/ygg-verify.sh wait /mut1.log` in the foreground. +# +# Mutants file (plain text; the text before the first `===` is ignored): +# === label-unique-in-the-file +# file: src/lib.rs (relative to workdir) +# test: cargo test add_ (optional; else $TEST) +# --- before +# a + b (exact text, may span lines; must occur ONCE) +# --- after +# a - b +# +# Each mutant is applied in the container's throwaway clone and reverted from +# git before the next, so no tree you care about is touched. The output is one +# row per mutant: label | outcome | killing test or detail. Outcomes: +# CAUGHT the test command failed AND a failing test name was parsed from it +# SURVIVED the test command passed with the mutation applied +# UNVIABLE BUILD failed +# TIMEOUT a run hit MUT_TIMEOUT (not a kill) +# NOAPPLY before text absent, ambiguous, or equal to after (no mutation made) +# ERROR the test command failed but no failing test was parsed (a build +# error without BUILD, a crash): not a kill +# Before any mutant, every distinct test command runs UNMUTATED. If one fails or +# runs zero tests the job aborts with `BASELINE RED`, because a red or empty +# baseline would make every mutant look killed or every one survive. +# Exit 0 means every mutant was CAUGHT or UNVIABLE and nothing else happened. +# +# DELETION LINE: this runner replaces the per-agent runmut/mutrun/strykrun +# scripts. It dies with tools/stopgap/ygg-verify.sh when `idunn verify` ships a +# typed mutation step. It reads tools/testparse.sh for failure names, the same +# parser the summary uses. +set -euo pipefail + +if [ "${1:-}" != --in-container ]; then + here=$(cd "$(dirname "$0")" && pwd) + repo=${1:?usage: mutation-table.sh [workdir]} + rev=${2:?revision}; image=${3:?image}; mfile=${4:?mutants file}; workdir=${5:-.} + [ -r "$mfile" ] || { echo "mutation-table: cannot read $mfile" >&2; exit 2; } + stage=$(mktemp -d) + trap 'rm -rf "$stage"' EXIT + cp "$here/testparse.sh" "$here/mutation-table.sh" "$stage/" + tr -d '\r' <"$mfile" >"$stage/mutants.txt" + bundle=$(tar -C "$stage" -cz testparse.sh mutation-table.sh mutants.txt | base64 | tr -d '\n') + cmd=$(printf 'mkdir -p /tmp/mt && echo %s | base64 -d | tar -xz -C /tmp/mt && WORKDIR=%q BUILD=%q TEST=%q MUT_TIMEOUT=%q bash /tmp/mt/mutation-table.sh --in-container' \ + "$bundle" "$workdir" "${BUILD:-}" "${TEST:-}" "${MUT_TIMEOUT:-600}") + "$here/stopgap/ygg-verify.sh" "$repo" "$rev" "$image" "$cmd" + exit $? +fi + +# ---- container side --------------------------------------------------------- +set +e +# shellcheck source=testparse.sh +. /tmp/mt/testparse.sh +cd "/src/${WORKDIR:-.}" || { echo "YGG-SUMMARY: BAD WORKDIR ${WORKDIR:-.}"; exit 2; } +M=/tmp/mt/m; rm -rf "$M"; mkdir -p "$M" +tmo=${MUT_TIMEOUT:-600} + +if ! perl -e ' + my ($mf, $out) = @ARGV; my ($n, $sec, %c, %seen) = (0, ""); + open my $f, "<", $mf or die "cannot open $mf\n"; + sub flush { + return unless exists $c{label}; + for my $k (qw(file before after)) { die "mutant $c{label}: no $k\n" unless defined $c{$k} } + my $d = "$out/$n"; mkdir $d; + for my $k (qw(label file test before after)) { + my $v = join("\n", ref $c{$k} ? @{$c{$k}} : ($c{$k} // "")); + open my $o, ">:raw", "$d/$k" or die; print $o $v; close $o; + } + } + while (<$f>) { + chomp; + if (/^=== (.+?)\s*$/) { flush(); die "duplicate label $1\n" if $seen{$1}++; %c = (label => $1); $n++; $sec = ""; next } + next unless exists $c{label}; + if ($sec eq "" && /^file: (.+?)\s*$/) { $c{file} = $1; next } + if ($sec eq "" && /^test: (.+?)\s*$/) { $c{test} = $1; next } + if (/^--- (before|after)\s*$/) { $sec = $1; $c{$sec} = []; next } + push @{$c{$sec}}, $_ if $sec; + } + flush(); die "no mutants in $mf\n" unless $n; + print "$n\n";' /tmp/mt/mutants.txt "$M" >"$M/count" 2>"$M/err"; then + echo "YGG-SUMMARY: BAD MUTANT FILE: $(cat "$M/err")"; exit 2 +fi +count=$(cat "$M/count") + +# run : prints the exit status; 124 means timeout +run() { timeout "$tmo" bash -o pipefail -c "$1" >"$2" 2>&1; echo $?; } + +declare -A based +for n in $(seq 1 "$count"); do + t=$(cat "$M/$n/test"); t=${t:-$TEST} + [ -n "$t" ] || { echo "YGG-SUMMARY: mutant $(cat "$M/$n/label") has no test: line and TEST is unset"; exit 2; } + [ -z "${based[$t]:-}" ] || continue + based[$t]=1 + rc=$(run "$t" "$M/base.log") + scan=$(tp_scan "$M/base.log") + tests=$(printf '%s\n' "$scan" | awk '/^FRAMEWORK/{for(i=1;i<=NF;i++){split($i,a,"=");if(a[1]=="passed"||a[1]=="failed")s+=a[2]}}END{print s+0}') + if [ "$rc" != 0 ]; then + echo "YGG-SUMMARY: BASELINE RED (exit $rc) for: $t"; tail -n 15 "$M/base.log" | sed 's/^/YGG-SUMMARY: /'; exit 2 + fi + if [ "$tests" -eq 0 ]; then + echo "YGG-SUMMARY: BASELINE RAN ZERO TESTS (or printed counts no parser reads) for: $t"; exit 2 + fi +done + +: >"$M/rows" +for n in $(seq 1 "$count"); do + label=$(cat "$M/$n/label"); file=$(cat "$M/$n/file"); t=$(cat "$M/$n/test"); t=${t:-$TEST} + out=''; detail='' + if ! detail=$(perl -e ' + local $/; my ($file, $bf, $af) = @ARGV; + open my $h, "<:raw", $file or do { print "cannot read $file\n"; exit 3 }; my $c = <$h>; close $h; + open my $b, "<:raw", $bf; my $before = <$b> // ""; open my $a, "<:raw", $af; my $after = <$a> // ""; + if (length($before) == 0) { print "empty before text\n"; exit 3 } + if ($before eq $after) { print "after equals before\n"; exit 3 } + my ($cnt, $p) = (0, 0); while (($p = index($c, $before, $p)) >= 0) { $cnt++; $p += length($before) } + if ($cnt == 0) { print "before text not found\n"; exit 3 } + if ($cnt > 1) { print "before text occurs $cnt times; make it unique\n"; exit 3 } + substr($c, index($c, $before), length($before)) = $after; + open my $o, ">:raw", $file or exit 3; print $o $c; close $o;' "$file" "$M/$n/before" "$M/$n/after" 2>&1); then + out=NOAPPLY + elif git diff --quiet -- "$file"; then + out=NOAPPLY; detail="applied, but git sees no change in $file" + else + out='' + if [ -n "${BUILD:-}" ]; then + rc=$(run "$BUILD" "$M/run.log") + if [ "$rc" = 124 ]; then out=TIMEOUT; detail="build exceeded ${tmo}s" + elif [ "$rc" != 0 ]; then out=UNVIABLE; detail="build exit $rc"; fi + fi + if [ -z "$out" ]; then + rc=$(run "$t" "$M/run.log") + if [ "$rc" = 0 ]; then out=SURVIVED; detail="tests passed with the mutation applied: $t" + elif [ "$rc" = 124 ]; then out=TIMEOUT; detail="test exceeded ${tmo}s: $t" + else + names=$(tp_scan "$M/run.log" | grep '^FAILED ' | awk '!s[$0]++' | sed 's/^FAILED //') + if [ -n "$names" ]; then out=CAUGHT; detail=$(printf '%s\n' "$names" | head -n 1); k=$(printf '%s\n' "$names" | wc -l); [ "$k" -le 1 ] || detail="$detail (+$((k - 1)) more)" + else out=ERROR; detail="exit $rc, no failing test parsed: $(tail -n 1 "$M/run.log" | cut -c1-120)"; fi + fi + fi + fi + printf '%s | %s | %s\n' "$label" "$out" "$detail" >>"$M/rows" + git checkout -q -- "$file" + if ! git diff --quiet -- "$file"; then echo "YGG-SUMMARY: RESTORE FAILED for $file after $label; stopping"; cat "$M/rows" | sed 's/^/YGG-SUMMARY: /'; exit 2; fi +done + +tally() { grep -c " | $1 | " "$M/rows" || true; } +bad=$(grep -vc -E ' \| (CAUGHT|UNVIABLE) \| ' "$M/rows" || true) +{ + echo "MUTATION $count mutants: caught=$(tally CAUGHT) survived=$(tally SURVIVED) unviable=$(tally UNVIABLE) timeout=$(tally TIMEOUT) noapply=$(tally NOAPPLY) error=$(tally ERROR) (baseline green, tests ran)" + grep -v ' | CAUGHT | ' "$M/rows" + grep ' | CAUGHT | ' "$M/rows" +} | sed 's/^/YGG-SUMMARY: /' +[ "$bad" -eq 0 ] diff --git a/tools/stopgap/ygg-verify.sh b/tools/stopgap/ygg-verify.sh index 8e35fb3..6f9ce10 100644 --- a/tools/stopgap/ygg-verify.sh +++ b/tools/stopgap/ygg-verify.sh @@ -83,6 +83,28 @@ # (e.g. DOCKER_ARGS='--sysctl net.ipv6.conf.all.disable_ipv6=0' or # DOCKER_ARGS='--network host'). Word-split by the remote shell, so quote a # flag's own value inside the string if it needs one. +# +# Summary and wait modes (agents use these; they save the polling calls): +# SUMMARY= ygg-verify.sh '' +# Runs the same job but keeps the whole output in and prints only +# a bounded summary (at most ~80 lines): the exit verdict, per-framework +# pass/fail/skip counts (cargo test, dotnet test, vitest, pytest), a ZERO +# TESTS RAN line when counts exist and are all zero, failing test names, and +# for a red job the failure sections and the log tail. When the job ends it +# writes .summary and then .done (holding the exit +# status). Use a fresh per job. Launch it as ONE background shell +# (run_in_background), then: +# ygg-verify.sh wait +# Blocks at most 270 s (WAIT_MAX can only lower that), prints the summary and +# exits with the job's status when the job is done. Otherwise prints +# `RUNNING s` and exits 99: call it again. Exit 98 means the +# launcher is gone with no verdict (killed outright) or never started. It +# checks a local file every 2 s; no remote call, no model turn, one wake. +# A job's own lines starting `YGG-SUMMARY: ` are always carried into the +# summary (tools/mutation-table.sh prints its table that way). The full log +# stays at ; open it only at a named failing test, with grep -n -A 40. +# Killing the launcher does not stop the remote container: its own TIMEOUT and +# IDLE watchdog does. # Images: # rust eureka-verify-rust: (rust 1.95, cargo-mutants, sccache; # every job shares the eureka-sccache volume at /sccache, keyed by @@ -97,11 +119,103 @@ # 'cargo test --workspace && git diff BASE HEAD > /tmp/d && cargo mutants --in-diff /tmp/d' set -euo pipefail +here=$(cd "$(dirname "$0")" && pwd) + +# Bounded summary of one job's log. $1 log, $2 exit status, $3 optional note. +ygg_summarize() { + local log=$1 status=$2 note=${3:-} word=RED scan fw names p f nf own=0 + [ "$status" = 0 ] && word=GREEN + # shellcheck source=../testparse.sh + . "$here/../testparse.sh" + grep -q '^YGG-SUMMARY: ' "$log" && own=1 + scan=$(tp_scan "$log") + fw=$(printf '%s\n' "$scan" | grep '^FRAMEWORK' || true) + names=$(printf '%s\n' "$scan" | grep '^FAILED ' | awk '!seen[$0]++' || true) + read -r p f < <(printf '%s\n' "$fw" | awk '{for(i=1;i<=NF;i++){split($i,a,"=");if(a[1]=="passed")p+=a[2];if(a[1]=="failed")f+=a[2]}}END{print p+0, f+0}') + nf=0; [ -z "$names" ] || nf=$(printf '%s\n' "$names" | wc -l) + { + echo "YGG-VERIFY exit $status $word${note:+ ($note)} log: $log" + grep -E '^ygg-verify: (TIMEOUT|IDLE|NO VERDICT)' "$log" || true + if [ "$status" = 0 ] && { [ "$nf" -gt 0 ] || [ "$f" -gt 0 ]; }; then + echo "MISMATCH: exit 0 but the log shows $nf failing test names and $f failed counts; do not trust the exit" + fi + [ -z "$fw" ] || printf '%s\n' "$fw" + if [ -z "$fw" ]; then [ "$own" = 1 ] || echo "no test counts recognised in the log (cargo, dotnet, vitest, pytest)" + elif [ $((p + f)) -eq 0 ]; then echo "ZERO TESTS RAN"; fi + if [ "$nf" -gt 0 ]; then + printf '%s\n' "$names" | awk 'NR<=15{print "FAILED " substr($0,8)} END{if(NR>15)print "... " NR-15 " more failing tests"}' + fi + { grep '^YGG-SUMMARY: ' "$log" || true; } | sed 's/^YGG-SUMMARY: //' + { grep -E '^(MISSED|TIMEOUT) |mutants tested' "$log" || true; } | awk 'NR<=20' + if [ "$status" != 0 ]; then + if [ "$own" = 1 ]; then echo "--- log tail"; { tail -n 8 "$log" | grep -v '^YGG-SUMMARY: '; } || true + else + [ "$nf" -gt 0 ] || [ "$f" -gt 0 ] || echo "exit non-zero with no failing test parsed: build error, crash, timeout or a non-test command" + secs=$(tp_sections "$log" 30 | awk 'NR<=40') + [ -z "$secs" ] || printf '%s\n' "$secs" + echo "--- log tail" + if [ -n "$secs" ]; then tail -n 8 "$log"; else tail -n 20 "$log"; fi + fi + elif [ -z "$fw" ] && [ "$own" = 0 ]; then + echo "--- log tail" + tail -n 5 "$log" + fi + } | awk 'NR<=80{print} NR==81{print "[summary truncated at 80 lines; full log is the path above]"}' +} + +if [ "${1:-}" = summarize ]; then # re-read a finished log: summarize + ygg_summarize "${2:?log}" "${3:?exit status}"; exit 0 +fi + +if [ "${1:-}" = wait ]; then + wlog=${2:?usage: ygg-verify.sh wait } + cap=${WAIT_MAX:-270}; [ "$cap" -le 270 ] || cap=270 + t0=$(date +%s) + while :; do + if [ -e "$wlog.done" ]; then + st=$(cat "$wlog.done") + if [ -s "$wlog.summary" ]; then cat "$wlog.summary"; else echo "ygg-verify: exit $st, no summary file; see $wlog"; fi + exit "$st" + fi + wpid=$(cat "$wlog.pid" 2>/dev/null || true) + el=$(( $(date +%s) - t0 )) + if [ -n "$wpid" ] && ! kill -0 "$wpid" 2>/dev/null && [ ! -e "$wlog.done" ]; then + echo "ygg-verify: DEAD - launcher $wpid is gone without a verdict; the job's outcome is unknown (log $wlog)"; exit 98 + fi + if [ -z "$wpid" ] && [ "$el" -ge 30 ]; then + echo "ygg-verify: NOT STARTED - no $wlog.pid after 30 s; was the job launched with SUMMARY=$wlog?"; exit 98 + fi + if [ "$el" -ge "$cap" ]; then + started=$(cat "$wlog.start" 2>/dev/null || echo "$t0") + echo "RUNNING $(( $(date +%s) - started ))s (log $wlog; last line: $(tail -n 1 "$wlog" 2>/dev/null | cut -c1-160))" + exit 99 + fi + sleep 2 + done +fi + +if [ -n "${SUMMARY:-}" ] && [ -z "${YGG_SUMMARY_INNER:-}" ]; then + slog=$SUMMARY; child='' + mkdir -p "$(dirname "$slog")" + rm -f "$slog.done" "$slog.summary" + : >"$slog"; date +%s >"$slog.start"; echo $$ >"$slog.pid" + ygg_finish() { # summary first, done-file last: a visible .done always has its summary + ygg_summarize "$slog" "$1" "${2:-}" >"$slog.summary.tmp" && mv "$slog.summary.tmp" "$slog.summary" || true + echo "$1" >"$slog.done.tmp"; mv "$slog.done.tmp" "$slog.done" + if [ -s "$slog.summary" ]; then cat "$slog.summary"; fi + exit "$1" + } + trap '[ -z "$child" ] || kill "$child" 2>/dev/null; ygg_finish 143 "launcher killed by a signal; the remote job may still be running"' TERM INT HUP + YGG_SUMMARY_INNER=1 "$0" "$@" >"$slog" 2>&1 & + child=$! + sst=0; wait "$child" || sst=$? + ygg_finish "$sst" +fi + repo=${1:?local repo}; rev=${2:?revision}; image=${3:?image}; cmd=${4:?command} host=${YGG_HOST:-ygg}; cpus=${CPUS:-4}; mem=${MEM:-6g}; slots=${SLOTS:-5} timeout_s=${TIMEOUT:-3600}; idle_s=${IDLE:-900} docker_args=${DOCKER_ARGS:-} -here=$(cd "$(dirname "$0")" && pwd) sshopts=(-o BatchMode=yes -o ServerAliveInterval=30 -o ServerAliveCountMax=4) sha=$(git -C "$repo" rev-parse --verify "$rev^{commit}") diff --git a/tools/test-tools.sh b/tools/test-tools.sh new file mode 100644 index 0000000..d6621b7 --- /dev/null +++ b/tools/test-tools.sh @@ -0,0 +1,38 @@ +#!/usr/bin/env bash +# Local, cheap checks of the verdict-bearing text parsers: tools/testparse.sh and +# the summary mode of tools/stopgap/ygg-verify.sh, over tools/fixtures/logs. +# cargo-*.log and mutation-table.log are real Yggdrasil runs (the tiny-crate +# fixture, 2026-10-01); dotnet, vitest, pytest, exit0 and no-verdict are +# hand-written. Run: bash tools/test-tools.sh (no network, no builds) +set -uo pipefail +here=$(cd "$(dirname "$0")" && pwd) +L=$here/fixtures/logs; V=$here/stopgap/ygg-verify.sh +fail=0 +# check [] +check() { + out=$(bash "$V" summarize "$L/$1" "$2") + printf '%s\n' "$out" | grep -qF -- "$3" || { echo "FAIL $1: missing: $3"; fail=1; return; } + if [ -n "${4:-}" ] && printf '%s\n' "$out" | grep -qF -- "$4"; then echo "FAIL $1: must not contain: $4"; fail=1; return; fi + [ "$(printf '%s\n' "$out" | wc -l)" -le 81 ] || { echo "FAIL $1: summary over 81 lines"; fail=1; return; } + echo "ok $1: $3" +} +check cargo-green.log 0 "exit 0 GREEN" "RED" +check cargo-green.log 0 "FRAMEWORK cargo passed=3 failed=0" +check cargo-red.log 101 "exit 101 RED" +check cargo-red.log 101 "FAILED tests::add_adds" +check cargo-red.log 101 "left: 5" +check cargo-zero-tests.log 0 "ZERO TESTS RAN" +check mutation-table.log 1 "clamp-positive | SURVIVED" +check mutation-table.log 1 "add-typo | UNVIABLE" +check mutation-table.log 1 "add-minus | CAUGHT | tests::add_adds" +check mutation-table.log 1 "caught=2 survived=1 unviable=1 timeout=0 noapply=1 error=0" "no test counts recognised" +check dotnet-red.log 1 "FRAMEWORK dotnet passed=11 failed=1 skipped=2" +check dotnet-red.log 1 "FAILED MyNs.AddTests.Subtracts" +check vitest-red.log 1 "FRAMEWORK vitest passed=10 failed=1 skipped=1" +check vitest-red.log 1 "FAILED src/add.test.ts > add > carries" +check pytest-red.log 1 "FRAMEWORK pytest passed=5 failed=1 skipped=1" +check pytest-red.log 1 "FAILED tests/test_add.py::test_carries" +check exit0-but-failed.log 0 "MISMATCH: exit 0 but the log shows" +check no-verdict.log 97 "NO VERDICT" +check no-verdict.log 97 "RED" "GREEN" +exit $fail diff --git a/tools/testparse.sh b/tools/testparse.sh new file mode 100644 index 0000000..11628de --- /dev/null +++ b/tools/testparse.sh @@ -0,0 +1,90 @@ +#!/usr/bin/env bash +# Sourced, not run. Reads a test-run log and says what it contains. +# +# tp_scan one line per fact: +# FRAMEWORK passed= failed= skipped= (summed over the log) +# FAILED (one per failing test) +# tp_sections [max-lines] cargo failure sections ("---- name stdout +# ----"), each cut to its last lines (default 30) +# +# Frameworks: cargo test, dotnet test, vitest/jest, pytest. Anything else is +# simply not recognised; the callers treat "recognised nothing" as a fact to +# print, never as success. +# +# Owner: tools/stopgap/ygg-verify.sh (summary mode) and tools/mutation-table.sh +# (kill attribution). One parser, so a summary and a kill table cannot disagree +# about what failed. Written for mawk: no match() groups, no gensub. + +tp_scan() { + awk ' + function num(s, a) { split(s, a, " "); return a[1] + 0 } + /^test result: / { # cargo, one line per test binary + line = $0; sub(/^test result: [A-Za-z]+\. /, "", line) + n = split(line, parts, "; ") + for (i = 1; i <= n; i++) { + v = num(parts[i]) + if (parts[i] ~ /passed/) cp += v + if (parts[i] ~ /failed/) cf += v + if (parts[i] ~ /ignored/) cs += v + } + cargo = 1; next + } + /^test .* \.\.\. FAILED$/ { name = $0; sub(/^test /, "", name); sub(/ \.\.\. FAILED$/, "", name); print "FAILED " name; next } + /^(Passed|Failed)! - Failed: *[0-9]+, Passed: *[0-9]+, Skipped: *[0-9]+, Total:/ { # dotnet + line = $0; sub(/^[A-Za-z]+! - /, "", line) + n = split(line, parts, ", ") + for (i = 1; i <= n; i++) { + v = parts[i]; sub(/^[A-Za-z]+: */, "", v); v += 0 + if (parts[i] ~ /^Failed:/) df += v + if (parts[i] ~ /^Passed:/) dp += v + if (parts[i] ~ /^Skipped:/) ds += v + } + dotnet = 1; next + } + /^[ \t]+Failed [^ ]+ \[/ { name = $0; sub(/^[ \t]+Failed /, "", name); sub(/ \[.*$/, "", name); print "FAILED " name; next } + /^ *Tests +[0-9]+ (failed|passed|skipped)/ { # vitest / jest + line = $0; sub(/^ *Tests +/, "", line); sub(/ *\(.*$/, "", line) + n = split(line, parts, " \\| ") + for (i = 1; i <= n; i++) { + v = num(parts[i]) + if (parts[i] ~ /failed/) vf += v + if (parts[i] ~ /passed/) vp += v + if (parts[i] ~ /skipped/) vs += v + } + vitest = 1; next + } + /^ *FAIL / { name = $0; sub(/^ *FAIL +/, "", name); print "FAILED " name; next } + /^=+ .*(passed|failed|no tests ran).* in [0-9.]+s/ { # pytest summary line + line = $0; gsub(/=/, "", line); sub(/ in [0-9.]+s.*$/, "", line) + n = split(line, parts, ", ") + for (i = 1; i <= n; i++) { + v = num(parts[i]) + if (parts[i] ~ /failed/) pf += v + if (parts[i] ~ /passed/) pp += v + if (parts[i] ~ /(skipped|deselected)/) ps += v + if (parts[i] ~ /errors?/) pf += v + } + pytest = 1; next + } + /^FAILED [^ ]+::/ { name = $2; print "FAILED " name; next } + END { + if (cargo) printf "FRAMEWORK cargo passed=%d failed=%d skipped=%d\n", cp, cf, cs + if (dotnet) printf "FRAMEWORK dotnet passed=%d failed=%d skipped=%d\n", dp, df, ds + if (vitest) printf "FRAMEWORK vitest passed=%d failed=%d skipped=%d\n", vp, vf, vs + if (pytest) printf "FRAMEWORK pytest passed=%d failed=%d skipped=%d\n", pp, pf, ps + }' "$1" +} + +tp_sections() { + awk -v max="${2:-30}" ' + function flush( i, start) { + if (!n) return + start = n > max ? n - max + 1 : 1 + for (i = start; i <= n; i++) print buf[i] + n = 0 + } + /^---- .* stdout ----$/ { flush(); n = 0; buf[++n] = $0; inside = 1; next } + inside && (/^failures:$/ || /^test result: /) { flush(); inside = 0; next } + inside { buf[++n] = $0 } + END { flush() }' "$1" +} -- 2.47.3 From 506680c26282b9a00217fdfc70d7e2ea45dec352 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 1 Oct 2026 15:09:08 +0200 Subject: [PATCH 02/16] Briefs and skill: summary/wait verify, call budgets, range dump, Forgejo contributions Co-Authored-By: Claude Sonnet 5.5 --- SKILL.md | 6 ++-- references/briefs.md | 65 ++++++++++++++++++++++++++++------------- references/changelog.md | 8 +++++ 3 files changed, 57 insertions(+), 22 deletions(-) diff --git a/SKILL.md b/SKILL.md index 651e36a..20f79bf 100644 --- a/SKILL.md +++ b/SKILL.md @@ -531,8 +531,10 @@ and reconciles the target doc with the Body. Scripts that agents share live in this repo under `tools/`. `tools/INDEX.md` lists each script, its consumer and the brief that names it; briefs name the exact script they want and agents do not browse. Contributions come by branch -and pull request, never a push to main. A script that verdicts depend on gets -a Soul pass before merge. A script whose brief no longer names it is pruned. +and Forgejo pull request (this repo lives on Yggdrasil's Forgejo), never a push +to main. A script that verdicts depend on gets a Soul pass before merge, and +`bash tools/test-tools.sh` passes first. The summary and wait modes of +`tools/stopgap/ygg-verify.sh` and `tools/mutation-table.sh` are such scripts. A script whose brief no longer names it is pruned. Scripts that know a project's layout stay in that project. ## Git and tooling rules (scars) diff --git a/references/briefs.md b/references/briefs.md index 2a5aca7..4447c27 100644 --- a/references/briefs.md +++ b/references/briefs.md @@ -98,7 +98,8 @@ path>. Do not map cuts, do not write code, do not commit; the root agent commits. Rehydrate (recipes: target in force, rulings in force, open questions and -follow-ups). Read enough of the Body to enumerate the persistent kinds. +follow-ups). Read enough of the Body to enumerate the persistent kinds. Orient +and probe through scripts, one call per question, output at most 150 lines. Give one table, a row per persistent kind: - Identity: what names it. Is the namespace stated? Is the name injective? Can @@ -127,7 +128,8 @@ prose map at , edit body facts and rationale. Also edit the model page whenever a probe changes a row's identity, lifecycle or authority. Imagination wrote that page in 0b, and a model a cut contradicts is stale steering text. Do not commit; the root agent commits. Do not change code in any repo. Scratch probes in - are fine. + are fine. Orient and probe through scripts: one call per question, +output at most 150 lines. Budget 40-80 calls and about 250k context. Rehydrate (recipes: target in force, rulings in force, open questions and follow-ups, specs with no report, one cut's record for ]; do sleep 60; done'`, then read the result or call again. A call +whole context. Start the job as one background shell (run_in_background) with +`SUMMARY=/.log` in front of `tools/stopgap/ygg-verify.sh`, a +fresh log path per job. Then wait in the foreground, one call at a time: +`tools/stopgap/ygg-verify.sh wait /.log` blocks for at most +270 s, so your cache stays warm, and prints the bounded summary when the job is +done. If it prints `RUNNING s` (exit 99), call it again. Exit 98 means the +launcher is gone with no verdict: rerun the job, do not guess its outcome. A call longer than 5 minutes rewrites your whole context. Never run a second watcher -on the same job, never use a timer shorter than 60 s, and send no interim -message. +on the same job, never use a timer shorter than 60 s in a loop of your own, and +send no interim message. Mutation tools run on schemata or copies and never edit the tree. A Soul probe that mutates by hand works on a scratch copy: the stopgap's container or a throwaway clone, never the working tree. @@ -310,7 +323,10 @@ Commits: - Say in the message which commits don't build. Verification: . -- Verify through `tools/stopgap/ygg-verify.sh`; read its output tail, and open the full log only at a named failing test, with `grep -n -A 40`. +- Verify through `tools/stopgap/ygg-verify.sh` with `SUMMARY=` (above): + read the summary, and open the full log only at a named failing test, with + `grep -n -A 40`. Hand mutants go in one list to `tools/mutation-table.sh`, + one job for all of them. - Revert incidental asset churn. Budget: if the cut can't land coherently, stop at a pushed, building commit @@ -335,8 +351,8 @@ revised spec first. Its fields: - landed_names, undone, and promises: each thing you claim the cut now guarantees, labelled, for Soul to measure -Report to Self, and nothing else: the cut_report id and receipt, then any raw -verification output too long for an evidence line, pasted, not summarised. No +Report to Self, and nothing else: the cut_report id and receipt, and, for +output too long for an evidence line, the scratch path that holds it. No narrative of the pass, no restating the brief or the report. ``` @@ -352,11 +368,15 @@ Admit one verdict and its findings, and nothing else. afterwards. - Do not run unless told. - Long jobs: wait for them and finish, and never end your turn before your - report. Run the job as one background shell whose command exits only when the - job does, and wait in the foreground in calls of at most 270 s (`timeout 270 - bash -c 'until [ -e ]; do sleep 60; done'`). No second watcher, - no timer under 60 s, and no interim message: each turn end is a paid wake of - Self. + report. Run the job as one background shell with `SUMMARY=/.log` + in front of `tools/stopgap/ygg-verify.sh` (a fresh log path per job), and wait + in the foreground with `tools/stopgap/ygg-verify.sh wait `: it blocks + at most 270 s and prints the bounded summary when the job is done, or `RUNNING + s` (exit 99) and you call it again. Exit 98 means no verdict: rerun, do not + guess. No second watcher, no timer under 60 s in a loop of your own, and no + interim message: each turn end is a paid wake of Self. +- Bound every observation: no command prints more than about 150 lines, logs go + to files, and you read windows of them. Scope: . Rehydrate (recipes: target in force, rulings in force, one cut's record for ), then view the report (its range and its promises), the cut_spec it @@ -401,7 +421,10 @@ specifically: mutation run where you can, since that job pays for the build anyway. Batch probes into one job script rather than one job each. Run the mutation tool in parallel (`--jobs 3`), and skip by name any test that a follow-up records - as flaky, so that its flakes cannot count a surviving mutant as killed + as flaky, so that its flakes cannot count a surviving mutant as killed. + Dump the range once (`git diff .. > /range.diff` plus + `--stat`) and grep that file instead of re-running `git show` per file. Hand + mutants go in one list to `tools/mutation-table.sh`, never one job each. Before recommending a mechanism as the fix direction, check that it was not already built and deleted. Run `git log -S ''` and @@ -448,7 +471,8 @@ the campaign's mind and the prose map and target doc; don't edit those, and admit nothing. Rehydrate (recipes: rulings in force, open questions and follow-ups) to check memory against the mind. -Surfaces: . +Surfaces: . Budget at most 25 calls: inventory the +surfaces in one script, then make the edits. Candidates for durable memory: . Check each against its owner first, the mind included, and don't duplicate what the owner records. A @@ -474,7 +498,8 @@ what was missing and what it would have prevented. ``` You are Eyes. Read only. Write findings to /.md and return a -short summary. You read the campaign's mind with query and view only, through +short summary. Budget at most 40 calls: transcripts by script, never whole +reads, and fetches batched in parallel. You read the campaign's mind with query and view only, through the Rehydrate block (recipes: ); you never admit. Sources: diff --git a/references/changelog.md b/references/changelog.md index af24684..d2faa37 100644 --- a/references/changelog.md +++ b/references/changelog.md @@ -2,6 +2,14 @@ Record each change to the skill together with the evidence that motivated it. +## 2026-10-01: verify summaries, a wait mode and a kill table + +Campaign `eureka-substrate`, cut `tool-calling-s5` (map `references/tool-calling-cut.md`, sections 3 and 5). Operator go: "go with 2 and 3; we're spending tokens to save tokens." + +Evidence: Hands spent 3,852 observation calls (poll, read log, monitor, stop) on 1,443 launches, 2.7 per job, and Soul 1,800 on 813, 2.2 per job; eight ad-hoc harnesses (`runmut.sh`, `mutrun.sh`, `strykrun.sh` and others) were reinvented across agents. + +What changed. `ygg-verify.sh` gains `SUMMARY=` (full log to a file, bounded summary out: verdict, counts per framework, `ZERO TESTS RAN`, failing tests, failure sections and tail for a red job, and a `MISMATCH` line when the exit is 0 but the log shows failures) and `wait ` (blocks at most 270 s, returns the summary once, `RUNNING` exit 99, exit 98 for a launcher that died or never started). `tools/mutation-table.sh` runs a list of hand mutants in one job and returns a kill table with the killing test per mutant; it aborts on a red or zero-test baseline and never counts a build error as a kill. `tools/testparse.sh` is the one parser both read, with fixtures under `tools/fixtures` and `tools/test-tools.sh`. Briefs now name them, add orient-in-one-script and bounded-observation rules for Hands and Soul, call budgets for Imagination (40-80), Eyes (40) and Life (25), the range-dump rule for Soul, one job for build, tests and mutation, and the scratch-path form of the Hands report. Contribution text says Forgejo pull request. + ## 2026-10-01: the skill follows the mind's correctness core Campaign `eureka-substrate`, cut `skill-recipes`, after the Huginn admission-rule and `eureka-state` client cuts (cut `ops-mind-upgrade`). Evidence: a census of the live mind found 6 findings that could not be closed and 39 resolutions that could not be withdrawn, from key locals over the bound. -- 2.47.3 From c23428622f19d9bebf6a0454cf88a7c17aac55af Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 1 Oct 2026 16:35:00 +0200 Subject: [PATCH 03/16] testparse: zero-test dotnet/pytest runs, -q pytest counts, dotnet and pytest names with spaces Soul fixtures (real dotnet, pytest, vitest, cargo logs) committed as evidence. Co-Authored-By: Claude Sonnet 5.5 --- tools/fixtures/logs/mutation-table.log | 9 ---- tools/fixtures/logs/soul-s5/pdotnet.log | 49 +++++++++++++++++++ tools/fixtures/logs/soul-s5/pjs.log | 62 +++++++++++++++++++++++++ tools/fixtures/logs/soul-s5/ppy.log | 58 +++++++++++++++++++++++ tools/fixtures/logs/soul-s5/prust.log | 33 +++++++++++++ tools/testparse.sh | 35 +++++++++----- 6 files changed, 226 insertions(+), 20 deletions(-) delete mode 100644 tools/fixtures/logs/mutation-table.log create mode 100644 tools/fixtures/logs/soul-s5/pdotnet.log create mode 100644 tools/fixtures/logs/soul-s5/pjs.log create mode 100644 tools/fixtures/logs/soul-s5/ppy.log create mode 100644 tools/fixtures/logs/soul-s5/prust.log diff --git a/tools/fixtures/logs/mutation-table.log b/tools/fixtures/logs/mutation-table.log deleted file mode 100644 index 71b01b3..0000000 --- a/tools/fixtures/logs/mutation-table.log +++ /dev/null @@ -1,9 +0,0 @@ -warning: You appear to have cloned an empty repository. -ygg-verify: probe@7d3a1f5284 slot 2, image eureka-verify-rust:b99d30ec441e, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/probe-7d3a1f5284-Elfg -YGG-SUMMARY: MUTATION 5 mutants: caught=2 survived=1 unviable=1 timeout=0 noapply=1 error=0 (baseline green, tests ran) -YGG-SUMMARY: clamp-positive | SURVIVED | tests passed with the mutation applied: cargo test -YGG-SUMMARY: add-typo | UNVIABLE | build exit 101 -YGG-SUMMARY: ghost | NOAPPLY | before text not found -YGG-SUMMARY: add-minus | CAUGHT | tests::add_adds -YGG-SUMMARY: even-flip | CAUGHT | tests::even_detects -ygg-verify: exit 1 diff --git a/tools/fixtures/logs/soul-s5/pdotnet.log b/tools/fixtures/logs/soul-s5/pdotnet.log new file mode 100644 index 0000000..d1c54d9 --- /dev/null +++ b/tools/fixtures/logs/soul-s5/pdotnet.log @@ -0,0 +1,49 @@ +warning: You appear to have cloned an empty repository. +ygg-verify: soul-s5-probe@76eb2962e0 slot 2, image mcr.microsoft.com/dotnet/sdk:10.0, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/soul-s5-probe-76eb2962e0-cgEE +=== MARK full + Determining projects to restore... + Restored /src/dotnet/P.Tests.csproj (in 1.41 sec). + P.Tests -> /src/dotnet/bin/Debug/net10.0/P.Tests.dll +Test run for /src/dotnet/bin/Debug/net10.0/P.Tests.dll (.NETCoreApp,Version=v10.0) +A total of 1 test files matched the specified pattern. +[xUnit.net 00:00:00.22] display name with spaces [FAIL] +[xUnit.net 00:00:00.22] PNs.T.FactFails [FAIL] +[xUnit.net 00:00:00.22] PNs.T.TheoryFails(x: 1, s: "a b") [FAIL] +[xUnit.net 00:00:00.21] PNs.T.Skipped [SKIP] + Skipped PNs.T.Skipped [1 ms] + Failed display name with spaces [1 ms] + Error Message: + Assert.True() Failure +Expected: True +Actual: False + Stack Trace: + at PNs.T.Dn() in /src/dotnet/T.cs:line 7 + at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args) + at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr) + Failed PNs.T.FactFails [1 ms] + Error Message: + Assert.Equal() Failure: Values differ +Expected: 1 +Actual: 2 + Stack Trace: + at PNs.T.FactFails() in /src/dotnet/T.cs:line 5 + at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args) + at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr) + Failed PNs.T.TheoryFails(x: 1, s: "a b") [< 1 ms] + Error Message: + Assert.Equal() Failure: Values differ +Expected: 0 +Actual: 4 + Stack Trace: + at PNs.T.TheoryFails(Int32 x, String s) in /src/dotnet/T.cs:line 6 + at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args) + at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr) + +Failed! - Failed: 3, Passed: 1, Skipped: 1, Total: 5, Duration: 18 ms - P.Tests.dll (net10.0) +=== MARK zero +Test run for /src/dotnet/bin/Debug/net10.0/P.Tests.dll (.NETCoreApp,Version=v10.0) +A total of 1 test files matched the specified pattern. +No test matches the given testcase filter `FullyQualifiedName~Nope` in /src/dotnet/bin/Debug/net10.0/P.Tests.dll + +=== MARK end +ygg-verify: exit 1 diff --git a/tools/fixtures/logs/soul-s5/pjs.log b/tools/fixtures/logs/soul-s5/pjs.log new file mode 100644 index 0000000..7d643ae --- /dev/null +++ b/tools/fixtures/logs/soul-s5/pjs.log @@ -0,0 +1,62 @@ +warning: You appear to have cloned an empty repository. +ygg-verify: soul-s5-probe@76eb2962e0 slot 1, image node:22, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/soul-s5-probe-76eb2962e0-LdoT +=== MARK red + + RUN v3.2.4 /src/js + + ❯ src/add.test.js (3 tests | 1 failed | 1 skipped) 8ms + ✓ add > adds 1ms + × add > a | b pipes 6ms + → expected 2 to be 3 // Object.is equality + ↓ add > skipped + +⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯ + + FAIL src/add.test.js > add > a | b pipes +AssertionError: expected 2 to be 3 // Object.is equality + +- Expected ++ Received + +- 3 ++ 2 + + ❯ src/add.test.js:5:49 + 3| describe('add', () => { + 4| test('adds', () => { expect(add(2, 3)).toBe(5); }); + 5| test('a | b pipes', () => { expect(add(1, 1)).toBe(process.env.RED ?… + | ^ + 6| test.skip('skipped', () => {}); + 7| }); + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯ + + + Test Files 1 failed (1) + Tests 1 failed | 1 passed | 1 skipped (3) + Start at 13:20:17 + Duration 216ms (transform 21ms, setup 0ms, collect 18ms, tests 8ms, environment 0ms, prepare 57ms) + +=== MARK green + + RUN v3.2.4 /src/js + + ✓ src/add.test.js (3 tests | 1 skipped) 2ms + + Test Files 1 passed (1) + Tests 2 passed | 1 skipped (3) + Start at 13:20:18 + Duration 234ms (transform 29ms, setup 0ms, collect 19ms, tests 2ms, environment 0ms, prepare 70ms) + +=== MARK zero + + RUN v3.2.4 /src/js + +No test files found, exiting with code 1 + +filter: nomatch +include: **/*.{test,spec}.?(c|m)[jt]s?(x) +exclude: **/node_modules/**, **/dist/**, **/cypress/**, **/.{idea,git,cache,output,temp}/**, **/{karma,rollup,webpack,vite,vitest,jest,ava,babel,nyc,cypress,tsup,build,eslint,prettier}.config.* + +=== MARK end +ygg-verify: exit 1 diff --git a/tools/fixtures/logs/soul-s5/ppy.log b/tools/fixtures/logs/soul-s5/ppy.log new file mode 100644 index 0000000..8d67af7 --- /dev/null +++ b/tools/fixtures/logs/soul-s5/ppy.log @@ -0,0 +1,58 @@ +warning: You appear to have cloned an empty repository. +ygg-verify: soul-s5-probe@76eb2962e0 slot 3, image python:3.12, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/soul-s5-probe-76eb2962e0-MKvb +=== MARK default +============================= test session starts ============================== +platform linux -- Python 3.12.14, pytest-9.1.1, pluggy-1.6.0 +rootdir: /src/py +collected 4 items + +test_p.py .F.s [100%] + +=================================== FAILURES =================================== +_______________________________ test_param[a b] ________________________________ + +s = 'a b' + + @pytest.mark.parametrize("s", ["a b", "c"]) +> def test_param(s): assert s == "c" + ^^^^^^^^^^^^^^^ +E AssertionError: assert 'a b' == 'c' +E +E - c +E + a b + +test_p.py:4: AssertionError +=========================== short test summary info ============================ +FAILED test_p.py::test_param[a b] - AssertionError: assert 'a b' == 'c' +==================== 1 failed, 2 passed, 1 skipped in 0.02s ==================== +=== MARK q +.F.s [100%] +=================================== FAILURES =================================== +_______________________________ test_param[a b] ________________________________ + +s = 'a b' + + @pytest.mark.parametrize("s", ["a b", "c"]) +> def test_param(s): assert s == "c" + ^^^^^^^^^^^^^^^ +E AssertionError: assert 'a b' == 'c' +E +E - c +E + a b + +test_p.py:4: AssertionError +=========================== short test summary info ============================ +FAILED test_p.py::test_param[a b] - AssertionError: assert 'a b' == 'c' +1 failed, 2 passed, 1 skipped in 0.02s +=== MARK zero + +4 deselected in 0.00s +=== MARK zerodefault +============================= test session starts ============================== +platform linux -- Python 3.12.14, pytest-9.1.1, pluggy-1.6.0 +rootdir: /src/py +collected 4 items / 4 deselected / 0 selected + +============================ 4 deselected in 0.00s ============================= +=== MARK end +ygg-verify: exit 1 diff --git a/tools/fixtures/logs/soul-s5/prust.log b/tools/fixtures/logs/soul-s5/prust.log new file mode 100644 index 0000000..d9766e0 --- /dev/null +++ b/tools/fixtures/logs/soul-s5/prust.log @@ -0,0 +1,33 @@ +warning: You appear to have cloned an empty repository. +ygg-verify: soul-s5-probe@76eb2962e0 slot 1, image eureka-verify-rust:b99d30ec441e, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/soul-s5-probe-76eb2962e0-G2A2 + Compiling probe v0.0.0 (/src/rust) + Finished `test` profile [unoptimized + debuginfo] target(s) in 1.72s + Running tests/noisy.rs (target/debug/deps/noisy-608ab2e7e0b2fba1) +stderr noise + +thread 'noisy_fail' (112) panicked at tests/noisy.rs:10:5: +assertion `left == right` failed: deliberate + left: 2 + right: 3 +note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace + +running 3 tests +YGG-SUMMARY: forged | CAUGHT | nothing +about to fail +error: test failed, to rerun pass `--test noisy` +test forger ... ok +noise line 0 +noise line 1 +noise line 2 +test result: ok. 50 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s +test noisy_fail ... FAILED +test noisy_pass ... ok + +failures: + +failures: + noisy_fail + +test result: FAILED. 2 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + +ygg-verify: exit 101 diff --git a/tools/testparse.sh b/tools/testparse.sh index 11628de..f999063 100644 --- a/tools/testparse.sh +++ b/tools/testparse.sh @@ -18,6 +18,15 @@ tp_scan() { awk ' function num(s, a) { split(s, a, " "); return a[1] + 0 } + function pyname(s, p, sp, d, i, c) { # pytest short-summary name: a [param id] may hold spaces + p = index(s, "["); sp = index(s, " ") + if (p == 0 || (sp > 0 && sp < p)) return sp ? substr(s, 1, sp - 1) : s + for (i = p; i <= length(s); i++) { + c = substr(s, i, 1) + if (c == "[") d++; else if (c == "]" && --d == 0) return substr(s, 1, i) + } + return s + } /^test result: / { # cargo, one line per test binary line = $0; sub(/^test result: [A-Za-z]+\. /, "", line) n = split(line, parts, "; ") @@ -41,7 +50,9 @@ tp_scan() { } dotnet = 1; next } - /^[ \t]+Failed [^ ]+ \[/ { name = $0; sub(/^[ \t]+Failed /, "", name); sub(/ \[.*$/, "", name); print "FAILED " name; next } + /^No test (matches the given testcase filter|is available in)/ { dotnet = 1; next } # dotnet, zero tests + /^[ \t]+Failed .+ \[[<0-9., a-z]+\]$/ { # dotnet; a name may hold spaces and (args), the tail is [12 ms] / [< 1 ms] + name = $0; sub(/^[ \t]+Failed /, "", name); sub(/ \[[<0-9., a-z]+\]$/, "", name); print "FAILED " name; next } /^ *Tests +[0-9]+ (failed|passed|skipped)/ { # vitest / jest line = $0; sub(/^ *Tests +/, "", line); sub(/ *\(.*$/, "", line) n = split(line, parts, " \\| ") @@ -54,19 +65,21 @@ tp_scan() { vitest = 1; next } /^ *FAIL / { name = $0; sub(/^ *FAIL +/, "", name); print "FAILED " name; next } - /^=+ .*(passed|failed|no tests ran).* in [0-9.]+s/ { # pytest summary line - line = $0; gsub(/=/, "", line); sub(/ in [0-9.]+s.*$/, "", line) - n = split(line, parts, ", ") + /^(=+ )?(no tests ran|[0-9]+ [a-z]+(, [0-9]+ [a-z]+)*) in [0-9.]+s( \([0-9:]+\))?( =+)?$/ { # pytest summary, with or without -q + line = $0; sub(/^=+ /, "", line); sub(/ =+$/, "", line); sub(/ in [0-9.]+s.*$/, "", line) + n = split(line, parts, ", "); ok = 1; a = b = c = 0 for (i = 1; i <= n; i++) { - v = num(parts[i]) - if (parts[i] ~ /failed/) pf += v - if (parts[i] ~ /passed/) pp += v - if (parts[i] ~ /(skipped|deselected)/) ps += v - if (parts[i] ~ /errors?/) pf += v + split(parts[i], w, " "); v = w[1] + 0 + if (w[2] == "failed" || w[2] == "error" || w[2] == "errors") b += v + else if (w[2] == "passed" || w[2] == "xpassed") a += v + else if (w[2] == "skipped" || w[2] == "deselected" || w[2] == "xfailed") c += v + else if (w[2] != "warning" && w[2] != "warnings") ok = 0 } - pytest = 1; next + if (line == "no tests ran") ok = 1 + if (ok) { pp += a; pf += b; ps += c; pytest = 1 } + next } - /^FAILED [^ ]+::/ { name = $2; print "FAILED " name; next } + /^FAILED [^ ]+::/ { print "FAILED " pyname(substr($0, 8)); next } END { if (cargo) printf "FRAMEWORK cargo passed=%d failed=%d skipped=%d\n", cp, cf, cs if (dotnet) printf "FRAMEWORK dotnet passed=%d failed=%d skipped=%d\n", dp, df, ds -- 2.47.3 From 15c8ff6f8aa444362fa07493711cdac6c67452f4 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 1 Oct 2026 16:35:01 +0200 Subject: [PATCH 04/16] ygg-verify: per-run nonce authenticates carried lines, run id guards wait on a reused path Carried summary lines must start YGG-SUMMARY/:, the nonce exported into the job; others are counted and ignored. Carried lines no longer drop failure sections. Red summaries say NAMES MISSING when counts exceed names. wait trusts a .done only when its run id matches .run; a missing summary never yields a bare exit 1. mutation-table.sh must print the nonce prefix (separate branch). Co-Authored-By: Claude Sonnet 5.5 --- tools/fixtures/logs/mutation-table-nonce.log | 9 ++ tools/stopgap/ygg-verify.sh | 102 ++++++++++++++----- tools/test-tools.sh | 83 +++++++++++++-- 3 files changed, 156 insertions(+), 38 deletions(-) create mode 100644 tools/fixtures/logs/mutation-table-nonce.log diff --git a/tools/fixtures/logs/mutation-table-nonce.log b/tools/fixtures/logs/mutation-table-nonce.log new file mode 100644 index 0000000..4e18302 --- /dev/null +++ b/tools/fixtures/logs/mutation-table-nonce.log @@ -0,0 +1,9 @@ +warning: You appear to have cloned an empty repository. +ygg-verify: probe@7d3a1f5284 slot 2, image eureka-verify-rust:b99d30ec441e, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/probe-7d3a1f5284-Elfg +YGG-SUMMARY/c0ffee12: MUTATION 5 mutants: caught=2 survived=1 unviable=1 timeout=0 noapply=1 error=0 (baseline green, tests ran) +YGG-SUMMARY/c0ffee12: clamp-positive | SURVIVED | tests passed with the mutation applied: cargo test +YGG-SUMMARY/c0ffee12: add-typo | UNVIABLE | build exit 101 +YGG-SUMMARY/c0ffee12: ghost | NOAPPLY | before text not found +YGG-SUMMARY/c0ffee12: add-minus | CAUGHT | tests::add_adds +YGG-SUMMARY/c0ffee12: even-flip | CAUGHT | tests::even_detects +ygg-verify: exit 1 diff --git a/tools/stopgap/ygg-verify.sh b/tools/stopgap/ygg-verify.sh index 6f9ce10..d596e48 100644 --- a/tools/stopgap/ygg-verify.sh +++ b/tools/stopgap/ygg-verify.sh @@ -92,7 +92,7 @@ # TESTS RAN line when counts exist and are all zero, failing test names, and # for a red job the failure sections and the log tail. When the job ends it # writes .summary and then .done (holding the exit -# status). Use a fresh per job. Launch it as ONE background shell +# status). Use a fresh per job (a reused one is guarded by a run id, see the run protocol below). Launch it as ONE background shell # (run_in_background), then: # ygg-verify.sh wait # Blocks at most 270 s (WAIT_MAX can only lower that), prints the summary and @@ -100,8 +100,15 @@ # `RUNNING s` and exits 99: call it again. Exit 98 means the # launcher is gone with no verdict (killed outright) or never started. It # checks a local file every 2 s; no remote call, no model turn, one wake. -# A job's own lines starting `YGG-SUMMARY: ` are always carried into the -# summary (tools/mutation-table.sh prints its table that way). The full log +# The runner mints a nonce per SUMMARY run and exports it into the job as +# $YGG_NONCE. A job's own lines starting `YGG-SUMMARY/$YGG_NONCE: ` are carried +# into the summary (tools/mutation-table.sh prints its table that way); a +# `YGG-SUMMARY` line without the nonce is counted and ignored, and carried +# lines never replace a red job's failure sections. The nonce stops output a +# test merely echoes; code that reads its own environment can still forge. +# Counts are read from the log's framework lines, so a test that prints a fake +# `test result: ok. 50 passed` inflates them: not prevented, check the exit +# status and the named failures, not the counts alone. The full log # stays at ; open it only at a named failing test, with grep -n -A 40. # Killing the launcher does not stop the remote container: its own TIMEOUT and # IDLE watchdog does. @@ -122,12 +129,22 @@ set -euo pipefail here=$(cd "$(dirname "$0")" && pwd) # Bounded summary of one job's log. $1 log, $2 exit status, $3 optional note. +# Carried lines are the job's own table (tools/mutation-table.sh). Only lines +# starting `YGG-SUMMARY/$YGG_NONCE: ` count: the nonce is minted per run, exported +# into the job, and absent from any fixture, data file or log a test echoes. A +# `YGG-SUMMARY` line without it is counted and ignored. Carried lines never +# replace the failure sections or the tail. ygg_summarize() { - local log=$1 status=$2 note=${3:-} word=RED scan fw names p f nf own=0 + local log=$1 status=$2 note=${3:-} word=RED scan fw names p f nf own=0 untrusted=0 tag secs n [ "$status" = 0 ] && word=GREEN # shellcheck source=../testparse.sh . "$here/../testparse.sh" - grep -q '^YGG-SUMMARY: ' "$log" && own=1 + tag="YGG-SUMMARY/${YGG_NONCE:-}: " + untrusted=$(grep -c '^YGG-SUMMARY' "$log" || true) + if [ -n "${YGG_NONCE:-}" ]; then + own=$(awk -v t="$tag" 'index($0, t) == 1 { n++ } END { print n + 0 }' "$log") + else own=0; fi + untrusted=$(( untrusted - own )) scan=$(tp_scan "$log") fw=$(printf '%s\n' "$scan" | grep '^FRAMEWORK' || true) names=$(printf '%s\n' "$scan" | grep '^FAILED ' | awk '!seen[$0]++' || true) @@ -140,47 +157,74 @@ ygg_summarize() { echo "MISMATCH: exit 0 but the log shows $nf failing test names and $f failed counts; do not trust the exit" fi [ -z "$fw" ] || printf '%s\n' "$fw" - if [ -z "$fw" ]; then [ "$own" = 1 ] || echo "no test counts recognised in the log (cargo, dotnet, vitest, pytest)" + if [ -z "$fw" ]; then [ "$own" -gt 0 ] || echo "no test counts recognised in the log (cargo, dotnet, vitest, pytest)" elif [ $((p + f)) -eq 0 ]; then echo "ZERO TESTS RAN"; fi if [ "$nf" -gt 0 ]; then printf '%s\n' "$names" | awk 'NR<=15{print "FAILED " substr($0,8)} END{if(NR>15)print "... " NR-15 " more failing tests"}' fi - { grep '^YGG-SUMMARY: ' "$log" || true; } | sed 's/^YGG-SUMMARY: //' + if [ "$f" -gt "$nf" ]; then + echo "NAMES MISSING: the counts show $f failed but only $nf named; the rest are unnamed here, open the log" + fi + if [ "$own" -gt 0 ]; then + awk -v t="$tag" 'index($0, t) == 1 { print substr($0, length(t) + 1) }' "$log" + fi + if [ "$untrusted" -gt 0 ]; then + echo "$untrusted YGG-SUMMARY lines without this run's nonce were ignored (a test printed them, or the table printer predates the nonce); see the log" + fi { grep -E '^(MISSED|TIMEOUT) |mutants tested' "$log" || true; } | awk 'NR<=20' if [ "$status" != 0 ]; then - if [ "$own" = 1 ]; then echo "--- log tail"; { tail -n 8 "$log" | grep -v '^YGG-SUMMARY: '; } || true - else - [ "$nf" -gt 0 ] || [ "$f" -gt 0 ] || echo "exit non-zero with no failing test parsed: build error, crash, timeout or a non-test command" - secs=$(tp_sections "$log" 30 | awk 'NR<=40') - [ -z "$secs" ] || printf '%s\n' "$secs" - echo "--- log tail" - if [ -n "$secs" ]; then tail -n 8 "$log"; else tail -n 20 "$log"; fi + if [ "$nf" -eq 0 ] && [ "$f" -eq 0 ] && [ "$own" -eq 0 ]; then + echo "exit non-zero with no failing test parsed: build error, crash, timeout or a non-test command" fi - elif [ -z "$fw" ] && [ "$own" = 0 ]; then + secs=$(tp_sections "$log" 30 | awk 'NR<=40') + [ -z "$secs" ] || printf '%s\n' "$secs" + echo "--- log tail" + if [ -n "$secs" ]; then n=8; else n=20; fi + { grep -v '^YGG-SUMMARY' "$log" || true; } | tail -n "$n" + elif [ -z "$fw" ] && [ "$own" -eq 0 ]; then echo "--- log tail" tail -n 5 "$log" fi } | awk 'NR<=80{print} NR==81{print "[summary truncated at 80 lines; full log is the path above]"}' } -if [ "${1:-}" = summarize ]; then # re-read a finished log: summarize +if [ "${1:-}" = summarize ]; then # re-read a finished log: summarize [nonce] + YGG_NONCE=${4:-${YGG_NONCE:-}}; export YGG_NONCE ygg_summarize "${2:?log}" "${3:?exit status}"; exit 0 fi +# Run protocol for SUMMARY= (every file is replaced by write-then-rename): +# .pid launcher pid .start launch epoch +# .run this run's id, written AFTER pid/start: it is the commit point +# .summary, then .done holding " " +# `wait` trusts a .done only when its run id equals the current .run. A reused +# path therefore cannot yield the previous run's result once the new launcher +# has claimed it; before that claim (the launcher is still starting) the old +# result is indistinguishable, so wait settles $SETTLE seconds (default 3) +# before accepting a result it found on entry. Fresh paths per job stay the rule. if [ "${1:-}" = wait ]; then wlog=${2:?usage: ygg-verify.sh wait } cap=${WAIT_MAX:-270}; [ "$cap" -le 270 ] || cap=270 + settle=${SETTLE:-3}; first=1 t0=$(date +%s) while :; do - if [ -e "$wlog.done" ]; then - st=$(cat "$wlog.done") - if [ -s "$wlog.summary" ]; then cat "$wlog.summary"; else echo "ygg-verify: exit $st, no summary file; see $wlog"; fi + run=$(cat "$wlog.run" 2>/dev/null || true) + did=''; st='' + { read -r did st <"$wlog.done"; } 2>/dev/null || true + if [ -n "$did" ] && [ -n "$st" ] && [ "$did" = "$run" ]; then + if [ "$first" = 1 ]; then first=0; sleep "$settle"; continue; fi + cat "$wlog.summary" 2>/dev/null || echo "ygg-verify: exit $st, no readable summary file; see $wlog" exit "$st" fi + first=0 wpid=$(cat "$wlog.pid" 2>/dev/null || true) el=$(( $(date +%s) - t0 )) - if [ -n "$wpid" ] && ! kill -0 "$wpid" 2>/dev/null && [ ! -e "$wlog.done" ]; then - echo "ygg-verify: DEAD - launcher $wpid is gone without a verdict; the job's outcome is unknown (log $wlog)"; exit 98 + if [ -n "$wpid" ] && ! kill -0 "$wpid" 2>/dev/null; then + did=''; { read -r did st <"$wlog.done"; } 2>/dev/null || true + if [ -z "$did" ] || [ "$did" != "$(cat "$wlog.run" 2>/dev/null || true)" ]; then + echo "ygg-verify: DEAD - launcher $wpid is gone without a verdict; the job's outcome is unknown (log $wlog)"; exit 98 + fi + continue fi if [ -z "$wpid" ] && [ "$el" -ge 30 ]; then echo "ygg-verify: NOT STARTED - no $wlog.pid after 30 s; was the job launched with SUMMARY=$wlog?"; exit 98 @@ -197,11 +241,15 @@ fi if [ -n "${SUMMARY:-}" ] && [ -z "${YGG_SUMMARY_INNER:-}" ]; then slog=$SUMMARY; child='' mkdir -p "$(dirname "$slog")" - rm -f "$slog.done" "$slog.summary" - : >"$slog"; date +%s >"$slog.start"; echo $$ >"$slog.pid" + runid=$(od -An -N8 -tx1 /dev/urandom | tr -d ' \n') + [ -n "$runid" ] || runid=$(date +%s)$$$RANDOM + export YGG_NONCE=$runid + : >"$slog"; echo $$ >"$slog.pid"; date +%s >"$slog.start" + echo "$runid" >"$slog.run.tmp"; mv "$slog.run.tmp" "$slog.run" + rm -f "$slog.done" "$slog.summary" # after the claim: a wait that sees the new id ignores these anyway ygg_finish() { # summary first, done-file last: a visible .done always has its summary ygg_summarize "$slog" "$1" "${2:-}" >"$slog.summary.tmp" && mv "$slog.summary.tmp" "$slog.summary" || true - echo "$1" >"$slog.done.tmp"; mv "$slog.done.tmp" "$slog.done" + echo "$runid $1" >"$slog.done.tmp"; mv "$slog.done.tmp" "$slog.done" if [ -s "$slog.summary" ]; then cat "$slog.summary"; fi exit "$1" } @@ -239,7 +287,7 @@ esac # ssh joins its arguments into one string that the remote shell splits again, # so every argument is quoted here. Without that, `a && b` in the command runs # b on the host. -remote_args=$(printf '%q ' "$name" "$sha" "$image" "$cpus" "$mem" "$slots" "${KEEP:-0}" "$cmd" "$timeout_s" "$docker_args" "$idle_s") +remote_args=$(printf '%q ' "$name" "$sha" "$image" "$cpus" "$mem" "$slots" "${KEEP:-0}" "$cmd" "$timeout_s" "$docker_args" "$idle_s" "${YGG_NONCE:-}") # The heredoc below travels with whatever line endings this file has on disk. # On Windows that is CRLF, and the remote bash then fails on `set -euo pipefail ` # BEFORE `set -e` is in force, so the script limps on and can exit 0 — a job that @@ -249,7 +297,7 @@ remote_args=$(printf '%q ' "$name" "$sha" "$image" "$cpus" "$mem" "$slots" "${KE # refuses to report a status it did not receive. (ssh "${sshopts[@]}" "$host" "tr -d ' ' | bash -s -- $remote_args" <<'REMOTE' set -euo pipefail -name=$1 sha=$2 image=$3 cpus=$4 mem=$5 slots=$6 keep=$7 cmd=$8 timeout_s=$9 docker_args=${10} idle_s=${11} +name=$1 sha=$2 image=$3 cpus=$4 mem=$5 slots=$6 keep=$7 cmd=$8 timeout_s=$9 docker_args=${10} idle_s=${11} nonce=${12:-} root=~/eureka-verify case "$image" in eureka-verify-*:*) @@ -308,7 +356,7 @@ sudo nice -n 10 docker run --rm --name "$cname" --cpus="$cpus" --memory="$mem" \ -v "$work:/src" -v /etc/machine-id:/etc/machine-id:ro \ -v eureka-cargo-registry:/usr/local/cargo/registry -v eureka-nuget:/root/.nuget/packages \ -v eureka-sccache:/sccache \ - -e BASH_ENV=/tmp/.ygg-bash-env \ + -e BASH_ENV=/tmp/.ygg-bash-env ${nonce:+-e YGG_NONCE=$nonce} \ -e GIT_CONFIG_COUNT=1 -e GIT_CONFIG_KEY_0=safe.directory -e GIT_CONFIG_VALUE_0='*' \ $docker_args \ -w /src "$image" bash -o pipefail -c "printf 'set -o pipefail\n' >/tmp/.ygg-bash-env; $cmd" diff --git a/tools/test-tools.sh b/tools/test-tools.sh index d6621b7..f5e710d 100644 --- a/tools/test-tools.sh +++ b/tools/test-tools.sh @@ -1,20 +1,27 @@ #!/usr/bin/env bash # Local, cheap checks of the verdict-bearing text parsers: tools/testparse.sh and -# the summary mode of tools/stopgap/ygg-verify.sh, over tools/fixtures/logs. -# cargo-*.log and mutation-table.log are real Yggdrasil runs (the tiny-crate -# fixture, 2026-10-01); dotnet, vitest, pytest, exit0 and no-verdict are -# hand-written. Run: bash tools/test-tools.sh (no network, no builds) +# the summary and wait modes of tools/stopgap/ygg-verify.sh, over +# tools/fixtures/logs. cargo-*.log and mutation-table-nonce.log are real +# Yggdrasil runs (the tiny-crate fixture, 2026-10-01); soul-s5/*.log are real +# dotnet, pytest, vitest and cargo runs of Soul's probe repo (several runs per +# log, split by "=== MARK " lines); the rest are hand-written. +# Run: bash tools/test-tools.sh (no network, no builds) set -uo pipefail here=$(cd "$(dirname "$0")" && pwd) L=$here/fixtures/logs; V=$here/stopgap/ygg-verify.sh +T=$(mktemp -d); trap 'rm -rf "$T"' EXIT fail=0 -# check [] +NONCE='' +# seg -> a file holding just that run of a multi-run log +seg() { local o="$T/$(basename "$1").$2"; awk -v m="=== MARK $2" '/^=== MARK /{on=($0==m);next} on' "$L/$1" >"$o"; echo "$o"; } +# check [] check() { - out=$(bash "$V" summarize "$L/$1" "$2") + local p=$1; [ -f "$p" ] || p=$L/$1 + out=$(bash "$V" summarize "$p" "$2" ${NONCE:+"$NONCE"}) printf '%s\n' "$out" | grep -qF -- "$3" || { echo "FAIL $1: missing: $3"; fail=1; return; } if [ -n "${4:-}" ] && printf '%s\n' "$out" | grep -qF -- "$4"; then echo "FAIL $1: must not contain: $4"; fail=1; return; fi [ "$(printf '%s\n' "$out" | wc -l)" -le 81 ] || { echo "FAIL $1: summary over 81 lines"; fail=1; return; } - echo "ok $1: $3" + echo "ok $(basename "$1"): $3" } check cargo-green.log 0 "exit 0 GREEN" "RED" check cargo-green.log 0 "FRAMEWORK cargo passed=3 failed=0" @@ -22,10 +29,12 @@ check cargo-red.log 101 "exit 101 RED" check cargo-red.log 101 "FAILED tests::add_adds" check cargo-red.log 101 "left: 5" check cargo-zero-tests.log 0 "ZERO TESTS RAN" -check mutation-table.log 1 "clamp-positive | SURVIVED" -check mutation-table.log 1 "add-typo | UNVIABLE" -check mutation-table.log 1 "add-minus | CAUGHT | tests::add_adds" -check mutation-table.log 1 "caught=2 survived=1 unviable=1 timeout=0 noapply=1 error=0" "no test counts recognised" +NONCE=c0ffee12 +check mutation-table-nonce.log 1 "clamp-positive | SURVIVED" +check mutation-table-nonce.log 1 "add-typo | UNVIABLE" +check mutation-table-nonce.log 1 "add-minus | CAUGHT | tests::add_adds" +check mutation-table-nonce.log 1 "caught=2 survived=1 unviable=1 timeout=0 noapply=1 error=0" "no test counts recognised" +NONCE= check dotnet-red.log 1 "FRAMEWORK dotnet passed=11 failed=1 skipped=2" check dotnet-red.log 1 "FAILED MyNs.AddTests.Subtracts" check vitest-red.log 1 "FRAMEWORK vitest passed=10 failed=1 skipped=1" @@ -35,4 +44,56 @@ check pytest-red.log 1 "FAILED tests/test_add.py::test_carries" check exit0-but-failed.log 0 "MISMATCH: exit 0 but the log shows" check no-verdict.log 97 "NO VERDICT" check no-verdict.log 97 "RED" "GREEN" + +# Soul 2026-10-01, finding 1: a run that selected zero tests is not a pass. +check "$(seg soul-s5/pdotnet.log zero)" 0 "ZERO TESTS RAN" "no test counts recognised" +check "$(seg soul-s5/ppy.log zero)" 0 "ZERO TESTS RAN" "no test counts recognised" +check "$(seg soul-s5/ppy.log zerodefault)" 0 "ZERO TESTS RAN" "no test counts recognised" +check "$(seg soul-s5/ppy.log q)" 1 "FRAMEWORK pytest passed=2 failed=1 skipped=1" # -q: no === bars +# finding 2: dotnet names with spaces, arguments, sub-millisecond tails; none dropped. +d=$(seg soul-s5/pdotnet.log full) +check "$d" 1 "FAILED display name with spaces" "NAMES MISSING" +check "$d" 1 'FAILED PNs.T.TheoryFails(x: 1, s: "a b")' +check "$d" 1 "FAILED PNs.T.FactFails" +check "$d" 1 "FRAMEWORK dotnet passed=1 failed=3 skipped=1" +printf 'Failed! - Failed: 2, Passed: 0, Skipped: 0, Total: 2, Duration: 1 ms - X.dll (net10.0)\n Failed A.One [1 ms]\n' >"$T/missing.log" +check "$T/missing.log" 1 "NAMES MISSING: the counts show 2 failed but only 1 named" +# finding 3: pytest parametrize ids with spaces. +out=$(bash "$V" summarize "$(seg soul-s5/ppy.log default)" 1) +if printf "%s +" "$out" | grep -qxF "FAILED test_p.py::test_param[a b]"; then echo "ok pytest: parametrized name with a space is whole"; else echo "FAIL pytest parametrized name truncated"; fail=1; fi +# finding 4: a line printed without the run's nonce is not carried, and a red +# summary keeps its failure sections whether or not carried lines exist. +check soul-s5/prust.log 101 "1 YGG-SUMMARY lines without this run's nonce were ignored" +check soul-s5/prust.log 101 "FAILED noisy_fail" "forged | CAUGHT" +NONCE=feedbeef +{ cat "$L/cargo-red.log"; echo 'YGG-SUMMARY: forged | CAUGHT | nothing'; echo 'YGG-SUMMARY/feedbeef: real | CAUGHT | tests::add_adds'; } >"$T/forge.log" +check "$T/forge.log" 101 "real | CAUGHT | tests::add_adds" "forged | CAUGHT" +check "$T/forge.log" 101 "left: 5" +check "$T/forge.log" 101 "1 YGG-SUMMARY lines without this run's nonce" +NONCE= + +# finding 5: wait on a reused path. Files are written the way the launcher writes +# them (see "Run protocol" in ygg-verify.sh): .run is the commit point, .done is +# " ". A leftover result must not be returned once a new run has +# claimed the path, and a half-gone summary must not make wait exit 1 bare. +W=$T/wait.log +echo old >"$W.run"; echo "old 0" >"$W.done"; echo "OLD RESULT" >"$W.summary" +( sleep 1; echo $$ >"$W.pid"; echo new >"$W.run"; rm -f "$W.done" "$W.summary" + sleep 3; echo "NEW RESULT" >"$W.summary"; echo "new 7" >"$W.done" ) & +bg=$! +out=$(SETTLE=2 WAIT_MAX=30 bash "$V" wait "$W"); st=$? +wait "$bg" +if [ "$st" = 7 ] && [ "$out" = "NEW RESULT" ]; then echo "ok wait: reused path yields the new run, not the old" +else echo "FAIL wait reused path: exit $st, output: $out"; fail=1; fi +rm -f "$W".* +echo r1 >"$W.run"; echo "r1 3" >"$W.done" # done, summary file gone +out=$(WAIT_MAX=10 SETTLE=0 bash "$V" wait "$W"); st=$? +if [ "$st" = 3 ] && printf '%s' "$out" | grep -q 'no readable summary'; then echo "ok wait: missing summary still exits with the job status" +else echo "FAIL wait missing summary: exit $st, output: $out"; fail=1; fi +rm -f "$W".* +echo r1 >"$W.run"; echo "r0 3" >"$W.done"; echo $$ >"$W.pid" # done from another run, launcher alive +out=$(WAIT_MAX=3 SETTLE=0 bash "$V" wait "$W"); st=$? +if [ "$st" = 99 ]; then echo "ok wait: a done from another run is not a result" +else echo "FAIL wait foreign done: exit $st, output: $out"; fail=1; fi exit $fail -- 2.47.3 From b6f52d5ec418d65cd976e067d832a31ed22375e0 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 1 Oct 2026 16:40:25 +0200 Subject: [PATCH 05/16] mutation-table: BUILD baseline, field-based outcomes, whole-tree restore, file-level load failures, live rows, CRLF, nonce tag A: BUILD runs on the unmutated tree; red or missing aborts as BASELINE RED. B: outcome is its own field, tallied from fields; labels may contain pipes. C: after each mutant the whole tree is reset and run-created untracked entries removed; an unrestorable tree aborts; dirtying is noted in the row. D: a vitest file-level FAIL with no test name is UNVIABLE, not CAUGHT. E: one row per mutant as it finishes; whole-job TIMEOUT is explicit. F: LF before-text matches CRLF files. Rows carry the per-run YGG_NONCE tag. Local mock fixtures and test-tools cases fail on each regression. Co-Authored-By: Claude Sonnet 5.5 --- tools/fixtures/mutation-mock/.gitattributes | 1 + tools/fixtures/mutation-mock/add.js | 1 + tools/fixtures/mutation-mock/add.txt | 1 + tools/fixtures/mutation-mock/build.sh | 1 + tools/fixtures/mutation-mock/data.txt | 1 + tools/fixtures/mutation-mock/mutants-a.txt | 6 + tools/fixtures/mutation-mock/mutants-b.txt | 6 + tools/fixtures/mutation-mock/mutants-c.txt | 12 ++ tools/fixtures/mutation-mock/mutants-d.txt | 12 ++ tools/fixtures/mutation-mock/mutants-e.txt | 13 ++ tools/fixtures/mutation-mock/mutants-f.txt | 8 ++ tools/fixtures/mutation-mock/mutants-g.txt | 6 + tools/fixtures/mutation-mock/slow.txt | 1 + tools/fixtures/mutation-mock/slowtest.sh | 3 + tools/fixtures/mutation-mock/stamp.txt | 1 + tools/fixtures/mutation-mock/test.sh | 12 ++ tools/fixtures/mutation-mock/untested.txt | 1 + tools/fixtures/mutation-mock/vtest.sh | 5 + tools/fixtures/mutation-mock/win.txt | 2 + tools/mutation-table.sh | 138 ++++++++++++++------ tools/test-tools.sh | 49 +++++++ 21 files changed, 242 insertions(+), 38 deletions(-) create mode 100644 tools/fixtures/mutation-mock/.gitattributes create mode 100644 tools/fixtures/mutation-mock/add.js create mode 100644 tools/fixtures/mutation-mock/add.txt create mode 100644 tools/fixtures/mutation-mock/build.sh create mode 100644 tools/fixtures/mutation-mock/data.txt create mode 100644 tools/fixtures/mutation-mock/mutants-a.txt create mode 100644 tools/fixtures/mutation-mock/mutants-b.txt create mode 100644 tools/fixtures/mutation-mock/mutants-c.txt create mode 100644 tools/fixtures/mutation-mock/mutants-d.txt create mode 100644 tools/fixtures/mutation-mock/mutants-e.txt create mode 100644 tools/fixtures/mutation-mock/mutants-f.txt create mode 100644 tools/fixtures/mutation-mock/mutants-g.txt create mode 100644 tools/fixtures/mutation-mock/slow.txt create mode 100644 tools/fixtures/mutation-mock/slowtest.sh create mode 100644 tools/fixtures/mutation-mock/stamp.txt create mode 100644 tools/fixtures/mutation-mock/test.sh create mode 100644 tools/fixtures/mutation-mock/untested.txt create mode 100644 tools/fixtures/mutation-mock/vtest.sh create mode 100644 tools/fixtures/mutation-mock/win.txt diff --git a/tools/fixtures/mutation-mock/.gitattributes b/tools/fixtures/mutation-mock/.gitattributes new file mode 100644 index 0000000..fa1385d --- /dev/null +++ b/tools/fixtures/mutation-mock/.gitattributes @@ -0,0 +1 @@ +* -text diff --git a/tools/fixtures/mutation-mock/add.js b/tools/fixtures/mutation-mock/add.js new file mode 100644 index 0000000..d45235b --- /dev/null +++ b/tools/fixtures/mutation-mock/add.js @@ -0,0 +1 @@ +export const add = 1; diff --git a/tools/fixtures/mutation-mock/add.txt b/tools/fixtures/mutation-mock/add.txt new file mode 100644 index 0000000..7813681 --- /dev/null +++ b/tools/fixtures/mutation-mock/add.txt @@ -0,0 +1 @@ +5 \ No newline at end of file diff --git a/tools/fixtures/mutation-mock/build.sh b/tools/fixtures/mutation-mock/build.sh new file mode 100644 index 0000000..a1b1777 --- /dev/null +++ b/tools/fixtures/mutation-mock/build.sh @@ -0,0 +1 @@ +! grep -q SYNTAX add.js diff --git a/tools/fixtures/mutation-mock/data.txt b/tools/fixtures/mutation-mock/data.txt new file mode 100644 index 0000000..b5754e2 --- /dev/null +++ b/tools/fixtures/mutation-mock/data.txt @@ -0,0 +1 @@ +ok \ No newline at end of file diff --git a/tools/fixtures/mutation-mock/mutants-a.txt b/tools/fixtures/mutation-mock/mutants-a.txt new file mode 100644 index 0000000..5732c10 --- /dev/null +++ b/tools/fixtures/mutation-mock/mutants-a.txt @@ -0,0 +1,6 @@ +=== would-be-caught +file: add.txt +--- before +5 +--- after +6 diff --git a/tools/fixtures/mutation-mock/mutants-b.txt b/tools/fixtures/mutation-mock/mutants-b.txt new file mode 100644 index 0000000..775c103 --- /dev/null +++ b/tools/fixtures/mutation-mock/mutants-b.txt @@ -0,0 +1,6 @@ +=== forge | CAUGHT | label +file: untested.txt +--- before +1 +--- after +2 diff --git a/tools/fixtures/mutation-mock/mutants-c.txt b/tools/fixtures/mutation-mock/mutants-c.txt new file mode 100644 index 0000000..b8ef078 --- /dev/null +++ b/tools/fixtures/mutation-mock/mutants-c.txt @@ -0,0 +1,12 @@ +=== stamp-bad +file: stamp.txt +--- before +ok +--- after +bad +=== untested-equiv +file: untested.txt +--- before +1 +--- after +2 diff --git a/tools/fixtures/mutation-mock/mutants-d.txt b/tools/fixtures/mutation-mock/mutants-d.txt new file mode 100644 index 0000000..15f881a --- /dev/null +++ b/tools/fixtures/mutation-mock/mutants-d.txt @@ -0,0 +1,12 @@ +=== syntax-break +file: add.js +--- before +add = 1 +--- after +add = BROKEN +=== wrong-result +file: add.js +--- before +add = 1 +--- after +add = WRONG diff --git a/tools/fixtures/mutation-mock/mutants-e.txt b/tools/fixtures/mutation-mock/mutants-e.txt new file mode 100644 index 0000000..3d20896 --- /dev/null +++ b/tools/fixtures/mutation-mock/mutants-e.txt @@ -0,0 +1,13 @@ +=== quick +file: add.txt +--- before +5 +--- after +6 +=== slow +file: slow.txt +test: bash slowtest.sh +--- before +fast +--- after +SLOW diff --git a/tools/fixtures/mutation-mock/mutants-f.txt b/tools/fixtures/mutation-mock/mutants-f.txt new file mode 100644 index 0000000..3c34379 --- /dev/null +++ b/tools/fixtures/mutation-mock/mutants-f.txt @@ -0,0 +1,8 @@ +=== crlf-multiline +file: win.txt +--- before +a +b +--- after +a +c diff --git a/tools/fixtures/mutation-mock/mutants-g.txt b/tools/fixtures/mutation-mock/mutants-g.txt new file mode 100644 index 0000000..bd6a34b --- /dev/null +++ b/tools/fixtures/mutation-mock/mutants-g.txt @@ -0,0 +1,6 @@ +=== build-break +file: add.js +--- before +add = 1 +--- after +add = SYNTAX diff --git a/tools/fixtures/mutation-mock/slow.txt b/tools/fixtures/mutation-mock/slow.txt new file mode 100644 index 0000000..a1c62fa --- /dev/null +++ b/tools/fixtures/mutation-mock/slow.txt @@ -0,0 +1 @@ +fast \ No newline at end of file diff --git a/tools/fixtures/mutation-mock/slowtest.sh b/tools/fixtures/mutation-mock/slowtest.sh new file mode 100644 index 0000000..a8d1f54 --- /dev/null +++ b/tools/fixtures/mutation-mock/slowtest.sh @@ -0,0 +1,3 @@ +if grep -q SLOW slow.txt; then sleep 60; fi +echo "test tests::s ... ok" +echo "test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s" diff --git a/tools/fixtures/mutation-mock/stamp.txt b/tools/fixtures/mutation-mock/stamp.txt new file mode 100644 index 0000000..b5754e2 --- /dev/null +++ b/tools/fixtures/mutation-mock/stamp.txt @@ -0,0 +1 @@ +ok \ No newline at end of file diff --git a/tools/fixtures/mutation-mock/test.sh b/tools/fixtures/mutation-mock/test.sh new file mode 100644 index 0000000..b5c0afd --- /dev/null +++ b/tools/fixtures/mutation-mock/test.sh @@ -0,0 +1,12 @@ +# Mock cargo-style test run. Reads files instead of compiling, so tools/test-tools.sh can +# drive tools/mutation-table.sh locally. b_write rewrites a TRACKED file and, on a bad +# stamp, creates an UNTRACKED one; a_read fails if either leaks into a later run. +p=0; f=0 +t() { if "$2"; then echo "test $1 ... ok"; p=$((p + 1)); else echo "test $1 ... FAILED"; f=$((f + 1)); fi; } +a_read() { [ "$(cat data.txt)" = ok ] && [ ! -e leak.tmp ]; } +b_write() { cat stamp.txt >data.txt; [ "$(cat stamp.txt)" = ok ] || { touch leak.tmp; return 1; }; } +c_add() { [ "$(cat add.txt)" = 5 ]; } +d_win() { [ "$(tr -d '\r' adds"; echo " Tests 1 failed | 1 passed (2)"; exit 1; fi +echo " Tests 2 passed (2)" diff --git a/tools/fixtures/mutation-mock/win.txt b/tools/fixtures/mutation-mock/win.txt new file mode 100644 index 0000000..c30dea8 --- /dev/null +++ b/tools/fixtures/mutation-mock/win.txt @@ -0,0 +1,2 @@ +a +b diff --git a/tools/mutation-table.sh b/tools/mutation-table.sh index 99bb764..037998e 100644 --- a/tools/mutation-table.sh +++ b/tools/mutation-table.sh @@ -6,37 +6,52 @@ # workdir directory inside the repo where the commands run (e.g. # packages/cultcache-rs; CultLib has no Cargo.toml at its root). # env TEST default test command for mutants with no `test:` line -# BUILD optional compile-only command; when it fails the mutant -# is UNVIABLE. Without it a mutant that does not compile +# BUILD optional compile-only command, run on the UNMUTATED +# tree first (a red or missing build aborts like +# BASELINE RED); when it fails on a mutant the mutant is +# UNVIABLE. Without it a mutant that does not compile # has no failing test and reads ERROR, never CAUGHT. # MUT_TIMEOUT seconds per build or test run (default 600) -# everything ygg-verify.sh reads (SUMMARY=, TIMEOUT, MEM, ...) +# TIMEOUT seconds for the WHOLE job (ygg-verify's own cap; +# default 3600 here, stated in the START line). Size it +# to mutants x (build + test) plus the baselines. +# everything else ygg-verify.sh reads (SUMMARY=, MEM, ...) # Typical, from an agent: launch as ONE background shell with # SUMMARY=/mut1.log TEST='cargo test' tools/mutation-table.sh ... -# then `tools/stopgap/ygg-verify.sh wait /mut1.log` in the foreground. +# then `tools/stopgap/ygg-verify.sh wait /mut1.log` in the foreground, +# with a Bash tool timeout of at least 280000 ms. # # Mutants file (plain text; the text before the first `===` is ignored): -# === label-unique-in-the-file +# === label-unique-in-the-file (any text; `|` is allowed) # file: src/lib.rs (relative to workdir) # test: cargo test add_ (optional; else $TEST) # --- before -# a + b (exact text, may span lines; must occur ONCE) +# a + b (exact text, may span lines; must occur ONCE; +# LF text also matches a CRLF file, a file +# with mixed endings may read NOAPPLY) # --- after # a - b # -# Each mutant is applied in the container's throwaway clone and reverted from -# git before the next, so no tree you care about is touched. The output is one -# row per mutant: label | outcome | killing test or detail. Outcomes: +# Each mutant is applied in the container's throwaway clone. After every mutant +# the WHOLE tree is reset to HEAD and any untracked entry the run created is +# removed (entries that existed after the baselines, such as target/, stay); if +# the tree cannot be restored the job stops. A run that changed other files says +# so in its row. One row prints as each mutant finishes, so a job timeout keeps +# the finished rows. Row (a `YGG-SUMMARY/$YGG_NONCE: ` line, plain `YGG-SUMMARY: ` when no nonce is set): [i/N] label | outcome | killing test or detail. The +# outcome is stored as its own field and tallied from fields, never read back +# from the rendered row. Outcomes: # CAUGHT the test command failed AND a failing test name was parsed from it # SURVIVED the test command passed with the mutation applied -# UNVIABLE BUILD failed +# UNVIABLE BUILD failed, or a test file failed to load (vitest `FAIL +# [ ]` with no test name): nothing ran, not a kill # TIMEOUT a run hit MUT_TIMEOUT (not a kill) # NOAPPLY before text absent, ambiguous, or equal to after (no mutation made) # ERROR the test command failed but no failing test was parsed (a build # error without BUILD, a crash): not a kill -# Before any mutant, every distinct test command runs UNMUTATED. If one fails or -# runs zero tests the job aborts with `BASELINE RED`, because a red or empty -# baseline would make every mutant look killed or every one survive. +# Before any mutant, every distinct test command and BUILD run UNMUTATED. If one +# fails or a test command runs zero tests the job aborts with `BASELINE RED` or +# `BASELINE RAN ZERO TESTS`, because a red or empty baseline would make every +# mutant look killed or every one survive. # Exit 0 means every mutant was CAUGHT or UNVIABLE and nothing else happened. # # DELETION LINE: this runner replaces the per-agent runmut/mutrun/strykrun @@ -55,18 +70,21 @@ if [ "${1:-}" != --in-container ]; then cp "$here/testparse.sh" "$here/mutation-table.sh" "$stage/" tr -d '\r' <"$mfile" >"$stage/mutants.txt" bundle=$(tar -C "$stage" -cz testparse.sh mutation-table.sh mutants.txt | base64 | tr -d '\n') - cmd=$(printf 'mkdir -p /tmp/mt && echo %s | base64 -d | tar -xz -C /tmp/mt && WORKDIR=%q BUILD=%q TEST=%q MUT_TIMEOUT=%q bash /tmp/mt/mutation-table.sh --in-container' \ - "$bundle" "$workdir" "${BUILD:-}" "${TEST:-}" "${MUT_TIMEOUT:-600}") - "$here/stopgap/ygg-verify.sh" "$repo" "$rev" "$image" "$cmd" + cmd=$(printf 'mkdir -p /tmp/mt && echo %s | base64 -d | tar -xz -C /tmp/mt && WORKDIR=%q BUILD=%q TEST=%q MUT_TIMEOUT=%q JOB_TIMEOUT=%q bash /tmp/mt/mutation-table.sh --in-container' \ + "$bundle" "$workdir" "${BUILD:-}" "${TEST:-}" "${MUT_TIMEOUT:-600}" "${TIMEOUT:-3600}") + TIMEOUT=${TIMEOUT:-3600} "$here/stopgap/ygg-verify.sh" "$repo" "$rev" "$image" "$cmd" exit $? fi # ---- container side --------------------------------------------------------- set +e +SRC=${MT_SRC:-/src}; MT_DIR=${MT_DIR:-/tmp/mt} # overridable only so tools/test-tools.sh can run this half locally against a mock +# ygg-verify.sh mints a nonce per run and carries only lines tagged with it; unset (a local run), the plain tag is used +SUMTAG="YGG-SUMMARY${YGG_NONCE:+/$YGG_NONCE}" # shellcheck source=testparse.sh -. /tmp/mt/testparse.sh -cd "/src/${WORKDIR:-.}" || { echo "YGG-SUMMARY: BAD WORKDIR ${WORKDIR:-.}"; exit 2; } -M=/tmp/mt/m; rm -rf "$M"; mkdir -p "$M" +. "$MT_DIR/testparse.sh" +cd "$SRC/${WORKDIR:-.}" || { echo "$SUMTAG: BAD WORKDIR ${WORKDIR:-.}"; exit 2; } +M="$MT_DIR/m"; rm -rf "$M"; mkdir -p "$M" tmo=${MUT_TIMEOUT:-600} if ! perl -e ' @@ -91,32 +109,66 @@ if ! perl -e ' push @{$c{$sec}}, $_ if $sec; } flush(); die "no mutants in $mf\n" unless $n; - print "$n\n";' /tmp/mt/mutants.txt "$M" >"$M/count" 2>"$M/err"; then - echo "YGG-SUMMARY: BAD MUTANT FILE: $(cat "$M/err")"; exit 2 + print "$n\n";' "$MT_DIR/mutants.txt" "$M" >"$M/count" 2>"$M/err"; then + echo "$SUMTAG: BAD MUTANT FILE: $(cat "$M/err")"; exit 2 fi count=$(cat "$M/count") +echo "$SUMTAG: START $count mutants; per-run timeout ${tmo}s; whole-job timeout ${JOB_TIMEOUT:-unknown}s; one row per mutant follows as it finishes, so a timeout keeps the finished rows" # run : prints the exit status; 124 means timeout run() { timeout "$tmo" bash -o pipefail -c "$1" >"$2" 2>&1; echo $?; } +# The tree is the whole repository, not only the mutated file. A test or build +# that rewrites a tracked file would otherwise carry its pollution into the next +# mutant (a false CAUGHT, 2026-10-01). After every run, tracked state is reset +# to HEAD and any untracked entry that was not there after the baselines is +# removed. Entries present after the baselines (build output such as target/) +# are the baseline and stay, so builds stay incremental. +tree_state() { git -C "$SRC" status --porcelain 2>/dev/null; } +base_state='' +# restore: leaves the tree equal to HEAD plus the baseline entries; non-zero if it cannot. +restore() { + git -C "$SRC" reset -q --hard 2>/dev/null + tree_state | while IFS= read -r line; do + grep -qxF -- "$line" <<<"$base_state" || rm -rf -- "$SRC/${line:3}" + done + [ "$(tree_state)" = "$base_state" ] +} +# dirtied : what a run changed beyond the mutated file and the baseline +dirtied() { + local rel; rel=$(git -C "$SRC" ls-files --full-name -- "$1" 2>/dev/null | head -n 1) + tree_state | grep -vxF -- "$base_state" | grep -vF -- " $rel" | sed 's/^...//' | head -n 3 | paste -sd, - +} + declare -A based for n in $(seq 1 "$count"); do t=$(cat "$M/$n/test"); t=${t:-$TEST} - [ -n "$t" ] || { echo "YGG-SUMMARY: mutant $(cat "$M/$n/label") has no test: line and TEST is unset"; exit 2; } + [ -n "$t" ] || { echo "$SUMTAG: mutant $(cat "$M/$n/label") has no test: line and TEST is unset"; exit 2; } [ -z "${based[$t]:-}" ] || continue based[$t]=1 rc=$(run "$t" "$M/base.log") scan=$(tp_scan "$M/base.log") tests=$(printf '%s\n' "$scan" | awk '/^FRAMEWORK/{for(i=1;i<=NF;i++){split($i,a,"=");if(a[1]=="passed"||a[1]=="failed")s+=a[2]}}END{print s+0}') if [ "$rc" != 0 ]; then - echo "YGG-SUMMARY: BASELINE RED (exit $rc) for: $t"; tail -n 15 "$M/base.log" | sed 's/^/YGG-SUMMARY: /'; exit 2 + echo "$SUMTAG: BASELINE RED (exit $rc) for: $t"; tail -n 15 "$M/base.log" | sed "s|^|$SUMTAG: |"; exit 2 fi if [ "$tests" -eq 0 ]; then - echo "YGG-SUMMARY: BASELINE RAN ZERO TESTS (or printed counts no parser reads) for: $t"; exit 2 + echo "$SUMTAG: BASELINE RAN ZERO TESTS (or printed counts no parser reads) for: $t"; exit 2 fi done +# BUILD runs unmutated too. A mistyped or missing build command fails every +# mutant, and a table of all-UNVIABLE rows reads as a fully killed suite. +if [ -n "${BUILD:-}" ]; then + rc=$(run "$BUILD" "$M/base.log") + if [ "$rc" != 0 ]; then + echo "$SUMTAG: BASELINE RED: BUILD failed on the unmutated tree (exit $rc; 124 is a timeout): $BUILD" + tail -n 15 "$M/base.log" | sed "s|^|$SUMTAG: |"; exit 2 + fi +fi +# Baseline runs may dirty tracked files; settle that before the first mutant. +git -C "$SRC" reset -q --hard 2>/dev/null +base_state=$(tree_state) -: >"$M/rows" for n in $(seq 1 "$count"); do label=$(cat "$M/$n/label"); file=$(cat "$M/$n/file"); t=$(cat "$M/$n/test"); t=${t:-$TEST} out=''; detail='' @@ -126,6 +178,8 @@ for n in $(seq 1 "$count"); do open my $b, "<:raw", $bf; my $before = <$b> // ""; open my $a, "<:raw", $af; my $after = <$a> // ""; if (length($before) == 0) { print "empty before text\n"; exit 3 } if ($before eq $after) { print "after equals before\n"; exit 3 } + # a CRLF file matches multi-line before text written with LF + if ($c =~ /\r\n/ && $before !~ /\r/) { s/\n/\r\n/g for $before, $after } my ($cnt, $p) = (0, 0); while (($p = index($c, $before, $p)) >= 0) { $cnt++; $p += length($before) } if ($cnt == 0) { print "before text not found\n"; exit 3 } if ($cnt > 1) { print "before text occurs $cnt times; make it unique\n"; exit 3 } @@ -135,7 +189,7 @@ for n in $(seq 1 "$count"); do elif git diff --quiet -- "$file"; then out=NOAPPLY; detail="applied, but git sees no change in $file" else - out='' + out=''; detail='' if [ -n "${BUILD:-}" ]; then rc=$(run "$BUILD" "$M/run.log") if [ "$rc" = 124 ]; then out=TIMEOUT; detail="build exceeded ${tmo}s" @@ -147,21 +201,29 @@ for n in $(seq 1 "$count"); do elif [ "$rc" = 124 ]; then out=TIMEOUT; detail="test exceeded ${tmo}s: $t" else names=$(tp_scan "$M/run.log" | grep '^FAILED ' | awk '!s[$0]++' | sed 's/^FAILED //') - if [ -n "$names" ]; then out=CAUGHT; detail=$(printf '%s\n' "$names" | head -n 1); k=$(printf '%s\n' "$names" | wc -l); [ "$k" -le 1 ] || detail="$detail (+$((k - 1)) more)" + # A file-level failure (vitest: "FAIL [ ]", no test name) is a + # module that did not load: a compile error, never a kill. + named=$(printf '%s\n' "$names" | grep -v -E '^[^ >]+ \[ .* \]$') + if [ -n "$named" ]; then + out=CAUGHT; detail=$(printf '%s\n' "$named" | head -n 1) + k=$(printf '%s\n' "$named" | wc -l); [ "$k" -le 1 ] || detail="$detail (+$((k - 1)) more)" + elif [ -n "$names" ]; then out=UNVIABLE; detail="a test file failed to load, no test ran: $(printf '%s\n' "$names" | head -n 1)" else out=ERROR; detail="exit $rc, no failing test parsed: $(tail -n 1 "$M/run.log" | cut -c1-120)"; fi fi fi + extra=$(dirtied "$file"); [ -z "$extra" ] || detail="$detail [run also changed: $extra; restored]" + fi + # The outcome is its own field. It is never recovered from rendered text, so a + # label or a test name containing " | CAUGHT | " cannot forge a kill. + printf '%s' "$out" >"$M/$n/outcome" + detail=${detail//$'\n'/ } + printf '%s: [%s/%s] %s | %s | %s\n' "$SUMTAG" "$n" "$count" "${label//|/\\|}" "$out" "$detail" + if ! restore; then + echo "$SUMTAG: RESTORE FAILED after $label: the tree differs from the baseline; stopping. Status: $(tree_state | head -n 5 | paste -sd';' -)"; exit 2 fi - printf '%s | %s | %s\n' "$label" "$out" "$detail" >>"$M/rows" - git checkout -q -- "$file" - if ! git diff --quiet -- "$file"; then echo "YGG-SUMMARY: RESTORE FAILED for $file after $label; stopping"; cat "$M/rows" | sed 's/^/YGG-SUMMARY: /'; exit 2; fi done -tally() { grep -c " | $1 | " "$M/rows" || true; } -bad=$(grep -vc -E ' \| (CAUGHT|UNVIABLE) \| ' "$M/rows" || true) -{ - echo "MUTATION $count mutants: caught=$(tally CAUGHT) survived=$(tally SURVIVED) unviable=$(tally UNVIABLE) timeout=$(tally TIMEOUT) noapply=$(tally NOAPPLY) error=$(tally ERROR) (baseline green, tests ran)" - grep -v ' | CAUGHT | ' "$M/rows" - grep ' | CAUGHT | ' "$M/rows" -} | sed 's/^/YGG-SUMMARY: /' -[ "$bad" -eq 0 ] +tally() { local c=0 f; for f in "$M"/*/outcome; do [ "$(cat "$f")" = "$1" ] && c=$((c + 1)); done; echo "$c"; } +caught=$(tally CAUGHT); unviable=$(tally UNVIABLE) +echo "$SUMTAG: MUTATION $count mutants: caught=$caught survived=$(tally SURVIVED) unviable=$unviable timeout=$(tally TIMEOUT) noapply=$(tally NOAPPLY) error=$(tally ERROR) (baseline green, tests ran)" +[ $((caught + unviable)) -eq "$count" ] diff --git a/tools/test-tools.sh b/tools/test-tools.sh index d6621b7..df73f9a 100644 --- a/tools/test-tools.sh +++ b/tools/test-tools.sh @@ -35,4 +35,53 @@ check pytest-red.log 1 "FAILED tests/test_add.py::test_carries" check exit0-but-failed.log 0 "MISMATCH: exit 0 but the log shows" check no-verdict.log 97 "NO VERDICT" check no-verdict.log 97 "RED" "GREEN" + +# ---- tools/mutation-table.sh, container half, against the mock in +# fixtures/mutation-mock (shell scripts that print cargo/vitest-shaped output). +# One case per finding of the 2026-10-01 Soul pass; each fails if it regresses. +mt() { # mt [BUILD]: sets mt_out, mt_rc, mt_repo + local root; root=$(mktemp -d); mt_repo=$root/repo; mkdir -p "$mt_repo" "$root/mt" + cp -r "$here/fixtures/mutation-mock/." "$mt_repo/" + git -C "$mt_repo" init -q && git -C "$mt_repo" add -A && git -C "$mt_repo" -c user.name=t -c user.email=t@t commit -qm mock + cp "$here/testparse.sh" "$root/mt/"; cp "$here/fixtures/mutation-mock/$1" "$root/mt/mutants.txt" + mt_out=$(MT_SRC=$mt_repo MT_DIR=$root/mt TEST=$2 BUILD=${3:-} MUT_TIMEOUT=${MT_TO:-15} timeout "${MT_WALL:-120}" bash "$here/mutation-table.sh" --in-container 2>&1) + mt_rc=$? +} +mtcheck() { # mtcheck ... + local name=$1 s; shift + for s in "$@"; do + case $s in + '!'*) if printf '%s\n' "$mt_out" | grep -qF -- "${s#!}"; then echo "FAIL mutation $name: must not contain: ${s#!}"; fail=1; return; fi ;; + *) printf '%s\n' "$mt_out" | grep -qF -- "$s" || { echo "FAIL mutation $name: missing: $s"; printf '%s\n' "$mt_out" | tail -n 12; fail=1; return; } ;; + esac + done + echo "ok mutation $name" +} +# A: a mistyped BUILD must abort, not read as a fully killed suite +mt mutants-a.txt 'bash test.sh' 'bash bulid.sh' +[ "$mt_rc" = 2 ] || { echo "FAIL mutation A: exit $mt_rc, want 2"; fail=1; } +mtcheck A-build-baseline "BASELINE RED: BUILD failed" "!UNVIABLE" +# B: a label that spells a kill must not forge one +mt mutants-b.txt 'bash test.sh' +[ "$mt_rc" = 1 ] || { echo "FAIL mutation B: exit $mt_rc, want 1"; fail=1; } +mtcheck B-label-forge "caught=0 survived=1" 'forge \| CAUGHT \| label | SURVIVED |' +# the nonce form ygg-verify.sh carries, and the plain form when no nonce is set +YGG_NONCE=n0nce1 mt mutants-b.txt 'bash test.sh' +mtcheck B-nonce "YGG-SUMMARY/n0nce1: [1/1] " "!YGG-SUMMARY: " +# C: a test that rewrites a tracked file or leaks an untracked one must not decide the next mutant +mt mutants-c.txt 'bash test.sh' +mtcheck C-pollution "stamp-bad | CAUGHT | tests::b_write" "also changed: data.txt,leak.tmp" "untested-equiv | SURVIVED" "!untested-equiv | CAUGHT" +[ -z "$(git -C "$mt_repo" status --porcelain)" ] || { echo "FAIL mutation C: tree not restored"; fail=1; } +# D: a test file that fails to load is not a kill; a named failing test is +mt mutants-d.txt 'bash vtest.sh' +mtcheck D-import-failure "syntax-break | UNVIABLE" "wrong-result | CAUGHT | add.js > adds" "caught=1 survived=0 unviable=1" +# E: rows print as mutants finish, so a job timeout keeps them +MT_WALL=8 mt mutants-e.txt 'bash test.sh' +mtcheck E-live-rows "[1/2] quick | CAUGHT" "!MUTATION" +# F: LF before-text matches a CRLF file +mt mutants-f.txt 'bash test.sh' +mtcheck F-crlf "crlf-multiline | CAUGHT | tests::d_win" +# a real BUILD failure on a mutant is still UNVIABLE +mt mutants-g.txt 'bash test.sh' 'bash build.sh' +mtcheck build-unviable "build-break | UNVIABLE | build exit 1" "unviable=1" exit $fail -- 2.47.3 From b02f577b395db9951ac04525f963a1b9339de666 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 1 Oct 2026 16:40:38 +0200 Subject: [PATCH 06/16] Fixtures: Soul's real-toolchain mutation probes (rust data.txt pollution, vitest import break) Co-Authored-By: Claude Sonnet 5.5 --- tools/fixtures/mutation-probe/.gitattributes | 1 + tools/fixtures/mutation-probe/README.txt | 8 +++++++ .../fixtures/mutation-probe/js/mutants-d.txt | 12 ++++++++++ tools/fixtures/mutation-probe/js/package.json | 1 + tools/fixtures/mutation-probe/js/src/add.js | 3 +++ .../mutation-probe/js/src/add.test.js | 7 ++++++ tools/fixtures/mutation-probe/rust/Cargo.toml | 6 +++++ tools/fixtures/mutation-probe/rust/data.txt | 1 + .../mutation-probe/rust/mutants-a.txt | 6 +++++ .../mutation-probe/rust/mutants-b.txt | 6 +++++ .../mutation-probe/rust/mutants-c.txt | 12 ++++++++++ tools/fixtures/mutation-probe/rust/src/lib.rs | 22 +++++++++++++++++++ 12 files changed, 85 insertions(+) create mode 100644 tools/fixtures/mutation-probe/.gitattributes create mode 100644 tools/fixtures/mutation-probe/README.txt create mode 100644 tools/fixtures/mutation-probe/js/mutants-d.txt create mode 100644 tools/fixtures/mutation-probe/js/package.json create mode 100644 tools/fixtures/mutation-probe/js/src/add.js create mode 100644 tools/fixtures/mutation-probe/js/src/add.test.js create mode 100644 tools/fixtures/mutation-probe/rust/Cargo.toml create mode 100644 tools/fixtures/mutation-probe/rust/data.txt create mode 100644 tools/fixtures/mutation-probe/rust/mutants-a.txt create mode 100644 tools/fixtures/mutation-probe/rust/mutants-b.txt create mode 100644 tools/fixtures/mutation-probe/rust/mutants-c.txt create mode 100644 tools/fixtures/mutation-probe/rust/src/lib.rs diff --git a/tools/fixtures/mutation-probe/.gitattributes b/tools/fixtures/mutation-probe/.gitattributes new file mode 100644 index 0000000..fa1385d --- /dev/null +++ b/tools/fixtures/mutation-probe/.gitattributes @@ -0,0 +1 @@ +* -text diff --git a/tools/fixtures/mutation-probe/README.txt b/tools/fixtures/mutation-probe/README.txt new file mode 100644 index 0000000..a76b4ae --- /dev/null +++ b/tools/fixtures/mutation-probe/README.txt @@ -0,0 +1,8 @@ +Real-toolchain probes for tools/mutation-table.sh (Soul's 2026-10-01 scratch repo, committed). +Run each on Yggdrasil through the stopgap; expected rows: + rust/mutants-a.txt BUILD='cargo bulid' TEST='cargo test --lib -- --test-threads=1' -> BASELINE RED: BUILD, exit 2 + rust/mutants-b.txt (same TEST, no BUILD) -> forge row SURVIVED, caught=0, exit 1 + rust/mutants-c.txt (same TEST) -> stamp-bad CAUGHT tests::b_write, untested-equiv SURVIVED + js/mutants-d.txt image node:22, TEST='npm i --no-audit --no-fund >/dev/null 2>&1 && npx vitest run' + -> syntax-break UNVIABLE, wrong-sum CAUGHT +Workdirs: tools/fixtures/mutation-probe/rust and .../js. diff --git a/tools/fixtures/mutation-probe/js/mutants-d.txt b/tools/fixtures/mutation-probe/js/mutants-d.txt new file mode 100644 index 0000000..5972666 --- /dev/null +++ b/tools/fixtures/mutation-probe/js/mutants-d.txt @@ -0,0 +1,12 @@ +=== syntax-break +file: src/add.js +--- before + return a + b; +--- after + return a +; +=== wrong-sum +file: src/add.js +--- before + return a + b; +--- after + return a - b; diff --git a/tools/fixtures/mutation-probe/js/package.json b/tools/fixtures/mutation-probe/js/package.json new file mode 100644 index 0000000..002e569 --- /dev/null +++ b/tools/fixtures/mutation-probe/js/package.json @@ -0,0 +1 @@ +{ "name": "p", "private": true, "type": "module", "devDependencies": { "vitest": "3.2.4" } } diff --git a/tools/fixtures/mutation-probe/js/src/add.js b/tools/fixtures/mutation-probe/js/src/add.js new file mode 100644 index 0000000..7d65831 --- /dev/null +++ b/tools/fixtures/mutation-probe/js/src/add.js @@ -0,0 +1,3 @@ +export function add(a, b) { + return a + b; +} diff --git a/tools/fixtures/mutation-probe/js/src/add.test.js b/tools/fixtures/mutation-probe/js/src/add.test.js new file mode 100644 index 0000000..0954490 --- /dev/null +++ b/tools/fixtures/mutation-probe/js/src/add.test.js @@ -0,0 +1,7 @@ +import { test, expect, describe } from 'vitest'; +import { add } from './add.js'; +describe('add', () => { + test('adds', () => { expect(add(2, 3)).toBe(5); }); + test('a | b pipes', () => { expect(add(1, 1)).toBe(process.env.RED ? 3 : 2); }); + test.skip('skipped', () => {}); +}); diff --git a/tools/fixtures/mutation-probe/rust/Cargo.toml b/tools/fixtures/mutation-probe/rust/Cargo.toml new file mode 100644 index 0000000..2cb5f6c --- /dev/null +++ b/tools/fixtures/mutation-probe/rust/Cargo.toml @@ -0,0 +1,6 @@ +[package] +name = "probe" +version = "0.0.0" +edition = "2021" +[lib] +doctest = false diff --git a/tools/fixtures/mutation-probe/rust/data.txt b/tools/fixtures/mutation-probe/rust/data.txt new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/tools/fixtures/mutation-probe/rust/data.txt @@ -0,0 +1 @@ +ok diff --git a/tools/fixtures/mutation-probe/rust/mutants-a.txt b/tools/fixtures/mutation-probe/rust/mutants-a.txt new file mode 100644 index 0000000..a1cf01c --- /dev/null +++ b/tools/fixtures/mutation-probe/rust/mutants-a.txt @@ -0,0 +1,6 @@ +=== add-minus +file: src/lib.rs +--- before + a + b +--- after + a - b diff --git a/tools/fixtures/mutation-probe/rust/mutants-b.txt b/tools/fixtures/mutation-probe/rust/mutants-b.txt new file mode 100644 index 0000000..c9880a1 --- /dev/null +++ b/tools/fixtures/mutation-probe/rust/mutants-b.txt @@ -0,0 +1,6 @@ +=== forge | CAUGHT | label +file: src/lib.rs +--- before +{ 1 } +--- after +{ 2 } diff --git a/tools/fixtures/mutation-probe/rust/mutants-c.txt b/tools/fixtures/mutation-probe/rust/mutants-c.txt new file mode 100644 index 0000000..7378567 --- /dev/null +++ b/tools/fixtures/mutation-probe/rust/mutants-c.txt @@ -0,0 +1,12 @@ +=== stamp-bad +file: src/lib.rs +--- before +{ "ok" } +--- after +{ "bad" } +=== untested-equiv +file: src/lib.rs +--- before +{ 1 } +--- after +{ 2 } diff --git a/tools/fixtures/mutation-probe/rust/src/lib.rs b/tools/fixtures/mutation-probe/rust/src/lib.rs new file mode 100644 index 0000000..1bb2278 --- /dev/null +++ b/tools/fixtures/mutation-probe/rust/src/lib.rs @@ -0,0 +1,22 @@ +pub fn stamp() -> &'static str { "ok" } +pub fn untested() -> i32 { 1 } +pub fn add(a: i32, b: i32) -> i32 { + a + b +} + +#[cfg(test)] +mod tests { + use super::*; + #[test] + fn a_read() { + let s = std::fs::read_to_string("data.txt").unwrap(); + assert_eq!(s.trim(), "ok"); + } + #[test] + fn b_write() { + std::fs::write("data.txt", stamp()).unwrap(); + assert_eq!(stamp(), "ok"); + } + #[test] + fn c_add() { assert_eq!(add(2, 3), 5); } +} -- 2.47.3 From 4bcd4348283e13fb2bd2602e8f51fc4a7013d12f Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 1 Oct 2026 16:43:26 +0200 Subject: [PATCH 07/16] mutation-table: name the mutated file from the workdir; briefs: wait calls take a 280000 ms Bash timeout The pollution note listed nothing when the workdir was below the repo root (ls-files ran from the root); the mock now runs from a subdirectory so that regresses visibly. Every brief/INDEX line showing ygg-verify wait now says to pass a Bash timeout of at least 280000 ms. Co-Authored-By: Claude Sonnet 5.5 --- references/briefs.md | 4 ++-- tools/INDEX.md | 4 ++-- tools/mutation-table.sh | 5 +++-- tools/test-tools.sh | 8 ++++---- 4 files changed, 11 insertions(+), 10 deletions(-) diff --git a/references/briefs.md b/references/briefs.md index 4447c27..e18b3d2 100644 --- a/references/briefs.md +++ b/references/briefs.md @@ -292,7 +292,7 @@ whole context. Start the job as one background shell (run_in_background) with `SUMMARY=/.log` in front of `tools/stopgap/ygg-verify.sh`, a fresh log path per job. Then wait in the foreground, one call at a time: `tools/stopgap/ygg-verify.sh wait /.log` blocks for at most -270 s, so your cache stays warm, and prints the bounded summary when the job is +270 s (run it with a Bash tool `timeout` of at least 280000 ms: the default 120 s backgrounds the call and costs a wake), so your cache stays warm, and prints the bounded summary when the job is done. If it prints `RUNNING s` (exit 99), call it again. Exit 98 means the launcher is gone with no verdict: rerun the job, do not guess its outcome. A call longer than 5 minutes rewrites your whole context. Never run a second watcher @@ -370,7 +370,7 @@ Admit one verdict and its findings, and nothing else. - Long jobs: wait for them and finish, and never end your turn before your report. Run the job as one background shell with `SUMMARY=/.log` in front of `tools/stopgap/ygg-verify.sh` (a fresh log path per job), and wait - in the foreground with `tools/stopgap/ygg-verify.sh wait `: it blocks + in the foreground with `tools/stopgap/ygg-verify.sh wait `: pass a Bash tool `timeout` of at least 280000 ms (the default 120 s backgrounds the call and costs a wake); it blocks at most 270 s and prints the bounded summary when the job is done, or `RUNNING s` (exit 99) and you call it again. Exit 98 means no verdict: rerun, do not guess. No second watcher, no timer under 60 s in a loop of your own, and no diff --git a/tools/INDEX.md b/tools/INDEX.md index 25efd0c..deb8c50 100644 --- a/tools/INDEX.md +++ b/tools/INDEX.md @@ -8,7 +8,7 @@ gets a Soul pass before merge. | script | purpose | consumer | named by | |---|---|---|---| -| `tools/stopgap/ygg-verify.sh` | Pushes an exact revision to a mirror on Yggdrasil and runs one command in a capped, niced container. `SUMMARY=` keeps the full log and prints a bounded summary (verdict, counts, ZERO TESTS RAN, failing tests); `wait ` blocks at most 270 s and returns the summary once. Dies with the stopgap when Idunn's verify transaction lands. | Hands, Soul, Imagination probes | `references/briefs.md` (Hands long jobs and verification, Soul long jobs), `SKILL.md` (Yggdrasil) | -| `tools/mutation-table.sh` | One verify job that runs a list of hand mutants (file, before, after, test) against a revision and returns a kill table: CAUGHT with the killing test, SURVIVED, UNVIABLE, TIMEOUT, NOAPPLY, ERROR. Refuses a red or zero-test baseline. Replaces per-agent runmut/mutrun/strykrun scripts. | Soul, Hands (scoped run) | `references/briefs.md` (Hands verification, Soul reruns) | +| `tools/stopgap/ygg-verify.sh` | Pushes an exact revision to a mirror on Yggdrasil and runs one command in a capped, niced container. `SUMMARY=` keeps the full log and prints a bounded summary (verdict, counts, ZERO TESTS RAN, failing tests); `wait ` blocks at most 270 s and returns the summary once; call it with a Bash tool `timeout` of at least 280000 ms. Dies with the stopgap when Idunn's verify transaction lands. | Hands, Soul, Imagination probes | `references/briefs.md` (Hands long jobs and verification, Soul long jobs), `SKILL.md` (Yggdrasil) | +| `tools/mutation-table.sh` | One verify job that runs a list of hand mutants (file, before, after, test) against a revision and returns a kill table: CAUGHT with the killing test, SURVIVED, UNVIABLE, TIMEOUT, NOAPPLY, ERROR. Refuses a red or zero-test baseline and a red BUILD; rows print live; the whole tree is restored after each mutant. Replaces per-agent runmut/mutrun/strykrun scripts. | Soul, Hands (scoped run) | `references/briefs.md` (Hands verification, Soul reruns) | | `tools/testparse.sh` | Sourced library: failing test names and pass/fail/skip counts from cargo, dotnet, vitest and pytest logs. Shared by the summary and the kill table. | the two scripts above | not named by a brief; it lives and dies with them | | `tools/test-tools.sh` | Local fixture check of the parsers and the summary over `tools/fixtures/logs` (real cargo and mutation logs, plus hand-written dotnet, vitest, pytest, exit-0-but-failed and no-verdict logs). Run it before merging a change to the scripts above. | whoever changes those scripts | `SKILL.md` (Shared tools) | diff --git a/tools/mutation-table.sh b/tools/mutation-table.sh index 037998e..a8580dd 100644 --- a/tools/mutation-table.sh +++ b/tools/mutation-table.sh @@ -136,8 +136,9 @@ restore() { } # dirtied : what a run changed beyond the mutated file and the baseline dirtied() { - local rel; rel=$(git -C "$SRC" ls-files --full-name -- "$1" 2>/dev/null | head -n 1) - tree_state | grep -vxF -- "$base_state" | grep -vF -- " $rel" | sed 's/^...//' | head -n 3 | paste -sd, - + local rel; rel=$(git ls-files --full-name -- "$1" 2>/dev/null | head -n 1) + [ -n "$rel" ] || return 0 # cannot name the mutated file: report nothing rather than everything + tree_state | grep -vxF -- "$base_state" | grep -vF -- " $rel" | sed "s/^...//" | head -n 3 | paste -sd, - } declare -A based diff --git a/tools/test-tools.sh b/tools/test-tools.sh index df73f9a..5f7d17f 100644 --- a/tools/test-tools.sh +++ b/tools/test-tools.sh @@ -37,14 +37,14 @@ check no-verdict.log 97 "NO VERDICT" check no-verdict.log 97 "RED" "GREEN" # ---- tools/mutation-table.sh, container half, against the mock in -# fixtures/mutation-mock (shell scripts that print cargo/vitest-shaped output). +# fixtures/mutation-mock, run from the subdirectory pkg/ (a workdir below the repo root) (shell scripts that print cargo/vitest-shaped output). # One case per finding of the 2026-10-01 Soul pass; each fails if it regresses. mt() { # mt [BUILD]: sets mt_out, mt_rc, mt_repo local root; root=$(mktemp -d); mt_repo=$root/repo; mkdir -p "$mt_repo" "$root/mt" - cp -r "$here/fixtures/mutation-mock/." "$mt_repo/" + mkdir -p "$mt_repo/pkg"; cp -r "$here/fixtures/mutation-mock/." "$mt_repo/pkg/" git -C "$mt_repo" init -q && git -C "$mt_repo" add -A && git -C "$mt_repo" -c user.name=t -c user.email=t@t commit -qm mock cp "$here/testparse.sh" "$root/mt/"; cp "$here/fixtures/mutation-mock/$1" "$root/mt/mutants.txt" - mt_out=$(MT_SRC=$mt_repo MT_DIR=$root/mt TEST=$2 BUILD=${3:-} MUT_TIMEOUT=${MT_TO:-15} timeout "${MT_WALL:-120}" bash "$here/mutation-table.sh" --in-container 2>&1) + mt_out=$(WORKDIR=pkg MT_SRC=$mt_repo MT_DIR=$root/mt TEST=$2 BUILD=${3:-} MUT_TIMEOUT=${MT_TO:-15} timeout "${MT_WALL:-120}" bash "$here/mutation-table.sh" --in-container 2>&1) mt_rc=$? } mtcheck() { # mtcheck ... @@ -70,7 +70,7 @@ YGG_NONCE=n0nce1 mt mutants-b.txt 'bash test.sh' mtcheck B-nonce "YGG-SUMMARY/n0nce1: [1/1] " "!YGG-SUMMARY: " # C: a test that rewrites a tracked file or leaks an untracked one must not decide the next mutant mt mutants-c.txt 'bash test.sh' -mtcheck C-pollution "stamp-bad | CAUGHT | tests::b_write" "also changed: data.txt,leak.tmp" "untested-equiv | SURVIVED" "!untested-equiv | CAUGHT" +mtcheck C-pollution "stamp-bad | CAUGHT | tests::b_write" "also changed: pkg/data.txt,pkg/leak.tmp" "untested-equiv | SURVIVED" "!untested-equiv | CAUGHT" [ -z "$(git -C "$mt_repo" status --porcelain)" ] || { echo "FAIL mutation C: tree not restored"; fail=1; } # D: a test file that fails to load is not a kill; a named failing test is mt mutants-d.txt 'bash vtest.sh' -- 2.47.3 From 62bc4d5c1d5868e81642fc0069647368bd299544 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 1 Oct 2026 17:36:49 +0200 Subject: [PATCH 08/16] mutation-table: Docker owns the per-mutant snapshot; classify jest and vitest file-level load failures MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The git-status restore could not see ignored files, so a test keeping state under target/ polluted the next mutant (an uncovered mutant read CAUGHT). The baselines now run once in one container whose filesystem holds the revision; that container is `docker commit`ted to a throwaway image and each mutant runs in a fresh --rm container from it. restore(), dirtied() and tree_state() are deleted. ygg-verify.sh gains RUN_ON_HOST=1 (the driver owns container topology; the watchdog kills by job label). testparse.sh: LOADFAIL for file-level FAIL lines (no " > "), jest "●" names and jest Tests: counts. Mock docker, jest and vitest-variant fixtures, soul2 reproducer. Co-Authored-By: Claude Sonnet 5.5 --- tools/fixtures/logs/jest-load-failure.log | 18 + tools/fixtures/logs/jest-red.log | 25 ++ .../logs/vitest-project-load-failure.log | 7 + tools/fixtures/logs/vitest-project-red.log | 6 + .../logs/vitest-spaces-load-failure.log | 6 + tools/fixtures/mutation-mock/.gitignore | 1 + tools/fixtures/mutation-mock/fakebin/docker | 44 +++ tools/fixtures/mutation-mock/fakebin/sudo | 2 + tools/fixtures/mutation-mock/jtest.sh | 10 + tools/fixtures/mutation-mock/mutants-h.txt | 12 + tools/fixtures/mutation-mock/test.sh | 6 +- .../fixtures/mutation-probe/soul2/.gitignore | 1 + .../fixtures/mutation-probe/soul2/Cargo.toml | 6 + .../fixtures/mutation-probe/soul2/mutants.txt | 18 + .../fixtures/mutation-probe/soul2/src/lib.rs | 32 ++ tools/mutation-table.sh | 367 ++++++++++-------- tools/stopgap/ygg-verify.sh | 56 ++- tools/test-tools.sh | 60 ++- tools/testparse.sh | 26 +- 19 files changed, 520 insertions(+), 183 deletions(-) create mode 100644 tools/fixtures/logs/jest-load-failure.log create mode 100644 tools/fixtures/logs/jest-red.log create mode 100644 tools/fixtures/logs/vitest-project-load-failure.log create mode 100644 tools/fixtures/logs/vitest-project-red.log create mode 100644 tools/fixtures/logs/vitest-spaces-load-failure.log create mode 100644 tools/fixtures/mutation-mock/.gitignore create mode 100644 tools/fixtures/mutation-mock/fakebin/docker create mode 100644 tools/fixtures/mutation-mock/fakebin/sudo create mode 100644 tools/fixtures/mutation-mock/jtest.sh create mode 100644 tools/fixtures/mutation-mock/mutants-h.txt create mode 100644 tools/fixtures/mutation-probe/soul2/.gitignore create mode 100644 tools/fixtures/mutation-probe/soul2/Cargo.toml create mode 100644 tools/fixtures/mutation-probe/soul2/mutants.txt create mode 100644 tools/fixtures/mutation-probe/soul2/src/lib.rs diff --git a/tools/fixtures/logs/jest-load-failure.log b/tools/fixtures/logs/jest-load-failure.log new file mode 100644 index 0000000..c04a4d2 --- /dev/null +++ b/tools/fixtures/logs/jest-load-failure.log @@ -0,0 +1,18 @@ +SYNTHETIC: hand-written in the format jest prints; not captured from a run. + FAIL src/add.test.js + ● Test suite failed to run + + Jest encountered an unexpected token + + SyntaxError: /work/src/add.js: Unexpected token (1:11) + + > 1 | add = BROKEN BROKEN + | ^ + + at Parser._raise (node_modules/@babel/parser/lib/index.js:571:17) + +Test Suites: 1 failed, 1 total +Tests: 0 total +Snapshots: 0 total +Time: 0.298 s +Ran all test suites. diff --git a/tools/fixtures/logs/jest-red.log b/tools/fixtures/logs/jest-red.log new file mode 100644 index 0000000..10a5b07 --- /dev/null +++ b/tools/fixtures/logs/jest-red.log @@ -0,0 +1,25 @@ +SYNTHETIC: hand-written in the format jest prints; not captured from a run. + FAIL src/add.test.js + add + ✓ adds (2 ms) + ✕ carries (3 ms) + ○ skipped subtracts + + ● add › carries + + expect(received).toBe(expected) // Object.is equality + + Expected: 10 + Received: 9 + + 4 | test('carries', () => { + > 5 | expect(add(5, 5)).toBe(10); + | ^ + + at Object. (src/add.test.js:5:21) + +Test Suites: 1 failed, 1 total +Tests: 1 failed, 1 skipped, 1 passed, 3 total +Snapshots: 0 total +Time: 0.412 s +Ran all test suites. diff --git a/tools/fixtures/logs/vitest-project-load-failure.log b/tools/fixtures/logs/vitest-project-load-failure.log new file mode 100644 index 0000000..97d9c00 --- /dev/null +++ b/tools/fixtures/logs/vitest-project-load-failure.log @@ -0,0 +1,7 @@ +SYNTHETIC: hand-written in the format vitest prints; not captured from a run. + FAIL |unit| src/add.test.ts [ src/add.test.ts ] +Error: Transform failed with 1 error: +/work/src/add.ts:1:11: ERROR: Unexpected "BROKEN" + + Test Files 1 failed (1) + Tests no tests diff --git a/tools/fixtures/logs/vitest-project-red.log b/tools/fixtures/logs/vitest-project-red.log new file mode 100644 index 0000000..060a129 --- /dev/null +++ b/tools/fixtures/logs/vitest-project-red.log @@ -0,0 +1,6 @@ +SYNTHETIC: hand-written in the format vitest prints; not captured from a run. + FAIL |unit| src/add.test.ts > add > carries +AssertionError: expected 9 to be 10 + + Test Files 1 failed (1) + Tests 1 failed | 1 passed (2) diff --git a/tools/fixtures/logs/vitest-spaces-load-failure.log b/tools/fixtures/logs/vitest-spaces-load-failure.log new file mode 100644 index 0000000..759c1f0 --- /dev/null +++ b/tools/fixtures/logs/vitest-spaces-load-failure.log @@ -0,0 +1,6 @@ +SYNTHETIC: hand-written in the format vitest prints; not captured from a run. + FAIL src/my tests/add.test.ts [ src/my tests/add.test.ts ] +Error: Transform failed with 1 error: + + Test Files 1 failed | 1 passed (2) + Tests 1 passed (1) diff --git a/tools/fixtures/mutation-mock/.gitignore b/tools/fixtures/mutation-mock/.gitignore new file mode 100644 index 0000000..6575205 --- /dev/null +++ b/tools/fixtures/mutation-mock/.gitignore @@ -0,0 +1 @@ +cache.txt diff --git a/tools/fixtures/mutation-mock/fakebin/docker b/tools/fixtures/mutation-mock/fakebin/docker new file mode 100644 index 0000000..fe2a356 --- /dev/null +++ b/tools/fixtures/mutation-mock/fakebin/docker @@ -0,0 +1,44 @@ +#!/usr/bin/env bash +# Mock docker for tools/test-tools.sh: a container is a directory, an image is a +# copy of one, so `run` from a committed image really is copy-on-write. Implements +# only what tools/mutation-table.sh --host calls: run, commit, cp, rm, rmi. The +# in-container paths /mt and /seed map to the mounted host directories, and the +# script's own /src and /tmp/mtm live inside the container directory. +FD=${FAKE_DOCKER:?}; mkdir -p "$FD/c" "$FD/i" +imgdir() { echo "$FD/i/${1//[:\/]/_}"; } +cmd=$1; shift +case $cmd in + run) + rm_after=0; name=''; mt=''; seed=/nonexistent; labels=0; envs=() + while [ $# -gt 0 ]; do + case $1 in + --rm) rm_after=1 ;; + --name) name=$2; shift ;; + --label) labels=$((labels + 1)); shift ;; + -v) case $2 in *:/mt:ro) mt=${2%:/mt:ro} ;; *:/seed:ro) seed=${2%:/seed:ro} ;; esac; shift ;; + -e) case $2 in BASH_ENV=*) ;; *) envs+=("$2") ;; esac; shift ;; + -w) shift ;; + --*=*) ;; + -*) echo "fake docker: unknown flag $1" >&2; exit 125 ;; + *) break ;; + esac + shift + done + image=$1; shift + [ "$labels" -ge 1 ] || { echo "fake docker: container without the job label" >&2; exit 125; } + c=$FD/c/$name; rm -rf "$c" + if [ -d "$(imgdir "$image")" ]; then cp -a "$(imgdir "$image")" "$c"; else mkdir -p "$c"; fi + echo "run $name $image" >>"$FD/log" + script=${*: -1}; script=${script//\/mt\//$mt/} + ( for e in "${envs[@]}"; do export "$e"; done + export MT_SRC=$c/src MT_M=$c/tmp/mtm MT_DIR=$mt MT_SEED=$seed + bash -o pipefail -c "$script" ) + rc=$? + [ "$rm_after" = 0 ] || rm -rf "$c" + exit $rc ;; + commit) [ "$1" != -q ] || shift; rm -rf "$(imgdir "$2")"; cp -a "$FD/c/$1" "$(imgdir "$2")"; echo "commit $1 $2" >>"$FD/log" ;; + cp) f=$FD/c/${1%%:*}${1#*:}; tar -C "$(dirname "$f")" -c "$(basename "$f")" ;; + rm) shift; for n in "$@"; do case $n in -*) ;; *) rm -rf "$FD/c/$n" ;; esac; done ;; + rmi) shift; for n in "$@"; do case $n in -*) ;; *) rm -rf "$(imgdir "$n")" ;; esac; done ;; + *) echo "fake docker: unsupported $cmd" >&2; exit 125 ;; +esac diff --git a/tools/fixtures/mutation-mock/fakebin/sudo b/tools/fixtures/mutation-mock/fakebin/sudo new file mode 100644 index 0000000..c219c21 --- /dev/null +++ b/tools/fixtures/mutation-mock/fakebin/sudo @@ -0,0 +1,2 @@ +#!/usr/bin/env bash +exec "$@" diff --git a/tools/fixtures/mutation-mock/jtest.sh b/tools/fixtures/mutation-mock/jtest.sh new file mode 100644 index 0000000..a1d7662 --- /dev/null +++ b/tools/fixtures/mutation-mock/jtest.sh @@ -0,0 +1,10 @@ +# Mock jest run (ignore for the log shapes: see tools/fixtures/logs/jest-*.log). BROKEN in +# add.js is a module that does not load (file header, "Test suite failed to run", no test +# named); WRONG is a real assertion failure with its "●" block. +if grep -q BROKEN add.js; then + echo " FAIL add.js"; echo " ● Test suite failed to run"; echo; echo "Test Suites: 1 failed, 1 total"; echo "Tests: 0 total"; exit 1 +fi +if grep -q WRONG add.js; then + echo " FAIL add.js"; echo " ✕ adds (2 ms)"; echo; echo " ● add › adds"; echo; echo "Tests: 1 failed, 1 passed, 2 total"; exit 1 +fi +echo "Tests: 2 passed, 2 total" diff --git a/tools/fixtures/mutation-mock/mutants-h.txt b/tools/fixtures/mutation-mock/mutants-h.txt new file mode 100644 index 0000000..de263ef --- /dev/null +++ b/tools/fixtures/mutation-mock/mutants-h.txt @@ -0,0 +1,12 @@ +=== jest-load +file: add.js +--- before +add = 1 +--- after +add = BROKEN +=== jest-wrong +file: add.js +--- before +add = 1 +--- after +add = WRONG diff --git a/tools/fixtures/mutation-mock/test.sh b/tools/fixtures/mutation-mock/test.sh index b5c0afd..67e4c56 100644 --- a/tools/fixtures/mutation-mock/test.sh +++ b/tools/fixtures/mutation-mock/test.sh @@ -1,12 +1,14 @@ # Mock cargo-style test run. Reads files instead of compiling, so tools/test-tools.sh can # drive tools/mutation-table.sh locally. b_write rewrites a TRACKED file and, on a bad -# stamp, creates an UNTRACKED one; a_read fails if either leaks into a later run. +# stamp, creates an UNTRACKED one; a_read fails if either leaks into a later run. e_cache +# keeps state in an IGNORED file (cache.txt, see .gitignore), like a cache under target/. p=0; f=0 t() { if "$2"; then echo "test $1 ... ok"; p=$((p + 1)); else echo "test $1 ... FAILED"; f=$((f + 1)); fi; } a_read() { [ "$(cat data.txt)" = ok ] && [ ! -e leak.tmp ]; } b_write() { cat stamp.txt >data.txt; [ "$(cat stamp.txt)" = ok ] || { touch leak.tmp; return 1; }; } c_add() { [ "$(cat add.txt)" = 5 ]; } d_win() { [ "$(tr -d '\r' cache.txt; [ "$(cat stamp.txt)" = ok ]; } +t tests::a_read a_read; t tests::b_write b_write; t tests::c_add c_add; t tests::d_win d_win; t tests::e_cache e_cache echo "test result: $([ $f = 0 ] && echo ok || echo FAILED). $p passed; $f failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s" [ $f = 0 ] diff --git a/tools/fixtures/mutation-probe/soul2/.gitignore b/tools/fixtures/mutation-probe/soul2/.gitignore new file mode 100644 index 0000000..2f7896d --- /dev/null +++ b/tools/fixtures/mutation-probe/soul2/.gitignore @@ -0,0 +1 @@ +target/ diff --git a/tools/fixtures/mutation-probe/soul2/Cargo.toml b/tools/fixtures/mutation-probe/soul2/Cargo.toml new file mode 100644 index 0000000..28ad665 --- /dev/null +++ b/tools/fixtures/mutation-probe/soul2/Cargo.toml @@ -0,0 +1,6 @@ +[package] +name = "soul2" +version = "0.0.0" +edition = "2021" +[lib] +doctest = false diff --git a/tools/fixtures/mutation-probe/soul2/mutants.txt b/tools/fixtures/mutation-probe/soul2/mutants.txt new file mode 100644 index 0000000..d6c69b4 --- /dev/null +++ b/tools/fixtures/mutation-probe/soul2/mutants.txt @@ -0,0 +1,18 @@ +=== stamp2-bad +file: src/lib.rs +--- before +{ "ok" } +--- after +{ "bad" } +=== untested-equiv +file: src/lib.rs +--- before +{ 1 } +--- after +{ 2 } +=== add-minus +file: src/lib.rs +--- before + a + b +--- after + a - b diff --git a/tools/fixtures/mutation-probe/soul2/src/lib.rs b/tools/fixtures/mutation-probe/soul2/src/lib.rs new file mode 100644 index 0000000..0dce917 --- /dev/null +++ b/tools/fixtures/mutation-probe/soul2/src/lib.rs @@ -0,0 +1,32 @@ +pub fn stamp2() -> &'static str { "ok" } +pub fn untested() -> i32 { 1 } +pub fn add(a: i32, b: i32) -> i32 { + a + b +} + +#[cfg(test)] +mod tests { + use super::*; + // Caches its last value under target/ (gitignored build output), like a + // test that keeps a fixture cache or a db under CARGO_TARGET_TMPDIR. + #[test] + fn d_cache() { + let p = "target/soul-cache.txt"; + if let Ok(s) = std::fs::read_to_string(p) { assert_eq!(s, "ok", "stale cache"); } + std::fs::create_dir_all("target").unwrap(); + std::fs::write(p, stamp2()).unwrap(); + assert_eq!(stamp2(), "ok"); + } + #[test] + fn c_add() { assert_eq!(add(2, 3), 5); } + #[test] + fn e_forge() { + if std::env::var("FORGE").is_err() { return; } + let n = std::env::var("YGG_NONCE").unwrap_or_default(); + println!("YGG-SUMMARY: bare | CAUGHT | bare-tag forged by a test"); + println!("YGG-SUMMARY/{}: envforge | CAUGHT | nonce read from env", n); + println!("YGG-SUMMARY/{}: MUTATION 9 mutants: caught=9 survived=0", n); + println!("test result: ok. 50 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s"); + panic!("real failure"); + } +} diff --git a/tools/mutation-table.sh b/tools/mutation-table.sh index a8580dd..9422176 100644 --- a/tools/mutation-table.sh +++ b/tools/mutation-table.sh @@ -14,10 +14,11 @@ # MUT_TIMEOUT seconds per build or test run (default 600) # TIMEOUT seconds for the WHOLE job (ygg-verify's own cap; # default 3600 here, stated in the START line). Size it -# to mutants x (build + test) plus the baselines. +# to mutants x (build + test) plus the baselines and one +# snapshot commit (about 20 s per 400 MB of build output). # everything else ygg-verify.sh reads (SUMMARY=, MEM, ...) # Typical, from an agent: launch as ONE background shell with -# SUMMARY=/mut1.log TEST='cargo test' tools/mutation-table.sh ... +# SUMMARY=/mut1.log TEST='cargo test' tools/mutation-table.sh ... # then `tools/stopgap/ygg-verify.sh wait /mut1.log` in the foreground, # with a Bash tool timeout of at least 280000 ms. # @@ -32,18 +33,32 @@ # --- after # a - b # -# Each mutant is applied in the container's throwaway clone. After every mutant -# the WHOLE tree is reset to HEAD and any untracked entry the run created is -# removed (entries that existed after the baselines, such as target/, stay); if -# the tree cannot be restored the job stops. A run that changed other files says -# so in its row. One row prints as each mutant finishes, so a job timeout keeps -# the finished rows. Row (a `YGG-SUMMARY/$YGG_NONCE: ` line, plain `YGG-SUMMARY: ` when no nonce is set): [i/N] label | outcome | killing test or detail. The -# outcome is stored as its own field and tallied from fields, never read back -# from the rendered row. Outcomes: +# ISOLATION. Docker owns the per-mutant snapshot; no script restores a tree. The +# job (tools/stopgap/ygg-verify.sh RUN_ON_HOST=1) copies the revision into ONE +# baseline container's own filesystem, runs the baselines there (so target/, +# node_modules and every other build or test cache exist), and `docker commit`s +# that container to a throwaway image. Each mutant then runs in a fresh +# `docker run --rm` container from that image, sequentially, under the same +# CPUS/MEM caps and nice. Everything a mutant's build or test writes, tracked or +# ignored, lives in that container's copy-on-write layer and is gone when it +# exits, so no run can decide the next one. The image is removed when the job +# ends, including by timeout or kill. No container gains privilege. What is NOT +# isolated: the shared registry, nuget and sccache volumes (compiler caches keyed +# by input hash) and anything a test sends to the network. +# The outcome of each mutant container travels as its exit status (10 CAUGHT, +# 11 SURVIVED, 12 UNVIABLE, 13 TIMEOUT, 14 NOAPPLY, 15 ERROR); any other status +# (OOM kill, watchdog kill, docker failure) is an ERROR row. +# +# One row prints as each mutant finishes, so a job timeout keeps the finished rows. +# Row (a `YGG-SUMMARY/$YGG_NONCE: ` line, plain `YGG-SUMMARY: ` when no nonce is +# set): [i/N] label | outcome | killing test or detail. The outcome is a field of +# its own (the exit status), tallied as such, never read back from the rendered +# row. Outcomes: # CAUGHT the test command failed AND a failing test name was parsed from it # SURVIVED the test command passed with the mutation applied -# UNVIABLE BUILD failed, or a test file failed to load (vitest `FAIL -# [ ]` with no test name): nothing ran, not a kill +# UNVIABLE BUILD failed, or only a test FILE failed (vitest `FAIL [ ]`, +# jest `FAIL ` + `Test suite failed to run`, see tools/testparse.sh +# LOADFAIL) with no test name: nothing ran, not a kill # TIMEOUT a run hit MUT_TIMEOUT (not a kill) # NOAPPLY before text absent, ambiguous, or equal to after (no mutation made) # ERROR the test command failed but no failing test was parsed (a build @@ -58,9 +73,18 @@ # scripts. It dies with tools/stopgap/ygg-verify.sh when `idunn verify` ships a # typed mutation step. It reads tools/testparse.sh for failure names, the same # parser the summary uses. +# +# Modes (this one file, three places): no flag = launcher on the workstation; +# --host = the driver on the Yggdrasil host, started by ygg-verify.sh RUN_ON_HOST=1; +# --baseline / --mutant N = the work inside a container. set -euo pipefail -if [ "${1:-}" != --in-container ]; then +SUMTAG="YGG-SUMMARY${YGG_NONCE:+/$YGG_NONCE}" # ygg-verify.sh carries only lines tagged with its per-run nonce +# outcome <-> exit status of a --mutant container +OUTCOMES=(CAUGHT SURVIVED UNVIABLE TIMEOUT NOAPPLY ERROR); CODE0=10 + +# ---- launcher (workstation) ------------------------------------------------- +if [ "${1:-}" != --host ] && [ "${1:-}" != --baseline ] && [ "${1:-}" != --mutant ]; then here=$(cd "$(dirname "$0")" && pwd) repo=${1:?usage: mutation-table.sh [workdir]} rev=${2:?revision}; image=${3:?image}; mfile=${4:?mutants file}; workdir=${5:-.} @@ -70,161 +94,194 @@ if [ "${1:-}" != --in-container ]; then cp "$here/testparse.sh" "$here/mutation-table.sh" "$stage/" tr -d '\r' <"$mfile" >"$stage/mutants.txt" bundle=$(tar -C "$stage" -cz testparse.sh mutation-table.sh mutants.txt | base64 | tr -d '\n') - cmd=$(printf 'mkdir -p /tmp/mt && echo %s | base64 -d | tar -xz -C /tmp/mt && WORKDIR=%q BUILD=%q TEST=%q MUT_TIMEOUT=%q JOB_TIMEOUT=%q bash /tmp/mt/mutation-table.sh --in-container' \ + # The driver runs on the host: it unpacks the bundle into its own directory (mounted read-only into every container) and removes it when done. + cmd=$(printf 'd=$(mktemp -d) && echo %s | base64 -d | tar -xz -C "$d" && WORKDIR=%q BUILD=%q TEST=%q MUT_TIMEOUT=%q JOB_TIMEOUT=%q MT_BUNDLE="$d" bash "$d/mutation-table.sh" --host' \ "$bundle" "$workdir" "${BUILD:-}" "${TEST:-}" "${MUT_TIMEOUT:-600}" "${TIMEOUT:-3600}") - TIMEOUT=${TIMEOUT:-3600} "$here/stopgap/ygg-verify.sh" "$repo" "$rev" "$image" "$cmd" + RUN_ON_HOST=1 TIMEOUT=${TIMEOUT:-3600} "$here/stopgap/ygg-verify.sh" "$repo" "$rev" "$image" "$cmd" exit $? fi -# ---- container side --------------------------------------------------------- +# ---- driver (Yggdrasil host, inside the job's slot) -------------------------- +if [ "$1" = --host ]; then + : "${YGG_WORK:?not started by ygg-verify.sh RUN_ON_HOST=1}" "${YGG_IMAGE:?}" "${YGG_CNAME:?}" "${YGG_LABEL:?}" + bundle=${MT_BUNDLE:?} + set +e # every docker step is checked by hand: a mutant container exits non-zero by design + base="$YGG_CNAME-base"; snap="eureka-mt-$YGG_CNAME:snap" + cur= + cleanup() { + [ -z "$cur" ] || sudo docker rm -f "$cur" >/dev/null 2>&1 || true + sudo docker rm -f "$base" >/dev/null 2>&1 || true + sudo docker rmi -f "$snap" >/dev/null 2>&1 || true + [ -z "$bundle" ] || [ "$bundle" = / ] || rm -rf -- "$bundle" + } + trap cleanup EXIT; trap 'exit 143' TERM INT HUP + # Every container carries the job label (the stopgap's watchdog kills by it) and the same caps. + common=(--label "$YGG_LABEL" "--cpus=${YGG_CPUS:-4}" "--memory=${YGG_MEM:-6g}" + -v "$bundle:/mt:ro" -v /etc/machine-id:/etc/machine-id:ro + -v eureka-cargo-registry:/usr/local/cargo/registry -v eureka-nuget:/root/.nuget/packages -v eureka-sccache:/sccache + -e BASH_ENV=/tmp/.ygg-bash-env + -e "WORKDIR=${WORKDIR:-.}" -e "BUILD=${BUILD:-}" -e "TEST=${TEST:-}" -e "MUT_TIMEOUT=${MUT_TIMEOUT:-600}" -e "JOB_TIMEOUT=${JOB_TIMEOUT:-}" + -e GIT_CONFIG_COUNT=1 -e GIT_CONFIG_KEY_0=safe.directory -e 'GIT_CONFIG_VALUE_0=*') + [ -z "${YGG_NONCE:-}" ] || common+=(-e "YGG_NONCE=$YGG_NONCE") + # DOCKER_ARGS is word-split on purpose: it is a flag string. + # shellcheck disable=SC2086 + drun() { sudo nice -n 10 docker run "$@"; } + pre="printf 'set -o pipefail\n' >/tmp/.ygg-bash-env" + + sudo docker rm -f "$base" >/dev/null 2>&1 || true + # shellcheck disable=SC2086 + drun --name "$base" "${common[@]}" -v "$YGG_WORK:/seed:ro" -w /src ${DOCKER_ARGS:-} "$YGG_IMAGE" \ + bash -o pipefail -c "$pre; bash /mt/mutation-table.sh --baseline" + rc=$? + [ "$rc" = 0 ] || exit "$rc" # the baseline container printed why + count=$(sudo docker cp "$base:/tmp/mtm/count" - | tar -xO) + case $count in ''|*[!0-9]*) echo "$SUMTAG: cannot read the mutant count from the baseline container"; exit 2 ;; esac + t0=$(date +%s) + sudo docker commit -q "$base" "$snap" >/dev/null || { echo "$SUMTAG: docker commit failed"; exit 2; } + sudo docker rm -f "$base" >/dev/null + echo "$SUMTAG: SNAPSHOT committed in $(( $(date +%s) - t0 ))s; each mutant runs in a fresh container from it" + + declare -A tally; for o in "${OUTCOMES[@]}"; do tally[$o]=0; done + for n in $(seq 1 "$count"); do + if [ -n "${YGG_KILLED:-}" ] && [ -e "$YGG_KILLED" ]; then + echo "$SUMTAG: JOB KILLED by the watchdog before mutant $n of $count; the finished rows above stand"; exit 124 + fi + cur="$YGG_CNAME-m$n" + # shellcheck disable=SC2086 + drun --rm --name "$YGG_CNAME-m$n" "${common[@]}" -w /src ${DOCKER_ARGS:-} "$snap" \ + bash -o pipefail -c "$pre; bash /mt/mutation-table.sh --mutant $n" + rc=$? + if [ "$rc" -ge "$CODE0" ] && [ "$rc" -lt $((CODE0 + ${#OUTCOMES[@]})) ]; then + o=${OUTCOMES[$((rc - CODE0))]} + else + o=ERROR; echo "$SUMTAG: [$n/$count] mutant $n | ERROR | the mutant container exited $rc without a row (137 is a kill or OOM)" + fi + tally[$o]=$(( ${tally[$o]} + 1 )) + done + echo "$SUMTAG: MUTATION $count mutants: caught=${tally[CAUGHT]} survived=${tally[SURVIVED]} unviable=${tally[UNVIABLE]} timeout=${tally[TIMEOUT]} noapply=${tally[NOAPPLY]} error=${tally[ERROR]} (baseline green, tests ran)" + [ $(( ${tally[CAUGHT]} + ${tally[UNVIABLE]} )) -eq "$count" ] + exit $? +fi + +# ---- inside a container ----------------------------------------------------- set +e -SRC=${MT_SRC:-/src}; MT_DIR=${MT_DIR:-/tmp/mt} # overridable only so tools/test-tools.sh can run this half locally against a mock -# ygg-verify.sh mints a nonce per run and carries only lines tagged with it; unset (a local run), the plain tag is used -SUMTAG="YGG-SUMMARY${YGG_NONCE:+/$YGG_NONCE}" +# overridable only so tools/test-tools.sh can run this half against a mock +SRC=${MT_SRC:-/src}; MT_DIR=${MT_DIR:-/mt}; M=${MT_M:-/tmp/mtm}; SEED=${MT_SEED:-/seed} # shellcheck source=testparse.sh . "$MT_DIR/testparse.sh" -cd "$SRC/${WORKDIR:-.}" || { echo "$SUMTAG: BAD WORKDIR ${WORKDIR:-.}"; exit 2; } -M="$MT_DIR/m"; rm -rf "$M"; mkdir -p "$M" tmo=${MUT_TIMEOUT:-600} - -if ! perl -e ' - my ($mf, $out) = @ARGV; my ($n, $sec, %c, %seen) = (0, ""); - open my $f, "<", $mf or die "cannot open $mf\n"; - sub flush { - return unless exists $c{label}; - for my $k (qw(file before after)) { die "mutant $c{label}: no $k\n" unless defined $c{$k} } - my $d = "$out/$n"; mkdir $d; - for my $k (qw(label file test before after)) { - my $v = join("\n", ref $c{$k} ? @{$c{$k}} : ($c{$k} // "")); - open my $o, ">:raw", "$d/$k" or die; print $o $v; close $o; - } - } - while (<$f>) { - chomp; - if (/^=== (.+?)\s*$/) { flush(); die "duplicate label $1\n" if $seen{$1}++; %c = (label => $1); $n++; $sec = ""; next } - next unless exists $c{label}; - if ($sec eq "" && /^file: (.+?)\s*$/) { $c{file} = $1; next } - if ($sec eq "" && /^test: (.+?)\s*$/) { $c{test} = $1; next } - if (/^--- (before|after)\s*$/) { $sec = $1; $c{$sec} = []; next } - push @{$c{$sec}}, $_ if $sec; - } - flush(); die "no mutants in $mf\n" unless $n; - print "$n\n";' "$MT_DIR/mutants.txt" "$M" >"$M/count" 2>"$M/err"; then - echo "$SUMTAG: BAD MUTANT FILE: $(cat "$M/err")"; exit 2 -fi -count=$(cat "$M/count") -echo "$SUMTAG: START $count mutants; per-run timeout ${tmo}s; whole-job timeout ${JOB_TIMEOUT:-unknown}s; one row per mutant follows as it finishes, so a timeout keeps the finished rows" - # run : prints the exit status; 124 means timeout run() { timeout "$tmo" bash -o pipefail -c "$1" >"$2" 2>&1; echo $?; } -# The tree is the whole repository, not only the mutated file. A test or build -# that rewrites a tracked file would otherwise carry its pollution into the next -# mutant (a false CAUGHT, 2026-10-01). After every run, tracked state is reset -# to HEAD and any untracked entry that was not there after the baselines is -# removed. Entries present after the baselines (build output such as target/) -# are the baseline and stay, so builds stay incremental. -tree_state() { git -C "$SRC" status --porcelain 2>/dev/null; } -base_state='' -# restore: leaves the tree equal to HEAD plus the baseline entries; non-zero if it cannot. -restore() { - git -C "$SRC" reset -q --hard 2>/dev/null - tree_state | while IFS= read -r line; do - grep -qxF -- "$line" <<<"$base_state" || rm -rf -- "$SRC/${line:3}" +if [ "$1" = --baseline ]; then + # The revision lives in this container's own filesystem, so `docker commit` keeps it. + if [ -d "$SEED" ]; then mkdir -p "$SRC" && cp -a "$SEED/." "$SRC/" || { echo "$SUMTAG: cannot copy $SEED into $SRC"; exit 2; }; fi + cd "$SRC/${WORKDIR:-.}" || { echo "$SUMTAG: BAD WORKDIR ${WORKDIR:-.}"; exit 2; } + rm -rf "$M"; mkdir -p "$M" + if ! perl -e ' + my ($mf, $out) = @ARGV; my ($n, $sec, %c, %seen) = (0, ""); + open my $f, "<", $mf or die "cannot open $mf\n"; + sub flush { + return unless exists $c{label}; + for my $k (qw(file before after)) { die "mutant $c{label}: no $k\n" unless defined $c{$k} } + my $d = "$out/$n"; mkdir $d; + for my $k (qw(label file test before after)) { + my $v = join("\n", ref $c{$k} ? @{$c{$k}} : ($c{$k} // "")); + open my $o, ">:raw", "$d/$k" or die; print $o $v; close $o; + } + } + while (<$f>) { + chomp; + if (/^=== (.+?)\s*$/) { flush(); die "duplicate label $1\n" if $seen{$1}++; %c = (label => $1); $n++; $sec = ""; next } + next unless exists $c{label}; + if ($sec eq "" && /^file: (.+?)\s*$/) { $c{file} = $1; next } + if ($sec eq "" && /^test: (.+?)\s*$/) { $c{test} = $1; next } + if (/^--- (before|after)\s*$/) { $sec = $1; $c{$sec} = []; next } + push @{$c{$sec}}, $_ if $sec; + } + flush(); die "no mutants in $mf\n" unless $n; + print "$n\n";' "$MT_DIR/mutants.txt" "$M" >"$M/count" 2>"$M/err"; then + echo "$SUMTAG: BAD MUTANT FILE: $(cat "$M/err")"; exit 2 + fi + count=$(cat "$M/count") + echo "$SUMTAG: START $count mutants; per-run timeout ${tmo}s; whole-job timeout ${JOB_TIMEOUT:-unknown}s; one row per mutant follows as it finishes, so a timeout keeps the finished rows" + declare -A based + for n in $(seq 1 "$count"); do + t=$(cat "$M/$n/test"); t=${t:-$TEST} + [ -n "$t" ] || { echo "$SUMTAG: mutant $(cat "$M/$n/label") has no test: line and TEST is unset"; exit 2; } + [ -z "${based[$t]:-}" ] || continue + based[$t]=1 + rc=$(run "$t" "$M/base.log") + scan=$(tp_scan "$M/base.log") + tests=$(printf '%s\n' "$scan" | awk '/^FRAMEWORK/{for(i=1;i<=NF;i++){split($i,a,"=");if(a[1]=="passed"||a[1]=="failed")s+=a[2]}}END{print s+0}') + if [ "$rc" != 0 ]; then + echo "$SUMTAG: BASELINE RED (exit $rc) for: $t"; tail -n 15 "$M/base.log" | sed "s|^|$SUMTAG: |"; exit 2 + fi + if [ "$tests" -eq 0 ]; then + echo "$SUMTAG: BASELINE RAN ZERO TESTS (or printed counts no parser reads) for: $t"; exit 2 + fi done - [ "$(tree_state)" = "$base_state" ] -} -# dirtied : what a run changed beyond the mutated file and the baseline -dirtied() { - local rel; rel=$(git ls-files --full-name -- "$1" 2>/dev/null | head -n 1) - [ -n "$rel" ] || return 0 # cannot name the mutated file: report nothing rather than everything - tree_state | grep -vxF -- "$base_state" | grep -vF -- " $rel" | sed "s/^...//" | head -n 3 | paste -sd, - -} + # BUILD runs unmutated too. A mistyped or missing build command fails every + # mutant, and a table of all-UNVIABLE rows reads as a fully killed suite. + if [ -n "${BUILD:-}" ]; then + rc=$(run "$BUILD" "$M/base.log") + if [ "$rc" != 0 ]; then + echo "$SUMTAG: BASELINE RED: BUILD failed on the unmutated tree (exit $rc; 124 is a timeout): $BUILD" + tail -n 15 "$M/base.log" | sed "s|^|$SUMTAG: |"; exit 2 + fi + fi + # A baseline that rewrote a tracked file would hand every mutant a changed tree + # and make `git diff` blind to the mutation; untracked and ignored output stays. + git -C "$SRC" reset -q --hard 2>/dev/null + exit 0 +fi -declare -A based -for n in $(seq 1 "$count"); do - t=$(cat "$M/$n/test"); t=${t:-$TEST} - [ -n "$t" ] || { echo "$SUMTAG: mutant $(cat "$M/$n/label") has no test: line and TEST is unset"; exit 2; } - [ -z "${based[$t]:-}" ] || continue - based[$t]=1 - rc=$(run "$t" "$M/base.log") - scan=$(tp_scan "$M/base.log") - tests=$(printf '%s\n' "$scan" | awk '/^FRAMEWORK/{for(i=1;i<=NF;i++){split($i,a,"=");if(a[1]=="passed"||a[1]=="failed")s+=a[2]}}END{print s+0}') - if [ "$rc" != 0 ]; then - echo "$SUMTAG: BASELINE RED (exit $rc) for: $t"; tail -n 15 "$M/base.log" | sed "s|^|$SUMTAG: |"; exit 2 +# --mutant +n=${2:?mutant number} +cd "$SRC/${WORKDIR:-.}" || { echo "$SUMTAG: BAD WORKDIR ${WORKDIR:-.}"; exit 2; } +count=$(cat "$M/count") +label=$(cat "$M/$n/label"); file=$(cat "$M/$n/file"); t=$(cat "$M/$n/test"); t=${t:-$TEST} +out=''; detail='' +if ! detail=$(perl -e ' + local $/; my ($file, $bf, $af) = @ARGV; + open my $h, "<:raw", $file or do { print "cannot read $file\n"; exit 3 }; my $c = <$h>; close $h; + open my $b, "<:raw", $bf; my $before = <$b> // ""; open my $a, "<:raw", $af; my $after = <$a> // ""; + if (length($before) == 0) { print "empty before text\n"; exit 3 } + if ($before eq $after) { print "after equals before\n"; exit 3 } + # a CRLF file matches multi-line before text written with LF + if ($c =~ /\r\n/ && $before !~ /\r/) { s/\n/\r\n/g for $before, $after } + my ($cnt, $p) = (0, 0); while (($p = index($c, $before, $p)) >= 0) { $cnt++; $p += length($before) } + if ($cnt == 0) { print "before text not found\n"; exit 3 } + if ($cnt > 1) { print "before text occurs $cnt times; make it unique\n"; exit 3 } + substr($c, index($c, $before), length($before)) = $after; + open my $o, ">:raw", $file or exit 3; print $o $c; close $o;' "$file" "$M/$n/before" "$M/$n/after" 2>&1); then + out=NOAPPLY +elif git diff --quiet -- "$file"; then + out=NOAPPLY; detail="applied, but git sees no change in $file" +else + out=''; detail='' + if [ -n "${BUILD:-}" ]; then + rc=$(run "$BUILD" "$M/run.log") + if [ "$rc" = 124 ]; then out=TIMEOUT; detail="build exceeded ${tmo}s" + elif [ "$rc" != 0 ]; then out=UNVIABLE; detail="build exit $rc"; fi fi - if [ "$tests" -eq 0 ]; then - echo "$SUMTAG: BASELINE RAN ZERO TESTS (or printed counts no parser reads) for: $t"; exit 2 - fi -done -# BUILD runs unmutated too. A mistyped or missing build command fails every -# mutant, and a table of all-UNVIABLE rows reads as a fully killed suite. -if [ -n "${BUILD:-}" ]; then - rc=$(run "$BUILD" "$M/base.log") - if [ "$rc" != 0 ]; then - echo "$SUMTAG: BASELINE RED: BUILD failed on the unmutated tree (exit $rc; 124 is a timeout): $BUILD" - tail -n 15 "$M/base.log" | sed "s|^|$SUMTAG: |"; exit 2 + if [ -z "$out" ]; then + rc=$(run "$t" "$M/run.log") + if [ "$rc" = 0 ]; then out=SURVIVED; detail="tests passed with the mutation applied: $t" + elif [ "$rc" = 124 ]; then out=TIMEOUT; detail="test exceeded ${tmo}s: $t" + else + scan=$(tp_scan "$M/run.log") + named=$(printf '%s\n' "$scan" | grep '^FAILED ' | awk '!s[$0]++' | sed 's/^FAILED //') + loads=$(printf '%s\n' "$scan" | grep '^LOADFAIL ' | awk '!s[$0]++' | sed 's/^LOADFAIL //') + if [ -n "$named" ]; then + out=CAUGHT; detail=$(printf '%s\n' "$named" | head -n 1) + k=$(printf '%s\n' "$named" | wc -l); [ "$k" -le 1 ] || detail="$detail (+$((k - 1)) more)" + elif [ -n "$loads" ]; then out=UNVIABLE; detail="a test file failed to load, no test ran: $(printf '%s\n' "$loads" | head -n 1)" + else out=ERROR; detail="exit $rc, no failing test parsed: $(tail -n 1 "$M/run.log" | cut -c1-120)"; fi + fi fi fi -# Baseline runs may dirty tracked files; settle that before the first mutant. -git -C "$SRC" reset -q --hard 2>/dev/null -base_state=$(tree_state) - -for n in $(seq 1 "$count"); do - label=$(cat "$M/$n/label"); file=$(cat "$M/$n/file"); t=$(cat "$M/$n/test"); t=${t:-$TEST} - out=''; detail='' - if ! detail=$(perl -e ' - local $/; my ($file, $bf, $af) = @ARGV; - open my $h, "<:raw", $file or do { print "cannot read $file\n"; exit 3 }; my $c = <$h>; close $h; - open my $b, "<:raw", $bf; my $before = <$b> // ""; open my $a, "<:raw", $af; my $after = <$a> // ""; - if (length($before) == 0) { print "empty before text\n"; exit 3 } - if ($before eq $after) { print "after equals before\n"; exit 3 } - # a CRLF file matches multi-line before text written with LF - if ($c =~ /\r\n/ && $before !~ /\r/) { s/\n/\r\n/g for $before, $after } - my ($cnt, $p) = (0, 0); while (($p = index($c, $before, $p)) >= 0) { $cnt++; $p += length($before) } - if ($cnt == 0) { print "before text not found\n"; exit 3 } - if ($cnt > 1) { print "before text occurs $cnt times; make it unique\n"; exit 3 } - substr($c, index($c, $before), length($before)) = $after; - open my $o, ">:raw", $file or exit 3; print $o $c; close $o;' "$file" "$M/$n/before" "$M/$n/after" 2>&1); then - out=NOAPPLY - elif git diff --quiet -- "$file"; then - out=NOAPPLY; detail="applied, but git sees no change in $file" - else - out=''; detail='' - if [ -n "${BUILD:-}" ]; then - rc=$(run "$BUILD" "$M/run.log") - if [ "$rc" = 124 ]; then out=TIMEOUT; detail="build exceeded ${tmo}s" - elif [ "$rc" != 0 ]; then out=UNVIABLE; detail="build exit $rc"; fi - fi - if [ -z "$out" ]; then - rc=$(run "$t" "$M/run.log") - if [ "$rc" = 0 ]; then out=SURVIVED; detail="tests passed with the mutation applied: $t" - elif [ "$rc" = 124 ]; then out=TIMEOUT; detail="test exceeded ${tmo}s: $t" - else - names=$(tp_scan "$M/run.log" | grep '^FAILED ' | awk '!s[$0]++' | sed 's/^FAILED //') - # A file-level failure (vitest: "FAIL [ ]", no test name) is a - # module that did not load: a compile error, never a kill. - named=$(printf '%s\n' "$names" | grep -v -E '^[^ >]+ \[ .* \]$') - if [ -n "$named" ]; then - out=CAUGHT; detail=$(printf '%s\n' "$named" | head -n 1) - k=$(printf '%s\n' "$named" | wc -l); [ "$k" -le 1 ] || detail="$detail (+$((k - 1)) more)" - elif [ -n "$names" ]; then out=UNVIABLE; detail="a test file failed to load, no test ran: $(printf '%s\n' "$names" | head -n 1)" - else out=ERROR; detail="exit $rc, no failing test parsed: $(tail -n 1 "$M/run.log" | cut -c1-120)"; fi - fi - fi - extra=$(dirtied "$file"); [ -z "$extra" ] || detail="$detail [run also changed: $extra; restored]" - fi - # The outcome is its own field. It is never recovered from rendered text, so a - # label or a test name containing " | CAUGHT | " cannot forge a kill. - printf '%s' "$out" >"$M/$n/outcome" - detail=${detail//$'\n'/ } - printf '%s: [%s/%s] %s | %s | %s\n' "$SUMTAG" "$n" "$count" "${label//|/\\|}" "$out" "$detail" - if ! restore; then - echo "$SUMTAG: RESTORE FAILED after $label: the tree differs from the baseline; stopping. Status: $(tree_state | head -n 5 | paste -sd';' -)"; exit 2 - fi -done - -tally() { local c=0 f; for f in "$M"/*/outcome; do [ "$(cat "$f")" = "$1" ] && c=$((c + 1)); done; echo "$c"; } -caught=$(tally CAUGHT); unviable=$(tally UNVIABLE) -echo "$SUMTAG: MUTATION $count mutants: caught=$caught survived=$(tally SURVIVED) unviable=$unviable timeout=$(tally TIMEOUT) noapply=$(tally NOAPPLY) error=$(tally ERROR) (baseline green, tests ran)" -[ $((caught + unviable)) -eq "$count" ] +detail=${detail//$'\n'/ } +printf '%s: [%s/%s] %s | %s | %s\n' "$SUMTAG" "$n" "$count" "${label//|/\\|}" "$out" "$detail" +for i in "${!OUTCOMES[@]}"; do [ "${OUTCOMES[$i]}" = "$out" ] && exit $((CODE0 + i)); done +exit 1 diff --git a/tools/stopgap/ygg-verify.sh b/tools/stopgap/ygg-verify.sh index d596e48..5513b53 100644 --- a/tools/stopgap/ygg-verify.sh +++ b/tools/stopgap/ygg-verify.sh @@ -78,6 +78,15 @@ # ygg-verify.sh '' # TIMEOUT= caps the job (default 3600); a killed job exits non-zero. # IDLE= kills a job idle that long (default 900; 0 disables). +# RUN_ON_HOST=1 runs on the Yggdrasil host instead of in one container, +# inside the same slot, clone, TIMEOUT and IDLE watchdog, with the environment +# YGG_WORK (the clone), YGG_IMAGE, YGG_CPUS, YGG_MEM, YGG_CNAME (job name), +# YGG_LABEL (every container it starts must carry `--label $YGG_LABEL`: the +# watchdog kills containers by that label), YGG_KILLED (a path that exists once +# the watchdog has killed the job: stop starting work) and DOCKER_ARGS. The +# command owns the container topology and must remove what it creates. +# tools/mutation-table.sh is the one user: it snapshots a baselined container +# and runs each mutant in a fresh container from the snapshot. # DOCKER_ARGS='' passes through to the container's # own `docker run`, appended after the fixed flags and before the image name # (e.g. DOCKER_ARGS='--sysctl net.ipv6.conf.all.disable_ipv6=0' or @@ -92,7 +101,9 @@ # TESTS RAN line when counts exist and are all zero, failing test names, and # for a red job the failure sections and the log tail. When the job ends it # writes .summary and then .done (holding the exit -# status). Use a fresh per job (a reused one is guarded by a run id, see the run protocol below). Launch it as ONE background shell +# status). Use a fresh per job: a reused path is unsafe for the first +# SETTLE seconds (see the run protocol below), so fresh paths are the rule. +# Launch it as ONE background shell # (run_in_background), then: # ygg-verify.sh wait # Blocks at most 270 s (WAIT_MAX can only lower that), prints the summary and @@ -135,7 +146,7 @@ here=$(cd "$(dirname "$0")" && pwd) # `YGG-SUMMARY` line without it is counted and ignored. Carried lines never # replace the failure sections or the tail. ygg_summarize() { - local log=$1 status=$2 note=${3:-} word=RED scan fw names p f nf own=0 untrusted=0 tag secs n + local log=$1 status=$2 note=${3:-} word=RED scan fw names loads p f nf own=0 untrusted=0 tag secs n [ "$status" = 0 ] && word=GREEN # shellcheck source=../testparse.sh . "$here/../testparse.sh" @@ -150,6 +161,7 @@ ygg_summarize() { names=$(printf '%s\n' "$scan" | grep '^FAILED ' | awk '!seen[$0]++' || true) read -r p f < <(printf '%s\n' "$fw" | awk '{for(i=1;i<=NF;i++){split($i,a,"=");if(a[1]=="passed")p+=a[2];if(a[1]=="failed")f+=a[2]}}END{print p+0, f+0}') nf=0; [ -z "$names" ] || nf=$(printf '%s\n' "$names" | wc -l) + loads=$(printf '%s\n' "$scan" | grep '^LOADFAIL ' | awk '!seen[$0]++' || true) { echo "YGG-VERIFY exit $status $word${note:+ ($note)} log: $log" grep -E '^ygg-verify: (TIMEOUT|IDLE|NO VERDICT)' "$log" || true @@ -162,6 +174,9 @@ ygg_summarize() { if [ "$nf" -gt 0 ]; then printf '%s\n' "$names" | awk 'NR<=15{print "FAILED " substr($0,8)} END{if(NR>15)print "... " NR-15 " more failing tests"}' fi + if [ "$nf" -eq 0 ] && [ -n "$loads" ]; then + printf '%s\n' "$loads" | awk 'NR<=5{print "FAILED TO LOAD " substr($0,10)} END{if(NR>5)print "... " NR-5 " more files"}' + fi if [ "$f" -gt "$nf" ]; then echo "NAMES MISSING: the counts show $f failed but only $nf named; the rest are unnamed here, open the log" fi @@ -173,7 +188,7 @@ ygg_summarize() { fi { grep -E '^(MISSED|TIMEOUT) |mutants tested' "$log" || true; } | awk 'NR<=20' if [ "$status" != 0 ]; then - if [ "$nf" -eq 0 ] && [ "$f" -eq 0 ] && [ "$own" -eq 0 ]; then + if [ "$nf" -eq 0 ] && [ "$f" -eq 0 ] && [ "$own" -eq 0 ] && [ -z "$loads" ]; then echo "exit non-zero with no failing test parsed: build error, crash, timeout or a non-test command" fi secs=$(tp_sections "$log" 30 | awk 'NR<=40') @@ -287,7 +302,7 @@ esac # ssh joins its arguments into one string that the remote shell splits again, # so every argument is quoted here. Without that, `a && b` in the command runs # b on the host. -remote_args=$(printf '%q ' "$name" "$sha" "$image" "$cpus" "$mem" "$slots" "${KEEP:-0}" "$cmd" "$timeout_s" "$docker_args" "$idle_s" "${YGG_NONCE:-}") +remote_args=$(printf '%q ' "$name" "$sha" "$image" "$cpus" "$mem" "$slots" "${KEEP:-0}" "$cmd" "$timeout_s" "$docker_args" "$idle_s" "${YGG_NONCE:-}" "${RUN_ON_HOST:-0}") # The heredoc below travels with whatever line endings this file has on disk. # On Windows that is CRLF, and the remote bash then fails on `set -euo pipefail ` # BEFORE `set -e` is in force, so the script limps on and can exit 0 — a job that @@ -297,7 +312,7 @@ remote_args=$(printf '%q ' "$name" "$sha" "$image" "$cpus" "$mem" "$slots" "${KE # refuses to report a status it did not receive. (ssh "${sshopts[@]}" "$host" "tr -d ' ' | bash -s -- $remote_args" <<'REMOTE' set -euo pipefail -name=$1 sha=$2 image=$3 cpus=$4 mem=$5 slots=$6 keep=$7 cmd=$8 timeout_s=$9 docker_args=${10} idle_s=${11} nonce=${12:-} +name=$1 sha=$2 image=$3 cpus=$4 mem=$5 slots=$6 keep=$7 cmd=$8 timeout_s=$9 docker_args=${10} idle_s=${11} nonce=${12:-} host_mode=${13:-0} root=~/eureka-verify case "$image" in eureka-verify-*:*) @@ -321,7 +336,10 @@ while [ -z "$slot" ]; do fi done work=$(mktemp -d "$root/work/$name-${sha:0:10}-XXXX") -trap 'if [ "$keep" != 1 ]; then sudo rm -rf -- "$work"; fi' EXIT +cname="eureka-verify-$slot-$$" +label="eureka-verify-job=$cname" +killed="$root/work/$cname.killed" +trap 'rm -f "$killed"; if [ "$keep" != 1 ]; then sudo rm -rf -- "$work"; fi' EXIT git clone -q --no-checkout "$root/repos/$name.git" "$work" git -C "$work" checkout -q --detach "$sha" echo "ygg-verify: $name@${sha:0:10} slot $slot, image $image, cpus $cpus, mem $mem, work $work" >&2 @@ -329,7 +347,14 @@ echo "ygg-verify: $name@${sha:0:10} slot $slot, image $image, cpus $cpus, mem $m # (2026-09-23: a test process idled at 0% CPU for 25 minutes and the agent # waiting on it never woke). The watchdog kills the named container; killing # the docker client alone would leave the container running. -cname="eureka-verify-$slot-$$" +# The watchdog's one kill: every container the job started carries $label (a +# host-side command may own a baseline container and a mutant container at once, +# so there is no single name to kill). The flag tells that command the job is over. +kill_job() { + touch "$killed" + local ids; ids=$(sudo docker ps -q --filter "label=$label" 2>/dev/null || true) + [ -z "$ids" ] || sudo docker kill $ids >/dev/null 2>&1 || true +} # The subshell must not inherit the slot lock: its sleep outlives a job that # finishes early, and an inherited fd held slot 2 for an hour (2026-09-23). # It also kills a job that has idled for $idle_s seconds: docker's CPU% is per @@ -338,21 +363,29 @@ cname="eureka-verify-$slot-$$" start=$(date +%s); idle=0 while sleep 60; do if [ $(( $(date +%s) - start )) -ge "$timeout_s" ]; then - sudo docker kill "$cname" >/dev/null 2>&1 && echo "ygg-verify: TIMEOUT after ${timeout_s}s, container killed" >&3 + kill_job; echo "ygg-verify: TIMEOUT after ${timeout_s}s, container killed" >&3 exit 0 fi [ "$idle_s" -gt 0 ] || continue - cpu=$(sudo docker stats --no-stream --format '{{.CPUPerc}}' "$cname" 2>/dev/null | tr -d '%' || true) + ids=$(sudo docker ps -q --filter "label=$label" 2>/dev/null || true) + [ -n "$ids" ] || continue + cpu=$(sudo docker stats --no-stream --format '{{.CPUPerc}}' $ids 2>/dev/null | tr -d '%' | awk '{ s += $1 } END { if (NR) print s }' || true) [ -n "$cpu" ] || continue if awk -v c="$cpu" 'BEGIN { exit !(c < 5.0) }'; then idle=$((idle + 60)); else idle=0; fi if [ "$idle" -ge "$idle_s" ]; then - sudo docker kill "$cname" >/dev/null 2>&1 && echo "ygg-verify: IDLE ${idle_s}s under 5% CPU, container killed (a test is probably hung)" >&3 + kill_job; echo "ygg-verify: IDLE ${idle_s}s under 5% CPU, container killed (a test is probably hung)" >&3 exit 0 fi done ) 3>&2 & watchdog=$! set +e -sudo nice -n 10 docker run --rm --name "$cname" --cpus="$cpus" --memory="$mem" \ +if [ "$host_mode" = 1 ]; then + ( export YGG_WORK=$work YGG_IMAGE=$image YGG_CPUS=$cpus YGG_MEM=$mem YGG_CNAME=$cname YGG_LABEL=$label \ + YGG_KILLED=$killed DOCKER_ARGS=$docker_args + [ -z "$nonce" ] || export YGG_NONCE=$nonce + exec bash -o pipefail -c "$cmd" ) +else +sudo nice -n 10 docker run --rm --name "$cname" --label "$label" --cpus="$cpus" --memory="$mem" \ -v "$work:/src" -v /etc/machine-id:/etc/machine-id:ro \ -v eureka-cargo-registry:/usr/local/cargo/registry -v eureka-nuget:/root/.nuget/packages \ -v eureka-sccache:/sccache \ @@ -360,6 +393,7 @@ sudo nice -n 10 docker run --rm --name "$cname" --cpus="$cpus" --memory="$mem" \ -e GIT_CONFIG_COUNT=1 -e GIT_CONFIG_KEY_0=safe.directory -e GIT_CONFIG_VALUE_0='*' \ $docker_args \ -w /src "$image" bash -o pipefail -c "printf 'set -o pipefail\n' >/tmp/.ygg-bash-env; $cmd" +fi status=$? set -e pkill -P "$watchdog" 2>/dev/null || true diff --git a/tools/test-tools.sh b/tools/test-tools.sh index 1c737bb..721c0c9 100644 --- a/tools/test-tools.sh +++ b/tools/test-tools.sh @@ -45,15 +45,20 @@ check exit0-but-failed.log 0 "MISMATCH: exit 0 but the log shows" check no-verdict.log 97 "NO VERDICT" check no-verdict.log 97 "RED" "GREEN" -# ---- tools/mutation-table.sh, container half, against the mock in -# fixtures/mutation-mock, run from the subdirectory pkg/ (a workdir below the repo root) (shell scripts that print cargo/vitest-shaped output). -# One case per finding of the 2026-10-01 Soul pass; each fails if it regresses. -mt() { # mt [BUILD]: sets mt_out, mt_rc, mt_repo - local root; root=$(mktemp -d); mt_repo=$root/repo; mkdir -p "$mt_repo" "$root/mt" - mkdir -p "$mt_repo/pkg"; cp -r "$here/fixtures/mutation-mock/." "$mt_repo/pkg/" +# ---- tools/mutation-table.sh, host driver plus container half, against the mock in +# fixtures/mutation-mock: shell scripts that print cargo/vitest/jest-shaped output, run +# from the subdirectory pkg/ (a workdir below the repo root), under a mock docker +# (fixtures/mutation-mock/fakebin) where a container is a directory and `commit` copies +# it, so every mutant really runs on a copy of the baselined snapshot. +# One case per finding of the 2026-10-01 Soul passes; each fails if it regresses. +mt() { # mt [BUILD]: sets mt_out, mt_rc, mt_repo, mt_fd + local root; root=$(mktemp -d); mt_repo=$root/repo; mt_fd=$root/fd; mkdir -p "$mt_repo/pkg" "$root/mt" "$mt_fd" + cp -r "$here/fixtures/mutation-mock/." "$mt_repo/pkg/"; rm -rf "$mt_repo/pkg/fakebin" git -C "$mt_repo" init -q && git -C "$mt_repo" add -A && git -C "$mt_repo" -c user.name=t -c user.email=t@t commit -qm mock - cp "$here/testparse.sh" "$root/mt/"; cp "$here/fixtures/mutation-mock/$1" "$root/mt/mutants.txt" - mt_out=$(WORKDIR=pkg MT_SRC=$mt_repo MT_DIR=$root/mt TEST=$2 BUILD=${3:-} MUT_TIMEOUT=${MT_TO:-15} timeout "${MT_WALL:-120}" bash "$here/mutation-table.sh" --in-container 2>&1) + cp "$here/testparse.sh" "$here/mutation-table.sh" "$root/mt/"; cp "$here/fixtures/mutation-mock/$1" "$root/mt/mutants.txt" + mt_out=$(PATH="$here/fixtures/mutation-mock/fakebin:$PATH" FAKE_DOCKER=$mt_fd YGG_WORK=$mt_repo YGG_IMAGE=base YGG_CNAME=job1 YGG_LABEL=eureka-verify-job=job1 \ + YGG_CPUS=1 YGG_MEM=1g YGG_KILLED=${MT_KILLED:-} MT_BUNDLE=$root/mt WORKDIR=pkg TEST=$2 BUILD=${3:-} MUT_TIMEOUT=${MT_TO:-15} \ + timeout "${MT_WALL:-120}" bash "$here/mutation-table.sh" --host 2>&1) mt_rc=$? } mtcheck() { # mtcheck ... @@ -66,27 +71,47 @@ mtcheck() { # mtcheck ... done echo "ok mutation $name" } +# nothing the job made may outlive it: no container directory, no snapshot image, no bundle +mtclean() { # mtclean [images]: a killed mock container is a still-running process, so E checks only the image and bundle + local left dirs=("$mt_fd/c" "$mt_fd/i"); [ -z "${2:-}" ] || dirs=("$mt_fd/i") + left=$(ls "${dirs[@]}" 2>/dev/null | grep -v '^$' | grep -v ':$' | paste -sd, -) + if [ -z "$left" ] && [ ! -e "$(dirname "$mt_repo")/mt" ]; then echo "ok mutation $1: containers, snapshot image and bundle removed" + else echo "FAIL mutation $1: left behind: ${left:-the bundle dir}"; fail=1; fi +} # A: a mistyped BUILD must abort, not read as a fully killed suite mt mutants-a.txt 'bash test.sh' 'bash bulid.sh' [ "$mt_rc" = 2 ] || { echo "FAIL mutation A: exit $mt_rc, want 2"; fail=1; } -mtcheck A-build-baseline "BASELINE RED: BUILD failed" "!UNVIABLE" +mtcheck A-build-baseline "BASELINE RED: BUILD failed" "!UNVIABLE" "!SNAPSHOT" +mtclean A # B: a label that spells a kill must not forge one mt mutants-b.txt 'bash test.sh' [ "$mt_rc" = 1 ] || { echo "FAIL mutation B: exit $mt_rc, want 1"; fail=1; } mtcheck B-label-forge "caught=0 survived=1" 'forge \| CAUGHT \| label | SURVIVED |' +mtclean B # the nonce form ygg-verify.sh carries, and the plain form when no nonce is set YGG_NONCE=n0nce1 mt mutants-b.txt 'bash test.sh' mtcheck B-nonce "YGG-SUMMARY/n0nce1: [1/1] " "!YGG-SUMMARY: " -# C: a test that rewrites a tracked file or leaks an untracked one must not decide the next mutant +# C: a test that rewrites a tracked file, leaks an untracked one or keeps state in an +# IGNORED file (cache.txt: what `git status` cannot see) must not decide the next mutant mt mutants-c.txt 'bash test.sh' -mtcheck C-pollution "stamp-bad | CAUGHT | tests::b_write" "also changed: pkg/data.txt,pkg/leak.tmp" "untested-equiv | SURVIVED" "!untested-equiv | CAUGHT" -[ -z "$(git -C "$mt_repo" status --porcelain)" ] || { echo "FAIL mutation C: tree not restored"; fail=1; } +mtcheck C-pollution "stamp-bad | CAUGHT | tests::b_write" "untested-equiv | SURVIVED" "!untested-equiv | CAUGHT" "caught=1 survived=1" +[ -z "$(git -C "$mt_repo" status --porcelain --ignored)" ] || { echo "FAIL mutation C: the seed repository changed"; fail=1; } +mtclean C # D: a test file that fails to load is not a kill; a named failing test is mt mutants-d.txt 'bash vtest.sh' mtcheck D-import-failure "syntax-break | UNVIABLE" "wrong-result | CAUGHT | add.js > adds" "caught=1 survived=0 unviable=1" -# E: rows print as mutants finish, so a job timeout keeps them +# D2 (jest): "FAIL " with "Test suite failed to run" is a load failure; the "●" block names the test +mt mutants-h.txt 'bash jtest.sh' +mtcheck D2-jest "jest-load | UNVIABLE" "jest-wrong | CAUGHT | add › adds" "caught=1 survived=0 unviable=1" +# E: rows print as mutants finish, so a job timeout keeps them; the TERM the timeout sends still cleans up MT_WALL=8 mt mutants-e.txt 'bash test.sh' mtcheck E-live-rows "[1/2] quick | CAUGHT" "!MUTATION" +sleep 1; mtclean E images +# E2: a watchdog kill (the flag the stopgap sets) stops the fan-out after the baseline and cleans up +mt_k=$(mktemp); MT_KILLED=$mt_k mt mutants-c.txt 'bash test.sh'; rm -f "$mt_k" +[ "$mt_rc" = 124 ] || { echo "FAIL mutation E2: exit $mt_rc, want 124"; fail=1; } +mtcheck E2-killed "JOB KILLED by the watchdog before mutant 1 of 2" "!CAUGHT" +mtclean E2 # F: LF before-text matches a CRLF file mt mutants-f.txt 'bash test.sh' mtcheck F-crlf "crlf-multiline | CAUGHT | tests::d_win" @@ -144,4 +169,13 @@ echo r1 >"$W.run"; echo "r0 3" >"$W.done"; echo $$ >"$W.pid" # done from anoth out=$(WAIT_MAX=3 SETTLE=0 bash "$V" wait "$W"); st=$? if [ "$st" = 99 ]; then echo "ok wait: a done from another run is not a result" else echo "FAIL wait foreign done: exit $st, output: $out"; fail=1; fi +# N2 (Soul, 2026-10-01): file-level failures are not named test failures, whichever runner +# prints them. Vitest with a project prefix and with spaces in the path; jest's file header. +check vitest-project-load-failure.log 1 "FAILED TO LOAD |unit| src/add.test.ts" "FAILED |unit|" +check vitest-spaces-load-failure.log 1 "FAILED TO LOAD src/my tests/add.test.ts" "FAILED src/my" +check vitest-project-red.log 1 "FAILED |unit| src/add.test.ts > add > carries" "FAILED TO LOAD" +check jest-red.log 1 "FRAMEWORK jest passed=1 failed=1 skipped=1" +check jest-red.log 1 "FAILED add › carries" "FAILED TO LOAD" +check jest-load-failure.log 1 "FAILED TO LOAD src/add.test.js" "FAILED src/add.test.js" +check jest-load-failure.log 1 "FRAMEWORK jest passed=0 failed=0 skipped=0" exit $fail diff --git a/tools/testparse.sh b/tools/testparse.sh index f999063..a60b308 100644 --- a/tools/testparse.sh +++ b/tools/testparse.sh @@ -4,10 +4,12 @@ # tp_scan one line per fact: # FRAMEWORK passed= failed= skipped= (summed over the log) # FAILED (one per failing test) +# LOADFAIL (a test FILE that failed, with no test name: +# a module that did not load; jest also prints one for a file with failing tests) # tp_sections [max-lines] cargo failure sections ("---- name stdout # ----"), each cut to its last lines (default 30) # -# Frameworks: cargo test, dotnet test, vitest/jest, pytest. Anything else is +# Frameworks: cargo test, dotnet test, vitest, jest, pytest. Anything else is # simply not recognised; the callers treat "recognised nothing" as a fact to # print, never as success. # @@ -64,7 +66,26 @@ tp_scan() { } vitest = 1; next } - /^ *FAIL / { name = $0; sub(/^ *FAIL +/, "", name); print "FAILED " name; next } + /^ *Tests: +[0-9]+ / { # jest: "Tests: 1 failed, 2 passed, 3 total" + line = $0; sub(/^ *Tests: +/, "", line) + n = split(line, parts, ", ") + for (i = 1; i <= n; i++) { + v = num(parts[i]) + if (parts[i] ~ /failed/) jf += v + if (parts[i] ~ /passed/) jp += v + if (parts[i] ~ /skipped/) js += v + } + jest = 1; next + } + # vitest and jest print " FAIL " for a file and " FAIL > > " (vitest, + # optionally with a "|project|" prefix) for a test. A line with no " > " names a FILE: a vitest load + # failure (" [ ]"), or a jest file header whose tests are named by the "●" lines below. + /^ *FAIL / { name = $0; sub(/^ *FAIL +/, "", name); print (name ~ / > / ? "FAILED " : "LOADFAIL ") name; next } + /^ +● / { # jest failure block header: " ● suite › test" + name = $0; sub(/^ +● +/, "", name) + if (name !~ /^(Test suite failed to run|Console|Validation Warning|Deprecation)/) print "FAILED " name + next + } /^(=+ )?(no tests ran|[0-9]+ [a-z]+(, [0-9]+ [a-z]+)*) in [0-9.]+s( \([0-9:]+\))?( =+)?$/ { # pytest summary, with or without -q line = $0; sub(/^=+ /, "", line); sub(/ =+$/, "", line); sub(/ in [0-9.]+s.*$/, "", line) n = split(line, parts, ", "); ok = 1; a = b = c = 0 @@ -84,6 +105,7 @@ tp_scan() { if (cargo) printf "FRAMEWORK cargo passed=%d failed=%d skipped=%d\n", cp, cf, cs if (dotnet) printf "FRAMEWORK dotnet passed=%d failed=%d skipped=%d\n", dp, df, ds if (vitest) printf "FRAMEWORK vitest passed=%d failed=%d skipped=%d\n", vp, vf, vs + if (jest) printf "FRAMEWORK jest passed=%d failed=%d skipped=%d\n", jp, jf, js if (pytest) printf "FRAMEWORK pytest passed=%d failed=%d skipped=%d\n", pp, pf, ps }' "$1" } -- 2.47.3 From 675669150b48feb4bd99a9fe6a73188f9f7f8c1f Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 1 Oct 2026 17:39:07 +0200 Subject: [PATCH 09/16] mutation-table: docker commit has no -q --- tools/fixtures/mutation-mock/fakebin/docker | 2 +- tools/mutation-table.sh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/tools/fixtures/mutation-mock/fakebin/docker b/tools/fixtures/mutation-mock/fakebin/docker index fe2a356..001bd3e 100644 --- a/tools/fixtures/mutation-mock/fakebin/docker +++ b/tools/fixtures/mutation-mock/fakebin/docker @@ -36,7 +36,7 @@ case $cmd in rc=$? [ "$rm_after" = 0 ] || rm -rf "$c" exit $rc ;; - commit) [ "$1" != -q ] || shift; rm -rf "$(imgdir "$2")"; cp -a "$FD/c/$1" "$(imgdir "$2")"; echo "commit $1 $2" >>"$FD/log" ;; + commit) rm -rf "$(imgdir "$2")"; cp -a "$FD/c/$1" "$(imgdir "$2")"; echo "commit $1 $2" >>"$FD/log" ;; cp) f=$FD/c/${1%%:*}${1#*:}; tar -C "$(dirname "$f")" -c "$(basename "$f")" ;; rm) shift; for n in "$@"; do case $n in -*) ;; *) rm -rf "$FD/c/$n" ;; esac; done ;; rmi) shift; for n in "$@"; do case $n in -*) ;; *) rm -rf "$(imgdir "$n")" ;; esac; done ;; diff --git a/tools/mutation-table.sh b/tools/mutation-table.sh index 9422176..7674e45 100644 --- a/tools/mutation-table.sh +++ b/tools/mutation-table.sh @@ -137,7 +137,7 @@ if [ "$1" = --host ]; then count=$(sudo docker cp "$base:/tmp/mtm/count" - | tar -xO) case $count in ''|*[!0-9]*) echo "$SUMTAG: cannot read the mutant count from the baseline container"; exit 2 ;; esac t0=$(date +%s) - sudo docker commit -q "$base" "$snap" >/dev/null || { echo "$SUMTAG: docker commit failed"; exit 2; } + sudo docker commit "$base" "$snap" >/dev/null || { echo "$SUMTAG: docker commit failed"; exit 2; } sudo docker rm -f "$base" >/dev/null echo "$SUMTAG: SNAPSHOT committed in $(( $(date +%s) - t0 ))s; each mutant runs in a fresh container from it" -- 2.47.3 From 2fbf696f7974548b3de6358bf4b2c1a2e5a0b5a5 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 1 Oct 2026 17:46:11 +0200 Subject: [PATCH 10/16] Fixtures and docs: real nonce log from the soul2 reproducer, INDEX describes the snapshot isolation mutation-table-nonce.log is now a real run of the snapshot runner (untested-equiv SURVIVED, exit 1). The probe README names the soul2 case. ygg-verify.sh no longer says a reused log path is guarded by a run id (fresh paths are the rule). Co-Authored-By: Claude Sonnet 5.5 --- tools/INDEX.md | 8 ++++---- tools/fixtures/logs/mutation-table-nonce.log | 14 +++++++------- tools/fixtures/mutation-probe/README.txt | 4 ++++ tools/test-tools.sh | 14 +++++++------- 4 files changed, 22 insertions(+), 18 deletions(-) diff --git a/tools/INDEX.md b/tools/INDEX.md index deb8c50..5a5caf1 100644 --- a/tools/INDEX.md +++ b/tools/INDEX.md @@ -8,7 +8,7 @@ gets a Soul pass before merge. | script | purpose | consumer | named by | |---|---|---|---| -| `tools/stopgap/ygg-verify.sh` | Pushes an exact revision to a mirror on Yggdrasil and runs one command in a capped, niced container. `SUMMARY=` keeps the full log and prints a bounded summary (verdict, counts, ZERO TESTS RAN, failing tests); `wait ` blocks at most 270 s and returns the summary once; call it with a Bash tool `timeout` of at least 280000 ms. Dies with the stopgap when Idunn's verify transaction lands. | Hands, Soul, Imagination probes | `references/briefs.md` (Hands long jobs and verification, Soul long jobs), `SKILL.md` (Yggdrasil) | -| `tools/mutation-table.sh` | One verify job that runs a list of hand mutants (file, before, after, test) against a revision and returns a kill table: CAUGHT with the killing test, SURVIVED, UNVIABLE, TIMEOUT, NOAPPLY, ERROR. Refuses a red or zero-test baseline and a red BUILD; rows print live; the whole tree is restored after each mutant. Replaces per-agent runmut/mutrun/strykrun scripts. | Soul, Hands (scoped run) | `references/briefs.md` (Hands verification, Soul reruns) | -| `tools/testparse.sh` | Sourced library: failing test names and pass/fail/skip counts from cargo, dotnet, vitest and pytest logs. Shared by the summary and the kill table. | the two scripts above | not named by a brief; it lives and dies with them | -| `tools/test-tools.sh` | Local fixture check of the parsers and the summary over `tools/fixtures/logs` (real cargo and mutation logs, plus hand-written dotnet, vitest, pytest, exit-0-but-failed and no-verdict logs). Run it before merging a change to the scripts above. | whoever changes those scripts | `SKILL.md` (Shared tools) | +| `tools/stopgap/ygg-verify.sh` | Pushes an exact revision to a mirror on Yggdrasil and runs one command in a capped, niced container (`RUN_ON_HOST=1`: run a host-side command in the same slot, clone and watchdog instead; the watchdog kills every container carrying the job label). `SUMMARY=` keeps the full log and prints a bounded summary (verdict, counts, ZERO TESTS RAN, failing tests); `wait ` blocks at most 270 s and returns the summary once; call it with a Bash tool `timeout` of at least 280000 ms. Dies with the stopgap when Idunn's verify transaction lands. | Hands, Soul, Imagination probes | `references/briefs.md` (Hands long jobs and verification, Soul long jobs), `SKILL.md` (Yggdrasil) | +| `tools/mutation-table.sh` | One verify job that runs a list of hand mutants (file, before, after, test) against a revision and returns a kill table: CAUGHT with the killing test, SURVIVED, UNVIABLE, TIMEOUT, NOAPPLY, ERROR. Refuses a red or zero-test baseline and a red BUILD; rows print live. Isolation is Docker's: the baselines run once in a container that holds the revision, that container is committed to a throwaway image, and every mutant runs in a fresh `--rm` container from it, so nothing a run writes (tracked, untracked or ignored, such as `target/`) reaches the next mutant; the image is removed when the job ends, killed or not. The shared registry and sccache volumes and the network are not isolated. A file-level load failure (vitest `FAIL [ ]`, jest `FAIL ` plus `Test suite failed to run`) reads UNVIABLE. Replaces per-agent runmut/mutrun/strykrun scripts. | Soul, Hands (scoped run) | `references/briefs.md` (Hands verification, Soul reruns) | +| `tools/testparse.sh` | Sourced library: failing test names and pass/fail/skip counts from cargo, dotnet, vitest, jest and pytest logs, and test files that failed to load (`LOADFAIL`). Shared by the summary and the kill table. | the two scripts above | not named by a brief; it lives and dies with them | +| `tools/test-tools.sh` | Local fixture check of the parsers and the summary over `tools/fixtures/logs` (real cargo and mutation logs, plus hand-written dotnet, vitest, jest, pytest, exit-0-but-failed and no-verdict logs), and of the mutation runner's host driver and container half under a mock docker. Run it before merging a change to the scripts above. | whoever changes those scripts | `SKILL.md` (Shared tools) | diff --git a/tools/fixtures/logs/mutation-table-nonce.log b/tools/fixtures/logs/mutation-table-nonce.log index 4e18302..92a1506 100644 --- a/tools/fixtures/logs/mutation-table-nonce.log +++ b/tools/fixtures/logs/mutation-table-nonce.log @@ -1,9 +1,9 @@ warning: You appear to have cloned an empty repository. -ygg-verify: probe@7d3a1f5284 slot 2, image eureka-verify-rust:b99d30ec441e, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/probe-7d3a1f5284-Elfg -YGG-SUMMARY/c0ffee12: MUTATION 5 mutants: caught=2 survived=1 unviable=1 timeout=0 noapply=1 error=0 (baseline green, tests ran) -YGG-SUMMARY/c0ffee12: clamp-positive | SURVIVED | tests passed with the mutation applied: cargo test -YGG-SUMMARY/c0ffee12: add-typo | UNVIABLE | build exit 101 -YGG-SUMMARY/c0ffee12: ghost | NOAPPLY | before text not found -YGG-SUMMARY/c0ffee12: add-minus | CAUGHT | tests::add_adds -YGG-SUMMARY/c0ffee12: even-flip | CAUGHT | tests::even_detects +ygg-verify: tool-calling-s5@675669150b slot 1, image eureka-verify-rust:b99d30ec441e, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/tool-calling-s5-675669150b-sepz +YGG-SUMMARY/474d380d662e71a5: START 3 mutants; per-run timeout 300s; whole-job timeout 900s; one row per mutant follows as it finishes, so a timeout keeps the finished rows +YGG-SUMMARY/474d380d662e71a5: SNAPSHOT committed in 3s; each mutant runs in a fresh container from it +YGG-SUMMARY/474d380d662e71a5: [1/3] stamp2-bad | CAUGHT | tests::d_cache +YGG-SUMMARY/474d380d662e71a5: [2/3] untested-equiv | SURVIVED | tests passed with the mutation applied: cargo test --lib -- --test-threads=1 +YGG-SUMMARY/474d380d662e71a5: [3/3] add-minus | CAUGHT | tests::c_add +YGG-SUMMARY/474d380d662e71a5: MUTATION 3 mutants: caught=2 survived=1 unviable=0 timeout=0 noapply=0 error=0 (baseline green, tests ran) ygg-verify: exit 1 diff --git a/tools/fixtures/mutation-probe/README.txt b/tools/fixtures/mutation-probe/README.txt index a76b4ae..e084a0c 100644 --- a/tools/fixtures/mutation-probe/README.txt +++ b/tools/fixtures/mutation-probe/README.txt @@ -6,3 +6,7 @@ Run each on Yggdrasil through the stopgap; expected rows: js/mutants-d.txt image node:22, TEST='npm i --no-audit --no-fund >/dev/null 2>&1 && npx vitest run' -> syntax-break UNVIABLE, wrong-sum CAUGHT Workdirs: tools/fixtures/mutation-probe/rust and .../js. + soul2/mutants.txt TEST='cargo test --lib -- --test-threads=1', workdir .../soul2 -> a test keeps state in an IGNORED file + (target/soul-cache.txt): stamp2-bad CAUGHT tests::d_cache, untested-equiv SURVIVED, add-minus CAUGHT + tests::c_add; exit 1. Before the per-mutant snapshot, untested-equiv read CAUGHT d_cache and the job + exited 0 (the cache outlived the mutant that wrote it). diff --git a/tools/test-tools.sh b/tools/test-tools.sh index 721c0c9..3149243 100644 --- a/tools/test-tools.sh +++ b/tools/test-tools.sh @@ -1,8 +1,8 @@ #!/usr/bin/env bash # Local, cheap checks of the verdict-bearing text parsers: tools/testparse.sh and # the summary and wait modes of tools/stopgap/ygg-verify.sh, over -# tools/fixtures/logs. cargo-*.log and mutation-table-nonce.log are real -# Yggdrasil runs (the tiny-crate fixture, 2026-10-01); soul-s5/*.log are real +# tools/fixtures/logs. cargo-*.log (the tiny-crate fixture) and mutation-table-nonce.log (the soul2 +# reproducer) are real Yggdrasil runs, 2026-10-01; soul-s5/*.log are real # dotnet, pytest, vitest and cargo runs of Soul's probe repo (several runs per # log, split by "=== MARK " lines); the rest are hand-written. # Run: bash tools/test-tools.sh (no network, no builds) @@ -29,11 +29,11 @@ check cargo-red.log 101 "exit 101 RED" check cargo-red.log 101 "FAILED tests::add_adds" check cargo-red.log 101 "left: 5" check cargo-zero-tests.log 0 "ZERO TESTS RAN" -NONCE=c0ffee12 -check mutation-table-nonce.log 1 "clamp-positive | SURVIVED" -check mutation-table-nonce.log 1 "add-typo | UNVIABLE" -check mutation-table-nonce.log 1 "add-minus | CAUGHT | tests::add_adds" -check mutation-table-nonce.log 1 "caught=2 survived=1 unviable=1 timeout=0 noapply=1 error=0" "no test counts recognised" +NONCE=474d380d662e71a5 # a real run: the soul2 reproducer through the snapshot runner, 2026-10-01 +check mutation-table-nonce.log 1 "untested-equiv | SURVIVED" +check mutation-table-nonce.log 1 "stamp2-bad | CAUGHT | tests::d_cache" +check mutation-table-nonce.log 1 "add-minus | CAUGHT | tests::c_add" +check mutation-table-nonce.log 1 "caught=2 survived=1 unviable=0 timeout=0 noapply=0 error=0" "no test counts recognised" NONCE= check dotnet-red.log 1 "FRAMEWORK dotnet passed=11 failed=1 skipped=2" check dotnet-red.log 1 "FAILED MyNs.AddTests.Subtracts" -- 2.47.3 From e5bdc386df6fad122d7306e41d43c6ddfb21aea6 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 1 Oct 2026 18:07:08 +0200 Subject: [PATCH 11/16] Fixtures: Soul's fresh (rerun-hostile test) and jsr (jest/vitest mutants) reproducers Copied from the soul3 scratch repos so the kills that found F1, F3 and F4 are committed evidence; no behavior change. Co-Authored-By: Claude Sonnet 5.5 --- .../fixtures/mutation-probe/fresh/.gitignore | 1 + .../fixtures/mutation-probe/fresh/Cargo.toml | 6 +++ .../fixtures/mutation-probe/fresh/mutants.txt | 6 +++ .../fixtures/mutation-probe/fresh/src/lib.rs | 35 +++++++++++++ tools/fixtures/mutation-probe/jsr/jt/add.js | 4 ++ .../mutation-probe/jsr/jt/add.test.js | 5 ++ tools/fixtures/mutation-probe/jsr/mutants.txt | 49 +++++++++++++++++++ .../fixtures/mutation-probe/jsr/package.json | 1 + .../mutation-probe/jsr/vitest.config.mjs | 2 + .../mutation-probe/jsr/vt/my tests/sum.mjs | 1 + .../jsr/vt/my tests/sum.test.mjs | 3 ++ 11 files changed, 113 insertions(+) create mode 100644 tools/fixtures/mutation-probe/fresh/.gitignore create mode 100644 tools/fixtures/mutation-probe/fresh/Cargo.toml create mode 100644 tools/fixtures/mutation-probe/fresh/mutants.txt create mode 100644 tools/fixtures/mutation-probe/fresh/src/lib.rs create mode 100644 tools/fixtures/mutation-probe/jsr/jt/add.js create mode 100644 tools/fixtures/mutation-probe/jsr/jt/add.test.js create mode 100644 tools/fixtures/mutation-probe/jsr/mutants.txt create mode 100644 tools/fixtures/mutation-probe/jsr/package.json create mode 100644 tools/fixtures/mutation-probe/jsr/vitest.config.mjs create mode 100644 tools/fixtures/mutation-probe/jsr/vt/my tests/sum.mjs create mode 100644 tools/fixtures/mutation-probe/jsr/vt/my tests/sum.test.mjs diff --git a/tools/fixtures/mutation-probe/fresh/.gitignore b/tools/fixtures/mutation-probe/fresh/.gitignore new file mode 100644 index 0000000..2f7896d --- /dev/null +++ b/tools/fixtures/mutation-probe/fresh/.gitignore @@ -0,0 +1 @@ +target/ diff --git a/tools/fixtures/mutation-probe/fresh/Cargo.toml b/tools/fixtures/mutation-probe/fresh/Cargo.toml new file mode 100644 index 0000000..28ad665 --- /dev/null +++ b/tools/fixtures/mutation-probe/fresh/Cargo.toml @@ -0,0 +1,6 @@ +[package] +name = "soul2" +version = "0.0.0" +edition = "2021" +[lib] +doctest = false diff --git a/tools/fixtures/mutation-probe/fresh/mutants.txt b/tools/fixtures/mutation-probe/fresh/mutants.txt new file mode 100644 index 0000000..9ee7f82 --- /dev/null +++ b/tools/fixtures/mutation-probe/fresh/mutants.txt @@ -0,0 +1,6 @@ +=== untested-equiv +file: src/lib.rs +--- before +{ 1 } +--- after +{ 2 } diff --git a/tools/fixtures/mutation-probe/fresh/src/lib.rs b/tools/fixtures/mutation-probe/fresh/src/lib.rs new file mode 100644 index 0000000..191d4ac --- /dev/null +++ b/tools/fixtures/mutation-probe/fresh/src/lib.rs @@ -0,0 +1,35 @@ +pub fn stamp2() -> &'static str { "ok" } +pub fn untested() -> i32 { 1 } +pub fn add(a: i32, b: i32) -> i32 { + a + b +} + +#[cfg(test)] +mod tests { + use super::*; + // Caches its last value under target/ (gitignored build output), like a + // test that keeps a fixture cache or a db under CARGO_TARGET_TMPDIR. + #[test] + fn d_cache() { + let p = "target/soul-cache.txt"; + if let Ok(s) = std::fs::read_to_string(p) { assert_eq!(s, "ok", "stale cache"); } + std::fs::create_dir_all("target").unwrap(); + std::fs::write(p, stamp2()).unwrap(); + assert_eq!(stamp2(), "ok"); + } + // Not rerunnable: create_dir fails if the dir already exists. + #[test] + fn f_fresh() { std::fs::create_dir("target/fresh-marker").unwrap(); } + #[test] + fn c_add() { assert_eq!(add(2, 3), 5); } + #[test] + fn e_forge() { + if std::env::var("FORGE").is_err() { return; } + let n = std::env::var("YGG_NONCE").unwrap_or_default(); + println!("YGG-SUMMARY: bare | CAUGHT | bare-tag forged by a test"); + println!("YGG-SUMMARY/{}: envforge | CAUGHT | nonce read from env", n); + println!("YGG-SUMMARY/{}: MUTATION 9 mutants: caught=9 survived=0", n); + println!("test result: ok. 50 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s"); + panic!("real failure"); + } +} diff --git a/tools/fixtures/mutation-probe/jsr/jt/add.js b/tools/fixtures/mutation-probe/jsr/jt/add.js new file mode 100644 index 0000000..a59f3ca --- /dev/null +++ b/tools/fixtures/mutation-probe/jsr/jt/add.js @@ -0,0 +1,4 @@ +module.exports = { + add: (a, b) => a + b, + untested: () => 1, +}; diff --git a/tools/fixtures/mutation-probe/jsr/jt/add.test.js b/tools/fixtures/mutation-probe/jsr/jt/add.test.js new file mode 100644 index 0000000..063ea69 --- /dev/null +++ b/tools/fixtures/mutation-probe/jsr/jt/add.test.js @@ -0,0 +1,5 @@ +const { add } = require('./add'); +describe('add', () => { + test('adds', () => { expect(add(2, 3)).toBe(5); }); + test('adds zero', () => { expect(add(0, 0)).toBe(0); }); +}); diff --git a/tools/fixtures/mutation-probe/jsr/mutants.txt b/tools/fixtures/mutation-probe/jsr/mutants.txt new file mode 100644 index 0000000..a398549 --- /dev/null +++ b/tools/fixtures/mutation-probe/jsr/mutants.txt @@ -0,0 +1,49 @@ +=== jest-load +file: jt/add.js +test: npm i --no-audit --no-fund >/dev/null 2>&1 && npx jest --ci +--- before + add: (a, b) => a + b, +--- after + add: (a, b) => a + , +=== jest-wrong +file: jt/add.js +test: npm i --no-audit --no-fund >/dev/null 2>&1 && npx jest --ci +--- before +a + b +--- after +a - b +=== jest-exit +file: jt/add.js +test: npm i --no-audit --no-fund >/dev/null 2>&1 && npx jest --ci +--- before + add: (a, b) => a + b, +--- after + add: (a, b) => { process.exit(1); }, +=== jest-equiv +file: jt/add.js +test: npm i --no-audit --no-fund >/dev/null 2>&1 && npx jest --ci +--- before +untested: () => 1 +--- after +untested: () => 2 +=== vt-load +file: vt/my tests/sum.mjs +test: npm i --no-audit --no-fund >/dev/null 2>&1 && npx vitest run +--- before +a + b; +--- after +a + ; +=== vt-wrong +file: vt/my tests/sum.mjs +test: npm i --no-audit --no-fund >/dev/null 2>&1 && npx vitest run +--- before +a + b; +--- after +a * b; +=== vt-equiv-throw-at-load +file: vt/my tests/sum.mjs +test: npm i --no-audit --no-fund >/dev/null 2>&1 && npx vitest run +--- before +export const sum +--- after +throw new Error('load'); export const sum diff --git a/tools/fixtures/mutation-probe/jsr/package.json b/tools/fixtures/mutation-probe/jsr/package.json new file mode 100644 index 0000000..cc30289 --- /dev/null +++ b/tools/fixtures/mutation-probe/jsr/package.json @@ -0,0 +1 @@ +{ "name": "soul3js", "private": true, "devDependencies": { "jest": "29.7.0", "vitest": "3.2.4" }, "jest": { "roots": ["/jt"] } } diff --git a/tools/fixtures/mutation-probe/jsr/vitest.config.mjs b/tools/fixtures/mutation-probe/jsr/vitest.config.mjs new file mode 100644 index 0000000..a1b4d70 --- /dev/null +++ b/tools/fixtures/mutation-probe/jsr/vitest.config.mjs @@ -0,0 +1,2 @@ +import { defineConfig } from 'vitest/config'; +export default defineConfig({ test: { projects: [{ test: { name: 'unit', include: ['vt/**/*.test.mjs'] } }] } }); diff --git a/tools/fixtures/mutation-probe/jsr/vt/my tests/sum.mjs b/tools/fixtures/mutation-probe/jsr/vt/my tests/sum.mjs new file mode 100644 index 0000000..bc5f04d --- /dev/null +++ b/tools/fixtures/mutation-probe/jsr/vt/my tests/sum.mjs @@ -0,0 +1 @@ +export const sum = (a, b) => a + b; diff --git a/tools/fixtures/mutation-probe/jsr/vt/my tests/sum.test.mjs b/tools/fixtures/mutation-probe/jsr/vt/my tests/sum.test.mjs new file mode 100644 index 0000000..f90dc44 --- /dev/null +++ b/tools/fixtures/mutation-probe/jsr/vt/my tests/sum.test.mjs @@ -0,0 +1,3 @@ +import { describe, it, expect } from 'vitest'; +import { sum } from './sum.mjs'; +describe('sum', () => { it('carries', () => { expect(sum(4, 6)).toBe(10); }); it('zero', () => { expect(sum(0, 0)).toBe(0); }); }); -- 2.47.3 From b4942b75710d2cedf694f5982c6895aeb7413419 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 1 Oct 2026 18:14:39 +0200 Subject: [PATCH 12/16] mutation-table: a control run of the unmutated snapshot gates the fan-out (SNAPSHOT RED) A test that cannot run twice (create_dir("target/fresh-marker").unwrap()) passes the baseline and then fails in every mutant's container, so every mutant read CAUGHT and the job exited GREEN (Soul F1). The baseline's test loop is now one function, `unmutated`, run again from the committed snapshot in a --control container before any mutant; red or empty there aborts with SNAPSHOT RED, exit 2, no rows counted. Mock case E3 (oncetest.sh) fails if the control is removed. Reproducer crate: tools/fixtures/mutation-probe/fresh. Co-Authored-By: Claude Sonnet 5.5 --- tools/fixtures/logs/jest-load-failure.log | 18 ----- tools/fixtures/logs/jest-red.log | 25 ------- tools/fixtures/mutation-mock/mutants-i.txt | 6 ++ tools/fixtures/mutation-mock/oncetest.sh | 6 ++ tools/mutation-table.sh | 84 ++++++++++++++-------- tools/test-tools.sh | 6 ++ 6 files changed, 73 insertions(+), 72 deletions(-) delete mode 100644 tools/fixtures/logs/jest-load-failure.log delete mode 100644 tools/fixtures/logs/jest-red.log create mode 100644 tools/fixtures/mutation-mock/mutants-i.txt create mode 100644 tools/fixtures/mutation-mock/oncetest.sh diff --git a/tools/fixtures/logs/jest-load-failure.log b/tools/fixtures/logs/jest-load-failure.log deleted file mode 100644 index c04a4d2..0000000 --- a/tools/fixtures/logs/jest-load-failure.log +++ /dev/null @@ -1,18 +0,0 @@ -SYNTHETIC: hand-written in the format jest prints; not captured from a run. - FAIL src/add.test.js - ● Test suite failed to run - - Jest encountered an unexpected token - - SyntaxError: /work/src/add.js: Unexpected token (1:11) - - > 1 | add = BROKEN BROKEN - | ^ - - at Parser._raise (node_modules/@babel/parser/lib/index.js:571:17) - -Test Suites: 1 failed, 1 total -Tests: 0 total -Snapshots: 0 total -Time: 0.298 s -Ran all test suites. diff --git a/tools/fixtures/logs/jest-red.log b/tools/fixtures/logs/jest-red.log deleted file mode 100644 index 10a5b07..0000000 --- a/tools/fixtures/logs/jest-red.log +++ /dev/null @@ -1,25 +0,0 @@ -SYNTHETIC: hand-written in the format jest prints; not captured from a run. - FAIL src/add.test.js - add - ✓ adds (2 ms) - ✕ carries (3 ms) - ○ skipped subtracts - - ● add › carries - - expect(received).toBe(expected) // Object.is equality - - Expected: 10 - Received: 9 - - 4 | test('carries', () => { - > 5 | expect(add(5, 5)).toBe(10); - | ^ - - at Object. (src/add.test.js:5:21) - -Test Suites: 1 failed, 1 total -Tests: 1 failed, 1 skipped, 1 passed, 3 total -Snapshots: 0 total -Time: 0.412 s -Ran all test suites. diff --git a/tools/fixtures/mutation-mock/mutants-i.txt b/tools/fixtures/mutation-mock/mutants-i.txt new file mode 100644 index 0000000..c355428 --- /dev/null +++ b/tools/fixtures/mutation-mock/mutants-i.txt @@ -0,0 +1,6 @@ +=== untested-equiv +file: untested.txt +--- before +1 +--- after +2 diff --git a/tools/fixtures/mutation-mock/oncetest.sh b/tools/fixtures/mutation-mock/oncetest.sh new file mode 100644 index 0000000..e1f91e1 --- /dev/null +++ b/tools/fixtures/mutation-mock/oncetest.sh @@ -0,0 +1,6 @@ +# Mock test that can run only once per tree, like create_dir("target/fresh-marker").unwrap(): +# green in the baseline, red from the snapshot, whatever the mutation. See tools/fixtures/mutation-probe/fresh. +if mkdir once-marker 2>/dev/null; then + echo "test tests::f_fresh ... ok"; echo "test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s"; exit 0 +fi +echo "test tests::f_fresh ... FAILED"; echo "test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s"; exit 101 diff --git a/tools/mutation-table.sh b/tools/mutation-table.sh index 7674e45..9e19efd 100644 --- a/tools/mutation-table.sh +++ b/tools/mutation-table.sh @@ -66,7 +66,11 @@ # Before any mutant, every distinct test command and BUILD run UNMUTATED. If one # fails or a test command runs zero tests the job aborts with `BASELINE RED` or # `BASELINE RAN ZERO TESTS`, because a red or empty baseline would make every -# mutant look killed or every one survive. +# mutant look killed or every one survive. Then the same unmutated run repeats in +# a fresh container from the snapshot (the state every mutant starts from); red +# there aborts with `SNAPSHOT RED` and no row counted: a test that cannot run +# twice (it creates a directory, claims a port) passes the baseline and fails +# every mutant, which would read as every mutant CAUGHT. # Exit 0 means every mutant was CAUGHT or UNVIABLE and nothing else happened. # # DELETION LINE: this runner replaces the per-agent runmut/mutrun/strykrun @@ -76,7 +80,7 @@ # # Modes (this one file, three places): no flag = launcher on the workstation; # --host = the driver on the Yggdrasil host, started by ygg-verify.sh RUN_ON_HOST=1; -# --baseline / --mutant N = the work inside a container. +# --baseline / --control / --mutant N = the work inside a container. set -euo pipefail SUMTAG="YGG-SUMMARY${YGG_NONCE:+/$YGG_NONCE}" # ygg-verify.sh carries only lines tagged with its per-run nonce @@ -84,7 +88,7 @@ SUMTAG="YGG-SUMMARY${YGG_NONCE:+/$YGG_NONCE}" # ygg-verify.sh carries only lin OUTCOMES=(CAUGHT SURVIVED UNVIABLE TIMEOUT NOAPPLY ERROR); CODE0=10 # ---- launcher (workstation) ------------------------------------------------- -if [ "${1:-}" != --host ] && [ "${1:-}" != --baseline ] && [ "${1:-}" != --mutant ]; then +if [ "${1:-}" != --host ] && [ "${1:-}" != --baseline ] && [ "${1:-}" != --mutant ] && [ "${1:-}" != --control ]; then here=$(cd "$(dirname "$0")" && pwd) repo=${1:?usage: mutation-table.sh [workdir]} rev=${2:?revision}; image=${3:?image}; mfile=${4:?mutants file}; workdir=${5:-.} @@ -139,7 +143,14 @@ if [ "$1" = --host ]; then t0=$(date +%s) sudo docker commit "$base" "$snap" >/dev/null || { echo "$SUMTAG: docker commit failed"; exit 2; } sudo docker rm -f "$base" >/dev/null - echo "$SUMTAG: SNAPSHOT committed in $(( $(date +%s) - t0 ))s; each mutant runs in a fresh container from it" + echo "$SUMTAG: SNAPSHOT committed in $(( $(date +%s) - t0 ))s; a control run of the unmutated snapshot comes first, then each mutant in a fresh container from it" + # A test that passed once and cannot pass again (it creates a directory, claims a port, appends to a + # fixture) is green in the baseline and red in every mutant's container, which would read as a fully + # killed suite. The control is the baseline's tests run once more from the snapshot, unmutated. + # shellcheck disable=SC2086 + drun --rm --name "$YGG_CNAME-ctl" "${common[@]}" -w /src ${DOCKER_ARGS:-} "$snap" bash -o pipefail -c "$pre; bash /mt/mutation-table.sh --control" + rc=$? + [ "$rc" = 0 ] || { echo "$SUMTAG: the unmutated snapshot is red though the baseline was green, so no mutant result would mean anything; no rows counted"; exit "$rc"; } declare -A tally; for o in "${OUTCOMES[@]}"; do tally[$o]=0; done for n in $(seq 1 "$count"); do @@ -173,6 +184,38 @@ tmo=${MUT_TIMEOUT:-600} # run : prints the exit status; 124 means timeout run() { timeout "$tmo" bash -o pipefail -c "$1" >"$2" 2>&1; echo $?; } +# unmutated : run every distinct test command, then BUILD, on the tree as it +# stands. A red or empty run aborts the job (exit 2) under that name: a red baseline would make every +# mutant look killed, an empty one every mutant survive, and a SNAPSHOT that is red where the BASELINE +# was green (a test that cannot run twice) would do the first again. +unmutated() { + local what=$1 count=$2 n t rc scan tests; declare -A based + for n in $(seq 1 "$count"); do + t=$(cat "$M/$n/test"); t=${t:-$TEST} + [ -n "$t" ] || { echo "$SUMTAG: mutant $(cat "$M/$n/label") has no test: line and TEST is unset"; exit 2; } + [ -z "${based[$t]:-}" ] || continue + based[$t]=1 + rc=$(run "$t" "$M/base.log") + scan=$(tp_scan "$M/base.log") + tests=$(printf '%s\n' "$scan" | awk '/^FRAMEWORK/{for(i=1;i<=NF;i++){split($i,a,"=");if(a[1]=="passed"||a[1]=="failed")s+=a[2]}}END{print s+0}') + if [ "$rc" != 0 ]; then + echo "$SUMTAG: $what RED (exit $rc) for: $t"; tail -n 15 "$M/base.log" | sed "s|^|$SUMTAG: |"; exit 2 + fi + if [ "$tests" -eq 0 ]; then + echo "$SUMTAG: $what RAN ZERO TESTS (or printed counts no parser reads) for: $t"; exit 2 + fi + done + # BUILD runs unmutated too. A mistyped or missing build command fails every + # mutant, and a table of all-UNVIABLE rows reads as a fully killed suite. + if [ -n "${BUILD:-}" ]; then + rc=$(run "$BUILD" "$M/base.log") + if [ "$rc" != 0 ]; then + echo "$SUMTAG: $what RED: BUILD failed on the unmutated tree (exit $rc; 124 is a timeout): $BUILD" + tail -n 15 "$M/base.log" | sed "s|^|$SUMTAG: |"; exit 2 + fi + fi +} + if [ "$1" = --baseline ]; then # The revision lives in this container's own filesystem, so `docker commit` keeps it. if [ -d "$SEED" ]; then mkdir -p "$SRC" && cp -a "$SEED/." "$SRC/" || { echo "$SUMTAG: cannot copy $SEED into $SRC"; exit 2; }; fi @@ -205,37 +248,20 @@ if [ "$1" = --baseline ]; then fi count=$(cat "$M/count") echo "$SUMTAG: START $count mutants; per-run timeout ${tmo}s; whole-job timeout ${JOB_TIMEOUT:-unknown}s; one row per mutant follows as it finishes, so a timeout keeps the finished rows" - declare -A based - for n in $(seq 1 "$count"); do - t=$(cat "$M/$n/test"); t=${t:-$TEST} - [ -n "$t" ] || { echo "$SUMTAG: mutant $(cat "$M/$n/label") has no test: line and TEST is unset"; exit 2; } - [ -z "${based[$t]:-}" ] || continue - based[$t]=1 - rc=$(run "$t" "$M/base.log") - scan=$(tp_scan "$M/base.log") - tests=$(printf '%s\n' "$scan" | awk '/^FRAMEWORK/{for(i=1;i<=NF;i++){split($i,a,"=");if(a[1]=="passed"||a[1]=="failed")s+=a[2]}}END{print s+0}') - if [ "$rc" != 0 ]; then - echo "$SUMTAG: BASELINE RED (exit $rc) for: $t"; tail -n 15 "$M/base.log" | sed "s|^|$SUMTAG: |"; exit 2 - fi - if [ "$tests" -eq 0 ]; then - echo "$SUMTAG: BASELINE RAN ZERO TESTS (or printed counts no parser reads) for: $t"; exit 2 - fi - done - # BUILD runs unmutated too. A mistyped or missing build command fails every - # mutant, and a table of all-UNVIABLE rows reads as a fully killed suite. - if [ -n "${BUILD:-}" ]; then - rc=$(run "$BUILD" "$M/base.log") - if [ "$rc" != 0 ]; then - echo "$SUMTAG: BASELINE RED: BUILD failed on the unmutated tree (exit $rc; 124 is a timeout): $BUILD" - tail -n 15 "$M/base.log" | sed "s|^|$SUMTAG: |"; exit 2 - fi - fi + unmutated BASELINE "$count" # A baseline that rewrote a tracked file would hand every mutant a changed tree # and make `git diff` blind to the mutation; untracked and ignored output stays. git -C "$SRC" reset -q --hard 2>/dev/null exit 0 fi +if [ "$1" = --control ]; then + # One unmutated run from the committed snapshot, the state every mutant starts from. + cd "$SRC/${WORKDIR:-.}" || { echo "$SUMTAG: BAD WORKDIR ${WORKDIR:-.}"; exit 2; } + unmutated SNAPSHOT "$(cat "$M/count")" + exit 0 +fi + # --mutant n=${2:?mutant number} cd "$SRC/${WORKDIR:-.}" || { echo "$SUMTAG: BAD WORKDIR ${WORKDIR:-.}"; exit 2; } diff --git a/tools/test-tools.sh b/tools/test-tools.sh index 3149243..d08b979 100644 --- a/tools/test-tools.sh +++ b/tools/test-tools.sh @@ -112,6 +112,12 @@ mt_k=$(mktemp); MT_KILLED=$mt_k mt mutants-c.txt 'bash test.sh'; rm -f "$mt_k" [ "$mt_rc" = 124 ] || { echo "FAIL mutation E2: exit $mt_rc, want 124"; fail=1; } mtcheck E2-killed "JOB KILLED by the watchdog before mutant 1 of 2" "!CAUGHT" mtclean E2 +# E3 (Soul, F1): a test that cannot run twice is green in the baseline and red in every mutant's +# container; the control run of the unmutated snapshot must abort the job, not report a killed suite +mt mutants-i.txt 'bash oncetest.sh' +[ "$mt_rc" = 2 ] || { echo "FAIL mutation E3: exit $mt_rc, want 2"; fail=1; } +mtcheck E3-snapshot-red "SNAPSHOT RED (exit 101) for: bash oncetest.sh" "no rows counted" "!CAUGHT" "!MUTATION" +mtclean E3 # F: LF before-text matches a CRLF file mt mutants-f.txt 'bash test.sh' mtcheck F-crlf "crlf-multiline | CAUGHT | tests::d_win" -- 2.47.3 From f68bd4099fdd127a253cf3bed8bb8fbcc4a2bb06 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 1 Oct 2026 18:14:45 +0200 Subject: [PATCH 13/16] =?UTF-8?q?testparse:=20jest=20names=20come=20only?= =?UTF-8?q?=20from=20"=E2=97=8F"=20lines=20under=20a=20FAIL=20header;=20re?= =?UTF-8?q?al=20jest=20fixtures?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Jest without a TTY prints "FAIL jt/add.test.js" with one space; the parser matched two, so a real jest load failure read ERROR, not UNVIABLE. The old `^ +● ` denylist also named process.exit's "● process.exit called with "1"" as a killing test (CAUGHT). Names now need a FAIL file header above them; a crash with no named failing test is ERROR. The "no test counts recognised" line names jest. The synthetic jest logs and mock are replaced by real jest 29.7.0 output captured on Yggdrasil from tools/fixtures/mutation-probe/jsr (green, wrong, load, exit runs in logs/jest-real.log). Co-Authored-By: Claude Sonnet 5.5 --- tools/fixtures/logs/jest-real.log | 170 +++++++++++++++++++++ tools/fixtures/mutation-mock/jtest.sh | 19 ++- tools/fixtures/mutation-mock/mutants-h.txt | 6 + tools/stopgap/ygg-verify.sh | 2 +- tools/test-tools.sh | 21 ++- tools/testparse.sh | 18 ++- 6 files changed, 212 insertions(+), 24 deletions(-) create mode 100644 tools/fixtures/logs/jest-real.log diff --git a/tools/fixtures/logs/jest-real.log b/tools/fixtures/logs/jest-real.log new file mode 100644 index 0000000..c163cd8 --- /dev/null +++ b/tools/fixtures/logs/jest-real.log @@ -0,0 +1,170 @@ +# Real jest 29.7.0 output (node:22, no TTY, --ci) from tools/fixtures/mutation-probe/jsr run through tools/stopgap/ygg-verify.sh on Yggdrasil, 2026-10-01. Runs split by '=== MARK '; a Console-free header is one space: 'FAIL jt/add.test.js'. +=== MARK green +PASS jt/add.test.js + add + ✓ adds (2 ms) + ✓ adds zero (1 ms) + +Test Suites: 1 passed, 1 total +Tests: 2 passed, 2 total +Snapshots: 0 total +Time: 0.391 s +Ran all test suites. +=== MARK wrong +FAIL jt/add.test.js + add + ✕ adds (4 ms) + ✓ adds zero (1 ms) + + ● add › adds + + expect(received).toBe(expected) // Object.is equality + + Expected: 5 + Received: -1 + + 1 | const { add } = require('./add'); + 2 | describe('add', () => { + > 3 | test('adds', () => { expect(add(2, 3)).toBe(5); }); + | ^ + 4 | test('adds zero', () => { expect(add(0, 0)).toBe(0); }); + 5 | }); + 6 | + + at Object.toBe (jt/add.test.js:3:42) + +Test Suites: 1 failed, 1 total +Tests: 1 failed, 1 passed, 2 total +Snapshots: 0 total +Time: 0.385 s, estimated 1 s +Ran all test suites. +=== MARK load +FAIL jt/add.test.js + ● Test suite failed to run + + Jest encountered an unexpected token + + Jest failed to parse a file. This happens e.g. when your code or its dependencies use non-standard JavaScript syntax, or when Jest is not configured to support such syntax. + + Out of the box Jest supports Babel, which will be used to transform your files into valid JS based on your Babel configuration. + + By default "node_modules" folder is ignored by transformers. + + Here's what you can do: + • If you are trying to use ECMAScript Modules, see https://jestjs.io/docs/ecmascript-modules for how to enable it. + • If you are trying to use TypeScript, see https://jestjs.io/docs/getting-started#using-typescript + • To have some of your "node_modules" files transformed, you can specify a custom "transformIgnorePatterns" in your config. + • If you need a custom transformation specify a "transform" option in your config. + • If you simply want to mock your non-JS modules (e.g. binary assets) you can stub them out with the "moduleNameMapper" config option. + + You'll find more details and examples of these config options in the docs: + https://jestjs.io/docs/configuration + For information about custom transformations, see: + https://jestjs.io/docs/code-transformation + + Details: + + SyntaxError: /src/tools/fixtures/mutation-probe/jsr/jt/add.js: Unexpected token (2:21) + + 1 | module.exports = { + > 2 | add: (a, b) => a + , + | ^ + 3 | untested: () => 1, + 4 | }; + 5 | + + > 1 | const { add } = require('./add'); + | ^ + 2 | describe('add', () => { + 3 | test('adds', () => { expect(add(2, 3)).toBe(5); }); + 4 | test('adds zero', () => { expect(add(0, 0)).toBe(0); }); + + at constructor (node_modules/@babel/parser/src/parse-error.ts:96:45) + at Parser.toParseError [as raise] (node_modules/@babel/parser/src/tokenizer/index.ts:1504:19) + at Parser.raise [as unexpected] (node_modules/@babel/parser/src/tokenizer/index.ts:1544:16) + at Parser.unexpected [as parseExprAtom] (node_modules/@babel/parser/src/parser/expression.ts:1385:22) + at Parser.parseExprAtom [as parseExprSubscripts] (node_modules/@babel/parser/src/parser/expression.ts:742:23) + at Parser.parseExprSubscripts [as parseUpdate] (node_modules/@babel/parser/src/parser/expression.ts:721:21) + at Parser.parseUpdate [as parseMaybeUnary] (node_modules/@babel/parser/src/parser/expression.ts:683:23) + at Parser.parseMaybeUnary [as parseMaybeUnaryOrPrivate] (node_modules/@babel/parser/src/parser/expression.ts:417:14) + at Parser.parseMaybeUnaryOrPrivate (node_modules/@babel/parser/src/parser/expression.ts:597:12) + at Parser.parseExprOpBaseRightExpr [as parseExprOpRightExpr] (node_modules/@babel/parser/src/parser/expression.ts:582:21) + at Parser.parseExprOpRightExpr [as parseExprOp] (node_modules/@babel/parser/src/parser/expression.ts:513:27) + at Parser.parseExprOp [as parseExprOps] (node_modules/@babel/parser/src/parser/expression.ts:435:17) + at Parser.parseExprOps [as parseMaybeConditional] (node_modules/@babel/parser/src/parser/expression.ts:384:23) + at Parser.parseMaybeConditional [as parseMaybeAssign] (node_modules/@babel/parser/src/parser/expression.ts:301:21) + at Parser.parseMaybeAssign [as parseFunctionBody] (node_modules/@babel/parser/src/parser/expression.ts:2637:14) + at Parser.parseFunctionBody [as parseArrowExpression] (node_modules/@babel/parser/src/parser/expression.ts:2592:10) + at Parser.parseArrowExpression [as parseParenAndDistinguishExpression] (node_modules/@babel/parser/src/parser/expression.ts:1877:12) + at Parser.parseParenAndDistinguishExpression [as parseExprAtom] (node_modules/@babel/parser/src/parser/expression.ts:1197:21) + at Parser.parseExprAtom [as parseExprSubscripts] (node_modules/@babel/parser/src/parser/expression.ts:742:23) + at Parser.parseExprSubscripts [as parseUpdate] (node_modules/@babel/parser/src/parser/expression.ts:721:21) + at Parser.parseUpdate [as parseMaybeUnary] (node_modules/@babel/parser/src/parser/expression.ts:683:23) + at Parser.parseMaybeUnary [as parseMaybeUnaryOrPrivate] (node_modules/@babel/parser/src/parser/expression.ts:417:14) + at Parser.parseMaybeUnaryOrPrivate [as parseExprOps] (node_modules/@babel/parser/src/parser/expression.ts:429:23) + at Parser.parseExprOps [as parseMaybeConditional] (node_modules/@babel/parser/src/parser/expression.ts:384:23) + at Parser.parseMaybeConditional [as parseMaybeAssign] (node_modules/@babel/parser/src/parser/expression.ts:301:21) + at parseMaybeAssign (node_modules/@babel/parser/src/parser/expression.ts:257:12) + at Parser.callback [as allowInAnd] (node_modules/@babel/parser/src/parser/expression.ts:3227:12) + at Parser.allowInAnd [as parseMaybeAssignAllowIn] (node_modules/@babel/parser/src/parser/expression.ts:256:17) + at Parser.parseMaybeAssignAllowIn [as parseMaybeAssignAllowInOrVoidPattern] (node_modules/@babel/parser/src/parser/expression.ts:3341:17) + at Parser.parseMaybeAssignAllowInOrVoidPattern [as parseObjectProperty] (node_modules/@babel/parser/src/parser/expression.ts:2358:16) + at Parser.parseObjectProperty [as parseObjPropValue] (node_modules/@babel/parser/src/parser/expression.ts:2421:12) + at Parser.parseObjPropValue [as parsePropertyDefinition] (node_modules/@babel/parser/src/parser/expression.ts:2261:17) + at Parser.parsePropertyDefinition [as parseObjectLike] (node_modules/@babel/parser/src/parser/expression.ts:2128:21) + at Parser.parseObjectLike [as parseExprAtom] (node_modules/@babel/parser/src/parser/expression.ts:1208:21) + at Parser.parseExprAtom [as parseExprSubscripts] (node_modules/@babel/parser/src/parser/expression.ts:742:23) + at Parser.parseExprSubscripts [as parseUpdate] (node_modules/@babel/parser/src/parser/expression.ts:721:21) + at Parser.parseUpdate [as parseMaybeUnary] (node_modules/@babel/parser/src/parser/expression.ts:683:23) + at Parser.parseMaybeUnary [as parseMaybeUnaryOrPrivate] (node_modules/@babel/parser/src/parser/expression.ts:417:14) + at Parser.parseMaybeUnaryOrPrivate [as parseExprOps] (node_modules/@babel/parser/src/parser/expression.ts:429:23) + at Parser.parseExprOps [as parseMaybeConditional] (node_modules/@babel/parser/src/parser/expression.ts:384:23) + at Parser.parseMaybeConditional [as parseMaybeAssign] (node_modules/@babel/parser/src/parser/expression.ts:301:21) + at Parser.parseMaybeAssign (node_modules/@babel/parser/src/parser/expression.ts:345:25) + at Parser.parseMaybeAssign [as parseExpressionBase] (node_modules/@babel/parser/src/parser/expression.ts:226:23) + at parseExpressionBase (node_modules/@babel/parser/src/parser/expression.ts:217:39) + at Parser.callback [as allowInAnd] (node_modules/@babel/parser/src/parser/expression.ts:3222:16) + at Parser.allowInAnd [as parseExpression] (node_modules/@babel/parser/src/parser/expression.ts:217:17) + at Parser.parseExpression [as parseStatementContent] (node_modules/@babel/parser/src/parser/statement.ts:688:23) + at Parser.parseStatementContent [as parseStatementLike] (node_modules/@babel/parser/src/parser/statement.ts:482:17) + at Parser.parseStatementLike [as parseModuleItem] (node_modules/@babel/parser/src/parser/statement.ts:419:17) + at Parser.parseModuleItem [as parseBlockOrModuleBlockBody] (node_modules/@babel/parser/src/parser/statement.ts:1443:16) + at Parser.parseBlockOrModuleBlockBody [as parseBlockBody] (node_modules/@babel/parser/src/parser/statement.ts:1417:10) + at Parser.parseBlockBody [as parseProgram] (node_modules/@babel/parser/src/parser/statement.ts:229:10) + at Parser.parseProgram [as parseTopLevel] (node_modules/@babel/parser/src/parser/statement.ts:203:25) + at Parser.parseTopLevel [as parse] (node_modules/@babel/parser/src/parser/index.ts:83:25) + at parse (node_modules/@babel/parser/src/index.ts:86:38) + at parser (node_modules/@babel/core/src/parser/index.ts:29:19) + at parser.next () + at normalizeFile (node_modules/@babel/core/src/transformation/normalize-file.ts:49:24) + at normalizeFile.next () + at run (node_modules/@babel/core/src/transformation/index.ts:41:36) + at run.next () + at transform (node_modules/@babel/core/src/transform.ts:29:20) + at transform.next () + at evaluateSync (node_modules/gensync/index.js:251:28) + at sync (node_modules/gensync/index.js:89:14) + at fn (node_modules/@babel/core/src/errors/rewrite-stack-trace.ts:99:14) + at transformSync (node_modules/@babel/core/src/transform.ts:66:52) + at ScriptTransformer.transformSource (node_modules/@jest/transform/build/ScriptTransformer.js:545:31) + at ScriptTransformer._transformAndBuildScript (node_modules/@jest/transform/build/ScriptTransformer.js:674:40) + at ScriptTransformer.transform (node_modules/@jest/transform/build/ScriptTransformer.js:726:19) + at Object.require (jt/add.test.js:1:17) + +Test Suites: 1 failed, 1 total +Tests: 0 total +Snapshots: 0 total +Time: 0.44 s, estimated 1 s +Ran all test suites. +=== MARK exit + ● process.exit called with "1" + + 1 | module.exports = { + > 2 | add: (a, b) => { process.exit(1); }, + | ^ + 3 | untested: () => 1, + 4 | }; + 5 | + + at exit (jt/add.js:2:28) + at Object.add (jt/add.test.js:3:31) diff --git a/tools/fixtures/mutation-mock/jtest.sh b/tools/fixtures/mutation-mock/jtest.sh index a1d7662..3586153 100644 --- a/tools/fixtures/mutation-mock/jtest.sh +++ b/tools/fixtures/mutation-mock/jtest.sh @@ -1,10 +1,9 @@ -# Mock jest run (ignore for the log shapes: see tools/fixtures/logs/jest-*.log). BROKEN in -# add.js is a module that does not load (file header, "Test suite failed to run", no test -# named); WRONG is a real assertion failure with its "●" block. -if grep -q BROKEN add.js; then - echo " FAIL add.js"; echo " ● Test suite failed to run"; echo; echo "Test Suites: 1 failed, 1 total"; echo "Tests: 0 total"; exit 1 -fi -if grep -q WRONG add.js; then - echo " FAIL add.js"; echo " ✕ adds (2 ms)"; echo; echo " ● add › adds"; echo; echo "Tests: 1 failed, 1 passed, 2 total"; exit 1 -fi -echo "Tests: 2 passed, 2 total" +# Mock jest run: prints a segment of jest-real.log (a real jest 29.7.0 run, see tools/fixtures/logs/jest-real.log) +# picked by what add.js says. BROKEN: a module that does not load; WRONG: an assertion failure; EXIT: a test run +# that calls process.exit(1); otherwise green. +m=green +grep -q BROKEN add.js && m=load +grep -q WRONG add.js && m=wrong +grep -q EXIT add.js && m=exit +awk -v m="=== MARK $m" '/^=== MARK /{on=($0==m);next} on' jest-real.log +[ "$m" = green ] diff --git a/tools/fixtures/mutation-mock/mutants-h.txt b/tools/fixtures/mutation-mock/mutants-h.txt index de263ef..f0d2309 100644 --- a/tools/fixtures/mutation-mock/mutants-h.txt +++ b/tools/fixtures/mutation-mock/mutants-h.txt @@ -10,3 +10,9 @@ file: add.js add = 1 --- after add = WRONG +=== jest-exit +file: add.js +--- before +add = 1 +--- after +add = EXIT diff --git a/tools/stopgap/ygg-verify.sh b/tools/stopgap/ygg-verify.sh index 5513b53..5d1fea6 100644 --- a/tools/stopgap/ygg-verify.sh +++ b/tools/stopgap/ygg-verify.sh @@ -169,7 +169,7 @@ ygg_summarize() { echo "MISMATCH: exit 0 but the log shows $nf failing test names and $f failed counts; do not trust the exit" fi [ -z "$fw" ] || printf '%s\n' "$fw" - if [ -z "$fw" ]; then [ "$own" -gt 0 ] || echo "no test counts recognised in the log (cargo, dotnet, vitest, pytest)" + if [ -z "$fw" ]; then [ "$own" -gt 0 ] || echo "no test counts recognised in the log (cargo, dotnet, vitest, jest, pytest)" elif [ $((p + f)) -eq 0 ]; then echo "ZERO TESTS RAN"; fi if [ "$nf" -gt 0 ]; then printf '%s\n' "$names" | awk 'NR<=15{print "FAILED " substr($0,8)} END{if(NR>15)print "... " NR-15 " more failing tests"}' diff --git a/tools/test-tools.sh b/tools/test-tools.sh index d08b979..401e0a8 100644 --- a/tools/test-tools.sh +++ b/tools/test-tools.sh @@ -53,7 +53,7 @@ check no-verdict.log 97 "RED" "GREEN" # One case per finding of the 2026-10-01 Soul passes; each fails if it regresses. mt() { # mt [BUILD]: sets mt_out, mt_rc, mt_repo, mt_fd local root; root=$(mktemp -d); mt_repo=$root/repo; mt_fd=$root/fd; mkdir -p "$mt_repo/pkg" "$root/mt" "$mt_fd" - cp -r "$here/fixtures/mutation-mock/." "$mt_repo/pkg/"; rm -rf "$mt_repo/pkg/fakebin" + cp -r "$here/fixtures/mutation-mock/." "$mt_repo/pkg/"; rm -rf "$mt_repo/pkg/fakebin"; cp "$L/jest-real.log" "$mt_repo/pkg/" git -C "$mt_repo" init -q && git -C "$mt_repo" add -A && git -C "$mt_repo" -c user.name=t -c user.email=t@t commit -qm mock cp "$here/testparse.sh" "$here/mutation-table.sh" "$root/mt/"; cp "$here/fixtures/mutation-mock/$1" "$root/mt/mutants.txt" mt_out=$(PATH="$here/fixtures/mutation-mock/fakebin:$PATH" FAKE_DOCKER=$mt_fd YGG_WORK=$mt_repo YGG_IMAGE=base YGG_CNAME=job1 YGG_LABEL=eureka-verify-job=job1 \ @@ -100,9 +100,10 @@ mtclean C # D: a test file that fails to load is not a kill; a named failing test is mt mutants-d.txt 'bash vtest.sh' mtcheck D-import-failure "syntax-break | UNVIABLE" "wrong-result | CAUGHT | add.js > adds" "caught=1 survived=0 unviable=1" -# D2 (jest): "FAIL " with "Test suite failed to run" is a load failure; the "●" block names the test +# D2 (jest, real output): "FAIL " with "Test suite failed to run" is a load failure; the "●" block names the +# test; a test run that calls process.exit prints a "●" line under no FAIL header, which names nothing (ERROR, not CAUGHT) mt mutants-h.txt 'bash jtest.sh' -mtcheck D2-jest "jest-load | UNVIABLE" "jest-wrong | CAUGHT | add › adds" "caught=1 survived=0 unviable=1" +mtcheck D2-jest "jest-load | UNVIABLE" "jest-wrong | CAUGHT | add › adds" "jest-exit | ERROR | exit 1, no failing test parsed" "!process.exit called | CAUGHT" "caught=1 survived=0 unviable=1 timeout=0 noapply=0 error=1" # E: rows print as mutants finish, so a job timeout keeps them; the TERM the timeout sends still cleans up MT_WALL=8 mt mutants-e.txt 'bash test.sh' mtcheck E-live-rows "[1/2] quick | CAUGHT" "!MUTATION" @@ -180,8 +181,14 @@ else echo "FAIL wait foreign done: exit $st, output: $out"; fail=1; fi check vitest-project-load-failure.log 1 "FAILED TO LOAD |unit| src/add.test.ts" "FAILED |unit|" check vitest-spaces-load-failure.log 1 "FAILED TO LOAD src/my tests/add.test.ts" "FAILED src/my" check vitest-project-red.log 1 "FAILED |unit| src/add.test.ts > add > carries" "FAILED TO LOAD" -check jest-red.log 1 "FRAMEWORK jest passed=1 failed=1 skipped=1" -check jest-red.log 1 "FAILED add › carries" "FAILED TO LOAD" -check jest-load-failure.log 1 "FAILED TO LOAD src/add.test.js" "FAILED src/add.test.js" -check jest-load-failure.log 1 "FRAMEWORK jest passed=0 failed=0 skipped=0" +# jest, real output (jest-real.log): one space after FAIL, no TTY. Names come from the "●" lines under a FAIL header only. +check "$(seg jest-real.log green)" 0 "FRAMEWORK jest passed=2 failed=0 skipped=0" "no test counts recognised" +check "$(seg jest-real.log wrong)" 1 "FRAMEWORK jest passed=1 failed=1 skipped=0" +check "$(seg jest-real.log wrong)" 1 "FAILED add › adds" "FAILED TO LOAD" +check "$(seg jest-real.log load)" 1 "FAILED TO LOAD jt/add.test.js" "FAILED jt/add.test.js" +check "$(seg jest-real.log load)" 1 "FRAMEWORK jest passed=0 failed=0 skipped=0" "FAILED Test suite" +# F3: process.exit(1) in a test run prints a "●" line under no FAIL header; it is a crash, not a named kill +check "$(seg jest-real.log exit)" 1 "exit non-zero with no failing test parsed" "FAILED process.exit" +# F4: the "nothing recognised" line names every framework the parser reads +check "$(seg jest-real.log exit)" 1 "no test counts recognised in the log (cargo, dotnet, vitest, jest, pytest)" exit $fail diff --git a/tools/testparse.sh b/tools/testparse.sh index a60b308..4bae433 100644 --- a/tools/testparse.sh +++ b/tools/testparse.sh @@ -77,13 +77,19 @@ tp_scan() { } jest = 1; next } - # vitest and jest print " FAIL " for a file and " FAIL > > " (vitest, - # optionally with a "|project|" prefix) for a test. A line with no " > " names a FILE: a vitest load - # failure (" [ ]"), or a jest file header whose tests are named by the "●" lines below. - /^ *FAIL / { name = $0; sub(/^ *FAIL +/, "", name); print (name ~ / > / ? "FAILED " : "LOADFAIL ") name; next } - /^ +● / { # jest failure block header: " ● suite › test" + # vitest and jest print "FAIL " for a file and " FAIL > > " (vitest, + # optionally with a "|project|" prefix) for a test. Vitest and a jest TTY pad the word with two spaces; + # jest without a TTY, which is how every job runs it, prints ONE ("FAIL jt/add.test.js"). A line with + # no " > " names a FILE: a vitest load failure (" [ ]"), or a jest file header whose + # tests are named by the "●" lines below it. + /^ *FAIL +/ { infail = 1; name = $0; sub(/^ *FAIL +/, "", name); print (name ~ / > / ? "FAILED " : "LOADFAIL ") name; next } + /^ *PASS +/ || /^Test Suites:/ { infail = 0 } + # jest failure block header " ● suite › test". It names a test only under a FAIL file header: a "●" + # elsewhere is output of the run itself (process.exit prints "● process.exit called with ..." with no + # FAIL header at all), and a crash that names no test stays unnamed, so callers read it as ERROR. + /^ +● / { name = $0; sub(/^ +● +/, "", name) - if (name !~ /^(Test suite failed to run|Console|Validation Warning|Deprecation)/) print "FAILED " name + if (infail && name !~ /^(Test suite failed to run|Console|Validation Warning|Deprecation)/) print "FAILED " name next } /^(=+ )?(no tests ran|[0-9]+ [a-z]+(, [0-9]+ [a-z]+)*) in [0-9.]+s( \([0-9:]+\))?( =+)?$/ { # pytest summary, with or without -q -- 2.47.3 From 4803a76dc2b86cfc664a9a90fa2769472fb42564 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 1 Oct 2026 18:25:35 +0200 Subject: [PATCH 14/16] ygg-verify: a RUN_ON_HOST job the watchdog killed exits 124, in its work directory, and its process group is signalled A RUN_ON_HOST job whose TIMEOUT fired still exited 0: kill_job only killed labelled containers and the status came from the host command (Soul F2). The host command now runs in its own process group (set -m), in the job's work directory (it ran in the ssh home), with stdin from /dev/null. kill_job sends the group TERM, then KILL 20 s later, and the verdict is forced to 124 when the watchdog killed the job, whatever the command exited with. The TIMEOUT/IDLE notice now prints before the kill so the 20 s grace cannot swallow it. Proved on Yggdrasil with the sleep-past-TIMEOUT probe (exit 124, cwd = the work dir, no sleep left) and with a command that traps TERM and exits 0 (still 124); probes recorded in tools/fixtures/mutation-probe/README.txt, with the expected rows for the fresh and jsr reproducers. Co-Authored-By: Claude Sonnet 5.5 --- tools/fixtures/mutation-probe/README.txt | 10 ++++++++ tools/stopgap/ygg-verify.sh | 30 ++++++++++++++++++------ 2 files changed, 33 insertions(+), 7 deletions(-) diff --git a/tools/fixtures/mutation-probe/README.txt b/tools/fixtures/mutation-probe/README.txt index e084a0c..98d9152 100644 --- a/tools/fixtures/mutation-probe/README.txt +++ b/tools/fixtures/mutation-probe/README.txt @@ -10,3 +10,13 @@ Workdirs: tools/fixtures/mutation-probe/rust and .../js. (target/soul-cache.txt): stamp2-bad CAUGHT tests::d_cache, untested-equiv SURVIVED, add-minus CAUGHT tests::c_add; exit 1. Before the per-mutant snapshot, untested-equiv read CAUGHT d_cache and the job exited 0 (the cache outlived the mutant that wrote it). + fresh/mutants.txt TEST='cargo test --lib -- --test-threads=1', workdir .../fresh (f_fresh creates a directory, so it cannot + run twice) -> `SNAPSHOT RED (exit 101) for: cargo test ...`, "no rows counted", exit 2. Before the control + run of the unmutated snapshot, the job read untested-equiv CAUGHT tests::f_fresh and exited 0. + jsr/mutants.txt workdir .../jsr, each mutant carries its own `test:` (jest 29.7.0 and vitest 3.2.4 from npm), image node:22 -> + jest-load UNVIABLE, jest-wrong CAUGHT add › adds, jest-exit ERROR (process.exit(1) names no test), + jest-equiv SURVIVED, vt-load UNVIABLE, vt-wrong CAUGHT, vt-equiv-throw-at-load UNVIABLE; exit 1. + (Before the fixes jest-load read ERROR and jest-exit read CAUGHT | process.exit called with "1".) +RUN_ON_HOST watchdog probe (tools/stopgap/ygg-verify.sh): RUN_ON_HOST=1 TIMEOUT=60 ygg-verify.sh HEAD rust + 'echo cwd=$PWD; sleep 300; echo hostdone' -> cwd is the job's work directory, `TIMEOUT after 60s`, exit 124, no `hostdone`, + no sleep left on the host. With trap "exit 0" TERM around the sleep the status is still 124 (it was 0). diff --git a/tools/stopgap/ygg-verify.sh b/tools/stopgap/ygg-verify.sh index 5d1fea6..ef28a07 100644 --- a/tools/stopgap/ygg-verify.sh +++ b/tools/stopgap/ygg-verify.sh @@ -339,7 +339,8 @@ work=$(mktemp -d "$root/work/$name-${sha:0:10}-XXXX") cname="eureka-verify-$slot-$$" label="eureka-verify-job=$cname" killed="$root/work/$cname.killed" -trap 'rm -f "$killed"; if [ "$keep" != 1 ]; then sudo rm -rf -- "$work"; fi' EXIT +hostpg="$root/work/$cname.hostpg" # RUN_ON_HOST: the command's process group id, for the watchdog +trap 'rm -f "$killed" "$hostpg"; if [ "$keep" != 1 ]; then sudo rm -rf -- "$work"; fi' EXIT git clone -q --no-checkout "$root/repos/$name.git" "$work" git -C "$work" checkout -q --detach "$sha" echo "ygg-verify: $name@${sha:0:10} slot $slot, image $image, cpus $cpus, mem $mem, work $work" >&2 @@ -350,10 +351,15 @@ echo "ygg-verify: $name@${sha:0:10} slot $slot, image $image, cpus $cpus, mem $m # The watchdog's one kill: every container the job started carries $label (a # host-side command may own a baseline container and a mutant container at once, # so there is no single name to kill). The flag tells that command the job is over. +# A RUN_ON_HOST command is also a process group of its own (set -m below): it gets TERM, +# then KILL 20 s later, so a command that starts no container (or starts one after the +# docker kill) dies too instead of running on past TIMEOUT (Soul, 2026-10-01). kill_job() { touch "$killed" - local ids; ids=$(sudo docker ps -q --filter "label=$label" 2>/dev/null || true) + local ids pg; ids=$(sudo docker ps -q --filter "label=$label" 2>/dev/null || true) [ -z "$ids" ] || sudo docker kill $ids >/dev/null 2>&1 || true + pg=$(cat "$hostpg" 2>/dev/null || true) + [ -z "$pg" ] || { kill -TERM -- "-$pg" 2>/dev/null || true; sleep 20; kill -KILL -- "-$pg" 2>/dev/null || true; } } # The subshell must not inherit the slot lock: its sleep outlives a job that # finishes early, and an inherited fd held slot 2 for an hour (2026-09-23). @@ -363,7 +369,7 @@ kill_job() { start=$(date +%s); idle=0 while sleep 60; do if [ $(( $(date +%s) - start )) -ge "$timeout_s" ]; then - kill_job; echo "ygg-verify: TIMEOUT after ${timeout_s}s, container killed" >&3 + echo "ygg-verify: TIMEOUT after ${timeout_s}s, container killed" >&3; kill_job exit 0 fi [ "$idle_s" -gt 0 ] || continue @@ -373,17 +379,27 @@ kill_job() { [ -n "$cpu" ] || continue if awk -v c="$cpu" 'BEGIN { exit !(c < 5.0) }'; then idle=$((idle + 60)); else idle=0; fi if [ "$idle" -ge "$idle_s" ]; then - kill_job; echo "ygg-verify: IDLE ${idle_s}s under 5% CPU, container killed (a test is probably hung)" >&3 + echo "ygg-verify: IDLE ${idle_s}s under 5% CPU, container killed (a test is probably hung)" >&3; kill_job exit 0 fi done ) 3>&2 & watchdog=$! set +e if [ "$host_mode" = 1 ]; then - ( export YGG_WORK=$work YGG_IMAGE=$image YGG_CPUS=$cpus YGG_MEM=$mem YGG_CNAME=$cname YGG_LABEL=$label \ + # Own process group (set -m), started in the job's work directory, so the watchdog can signal all of it. + set -m + ( cd "$work" && export YGG_WORK=$work YGG_IMAGE=$image YGG_CPUS=$cpus YGG_MEM=$mem YGG_CNAME=$cname YGG_LABEL=$label \ YGG_KILLED=$killed DOCKER_ARGS=$docker_args [ -z "$nonce" ] || export YGG_NONCE=$nonce - exec bash -o pipefail -c "$cmd" ) + exec bash -o pipefail -c "$cmd" ) "$hostpg" + wait "$hostpid" + status=$? + # The watchdog killed the job: that is the verdict, whatever the command exited with (a host + # command's status comes from the host command, not from a killed container). + if [ -e "$killed" ]; then status=124; fi else sudo nice -n 10 docker run --rm --name "$cname" --label "$label" --cpus="$cpus" --memory="$mem" \ -v "$work:/src" -v /etc/machine-id:/etc/machine-id:ro \ @@ -393,8 +409,8 @@ sudo nice -n 10 docker run --rm --name "$cname" --label "$label" --cpus="$cpus" -e GIT_CONFIG_COUNT=1 -e GIT_CONFIG_KEY_0=safe.directory -e GIT_CONFIG_VALUE_0='*' \ $docker_args \ -w /src "$image" bash -o pipefail -c "printf 'set -o pipefail\n' >/tmp/.ygg-bash-env; $cmd" -fi status=$? +fi set -e pkill -P "$watchdog" 2>/dev/null || true kill "$watchdog" 2>/dev/null || true -- 2.47.3 From 838b8594236d512662315fe6b787d782002d42e5 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 1 Oct 2026 18:25:36 +0200 Subject: [PATCH 15/16] mutation-table: delete the dead cur branch of the cleanup trap The --rm mutant container is gone by the time the trap runs, so `docker rm -f $cur` never had a container to remove. The stopgap's watchdog kills every job-labelled container before it signals the driver, so nothing needs tracking here. Also restores a line continuation the control-run edit had collapsed. Co-Authored-By: Claude Sonnet 5.5 --- tools/mutation-table.sh | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/tools/mutation-table.sh b/tools/mutation-table.sh index 9e19efd..036c6a6 100644 --- a/tools/mutation-table.sh +++ b/tools/mutation-table.sh @@ -111,9 +111,9 @@ if [ "$1" = --host ]; then bundle=${MT_BUNDLE:?} set +e # every docker step is checked by hand: a mutant container exits non-zero by design base="$YGG_CNAME-base"; snap="eureka-mt-$YGG_CNAME:snap" - cur= + # A TERM waits for the foreground container (bounded by MUT_TIMEOUT); the stopgap's watchdog kills every + # job-labelled container before it signals this driver, so cleanup has no running mutant to stop. cleanup() { - [ -z "$cur" ] || sudo docker rm -f "$cur" >/dev/null 2>&1 || true sudo docker rm -f "$base" >/dev/null 2>&1 || true sudo docker rmi -f "$snap" >/dev/null 2>&1 || true [ -z "$bundle" ] || [ "$bundle" = / ] || rm -rf -- "$bundle" @@ -148,7 +148,8 @@ if [ "$1" = --host ]; then # fixture) is green in the baseline and red in every mutant's container, which would read as a fully # killed suite. The control is the baseline's tests run once more from the snapshot, unmutated. # shellcheck disable=SC2086 - drun --rm --name "$YGG_CNAME-ctl" "${common[@]}" -w /src ${DOCKER_ARGS:-} "$snap" bash -o pipefail -c "$pre; bash /mt/mutation-table.sh --control" + drun --rm --name "$YGG_CNAME-ctl" "${common[@]}" -w /src ${DOCKER_ARGS:-} "$snap" \ + bash -o pipefail -c "$pre; bash /mt/mutation-table.sh --control" rc=$? [ "$rc" = 0 ] || { echo "$SUMTAG: the unmutated snapshot is red though the baseline was green, so no mutant result would mean anything; no rows counted"; exit "$rc"; } @@ -157,7 +158,6 @@ if [ "$1" = --host ]; then if [ -n "${YGG_KILLED:-}" ] && [ -e "$YGG_KILLED" ]; then echo "$SUMTAG: JOB KILLED by the watchdog before mutant $n of $count; the finished rows above stand"; exit 124 fi - cur="$YGG_CNAME-m$n" # shellcheck disable=SC2086 drun --rm --name "$YGG_CNAME-m$n" "${common[@]}" -w /src ${DOCKER_ARGS:-} "$snap" \ bash -o pipefail -c "$pre; bash /mt/mutation-table.sh --mutant $n" -- 2.47.3 From 8c14392d07deac67486db7d85968d3e218e7c955 Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Thu, 1 Oct 2026 18:39:36 +0200 Subject: [PATCH 16/16] Tool-calling map: record the follow-ups from the tools' Soul passes Co-Authored-By: Claude Opus 5.5 --- references/tool-calling-cut.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/references/tool-calling-cut.md b/references/tool-calling-cut.md index 56d4b8f..5e8b733 100644 --- a/references/tool-calling-cut.md +++ b/references/tool-calling-cut.md @@ -441,3 +441,9 @@ Verbatim: "Q1 A, Q2 yes, Q3 agreed, Q4 B, Q5 yes, but see Epiphany's approach wh ## 10. Q4 superseded (operator, 2026-10-01) Q4 is now **A**: the Eureka repo, prose and `tools/`, moves to Yggdrasil's Forgejo and is shipped open source with contributions welcome. See `faculty-philosophy-cut.md` "Rulings" Q3 for the verbatim reason. Contributions use Forgejo PRs. Whether the GitHub copy is archived, mirrored or deleted is still open; ask the operator. + +## 11. Follow-ups from the tools' Soul passes (2026-10-01) +The `tools/` work merged after four Soul passes (last at `838b859`). Recorded, not blocking: +- **Jest `infail` scope.** `infail` resets only on `PASS` or `Test Suites:`, so a crash after an earlier failing file is named as a failing test. The verdict stays right; the name is wrong. Reset at a blank line or the next non-indented non-`●` line. +- **No automated guard for the RUN_ON_HOST watchdog (exit 124).** It is proven only by the probe recipe in `tools/fixtures/mutation-probe/README.txt`; `tools/test-tools.sh` never starts the stopgap, so reverting the fix stays green. +- **Shared volumes are outside the isolation.** The cargo registry, nuget and `eureka-sccache` volumes carry state across the control and mutant containers. A test that counts into one could pass the control and read CAUGHT later. The runner header names the limit; no probe was built. -- 2.47.3