GameCult/Eureka
Watch
9
0
Fork
You've already forked Eureka
 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

Tool-calling cut §5: summary/wait verify, snapshot mutation runner #1

Merged
eureka-admin merged 17 commits from hands/tool-calling-s5 into main 2026-10-01 16:39:49 +00:00
Conversation 0 Commits 17 Files changed 82 +1935 -35
82 changed files with 1935 additions and 35 deletions
Download patch file Download diff file Expand all files Collapse all files

6
SKILL.md
View file

@ -531,8 +531,10 @@ and reconciles the target doc with the Body.
Scripts that agents share live in this repo under `tools/`. `tools/INDEX.md`
lists each script, its consumer and the brief that names it; briefs name the
exact script they want and agents do not browse. Contributions come by branch
and pull request, never a push to main. A script that verdicts depend on gets
a Soul pass before merge. A script whose brief no longer names it is pruned.
and Forgejo pull request (this repo lives on Yggdrasil's Forgejo), never a push
to main. A script that verdicts depend on gets a Soul pass before merge, and
`bash tools/test-tools.sh` passes first. The summary and wait modes of
`tools/stopgap/ygg-verify.sh` and `tools/mutation-table.sh` are such scripts. A script whose brief no longer names it is pruned.
Scripts that know a project's layout stay in that project.
## Git and tooling rules (scars)

65
references/briefs.md
View file

@ -98,7 +98,8 @@ path>. Do not map cuts, do not write code, do not commit; the root agent
commits.
Rehydrate (recipes: target in force, rulings in force, open questions and
follow-ups). Read enough of the Body to enumerate the persistent kinds.
follow-ups). Read enough of the Body to enumerate the persistent kinds. Orient
and probe through scripts, one call per question, output at most 150 lines.
Give one table, a row per persistent kind:
- Identity: what names it. Is the namespace stated? Is the name injective? Can
@ -127,7 +128,8 @@ prose map at <path>, edit body facts and rationale. Also edit the model page
whenever a probe changes a row's identity, lifecycle or authority. Imagination
wrote that page in 0b, and a model a cut contradicts is stale steering text.
Do not commit; the root agent commits. Do not change code in any repo. Scratch probes in
<scratchpad> are fine.
<scratchpad> are fine. Orient and probe through scripts: one call per question,
output at most 150 lines. Budget 40-80 calls and about 250k context.
Rehydrate (recipes: target in force, rulings in force, open questions and
follow-ups, specs with no report, one cut's record for <cuts that constrain
@ -258,7 +260,15 @@ one head (SKILL.md, step 3); keep it fitting:
- **Verify once, at the end.** Not after every edit. A full mutation sweep and
a batchmode compile per edit is most of a long run's spend and proves nothing
the final sweep will not.
the final sweep will not. One job carries the build, the scoped tests and the
mutation run, so the queue and the compile are paid once.
- **Orient in one script.** One Bash call prints git status, the base and the
spec's anchor ranges (`sed -n`) for every file the spec names. Independent
reads, views and greps go in one parallel message.
- **Bound every observation.** No command prints more than about 150 lines.
Logs go to a file; read `tail -n 60` or `grep -n -m`, and read windows of a
file, never the whole of anything over about 300 lines. Reading a file twice
is a smell: write what you need to keep to a scratch notes file.
- **Read the spec by id, not the map.** The cut_spec is bounded and holds only
this cut; a worker that reads 900 lines of map to find 60 has spent its
budget before it starts.
@ -278,13 +288,16 @@ one head (SKILL.md, step 3); keep it fitting:
Long jobs: wait for them and finish, and never end your turn before your
report. Each turn end reaches Self as a notification and costs a read of its
whole context. Start the job as one background shell whose command exits only
when the job does. Then wait in the foreground, one call at a time, each at
most 270 s so that your cache stays warm: `timeout 270 bash -c 'until [ -e
<done-file> ]; do sleep 60; done'`, then read the result or call again. A call
whole context. Start the job as one background shell (run_in_background) with
`SUMMARY=<scratchpad>/<job>.log` in front of `tools/stopgap/ygg-verify.sh`, a
fresh log path per job. Then wait in the foreground, one call at a time:
`tools/stopgap/ygg-verify.sh wait <scratchpad>/<job>.log` blocks for at most
270 s (run it with a Bash tool `timeout` of at least 280000 ms: the default 120 s backgrounds the call and costs a wake), so your cache stays warm, and prints the bounded summary when the job is
done. If it prints `RUNNING <n>s` (exit 99), call it again. Exit 98 means the
launcher is gone with no verdict: rerun the job, do not guess its outcome. A call
longer than 5 minutes rewrites your whole context. Never run a second watcher
on the same job, never use a timer shorter than 60 s, and send no interim
message.
on the same job, never use a timer shorter than 60 s in a loop of your own, and
send no interim message.
Mutation tools run on schemata or copies and never edit the tree. A Soul
probe that mutates by hand works on a scratch copy: the stopgap's container
or a throwaway clone, never the working tree.
@ -310,7 +323,10 @@ Commits:
- Say in the message which commits don't build.
Verification: <exact builds, tests, captures, negative greps>.
- Verify through `tools/stopgap/ygg-verify.sh`; read its output tail, and open the full log only at a named failing test, with `grep -n -A 40`.
- Verify through `tools/stopgap/ygg-verify.sh` with `SUMMARY=<log>` (above):
read the summary, and open the full log only at a named failing test, with
`grep -n -A 40`. Hand mutants go in one list to `tools/mutation-table.sh`,
one job for all of them.
- Revert incidental asset churn.
Budget: if the cut can't land coherently, stop at a pushed, building commit
@ -335,8 +351,8 @@ revised spec first. Its fields:
- landed_names, undone, and promises: each thing you claim the cut now
guarantees, labelled, for Soul to measure
Report to Self, and nothing else: the cut_report id and receipt, then any raw
verification output too long for an evidence line, pasted, not summarised. No
Report to Self, and nothing else: the cut_report id and receipt, and, for
output too long for an evidence line, the scratch path that holds it. No
narrative of the pass, no restating the brief or the report.
```
@ -352,11 +368,15 @@ Admit one verdict and its findings, and nothing else.
afterwards.
- Do not run <expensive runtime> unless told.
- Long jobs: wait for them and finish, and never end your turn before your
report. Run the job as one background shell whose command exits only when the
job does, and wait in the foreground in calls of at most 270 s (`timeout 270
bash -c 'until [ -e <done-file> ]; do sleep 60; done'`). No second watcher,
no timer under 60 s, and no interim message: each turn end is a paid wake of
Self.
report. Run the job as one background shell with `SUMMARY=<scratchpad>/<job>.log`
in front of `tools/stopgap/ygg-verify.sh` (a fresh log path per job), and wait
in the foreground with `tools/stopgap/ygg-verify.sh wait <that log>`: pass a Bash tool `timeout` of at least 280000 ms (the default 120 s backgrounds the call and costs a wake); it blocks
at most 270 s and prints the bounded summary when the job is done, or `RUNNING
<n>s` (exit 99) and you call it again. Exit 98 means no verdict: rerun, do not
guess. No second watcher, no timer under 60 s in a loop of your own, and no
interim message: each turn end is a paid wake of Self.
- Bound every observation: no command prints more than about 150 lines, logs go
to files, and you read windows of them.
Scope: <cut_report id>. Rehydrate (recipes: target in force, rulings in force,
one cut's record for <cut>), then view the report (its range and its promises), the cut_spec it
@ -401,7 +421,10 @@ specifically:
mutation run where you can, since that job pays for the build anyway. Batch
probes into one job script rather than one job each. Run the mutation tool
in parallel (`--jobs 3`), and skip by name any test that a follow-up records
as flaky, so that its flakes cannot count a surviving mutant as killed
as flaky, so that its flakes cannot count a surviving mutant as killed.
Dump the range once (`git diff <base>..<head> > <scratchpad>/range.diff` plus
`--stat`) and grep that file instead of re-running `git show` per file. Hand
mutants go in one list to `tools/mutation-table.sh`, never one job each.
Before recommending a mechanism as the fix direction, check that it was not
already built and deleted. Run `git log -S '<distinctive identifier>'` and
@ -448,7 +471,8 @@ the campaign's mind and the prose map and target doc; don't edit those, and
admit nothing. Rehydrate (recipes: rulings in force, open questions and
follow-ups) to check memory against the mind.
Surfaces: <memory dirs and files>.
Surfaces: <memory dirs and files>. Budget at most 25 calls: inventory the
surfaces in one script, then make the edits.
Candidates for durable memory: <ruling ids, scars>. Check each against its
owner first, the mind included, and don't duplicate what the owner records. A
@ -474,7 +498,8 @@ what was missing and what it would have prevented.
```
You are Eyes. Read only. Write findings to <scratchpad>/<name>.md and return a
short summary. You read the campaign's mind with query and view only, through
short summary. Budget at most 40 calls: transcripts by script, never whole
reads, and fetches batched in parallel. You read the campaign's mind with query and view only, through
the Rehydrate block (recipes: <as needed>); you never admit.
Sources:

8
references/changelog.md
View file

@ -2,6 +2,14 @@
Record each change to the skill together with the evidence that motivated it.
## 2026-10-01: verify summaries, a wait mode and a kill table
Campaign `eureka-substrate`, cut `tool-calling-s5` (map `references/tool-calling-cut.md`, sections 3 and 5). Operator go: "go with 2 and 3; we're spending tokens to save tokens."
Evidence: Hands spent 3,852 observation calls (poll, read log, monitor, stop) on 1,443 launches, 2.7 per job, and Soul 1,800 on 813, 2.2 per job; eight ad-hoc harnesses (`runmut.sh`, `mutrun.sh`, `strykrun.sh` and others) were reinvented across agents.
What changed. `ygg-verify.sh` gains `SUMMARY=<log>` (full log to a file, bounded summary out: verdict, counts per framework, `ZERO TESTS RAN`, failing tests, failure sections and tail for a red job, and a `MISMATCH` line when the exit is 0 but the log shows failures) and `wait <log>` (blocks at most 270 s, returns the summary once, `RUNNING` exit 99, exit 98 for a launcher that died or never started). `tools/mutation-table.sh` runs a list of hand mutants in one job and returns a kill table with the killing test per mutant; it aborts on a red or zero-test baseline and never counts a build error as a kill. `tools/testparse.sh` is the one parser both read, with fixtures under `tools/fixtures` and `tools/test-tools.sh`. Briefs now name them, add orient-in-one-script and bounded-observation rules for Hands and Soul, call budgets for Imagination (40-80), Eyes (40) and Life (25), the range-dump rule for Soul, one job for build, tests and mutation, and the scratch-path form of the Hands report. Contribution text says Forgejo pull request.
## 2026-10-01: the skill follows the mind's correctness core
Campaign `eureka-substrate`, cut `skill-recipes`, after the Huginn admission-rule and `eureka-state` client cuts (cut `ops-mind-upgrade`). Evidence: a census of the live mind found 6 findings that could not be closed and 39 resolutions that could not be withdrawn, from key locals over the bound.

6
references/tool-calling-cut.md
View file

@ -441,3 +441,9 @@ Verbatim: "Q1 A, Q2 yes, Q3 agreed, Q4 B, Q5 yes, but see Epiphany's approach wh
## 10. Q4 superseded (operator, 2026-10-01)
Q4 is now **A**: the Eureka repo, prose and `tools/`, moves to Yggdrasil's Forgejo and is shipped open source with contributions welcome. See `faculty-philosophy-cut.md` "Rulings" Q3 for the verbatim reason. Contributions use Forgejo PRs. Whether the GitHub copy is archived, mirrored or deleted is still open; ask the operator.
## 11. Follow-ups from the tools' Soul passes (2026-10-01)
The `tools/` work merged after four Soul passes (last at `838b859`). Recorded, not blocking:
- **Jest `infail` scope.** `infail` resets only on `PASS` or `Test Suites:`, so a crash after an earlier failing file is named as a failing test. The verdict stays right; the name is wrong. Reset at a blank line or the next non-indented non-`●` line.
- **No automated guard for the RUN_ON_HOST watchdog (exit 124).** It is proven only by the probe recipe in `tools/fixtures/mutation-probe/README.txt`; `tools/test-tools.sh` never starts the stopgap, so reverting the fix stays green.
- **Shared volumes are outside the isolation.** The cargo registry, nuget and `eureka-sccache` volumes carry state across the control and mutant containers. A test that counts into one could pass the control and read CAUGHT later. The runner header names the limit; no probe was built.

10
tools/INDEX.md
View file

@ -2,9 +2,13 @@
One line per script. Briefs name the exact script they want; agents do not
browse. A line whose consuming brief no longer names the script is dead and is
deleted. Contributions come by branch and pull request. A script that verdicts
depend on gets a Soul pass before merge.
deleted. Contributions come by branch and Forgejo pull request (this repo lives on
Yggdrasil's Forgejo), never a push to main. A script that verdicts depend on
gets a Soul pass before merge.
| script | purpose | consumer | named by |
|---|---|---|---|
| `tools/stopgap/ygg-verify.sh` | Pushes an exact revision to a mirror on Yggdrasil and runs one command in a capped, niced container. Dies with the stopgap when Idunn's verify transaction lands. | Hands, Soul, Imagination probes | `references/briefs.md` (Hands and Soul verification), `SKILL.md` (Yggdrasil) |
| `tools/stopgap/ygg-verify.sh` | Pushes an exact revision to a mirror on Yggdrasil and runs one command in a capped, niced container (`RUN_ON_HOST=1`: run a host-side command in the same slot, clone and watchdog instead; the watchdog kills every container carrying the job label). `SUMMARY=<log>` keeps the full log and prints a bounded summary (verdict, counts, ZERO TESTS RAN, failing tests); `wait <log>` blocks at most 270 s and returns the summary once; call it with a Bash tool `timeout` of at least 280000 ms. Dies with the stopgap when Idunn's verify transaction lands. | Hands, Soul, Imagination probes | `references/briefs.md` (Hands long jobs and verification, Soul long jobs), `SKILL.md` (Yggdrasil) |
| `tools/mutation-table.sh` | One verify job that runs a list of hand mutants (file, before, after, test) against a revision and returns a kill table: CAUGHT with the killing test, SURVIVED, UNVIABLE, TIMEOUT, NOAPPLY, ERROR. Refuses a red or zero-test baseline and a red BUILD; rows print live. Isolation is Docker's: the baselines run once in a container that holds the revision, that container is committed to a throwaway image, and every mutant runs in a fresh `--rm` container from it, so nothing a run writes (tracked, untracked or ignored, such as `target/`) reaches the next mutant; the image is removed when the job ends, killed or not. The shared registry and sccache volumes and the network are not isolated. A file-level load failure (vitest `FAIL <file> [ <file> ]`, jest `FAIL <file>` plus `Test suite failed to run`) reads UNVIABLE. Replaces per-agent runmut/mutrun/strykrun scripts. | Soul, Hands (scoped run) | `references/briefs.md` (Hands verification, Soul reruns) |
| `tools/testparse.sh` | Sourced library: failing test names and pass/fail/skip counts from cargo, dotnet, vitest, jest and pytest logs, and test files that failed to load (`LOADFAIL`). Shared by the summary and the kill table. | the two scripts above | not named by a brief; it lives and dies with them |
| `tools/test-tools.sh` | Local fixture check of the parsers and the summary over `tools/fixtures/logs` (real cargo and mutation logs, plus hand-written dotnet, vitest, jest, pytest, exit-0-but-failed and no-verdict logs), and of the mutation runner's host driver and container half under a mock docker. Run it before merging a change to the scripts above. | whoever changes those scripts | `SKILL.md` (Shared tools) |

20
tools/fixtures/logs/cargo-green.log Normal file
View file

@ -0,0 +1,20 @@
warning: You appear to have cloned an empty repository.
ygg-verify: probe@7d3a1f5284 slot 1, image eureka-verify-rust:b99d30ec441e, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/probe-7d3a1f5284-Pvbq
Compiling tiny-crate v0.0.0 (/src)
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.35s
Running unittests src/lib.rs (target/debug/deps/tiny_crate-7b09195d0b4b7f00)
Doc-tests tiny_crate
running 3 tests
test tests::add_adds ... ok
test tests::even_detects ... ok
test tests::clamp_negative ... ok
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
ygg-verify: exit 0

29
tools/fixtures/logs/cargo-red.log Normal file
View file

@ -0,0 +1,29 @@
warning: You appear to have cloned an empty repository.
ygg-verify: probe@7d3a1f5284 slot 2, image eureka-verify-rust:b99d30ec441e, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/probe-7d3a1f5284-axtF
Compiling tiny-crate v0.0.0 (/src)
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.34s
Running unittests src/lib.rs (target/debug/deps/tiny_crate-7b09195d0b4b7f00)
running 3 tests
test tests::add_adds ... FAILED
error: test failed, to rerun pass `--lib`
test tests::clamp_negative ... ok
test tests::even_detects ... ok
failures:
---- tests::add_adds stdout ----
thread 'tests::add_adds' (113) panicked at src/lib.rs:19:9:
assertion `left == right` failed
left: 5
right: 6
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
failures:
tests::add_adds
test result: FAILED. 2 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
ygg-verify: exit 101

11
tools/fixtures/logs/cargo-zero-tests.log Normal file
View file

@ -0,0 +1,11 @@
warning: You appear to have cloned an empty repository.
ygg-verify: probe@7d3a1f5284 slot 1, image eureka-verify-rust:b99d30ec441e, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/probe-7d3a1f5284-OoDx
Compiling tiny-crate v0.0.0 (/src)
Finished `test` profile [unoptimized + debuginfo] target(s) in 0.32s
Running unittests src/lib.rs (target/debug/deps/tiny_crate-7b09195d0b4b7f00)
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.00s
ygg-verify: exit 0

6
tools/fixtures/logs/dotnet-red.log Normal file
View file

@ -0,0 +1,6 @@
SYNTHETIC: hand-written in the format dotnet test prints; not captured from a run.
Passed MyNs.AddTests.Adds [2 ms]
Failed MyNs.AddTests.Subtracts [12 ms]
Error Message:
Assert.Equal() Failure: Expected 1, Actual 2
Failed! - Failed: 1, Passed: 11, Skipped: 2, Total: 14, Duration: 1 s - MyNs.Tests.dll (net10.0)

3
tools/fixtures/logs/exit0-but-failed.log Normal file
View file

@ -0,0 +1,3 @@
SYNTHETIC: a command whose last statement hid a failing test run (the exit status is 0 in the test).
test tests::add_adds ... FAILED
test result: FAILED. 2 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

170
tools/fixtures/logs/jest-real.log Normal file
View file

@ -0,0 +1,170 @@
# Real jest 29.7.0 output (node:22, no TTY, --ci) from tools/fixtures/mutation-probe/jsr run through tools/stopgap/ygg-verify.sh on Yggdrasil, 2026-10-01. Runs split by '=== MARK <name>'; a Console-free header is one space: 'FAIL jt/add.test.js'.
=== MARK green
PASS jt/add.test.js
add
✓ adds (2 ms)
✓ adds zero (1 ms)
Test Suites: 1 passed, 1 total
Tests: 2 passed, 2 total
Snapshots: 0 total
Time: 0.391 s
Ran all test suites.
=== MARK wrong
FAIL jt/add.test.js
add
✕ adds (4 ms)
✓ adds zero (1 ms)
● add › adds
expect(received).toBe(expected) // Object.is equality
Expected: 5
Received: -1
1 | const { add } = require('./add');
2 | describe('add', () => {
> 3 | test('adds', () => { expect(add(2, 3)).toBe(5); });
| ^
4 | test('adds zero', () => { expect(add(0, 0)).toBe(0); });
5 | });
6 |
at Object.toBe (jt/add.test.js:3:42)
Test Suites: 1 failed, 1 total
Tests: 1 failed, 1 passed, 2 total
Snapshots: 0 total
Time: 0.385 s, estimated 1 s
Ran all test suites.
=== MARK load
FAIL jt/add.test.js
● Test suite failed to run
Jest encountered an unexpected token
Jest failed to parse a file. This happens e.g. when your code or its dependencies use non-standard JavaScript syntax, or when Jest is not configured to support such syntax.
Out of the box Jest supports Babel, which will be used to transform your files into valid JS based on your Babel configuration.
By default "node_modules" folder is ignored by transformers.
Here's what you can do:
• If you are trying to use ECMAScript Modules, see https://jestjs.io/docs/ecmascript-modules for how to enable it.
• If you are trying to use TypeScript, see https://jestjs.io/docs/getting-started#using-typescript
• To have some of your "node_modules" files transformed, you can specify a custom "transformIgnorePatterns" in your config.
• If you need a custom transformation specify a "transform" option in your config.
• If you simply want to mock your non-JS modules (e.g. binary assets) you can stub them out with the "moduleNameMapper" config option.
You'll find more details and examples of these config options in the docs:
https://jestjs.io/docs/configuration
For information about custom transformations, see:
https://jestjs.io/docs/code-transformation
Details:
SyntaxError: /src/tools/fixtures/mutation-probe/jsr/jt/add.js: Unexpected token (2:21)
1 | module.exports = {
> 2 | add: (a, b) => a + ,
| ^
3 | untested: () => 1,
4 | };
5 |
> 1 | const { add } = require('./add');
| ^
2 | describe('add', () => {
3 | test('adds', () => { expect(add(2, 3)).toBe(5); });
4 | test('adds zero', () => { expect(add(0, 0)).toBe(0); });
at constructor (node_modules/@babel/parser/src/parse-error.ts:96:45)
at Parser.toParseError [as raise] (node_modules/@babel/parser/src/tokenizer/index.ts:1504:19)
at Parser.raise [as unexpected] (node_modules/@babel/parser/src/tokenizer/index.ts:1544:16)
at Parser.unexpected [as parseExprAtom] (node_modules/@babel/parser/src/parser/expression.ts:1385:22)
at Parser.parseExprAtom [as parseExprSubscripts] (node_modules/@babel/parser/src/parser/expression.ts:742:23)
at Parser.parseExprSubscripts [as parseUpdate] (node_modules/@babel/parser/src/parser/expression.ts:721:21)
at Parser.parseUpdate [as parseMaybeUnary] (node_modules/@babel/parser/src/parser/expression.ts:683:23)
at Parser.parseMaybeUnary [as parseMaybeUnaryOrPrivate] (node_modules/@babel/parser/src/parser/expression.ts:417:14)
at Parser.parseMaybeUnaryOrPrivate (node_modules/@babel/parser/src/parser/expression.ts:597:12)
at Parser.parseExprOpBaseRightExpr [as parseExprOpRightExpr] (node_modules/@babel/parser/src/parser/expression.ts:582:21)
at Parser.parseExprOpRightExpr [as parseExprOp] (node_modules/@babel/parser/src/parser/expression.ts:513:27)
at Parser.parseExprOp [as parseExprOps] (node_modules/@babel/parser/src/parser/expression.ts:435:17)
at Parser.parseExprOps [as parseMaybeConditional] (node_modules/@babel/parser/src/parser/expression.ts:384:23)
at Parser.parseMaybeConditional [as parseMaybeAssign] (node_modules/@babel/parser/src/parser/expression.ts:301:21)
at Parser.parseMaybeAssign [as parseFunctionBody] (node_modules/@babel/parser/src/parser/expression.ts:2637:14)
at Parser.parseFunctionBody [as parseArrowExpression] (node_modules/@babel/parser/src/parser/expression.ts:2592:10)
at Parser.parseArrowExpression [as parseParenAndDistinguishExpression] (node_modules/@babel/parser/src/parser/expression.ts:1877:12)
at Parser.parseParenAndDistinguishExpression [as parseExprAtom] (node_modules/@babel/parser/src/parser/expression.ts:1197:21)
at Parser.parseExprAtom [as parseExprSubscripts] (node_modules/@babel/parser/src/parser/expression.ts:742:23)
at Parser.parseExprSubscripts [as parseUpdate] (node_modules/@babel/parser/src/parser/expression.ts:721:21)
at Parser.parseUpdate [as parseMaybeUnary] (node_modules/@babel/parser/src/parser/expression.ts:683:23)
at Parser.parseMaybeUnary [as parseMaybeUnaryOrPrivate] (node_modules/@babel/parser/src/parser/expression.ts:417:14)
at Parser.parseMaybeUnaryOrPrivate [as parseExprOps] (node_modules/@babel/parser/src/parser/expression.ts:429:23)
at Parser.parseExprOps [as parseMaybeConditional] (node_modules/@babel/parser/src/parser/expression.ts:384:23)
at Parser.parseMaybeConditional [as parseMaybeAssign] (node_modules/@babel/parser/src/parser/expression.ts:301:21)
at parseMaybeAssign (node_modules/@babel/parser/src/parser/expression.ts:257:12)
at Parser.callback [as allowInAnd] (node_modules/@babel/parser/src/parser/expression.ts:3227:12)
at Parser.allowInAnd [as parseMaybeAssignAllowIn] (node_modules/@babel/parser/src/parser/expression.ts:256:17)
at Parser.parseMaybeAssignAllowIn [as parseMaybeAssignAllowInOrVoidPattern] (node_modules/@babel/parser/src/parser/expression.ts:3341:17)
at Parser.parseMaybeAssignAllowInOrVoidPattern [as parseObjectProperty] (node_modules/@babel/parser/src/parser/expression.ts:2358:16)
at Parser.parseObjectProperty [as parseObjPropValue] (node_modules/@babel/parser/src/parser/expression.ts:2421:12)
at Parser.parseObjPropValue [as parsePropertyDefinition] (node_modules/@babel/parser/src/parser/expression.ts:2261:17)
at Parser.parsePropertyDefinition [as parseObjectLike] (node_modules/@babel/parser/src/parser/expression.ts:2128:21)
at Parser.parseObjectLike [as parseExprAtom] (node_modules/@babel/parser/src/parser/expression.ts:1208:21)
at Parser.parseExprAtom [as parseExprSubscripts] (node_modules/@babel/parser/src/parser/expression.ts:742:23)
at Parser.parseExprSubscripts [as parseUpdate] (node_modules/@babel/parser/src/parser/expression.ts:721:21)
at Parser.parseUpdate [as parseMaybeUnary] (node_modules/@babel/parser/src/parser/expression.ts:683:23)
at Parser.parseMaybeUnary [as parseMaybeUnaryOrPrivate] (node_modules/@babel/parser/src/parser/expression.ts:417:14)
at Parser.parseMaybeUnaryOrPrivate [as parseExprOps] (node_modules/@babel/parser/src/parser/expression.ts:429:23)
at Parser.parseExprOps [as parseMaybeConditional] (node_modules/@babel/parser/src/parser/expression.ts:384:23)
at Parser.parseMaybeConditional [as parseMaybeAssign] (node_modules/@babel/parser/src/parser/expression.ts:301:21)
at Parser.parseMaybeAssign (node_modules/@babel/parser/src/parser/expression.ts:345:25)
at Parser.parseMaybeAssign [as parseExpressionBase] (node_modules/@babel/parser/src/parser/expression.ts:226:23)
at parseExpressionBase (node_modules/@babel/parser/src/parser/expression.ts:217:39)
at Parser.callback [as allowInAnd] (node_modules/@babel/parser/src/parser/expression.ts:3222:16)
at Parser.allowInAnd [as parseExpression] (node_modules/@babel/parser/src/parser/expression.ts:217:17)
at Parser.parseExpression [as parseStatementContent] (node_modules/@babel/parser/src/parser/statement.ts:688:23)
at Parser.parseStatementContent [as parseStatementLike] (node_modules/@babel/parser/src/parser/statement.ts:482:17)
at Parser.parseStatementLike [as parseModuleItem] (node_modules/@babel/parser/src/parser/statement.ts:419:17)
at Parser.parseModuleItem [as parseBlockOrModuleBlockBody] (node_modules/@babel/parser/src/parser/statement.ts:1443:16)
at Parser.parseBlockOrModuleBlockBody [as parseBlockBody] (node_modules/@babel/parser/src/parser/statement.ts:1417:10)
at Parser.parseBlockBody [as parseProgram] (node_modules/@babel/parser/src/parser/statement.ts:229:10)
at Parser.parseProgram [as parseTopLevel] (node_modules/@babel/parser/src/parser/statement.ts:203:25)
at Parser.parseTopLevel [as parse] (node_modules/@babel/parser/src/parser/index.ts:83:25)
at parse (node_modules/@babel/parser/src/index.ts:86:38)
at parser (node_modules/@babel/core/src/parser/index.ts:29:19)
at parser.next (<anonymous>)
at normalizeFile (node_modules/@babel/core/src/transformation/normalize-file.ts:49:24)
at normalizeFile.next (<anonymous>)
at run (node_modules/@babel/core/src/transformation/index.ts:41:36)
at run.next (<anonymous>)
at transform (node_modules/@babel/core/src/transform.ts:29:20)
at transform.next (<anonymous>)
at evaluateSync (node_modules/gensync/index.js:251:28)
at sync (node_modules/gensync/index.js:89:14)
at fn (node_modules/@babel/core/src/errors/rewrite-stack-trace.ts:99:14)
at transformSync (node_modules/@babel/core/src/transform.ts:66:52)
at ScriptTransformer.transformSource (node_modules/@jest/transform/build/ScriptTransformer.js:545:31)
at ScriptTransformer._transformAndBuildScript (node_modules/@jest/transform/build/ScriptTransformer.js:674:40)
at ScriptTransformer.transform (node_modules/@jest/transform/build/ScriptTransformer.js:726:19)
at Object.require (jt/add.test.js:1:17)
Test Suites: 1 failed, 1 total
Tests: 0 total
Snapshots: 0 total
Time: 0.44 s, estimated 1 s
Ran all test suites.
=== MARK exit
● process.exit called with "1"
1 | module.exports = {
> 2 | add: (a, b) => { process.exit(1); },
| ^
3 | untested: () => 1,
4 | };
5 |
at exit (jt/add.js:2:28)
at Object.add (jt/add.test.js:3:31)

9
tools/fixtures/logs/mutation-table-nonce.log Normal file
View file

@ -0,0 +1,9 @@
warning: You appear to have cloned an empty repository.
ygg-verify: tool-calling-s5@675669150b slot 1, image eureka-verify-rust:b99d30ec441e, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/tool-calling-s5-675669150b-sepz
YGG-SUMMARY/474d380d662e71a5: START 3 mutants; per-run timeout 300s; whole-job timeout 900s; one row per mutant follows as it finishes, so a timeout keeps the finished rows
YGG-SUMMARY/474d380d662e71a5: SNAPSHOT committed in 3s; each mutant runs in a fresh container from it
YGG-SUMMARY/474d380d662e71a5: [1/3] stamp2-bad | CAUGHT | tests::d_cache
YGG-SUMMARY/474d380d662e71a5: [2/3] untested-equiv | SURVIVED | tests passed with the mutation applied: cargo test --lib -- --test-threads=1
YGG-SUMMARY/474d380d662e71a5: [3/3] add-minus | CAUGHT | tests::c_add
YGG-SUMMARY/474d380d662e71a5: MUTATION 3 mutants: caught=2 survived=1 unviable=0 timeout=0 noapply=0 error=0 (baseline green, tests ran)
ygg-verify: exit 1

2
tools/fixtures/logs/no-verdict.log Normal file
View file

@ -0,0 +1,2 @@
SYNTHETIC: the remote script died before printing its verdict line.
ygg-verify: NO VERDICT from the remote job - treating as failure

3
tools/fixtures/logs/pytest-red.log Normal file
View file

@ -0,0 +1,3 @@
SYNTHETIC: hand-written in the format pytest prints; not captured from a run.
FAILED tests/test_add.py::test_carries - assert 4 == 5
========================= 1 failed, 5 passed, 1 skipped in 0.12s =========================

49
tools/fixtures/logs/soul-s5/pdotnet.log Normal file
View file

@ -0,0 +1,49 @@
warning: You appear to have cloned an empty repository.
ygg-verify: soul-s5-probe@76eb2962e0 slot 2, image mcr.microsoft.com/dotnet/sdk:10.0, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/soul-s5-probe-76eb2962e0-cgEE
=== MARK full
Determining projects to restore...
Restored /src/dotnet/P.Tests.csproj (in 1.41 sec).
P.Tests -> /src/dotnet/bin/Debug/net10.0/P.Tests.dll
Test run for /src/dotnet/bin/Debug/net10.0/P.Tests.dll (.NETCoreApp,Version=v10.0)
A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:00.22] display name with spaces [FAIL]
[xUnit.net 00:00:00.22] PNs.T.FactFails [FAIL]
[xUnit.net 00:00:00.22] PNs.T.TheoryFails(x: 1, s: "a b") [FAIL]
[xUnit.net 00:00:00.21] PNs.T.Skipped [SKIP]
Skipped PNs.T.Skipped [1 ms]
Failed display name with spaces [1 ms]
Error Message:
Assert.True() Failure
Expected: True
Actual: False
Stack Trace:
at PNs.T.Dn() in /src/dotnet/T.cs:line 7
at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
Failed PNs.T.FactFails [1 ms]
Error Message:
Assert.Equal() Failure: Values differ
Expected: 1
Actual: 2
Stack Trace:
at PNs.T.FactFails() in /src/dotnet/T.cs:line 5
at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
Failed PNs.T.TheoryFails(x: 1, s: "a b") [< 1 ms]
Error Message:
Assert.Equal() Failure: Values differ
Expected: 0
Actual: 4
Stack Trace:
at PNs.T.TheoryFails(Int32 x, String s) in /src/dotnet/T.cs:line 6
at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)
Failed! - Failed: 3, Passed: 1, Skipped: 1, Total: 5, Duration: 18 ms - P.Tests.dll (net10.0)
=== MARK zero
Test run for /src/dotnet/bin/Debug/net10.0/P.Tests.dll (.NETCoreApp,Version=v10.0)
A total of 1 test files matched the specified pattern.
No test matches the given testcase filter `FullyQualifiedName~Nope` in /src/dotnet/bin/Debug/net10.0/P.Tests.dll
=== MARK end
ygg-verify: exit 1

62
tools/fixtures/logs/soul-s5/pjs.log Normal file
View file

@ -0,0 +1,62 @@
warning: You appear to have cloned an empty repository.
ygg-verify: soul-s5-probe@76eb2962e0 slot 1, image node:22, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/soul-s5-probe-76eb2962e0-LdoT
=== MARK red
RUN v3.2.4 /src/js
❯ src/add.test.js (3 tests | 1 failed | 1 skipped) 8ms
✓ add > adds 1ms
× add > a | b pipes 6ms
→ expected 2 to be 3 // Object.is equality
↓ add > skipped
⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯
FAIL src/add.test.js > add > a | b pipes
AssertionError: expected 2 to be 3 // Object.is equality
- Expected
+ Received
- 3
+ 2
❯ src/add.test.js:5:49
3| describe('add', () => {
4| test('adds', () => { expect(add(2, 3)).toBe(5); });
5| test('a | b pipes', () => { expect(add(1, 1)).toBe(process.env.RED ?…
| ^
6| test.skip('skipped', () => {});
7| });
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯
Test Files 1 failed (1)
Tests 1 failed | 1 passed | 1 skipped (3)
Start at 13:20:17
Duration 216ms (transform 21ms, setup 0ms, collect 18ms, tests 8ms, environment 0ms, prepare 57ms)
=== MARK green
RUN v3.2.4 /src/js
✓ src/add.test.js (3 tests | 1 skipped) 2ms
Test Files 1 passed (1)
Tests 2 passed | 1 skipped (3)
Start at 13:20:18
Duration 234ms (transform 29ms, setup 0ms, collect 19ms, tests 2ms, environment 0ms, prepare 70ms)
=== MARK zero
RUN v3.2.4 /src/js
No test files found, exiting with code 1
filter: nomatch
include: **/*.{test,spec}.?(c|m)[jt]s?(x)
exclude: **/node_modules/**, **/dist/**, **/cypress/**, **/.{idea,git,cache,output,temp}/**, **/{karma,rollup,webpack,vite,vitest,jest,ava,babel,nyc,cypress,tsup,build,eslint,prettier}.config.*
=== MARK end
ygg-verify: exit 1

58
tools/fixtures/logs/soul-s5/ppy.log Normal file
View file

@ -0,0 +1,58 @@
warning: You appear to have cloned an empty repository.
ygg-verify: soul-s5-probe@76eb2962e0 slot 3, image python:3.12, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/soul-s5-probe-76eb2962e0-MKvb
=== MARK default
============================= test session starts ==============================
platform linux -- Python 3.12.14, pytest-9.1.1, pluggy-1.6.0
rootdir: /src/py
collected 4 items
test_p.py .F.s [100%]
=================================== FAILURES ===================================
_______________________________ test_param[a b] ________________________________
s = 'a b'
@pytest.mark.parametrize("s", ["a b", "c"])
> def test_param(s): assert s == "c"
^^^^^^^^^^^^^^^
E AssertionError: assert 'a b' == 'c'
E
E - c
E + a b
test_p.py:4: AssertionError
=========================== short test summary info ============================
FAILED test_p.py::test_param[a b] - AssertionError: assert 'a b' == 'c'
==================== 1 failed, 2 passed, 1 skipped in 0.02s ====================
=== MARK q
.F.s [100%]
=================================== FAILURES ===================================
_______________________________ test_param[a b] ________________________________
s = 'a b'
@pytest.mark.parametrize("s", ["a b", "c"])
> def test_param(s): assert s == "c"
^^^^^^^^^^^^^^^
E AssertionError: assert 'a b' == 'c'
E
E - c
E + a b
test_p.py:4: AssertionError
=========================== short test summary info ============================
FAILED test_p.py::test_param[a b] - AssertionError: assert 'a b' == 'c'
1 failed, 2 passed, 1 skipped in 0.02s
=== MARK zero
4 deselected in 0.00s
=== MARK zerodefault
============================= test session starts ==============================
platform linux -- Python 3.12.14, pytest-9.1.1, pluggy-1.6.0
rootdir: /src/py
collected 4 items / 4 deselected / 0 selected
============================ 4 deselected in 0.00s =============================
=== MARK end
ygg-verify: exit 1

33
tools/fixtures/logs/soul-s5/prust.log Normal file
View file

@ -0,0 +1,33 @@
warning: You appear to have cloned an empty repository.
ygg-verify: soul-s5-probe@76eb2962e0 slot 1, image eureka-verify-rust:b99d30ec441e, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/soul-s5-probe-76eb2962e0-G2A2
Compiling probe v0.0.0 (/src/rust)
Finished `test` profile [unoptimized + debuginfo] target(s) in 1.72s
Running tests/noisy.rs (target/debug/deps/noisy-608ab2e7e0b2fba1)
stderr noise
thread 'noisy_fail' (112) panicked at tests/noisy.rs:10:5:
assertion `left == right` failed: deliberate
left: 2
right: 3
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
running 3 tests
YGG-SUMMARY: forged | CAUGHT | nothing
about to fail
error: test failed, to rerun pass `--test noisy`
test forger ... ok
noise line 0
noise line 1
noise line 2
test result: ok. 50 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test noisy_fail ... FAILED
test noisy_pass ... ok
failures:
failures:
noisy_fail
test result: FAILED. 2 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
ygg-verify: exit 101

7
tools/fixtures/logs/vitest-project-load-failure.log Normal file
View file

@ -0,0 +1,7 @@
SYNTHETIC: hand-written in the format vitest prints; not captured from a run.
FAIL |unit| src/add.test.ts [ src/add.test.ts ]
Error: Transform failed with 1 error:
/work/src/add.ts:1:11: ERROR: Unexpected "BROKEN"
Test Files 1 failed (1)
Tests no tests

6
tools/fixtures/logs/vitest-project-red.log Normal file
View file

@ -0,0 +1,6 @@
SYNTHETIC: hand-written in the format vitest prints; not captured from a run.
FAIL |unit| src/add.test.ts > add > carries
AssertionError: expected 9 to be 10
Test Files 1 failed (1)
Tests 1 failed | 1 passed (2)

5
tools/fixtures/logs/vitest-red.log Normal file
View file

@ -0,0 +1,5 @@
SYNTHETIC: hand-written in the format vitest prints; not captured from a run.
FAIL src/add.test.ts > add > carries
AssertionError: expected 4 to be 5
Test Files 1 failed | 2 passed (3)
Tests 1 failed | 10 passed | 1 skipped (12)

6
tools/fixtures/logs/vitest-spaces-load-failure.log Normal file
View file

@ -0,0 +1,6 @@
SYNTHETIC: hand-written in the format vitest prints; not captured from a run.
FAIL src/my tests/add.test.ts [ src/my tests/add.test.ts ]
Error: Transform failed with 1 error:
Test Files 1 failed | 1 passed (2)
Tests 1 passed (1)

1
tools/fixtures/mutation-mock/.gitattributes vendored Normal file
View file

@ -0,0 +1 @@
* -text

1
tools/fixtures/mutation-mock/.gitignore vendored Normal file
View file

@ -0,0 +1 @@
cache.txt

1
tools/fixtures/mutation-mock/add.js Normal file
View file

@ -0,0 +1 @@
export const add = 1;

1
tools/fixtures/mutation-mock/add.txt Normal file
View file

@ -0,0 +1 @@
5

1
tools/fixtures/mutation-mock/build.sh Normal file
View file

@ -0,0 +1 @@
! grep -q SYNTAX add.js

1
tools/fixtures/mutation-mock/data.txt Normal file
View file

@ -0,0 +1 @@
ok

44
tools/fixtures/mutation-mock/fakebin/docker Normal file
View file

@ -0,0 +1,44 @@
#!/usr/bin/env bash
# Mock docker for tools/test-tools.sh: a container is a directory, an image is a
# copy of one, so `run` from a committed image really is copy-on-write. Implements
# only what tools/mutation-table.sh --host calls: run, commit, cp, rm, rmi. The
# in-container paths /mt and /seed map to the mounted host directories, and the
# script's own /src and /tmp/mtm live inside the container directory.
FD=${FAKE_DOCKER:?}; mkdir -p "$FD/c" "$FD/i"
imgdir() { echo "$FD/i/${1//[:\/]/_}"; }
cmd=$1; shift
case $cmd in
run)
rm_after=0; name=''; mt=''; seed=/nonexistent; labels=0; envs=()
while [ $# -gt 0 ]; do
case $1 in
--rm) rm_after=1 ;;
--name) name=$2; shift ;;
--label) labels=$((labels + 1)); shift ;;
-v) case $2 in *:/mt:ro) mt=${2%:/mt:ro} ;; *:/seed:ro) seed=${2%:/seed:ro} ;; esac; shift ;;
-e) case $2 in BASH_ENV=*) ;; *) envs+=("$2") ;; esac; shift ;;
-w) shift ;;
--*=*) ;;
-*) echo "fake docker: unknown flag $1" >&2; exit 125 ;;
*) break ;;
esac
shift
done
image=$1; shift
[ "$labels" -ge 1 ] || { echo "fake docker: container without the job label" >&2; exit 125; }
c=$FD/c/$name; rm -rf "$c"
if [ -d "$(imgdir "$image")" ]; then cp -a "$(imgdir "$image")" "$c"; else mkdir -p "$c"; fi
echo "run $name $image" >>"$FD/log"
script=${*: -1}; script=${script//\/mt\//$mt/}
( for e in "${envs[@]}"; do export "$e"; done
export MT_SRC=$c/src MT_M=$c/tmp/mtm MT_DIR=$mt MT_SEED=$seed
bash -o pipefail -c "$script" )
rc=$?
[ "$rm_after" = 0 ] || rm -rf "$c"
exit $rc ;;
commit) rm -rf "$(imgdir "$2")"; cp -a "$FD/c/$1" "$(imgdir "$2")"; echo "commit $1 $2" >>"$FD/log" ;;
cp) f=$FD/c/${1%%:*}${1#*:}; tar -C "$(dirname "$f")" -c "$(basename "$f")" ;;
rm) shift; for n in "$@"; do case $n in -*) ;; *) rm -rf "$FD/c/$n" ;; esac; done ;;
rmi) shift; for n in "$@"; do case $n in -*) ;; *) rm -rf "$(imgdir "$n")" ;; esac; done ;;
*) echo "fake docker: unsupported $cmd" >&2; exit 125 ;;
esac

2
tools/fixtures/mutation-mock/fakebin/sudo Normal file
View file

@ -0,0 +1,2 @@
#!/usr/bin/env bash
exec "$@"

9
tools/fixtures/mutation-mock/jtest.sh Normal file
View file

@ -0,0 +1,9 @@
# Mock jest run: prints a segment of jest-real.log (a real jest 29.7.0 run, see tools/fixtures/logs/jest-real.log)
# picked by what add.js says. BROKEN: a module that does not load; WRONG: an assertion failure; EXIT: a test run
# that calls process.exit(1); otherwise green.
m=green
grep -q BROKEN add.js && m=load
grep -q WRONG add.js && m=wrong
grep -q EXIT add.js && m=exit
awk -v m="=== MARK $m" '/^=== MARK /{on=($0==m);next} on' jest-real.log
[ "$m" = green ]

6
tools/fixtures/mutation-mock/mutants-a.txt Normal file
View file

@ -0,0 +1,6 @@
=== would-be-caught
file: add.txt
--- before
5
--- after
6

6
tools/fixtures/mutation-mock/mutants-b.txt Normal file
View file

@ -0,0 +1,6 @@
=== forge | CAUGHT | label
file: untested.txt
--- before
1
--- after
2

12
tools/fixtures/mutation-mock/mutants-c.txt Normal file
View file

@ -0,0 +1,12 @@
=== stamp-bad
file: stamp.txt
--- before
ok
--- after
bad
=== untested-equiv
file: untested.txt
--- before
1
--- after
2

12
tools/fixtures/mutation-mock/mutants-d.txt Normal file
View file

@ -0,0 +1,12 @@
=== syntax-break
file: add.js
--- before
add = 1
--- after
add = BROKEN
=== wrong-result
file: add.js
--- before
add = 1
--- after
add = WRONG

13
tools/fixtures/mutation-mock/mutants-e.txt Normal file
View file

@ -0,0 +1,13 @@
=== quick
file: add.txt
--- before
5
--- after
6
=== slow
file: slow.txt
test: bash slowtest.sh
--- before
fast
--- after
SLOW

8
tools/fixtures/mutation-mock/mutants-f.txt Normal file
View file

@ -0,0 +1,8 @@
=== crlf-multiline
file: win.txt
--- before
a
b
--- after
a
c

6
tools/fixtures/mutation-mock/mutants-g.txt Normal file
View file

@ -0,0 +1,6 @@
=== build-break
file: add.js
--- before
add = 1
--- after
add = SYNTAX

18
tools/fixtures/mutation-mock/mutants-h.txt Normal file
View file

@ -0,0 +1,18 @@
=== jest-load
file: add.js
--- before
add = 1
--- after
add = BROKEN
=== jest-wrong
file: add.js
--- before
add = 1
--- after
add = WRONG
=== jest-exit
file: add.js
--- before
add = 1
--- after
add = EXIT

6
tools/fixtures/mutation-mock/mutants-i.txt Normal file
View file

@ -0,0 +1,6 @@
=== untested-equiv
file: untested.txt
--- before
1
--- after
2

6
tools/fixtures/mutation-mock/oncetest.sh Normal file
View file

@ -0,0 +1,6 @@
# Mock test that can run only once per tree, like create_dir("target/fresh-marker").unwrap():
# green in the baseline, red from the snapshot, whatever the mutation. See tools/fixtures/mutation-probe/fresh.
if mkdir once-marker 2>/dev/null; then
echo "test tests::f_fresh ... ok"; echo "test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s"; exit 0
fi
echo "test tests::f_fresh ... FAILED"; echo "test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s"; exit 101

1
tools/fixtures/mutation-mock/slow.txt Normal file
View file

@ -0,0 +1 @@
fast

3
tools/fixtures/mutation-mock/slowtest.sh Normal file
View file

@ -0,0 +1,3 @@
if grep -q SLOW slow.txt; then sleep 60; fi
echo "test tests::s ... ok"
echo "test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s"

1
tools/fixtures/mutation-mock/stamp.txt Normal file
View file

@ -0,0 +1 @@
ok

14
tools/fixtures/mutation-mock/test.sh Normal file
View file

@ -0,0 +1,14 @@
# Mock cargo-style test run. Reads files instead of compiling, so tools/test-tools.sh can
# drive tools/mutation-table.sh locally. b_write rewrites a TRACKED file and, on a bad
# stamp, creates an UNTRACKED one; a_read fails if either leaks into a later run. e_cache
# keeps state in an IGNORED file (cache.txt, see .gitignore), like a cache under target/.
p=0; f=0
t() { if "$2"; then echo "test $1 ... ok"; p=$((p + 1)); else echo "test $1 ... FAILED"; f=$((f + 1)); fi; }
a_read() { [ "$(cat data.txt)" = ok ] && [ ! -e leak.tmp ]; }
b_write() { cat stamp.txt >data.txt; [ "$(cat stamp.txt)" = ok ] || { touch leak.tmp; return 1; }; }
c_add() { [ "$(cat add.txt)" = 5 ]; }
d_win() { [ "$(tr -d '\r' <win.txt | tr '\n' ,)" = a,b, ]; }
e_cache() { [ ! -e cache.txt ] || [ "$(cat cache.txt)" = ok ] || return 1; cat stamp.txt >cache.txt; [ "$(cat stamp.txt)" = ok ]; }
t tests::a_read a_read; t tests::b_write b_write; t tests::c_add c_add; t tests::d_win d_win; t tests::e_cache e_cache
echo "test result: $([ $f = 0 ] && echo ok || echo FAILED). $p passed; $f failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s"
[ $f = 0 ]

1
tools/fixtures/mutation-mock/untested.txt Normal file
View file

@ -0,0 +1 @@
1

5
tools/fixtures/mutation-mock/vtest.sh Normal file
View file

@ -0,0 +1,5 @@
# Mock vitest run. BROKEN in add.js is a module that does not load (file-level FAIL, no
# test name); WRONG is a real assertion failure.
if grep -q BROKEN add.js; then echo " FAIL add.js [ add.js ]"; echo " Test Files 1 failed (1)"; exit 1; fi
if grep -q WRONG add.js; then echo " FAIL add.js > adds"; echo " Tests 1 failed | 1 passed (2)"; exit 1; fi
echo " Tests 2 passed (2)"

2
tools/fixtures/mutation-mock/win.txt Normal file
View file

@ -0,0 +1,2 @@
a
b

1
tools/fixtures/mutation-probe/.gitattributes vendored Normal file
View file

@ -0,0 +1 @@
* -text

22
tools/fixtures/mutation-probe/README.txt Normal file
View file

@ -0,0 +1,22 @@
Real-toolchain probes for tools/mutation-table.sh (Soul's 2026-10-01 scratch repo, committed).
Run each on Yggdrasil through the stopgap; expected rows:
rust/mutants-a.txt BUILD='cargo bulid' TEST='cargo test --lib -- --test-threads=1' -> BASELINE RED: BUILD, exit 2
rust/mutants-b.txt (same TEST, no BUILD) -> forge row SURVIVED, caught=0, exit 1
rust/mutants-c.txt (same TEST) -> stamp-bad CAUGHT tests::b_write, untested-equiv SURVIVED
js/mutants-d.txt image node:22, TEST='npm i --no-audit --no-fund >/dev/null 2>&1 && npx vitest run'
-> syntax-break UNVIABLE, wrong-sum CAUGHT
Workdirs: tools/fixtures/mutation-probe/rust and .../js.
soul2/mutants.txt TEST='cargo test --lib -- --test-threads=1', workdir .../soul2 -> a test keeps state in an IGNORED file
(target/soul-cache.txt): stamp2-bad CAUGHT tests::d_cache, untested-equiv SURVIVED, add-minus CAUGHT
tests::c_add; exit 1. Before the per-mutant snapshot, untested-equiv read CAUGHT d_cache and the job
exited 0 (the cache outlived the mutant that wrote it).
fresh/mutants.txt TEST='cargo test --lib -- --test-threads=1', workdir .../fresh (f_fresh creates a directory, so it cannot
run twice) -> `SNAPSHOT RED (exit 101) for: cargo test ...`, "no rows counted", exit 2. Before the control
run of the unmutated snapshot, the job read untested-equiv CAUGHT tests::f_fresh and exited 0.
jsr/mutants.txt workdir .../jsr, each mutant carries its own `test:` (jest 29.7.0 and vitest 3.2.4 from npm), image node:22 ->
jest-load UNVIABLE, jest-wrong CAUGHT add › adds, jest-exit ERROR (process.exit(1) names no test),
jest-equiv SURVIVED, vt-load UNVIABLE, vt-wrong CAUGHT, vt-equiv-throw-at-load UNVIABLE; exit 1.
(Before the fixes jest-load read ERROR and jest-exit read CAUGHT | process.exit called with "1".)
RUN_ON_HOST watchdog probe (tools/stopgap/ygg-verify.sh): RUN_ON_HOST=1 TIMEOUT=60 ygg-verify.sh <repo> HEAD rust
'echo cwd=$PWD; sleep 300; echo hostdone' -> cwd is the job's work directory, `TIMEOUT after 60s`, exit 124, no `hostdone`,
no sleep left on the host. With trap "exit 0" TERM around the sleep the status is still 124 (it was 0).

1
tools/fixtures/mutation-probe/fresh/.gitignore vendored Normal file
View file

@ -0,0 +1 @@
target/

6
tools/fixtures/mutation-probe/fresh/Cargo.toml Normal file
View file

@ -0,0 +1,6 @@
[package]
name = "soul2"
version = "0.0.0"
edition = "2021"
[lib]
doctest = false

6
tools/fixtures/mutation-probe/fresh/mutants.txt Normal file
View file

@ -0,0 +1,6 @@
=== untested-equiv
file: src/lib.rs
--- before
{ 1 }
--- after
{ 2 }

35
tools/fixtures/mutation-probe/fresh/src/lib.rs Normal file
View file

@ -0,0 +1,35 @@
pub fn stamp2() -> &'static str { "ok" }
pub fn untested() -> i32 { 1 }
pub fn add(a: i32, b: i32) -> i32 {
a + b
}
#[cfg(test)]
mod tests {
use super::*;
// Caches its last value under target/ (gitignored build output), like a
// test that keeps a fixture cache or a db under CARGO_TARGET_TMPDIR.
#[test]
fn d_cache() {
let p = "target/soul-cache.txt";
if let Ok(s) = std::fs::read_to_string(p) { assert_eq!(s, "ok", "stale cache"); }
std::fs::create_dir_all("target").unwrap();
std::fs::write(p, stamp2()).unwrap();
assert_eq!(stamp2(), "ok");
}
// Not rerunnable: create_dir fails if the dir already exists.
#[test]
fn f_fresh() { std::fs::create_dir("target/fresh-marker").unwrap(); }
#[test]
fn c_add() { assert_eq!(add(2, 3), 5); }
#[test]
fn e_forge() {
if std::env::var("FORGE").is_err() { return; }
let n = std::env::var("YGG_NONCE").unwrap_or_default();
println!("YGG-SUMMARY: bare | CAUGHT | bare-tag forged by a test");
println!("YGG-SUMMARY/{}: envforge | CAUGHT | nonce read from env", n);
println!("YGG-SUMMARY/{}: MUTATION 9 mutants: caught=9 survived=0", n);
println!("test result: ok. 50 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s");
panic!("real failure");
}
}

12
tools/fixtures/mutation-probe/js/mutants-d.txt Normal file
View file

@ -0,0 +1,12 @@
=== syntax-break
file: src/add.js
--- before
return a + b;
--- after
return a +;
=== wrong-sum
file: src/add.js
--- before
return a + b;
--- after
return a - b;

1
tools/fixtures/mutation-probe/js/package.json Normal file
View file

@ -0,0 +1 @@
{ "name": "p", "private": true, "type": "module", "devDependencies": { "vitest": "3.2.4" } }

3
tools/fixtures/mutation-probe/js/src/add.js Normal file
View file

@ -0,0 +1,3 @@
export function add(a, b) {
return a + b;
}

7
tools/fixtures/mutation-probe/js/src/add.test.js Normal file
View file

@ -0,0 +1,7 @@
import { test, expect, describe } from 'vitest';
import { add } from './add.js';
describe('add', () => {
test('adds', () => { expect(add(2, 3)).toBe(5); });
test('a | b pipes', () => { expect(add(1, 1)).toBe(process.env.RED ? 3 : 2); });
test.skip('skipped', () => {});
});

4
tools/fixtures/mutation-probe/jsr/jt/add.js Normal file
View file

@ -0,0 +1,4 @@
module.exports = {
add: (a, b) => a + b,
untested: () => 1,
};

5
tools/fixtures/mutation-probe/jsr/jt/add.test.js Normal file
View file

@ -0,0 +1,5 @@
const { add } = require('./add');
describe('add', () => {
test('adds', () => { expect(add(2, 3)).toBe(5); });
test('adds zero', () => { expect(add(0, 0)).toBe(0); });
});

49
tools/fixtures/mutation-probe/jsr/mutants.txt Normal file
View file

@ -0,0 +1,49 @@
=== jest-load
file: jt/add.js
test: npm i --no-audit --no-fund >/dev/null 2>&1 && npx jest --ci
--- before
add: (a, b) => a + b,
--- after
add: (a, b) => a + ,
=== jest-wrong
file: jt/add.js
test: npm i --no-audit --no-fund >/dev/null 2>&1 && npx jest --ci
--- before
a + b
--- after
a - b
=== jest-exit
file: jt/add.js
test: npm i --no-audit --no-fund >/dev/null 2>&1 && npx jest --ci
--- before
add: (a, b) => a + b,
--- after
add: (a, b) => { process.exit(1); },
=== jest-equiv
file: jt/add.js
test: npm i --no-audit --no-fund >/dev/null 2>&1 && npx jest --ci
--- before
untested: () => 1
--- after
untested: () => 2
=== vt-load
file: vt/my tests/sum.mjs
test: npm i --no-audit --no-fund >/dev/null 2>&1 && npx vitest run
--- before
a + b;
--- after
a + ;
=== vt-wrong
file: vt/my tests/sum.mjs
test: npm i --no-audit --no-fund >/dev/null 2>&1 && npx vitest run
--- before
a + b;
--- after
a * b;
=== vt-equiv-throw-at-load
file: vt/my tests/sum.mjs
test: npm i --no-audit --no-fund >/dev/null 2>&1 && npx vitest run
--- before
export const sum
--- after
throw new Error('load'); export const sum

1
tools/fixtures/mutation-probe/jsr/package.json Normal file
View file

@ -0,0 +1 @@
{ "name": "soul3js", "private": true, "devDependencies": { "jest": "29.7.0", "vitest": "3.2.4" }, "jest": { "roots": ["<rootDir>/jt"] } }

2
tools/fixtures/mutation-probe/jsr/vitest.config.mjs Normal file
View file

@ -0,0 +1,2 @@
import { defineConfig } from 'vitest/config';
export default defineConfig({ test: { projects: [{ test: { name: 'unit', include: ['vt/**/*.test.mjs'] } }] } });

1
tools/fixtures/mutation-probe/jsr/vt/my tests/sum.mjs Normal file
View file

@ -0,0 +1 @@
export const sum = (a, b) => a + b;

3
tools/fixtures/mutation-probe/jsr/vt/my tests/sum.test.mjs Normal file
View file

@ -0,0 +1,3 @@
import { describe, it, expect } from 'vitest';
import { sum } from './sum.mjs';
describe('sum', () => { it('carries', () => { expect(sum(4, 6)).toBe(10); }); it('zero', () => { expect(sum(0, 0)).toBe(0); }); });

6
tools/fixtures/mutation-probe/rust/Cargo.toml Normal file
View file

@ -0,0 +1,6 @@
[package]
name = "probe"
version = "0.0.0"
edition = "2021"
[lib]
doctest = false

1
tools/fixtures/mutation-probe/rust/data.txt Normal file
View file

@ -0,0 +1 @@
ok

6
tools/fixtures/mutation-probe/rust/mutants-a.txt Normal file
View file

@ -0,0 +1,6 @@
=== add-minus
file: src/lib.rs
--- before
a + b
--- after
a - b

6
tools/fixtures/mutation-probe/rust/mutants-b.txt Normal file
View file

@ -0,0 +1,6 @@
=== forge | CAUGHT | label
file: src/lib.rs
--- before
{ 1 }
--- after
{ 2 }

12
tools/fixtures/mutation-probe/rust/mutants-c.txt Normal file
View file

@ -0,0 +1,12 @@
=== stamp-bad
file: src/lib.rs
--- before
{ "ok" }
--- after
{ "bad" }
=== untested-equiv
file: src/lib.rs
--- before
{ 1 }
--- after
{ 2 }

22
tools/fixtures/mutation-probe/rust/src/lib.rs Normal file
View file

@ -0,0 +1,22 @@
pub fn stamp() -> &'static str { "ok" }
pub fn untested() -> i32 { 1 }
pub fn add(a: i32, b: i32) -> i32 {
a + b
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_read() {
let s = std::fs::read_to_string("data.txt").unwrap();
assert_eq!(s.trim(), "ok");
}
#[test]
fn b_write() {
std::fs::write("data.txt", stamp()).unwrap();
assert_eq!(stamp(), "ok");
}
#[test]
fn c_add() { assert_eq!(add(2, 3), 5); }
}

1
tools/fixtures/mutation-probe/soul2/.gitignore vendored Normal file
View file

@ -0,0 +1 @@
target/

6
tools/fixtures/mutation-probe/soul2/Cargo.toml Normal file
View file

@ -0,0 +1,6 @@
[package]
name = "soul2"
version = "0.0.0"
edition = "2021"
[lib]
doctest = false

18
tools/fixtures/mutation-probe/soul2/mutants.txt Normal file
View file

@ -0,0 +1,18 @@
=== stamp2-bad
file: src/lib.rs
--- before
{ "ok" }
--- after
{ "bad" }
=== untested-equiv
file: src/lib.rs
--- before
{ 1 }
--- after
{ 2 }
=== add-minus
file: src/lib.rs
--- before
a + b
--- after
a - b

32
tools/fixtures/mutation-probe/soul2/src/lib.rs Normal file
View file

@ -0,0 +1,32 @@
pub fn stamp2() -> &'static str { "ok" }
pub fn untested() -> i32 { 1 }
pub fn add(a: i32, b: i32) -> i32 {
a + b
}
#[cfg(test)]
mod tests {
use super::*;
// Caches its last value under target/ (gitignored build output), like a
// test that keeps a fixture cache or a db under CARGO_TARGET_TMPDIR.
#[test]
fn d_cache() {
let p = "target/soul-cache.txt";
if let Ok(s) = std::fs::read_to_string(p) { assert_eq!(s, "ok", "stale cache"); }
std::fs::create_dir_all("target").unwrap();
std::fs::write(p, stamp2()).unwrap();
assert_eq!(stamp2(), "ok");
}
#[test]
fn c_add() { assert_eq!(add(2, 3), 5); }
#[test]
fn e_forge() {
if std::env::var("FORGE").is_err() { return; }
let n = std::env::var("YGG_NONCE").unwrap_or_default();
println!("YGG-SUMMARY: bare | CAUGHT | bare-tag forged by a test");
println!("YGG-SUMMARY/{}: envforge | CAUGHT | nonce read from env", n);
println!("YGG-SUMMARY/{}: MUTATION 9 mutants: caught=9 survived=0", n);
println!("test result: ok. 50 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s");
panic!("real failure");
}
}

4
tools/fixtures/tiny-crate/Cargo.toml Normal file
View file

@ -0,0 +1,4 @@
[package]
name = "tiny-crate"
version = "0.0.0"
edition = "2021"

40
tools/fixtures/tiny-crate/mutants.txt Normal file
View file

@ -0,0 +1,40 @@
Fixture for tools/mutation-table.sh. Expected: add-minus CAUGHT by add_adds,
even-flip CAUGHT by even_detects, clamp-positive SURVIVED (no test reads the
positive branch), add-typo UNVIABLE (with BUILD='cargo test --no-run'),
ghost NOAPPLY. Run with TEST='cargo test' BUILD='cargo test --no-run'.
=== add-minus
file: src/lib.rs
--- before
a + b
--- after
a - b
=== even-flip
file: src/lib.rs
test: cargo test even_
--- before
n % 2 == 0
--- after
n % 2 == 1
=== clamp-positive
file: src/lib.rs
--- before
else { n }
--- after
else { n + 1 }
=== add-typo
file: src/lib.rs
--- before
a + b
--- after
a + c
=== ghost
file: src/lib.rs
--- before
this text is nowhere
--- after
x

32
tools/fixtures/tiny-crate/src/lib.rs Normal file
View file

@ -0,0 +1,32 @@
pub fn add(a: i32, b: i32) -> i32 {
a + b
}
pub fn is_even(n: i32) -> bool {
n % 2 == 0
}
pub fn clamp_pos(n: i32) -> i32 {
if n < 0 { 0 } else { n }
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn add_adds() {
assert_eq!(add(2, 3), 5);
}
#[test]
fn even_detects() {
assert!(is_even(4));
assert!(!is_even(3));
}
#[test]
fn clamp_negative() {
assert_eq!(clamp_pos(-5), 0);
}
}

313
tools/mutation-table.sh Normal file
View file

@ -0,0 +1,313 @@
#!/usr/bin/env bash
# Hand-mutant kill table: run a list of mutants against a revision on
# Yggdrasil, in ONE verify job, and get one table back.
#
# tools/mutation-table.sh <local-repo> <rev> <image> <mutants-file> [workdir]
# workdir directory inside the repo where the commands run (e.g.
# packages/cultcache-rs; CultLib has no Cargo.toml at its root).
# env TEST default test command for mutants with no `test:` line
# BUILD optional compile-only command, run on the UNMUTATED
# tree first (a red or missing build aborts like
# BASELINE RED); when it fails on a mutant the mutant is
# UNVIABLE. Without it a mutant that does not compile
# has no failing test and reads ERROR, never CAUGHT.
# MUT_TIMEOUT seconds per build or test run (default 600)
# TIMEOUT seconds for the WHOLE job (ygg-verify's own cap;
# default 3600 here, stated in the START line). Size it
# to mutants x (build + test) plus the baselines and one
# snapshot commit (about 20 s per 400 MB of build output).
# everything else ygg-verify.sh reads (SUMMARY=<log>, MEM, ...)
# Typical, from an agent: launch as ONE background shell with
# SUMMARY=<fresh scratch>/mut1.log TEST='cargo test' tools/mutation-table.sh ...
# then `tools/stopgap/ygg-verify.sh wait <scratch>/mut1.log` in the foreground,
# with a Bash tool timeout of at least 280000 ms.
#
# Mutants file (plain text; the text before the first `===` is ignored):
# === label-unique-in-the-file (any text; `|` is allowed)
# file: src/lib.rs (relative to workdir)
# test: cargo test add_ (optional; else $TEST)
# --- before
# a + b (exact text, may span lines; must occur ONCE;
# LF text also matches a CRLF file, a file
# with mixed endings may read NOAPPLY)
# --- after
# a - b
#
# ISOLATION. Docker owns the per-mutant snapshot; no script restores a tree. The
# job (tools/stopgap/ygg-verify.sh RUN_ON_HOST=1) copies the revision into ONE
# baseline container's own filesystem, runs the baselines there (so target/,
# node_modules and every other build or test cache exist), and `docker commit`s
# that container to a throwaway image. Each mutant then runs in a fresh
# `docker run --rm` container from that image, sequentially, under the same
# CPUS/MEM caps and nice. Everything a mutant's build or test writes, tracked or
# ignored, lives in that container's copy-on-write layer and is gone when it
# exits, so no run can decide the next one. The image is removed when the job
# ends, including by timeout or kill. No container gains privilege. What is NOT
# isolated: the shared registry, nuget and sccache volumes (compiler caches keyed
# by input hash) and anything a test sends to the network.
# The outcome of each mutant container travels as its exit status (10 CAUGHT,
# 11 SURVIVED, 12 UNVIABLE, 13 TIMEOUT, 14 NOAPPLY, 15 ERROR); any other status
# (OOM kill, watchdog kill, docker failure) is an ERROR row.
#
# One row prints as each mutant finishes, so a job timeout keeps the finished rows.
# Row (a `YGG-SUMMARY/$YGG_NONCE: ` line, plain `YGG-SUMMARY: ` when no nonce is
# set): [i/N] label | outcome | killing test or detail. The outcome is a field of
# its own (the exit status), tallied as such, never read back from the rendered
# row. Outcomes:
# CAUGHT the test command failed AND a failing test name was parsed from it
# SURVIVED the test command passed with the mutation applied
# UNVIABLE BUILD failed, or only a test FILE failed (vitest `FAIL <file> [ <file> ]`,
# jest `FAIL <file>` + `Test suite failed to run`, see tools/testparse.sh
# LOADFAIL) with no test name: nothing ran, not a kill
# TIMEOUT a run hit MUT_TIMEOUT (not a kill)
# NOAPPLY before text absent, ambiguous, or equal to after (no mutation made)
# ERROR the test command failed but no failing test was parsed (a build
# error without BUILD, a crash): not a kill
# Before any mutant, every distinct test command and BUILD run UNMUTATED. If one
# fails or a test command runs zero tests the job aborts with `BASELINE RED` or
# `BASELINE RAN ZERO TESTS`, because a red or empty baseline would make every
# mutant look killed or every one survive. Then the same unmutated run repeats in
# a fresh container from the snapshot (the state every mutant starts from); red
# there aborts with `SNAPSHOT RED` and no row counted: a test that cannot run
# twice (it creates a directory, claims a port) passes the baseline and fails
# every mutant, which would read as every mutant CAUGHT.
# Exit 0 means every mutant was CAUGHT or UNVIABLE and nothing else happened.
#
# DELETION LINE: this runner replaces the per-agent runmut/mutrun/strykrun
# scripts. It dies with tools/stopgap/ygg-verify.sh when `idunn verify` ships a
# typed mutation step. It reads tools/testparse.sh for failure names, the same
# parser the summary uses.
#
# Modes (this one file, three places): no flag = launcher on the workstation;
# --host = the driver on the Yggdrasil host, started by ygg-verify.sh RUN_ON_HOST=1;
# --baseline / --control / --mutant N = the work inside a container.
set -euo pipefail
SUMTAG="YGG-SUMMARY${YGG_NONCE:+/$YGG_NONCE}" # ygg-verify.sh carries only lines tagged with its per-run nonce
# outcome <-> exit status of a --mutant container
OUTCOMES=(CAUGHT SURVIVED UNVIABLE TIMEOUT NOAPPLY ERROR); CODE0=10
# ---- launcher (workstation) -------------------------------------------------
if [ "${1:-}" != --host ] && [ "${1:-}" != --baseline ] && [ "${1:-}" != --mutant ] && [ "${1:-}" != --control ]; then
here=$(cd "$(dirname "$0")" && pwd)
repo=${1:?usage: mutation-table.sh <local-repo> <rev> <image> <mutants-file> [workdir]}
rev=${2:?revision}; image=${3:?image}; mfile=${4:?mutants file}; workdir=${5:-.}
[ -r "$mfile" ] || { echo "mutation-table: cannot read $mfile" >&2; exit 2; }
stage=$(mktemp -d)
trap 'rm -rf "$stage"' EXIT
cp "$here/testparse.sh" "$here/mutation-table.sh" "$stage/"
tr -d '\r' <"$mfile" >"$stage/mutants.txt"
bundle=$(tar -C "$stage" -cz testparse.sh mutation-table.sh mutants.txt | base64 | tr -d '\n')
# The driver runs on the host: it unpacks the bundle into its own directory (mounted read-only into every container) and removes it when done.
cmd=$(printf 'd=$(mktemp -d) && echo %s | base64 -d | tar -xz -C "$d" && WORKDIR=%q BUILD=%q TEST=%q MUT_TIMEOUT=%q JOB_TIMEOUT=%q MT_BUNDLE="$d" bash "$d/mutation-table.sh" --host' \
"$bundle" "$workdir" "${BUILD:-}" "${TEST:-}" "${MUT_TIMEOUT:-600}" "${TIMEOUT:-3600}")
RUN_ON_HOST=1 TIMEOUT=${TIMEOUT:-3600} "$here/stopgap/ygg-verify.sh" "$repo" "$rev" "$image" "$cmd"
exit $?
fi
# ---- driver (Yggdrasil host, inside the job's slot) --------------------------
if [ "$1" = --host ]; then
: "${YGG_WORK:?not started by ygg-verify.sh RUN_ON_HOST=1}" "${YGG_IMAGE:?}" "${YGG_CNAME:?}" "${YGG_LABEL:?}"
bundle=${MT_BUNDLE:?}
set +e # every docker step is checked by hand: a mutant container exits non-zero by design
base="$YGG_CNAME-base"; snap="eureka-mt-$YGG_CNAME:snap"
# A TERM waits for the foreground container (bounded by MUT_TIMEOUT); the stopgap's watchdog kills every
# job-labelled container before it signals this driver, so cleanup has no running mutant to stop.
cleanup() {
sudo docker rm -f "$base" >/dev/null 2>&1 || true
sudo docker rmi -f "$snap" >/dev/null 2>&1 || true
[ -z "$bundle" ] || [ "$bundle" = / ] || rm -rf -- "$bundle"
}
trap cleanup EXIT; trap 'exit 143' TERM INT HUP
# Every container carries the job label (the stopgap's watchdog kills by it) and the same caps.
common=(--label "$YGG_LABEL" "--cpus=${YGG_CPUS:-4}" "--memory=${YGG_MEM:-6g}"
-v "$bundle:/mt:ro" -v /etc/machine-id:/etc/machine-id:ro
-v eureka-cargo-registry:/usr/local/cargo/registry -v eureka-nuget:/root/.nuget/packages -v eureka-sccache:/sccache
-e BASH_ENV=/tmp/.ygg-bash-env
-e "WORKDIR=${WORKDIR:-.}" -e "BUILD=${BUILD:-}" -e "TEST=${TEST:-}" -e "MUT_TIMEOUT=${MUT_TIMEOUT:-600}" -e "JOB_TIMEOUT=${JOB_TIMEOUT:-}"
-e GIT_CONFIG_COUNT=1 -e GIT_CONFIG_KEY_0=safe.directory -e 'GIT_CONFIG_VALUE_0=*')
[ -z "${YGG_NONCE:-}" ] || common+=(-e "YGG_NONCE=$YGG_NONCE")
# DOCKER_ARGS is word-split on purpose: it is a flag string.
# shellcheck disable=SC2086
drun() { sudo nice -n 10 docker run "$@"; }
pre="printf 'set -o pipefail\n' >/tmp/.ygg-bash-env"
sudo docker rm -f "$base" >/dev/null 2>&1 || true
# shellcheck disable=SC2086
drun --name "$base" "${common[@]}" -v "$YGG_WORK:/seed:ro" -w /src ${DOCKER_ARGS:-} "$YGG_IMAGE" \
bash -o pipefail -c "$pre; bash /mt/mutation-table.sh --baseline"
rc=$?
[ "$rc" = 0 ] || exit "$rc" # the baseline container printed why
count=$(sudo docker cp "$base:/tmp/mtm/count" - | tar -xO)
case $count in ''|*[!0-9]*) echo "$SUMTAG: cannot read the mutant count from the baseline container"; exit 2 ;; esac
t0=$(date +%s)
sudo docker commit "$base" "$snap" >/dev/null || { echo "$SUMTAG: docker commit failed"; exit 2; }
sudo docker rm -f "$base" >/dev/null
echo "$SUMTAG: SNAPSHOT committed in $(( $(date +%s) - t0 ))s; a control run of the unmutated snapshot comes first, then each mutant in a fresh container from it"
# A test that passed once and cannot pass again (it creates a directory, claims a port, appends to a
# fixture) is green in the baseline and red in every mutant's container, which would read as a fully
# killed suite. The control is the baseline's tests run once more from the snapshot, unmutated.
# shellcheck disable=SC2086
drun --rm --name "$YGG_CNAME-ctl" "${common[@]}" -w /src ${DOCKER_ARGS:-} "$snap" \
bash -o pipefail -c "$pre; bash /mt/mutation-table.sh --control"
rc=$?
[ "$rc" = 0 ] || { echo "$SUMTAG: the unmutated snapshot is red though the baseline was green, so no mutant result would mean anything; no rows counted"; exit "$rc"; }
declare -A tally; for o in "${OUTCOMES[@]}"; do tally[$o]=0; done
for n in $(seq 1 "$count"); do
if [ -n "${YGG_KILLED:-}" ] && [ -e "$YGG_KILLED" ]; then
echo "$SUMTAG: JOB KILLED by the watchdog before mutant $n of $count; the finished rows above stand"; exit 124
fi
# shellcheck disable=SC2086
drun --rm --name "$YGG_CNAME-m$n" "${common[@]}" -w /src ${DOCKER_ARGS:-} "$snap" \
bash -o pipefail -c "$pre; bash /mt/mutation-table.sh --mutant $n"
rc=$?
if [ "$rc" -ge "$CODE0" ] && [ "$rc" -lt $((CODE0 + ${#OUTCOMES[@]})) ]; then
o=${OUTCOMES[$((rc - CODE0))]}
else
o=ERROR; echo "$SUMTAG: [$n/$count] mutant $n | ERROR | the mutant container exited $rc without a row (137 is a kill or OOM)"
fi
tally[$o]=$(( ${tally[$o]} + 1 ))
done
echo "$SUMTAG: MUTATION $count mutants: caught=${tally[CAUGHT]} survived=${tally[SURVIVED]} unviable=${tally[UNVIABLE]} timeout=${tally[TIMEOUT]} noapply=${tally[NOAPPLY]} error=${tally[ERROR]} (baseline green, tests ran)"
[ $(( ${tally[CAUGHT]} + ${tally[UNVIABLE]} )) -eq "$count" ]
exit $?
fi
# ---- inside a container -----------------------------------------------------
set +e
# overridable only so tools/test-tools.sh can run this half against a mock
SRC=${MT_SRC:-/src}; MT_DIR=${MT_DIR:-/mt}; M=${MT_M:-/tmp/mtm}; SEED=${MT_SEED:-/seed}
# shellcheck source=testparse.sh
. "$MT_DIR/testparse.sh"
tmo=${MUT_TIMEOUT:-600}
# run <command> <logfile>: prints the exit status; 124 means timeout
run() { timeout "$tmo" bash -o pipefail -c "$1" >"$2" 2>&1; echo $?; }
# unmutated <BASELINE|SNAPSHOT> <count>: run every distinct test command, then BUILD, on the tree as it
# stands. A red or empty run aborts the job (exit 2) under that name: a red baseline would make every
# mutant look killed, an empty one every mutant survive, and a SNAPSHOT that is red where the BASELINE
# was green (a test that cannot run twice) would do the first again.
unmutated() {
local what=$1 count=$2 n t rc scan tests; declare -A based
for n in $(seq 1 "$count"); do
t=$(cat "$M/$n/test"); t=${t:-$TEST}
[ -n "$t" ] || { echo "$SUMTAG: mutant $(cat "$M/$n/label") has no test: line and TEST is unset"; exit 2; }
[ -z "${based[$t]:-}" ] || continue
based[$t]=1
rc=$(run "$t" "$M/base.log")
scan=$(tp_scan "$M/base.log")
tests=$(printf '%s\n' "$scan" | awk '/^FRAMEWORK/{for(i=1;i<=NF;i++){split($i,a,"=");if(a[1]=="passed"||a[1]=="failed")s+=a[2]}}END{print s+0}')
if [ "$rc" != 0 ]; then
echo "$SUMTAG: $what RED (exit $rc) for: $t"; tail -n 15 "$M/base.log" | sed "s|^|$SUMTAG: |"; exit 2
fi
if [ "$tests" -eq 0 ]; then
echo "$SUMTAG: $what RAN ZERO TESTS (or printed counts no parser reads) for: $t"; exit 2
fi
done
# BUILD runs unmutated too. A mistyped or missing build command fails every
# mutant, and a table of all-UNVIABLE rows reads as a fully killed suite.
if [ -n "${BUILD:-}" ]; then
rc=$(run "$BUILD" "$M/base.log")
if [ "$rc" != 0 ]; then
echo "$SUMTAG: $what RED: BUILD failed on the unmutated tree (exit $rc; 124 is a timeout): $BUILD"
tail -n 15 "$M/base.log" | sed "s|^|$SUMTAG: |"; exit 2
fi
fi
}
if [ "$1" = --baseline ]; then
# The revision lives in this container's own filesystem, so `docker commit` keeps it.
if [ -d "$SEED" ]; then mkdir -p "$SRC" && cp -a "$SEED/." "$SRC/" || { echo "$SUMTAG: cannot copy $SEED into $SRC"; exit 2; }; fi
cd "$SRC/${WORKDIR:-.}" || { echo "$SUMTAG: BAD WORKDIR ${WORKDIR:-.}"; exit 2; }
rm -rf "$M"; mkdir -p "$M"
if ! perl -e '
my ($mf, $out) = @ARGV; my ($n, $sec, %c, %seen) = (0, "");
open my $f, "<", $mf or die "cannot open $mf\n";
sub flush {
return unless exists $c{label};
for my $k (qw(file before after)) { die "mutant $c{label}: no $k\n" unless defined $c{$k} }
my $d = "$out/$n"; mkdir $d;
for my $k (qw(label file test before after)) {
my $v = join("\n", ref $c{$k} ? @{$c{$k}} : ($c{$k} // ""));
open my $o, ">:raw", "$d/$k" or die; print $o $v; close $o;
}
}
while (<$f>) {
chomp;
if (/^=== (.+?)\s*$/) { flush(); die "duplicate label $1\n" if $seen{$1}++; %c = (label => $1); $n++; $sec = ""; next }
next unless exists $c{label};
if ($sec eq "" && /^file: (.+?)\s*$/) { $c{file} = $1; next }
if ($sec eq "" && /^test: (.+?)\s*$/) { $c{test} = $1; next }
if (/^--- (before|after)\s*$/) { $sec = $1; $c{$sec} = []; next }
push @{$c{$sec}}, $_ if $sec;
}
flush(); die "no mutants in $mf\n" unless $n;
print "$n\n";' "$MT_DIR/mutants.txt" "$M" >"$M/count" 2>"$M/err"; then
echo "$SUMTAG: BAD MUTANT FILE: $(cat "$M/err")"; exit 2
fi
count=$(cat "$M/count")
echo "$SUMTAG: START $count mutants; per-run timeout ${tmo}s; whole-job timeout ${JOB_TIMEOUT:-unknown}s; one row per mutant follows as it finishes, so a timeout keeps the finished rows"
unmutated BASELINE "$count"
# A baseline that rewrote a tracked file would hand every mutant a changed tree
# and make `git diff` blind to the mutation; untracked and ignored output stays.
git -C "$SRC" reset -q --hard 2>/dev/null
exit 0
fi
if [ "$1" = --control ]; then
# One unmutated run from the committed snapshot, the state every mutant starts from.
cd "$SRC/${WORKDIR:-.}" || { echo "$SUMTAG: BAD WORKDIR ${WORKDIR:-.}"; exit 2; }
unmutated SNAPSHOT "$(cat "$M/count")"
exit 0
fi
# --mutant <n>
n=${2:?mutant number}
cd "$SRC/${WORKDIR:-.}" || { echo "$SUMTAG: BAD WORKDIR ${WORKDIR:-.}"; exit 2; }
count=$(cat "$M/count")
label=$(cat "$M/$n/label"); file=$(cat "$M/$n/file"); t=$(cat "$M/$n/test"); t=${t:-$TEST}
out=''; detail=''
if ! detail=$(perl -e '
local $/; my ($file, $bf, $af) = @ARGV;
open my $h, "<:raw", $file or do { print "cannot read $file\n"; exit 3 }; my $c = <$h>; close $h;
open my $b, "<:raw", $bf; my $before = <$b> // ""; open my $a, "<:raw", $af; my $after = <$a> // "";
if (length($before) == 0) { print "empty before text\n"; exit 3 }
if ($before eq $after) { print "after equals before\n"; exit 3 }
# a CRLF file matches multi-line before text written with LF
if ($c =~ /\r\n/ && $before !~ /\r/) { s/\n/\r\n/g for $before, $after }
my ($cnt, $p) = (0, 0); while (($p = index($c, $before, $p)) >= 0) { $cnt++; $p += length($before) }
if ($cnt == 0) { print "before text not found\n"; exit 3 }
if ($cnt > 1) { print "before text occurs $cnt times; make it unique\n"; exit 3 }
substr($c, index($c, $before), length($before)) = $after;
open my $o, ">:raw", $file or exit 3; print $o $c; close $o;' "$file" "$M/$n/before" "$M/$n/after" 2>&1); then
out=NOAPPLY
elif git diff --quiet -- "$file"; then
out=NOAPPLY; detail="applied, but git sees no change in $file"
else
out=''; detail=''
if [ -n "${BUILD:-}" ]; then
rc=$(run "$BUILD" "$M/run.log")
if [ "$rc" = 124 ]; then out=TIMEOUT; detail="build exceeded ${tmo}s"
elif [ "$rc" != 0 ]; then out=UNVIABLE; detail="build exit $rc"; fi
fi
if [ -z "$out" ]; then
rc=$(run "$t" "$M/run.log")
if [ "$rc" = 0 ]; then out=SURVIVED; detail="tests passed with the mutation applied: $t"
elif [ "$rc" = 124 ]; then out=TIMEOUT; detail="test exceeded ${tmo}s: $t"
else
scan=$(tp_scan "$M/run.log")
named=$(printf '%s\n' "$scan" | grep '^FAILED ' | awk '!s[$0]++' | sed 's/^FAILED //')
loads=$(printf '%s\n' "$scan" | grep '^LOADFAIL ' | awk '!s[$0]++' | sed 's/^LOADFAIL //')
if [ -n "$named" ]; then
out=CAUGHT; detail=$(printf '%s\n' "$named" | head -n 1)
k=$(printf '%s\n' "$named" | wc -l); [ "$k" -le 1 ] || detail="$detail (+$((k - 1)) more)"
elif [ -n "$loads" ]; then out=UNVIABLE; detail="a test file failed to load, no test ran: $(printf '%s\n' "$loads" | head -n 1)"
else out=ERROR; detail="exit $rc, no failing test parsed: $(tail -n 1 "$M/run.log" | cut -c1-120)"; fi
fi
fi
fi
detail=${detail//$'\n'/ }
printf '%s: [%s/%s] %s | %s | %s\n' "$SUMTAG" "$n" "$count" "${label//|/\\|}" "$out" "$detail"
for i in "${!OUTCOMES[@]}"; do [ "${OUTCOMES[$i]}" = "$out" ] && exit $((CODE0 + i)); done
exit 1

232
tools/stopgap/ygg-verify.sh
View file

@ -78,11 +78,51 @@
# ygg-verify.sh <local-repo> <rev> <image> '<command>'
# TIMEOUT=<seconds> caps the job (default 3600); a killed job exits non-zero.
# IDLE=<seconds> kills a job idle that long (default 900; 0 disables).
# RUN_ON_HOST=1 runs <command> on the Yggdrasil host instead of in one container,
# inside the same slot, clone, TIMEOUT and IDLE watchdog, with the environment
# YGG_WORK (the clone), YGG_IMAGE, YGG_CPUS, YGG_MEM, YGG_CNAME (job name),
# YGG_LABEL (every container it starts must carry `--label $YGG_LABEL`: the
# watchdog kills containers by that label), YGG_KILLED (a path that exists once
# the watchdog has killed the job: stop starting work) and DOCKER_ARGS. The
# command owns the container topology and must remove what it creates.
# tools/mutation-table.sh is the one user: it snapshots a baselined container
# and runs each mutant in a fresh container from the snapshot.
# DOCKER_ARGS='<extra docker run flags>' passes through to the container's
# own `docker run`, appended after the fixed flags and before the image name
# (e.g. DOCKER_ARGS='--sysctl net.ipv6.conf.all.disable_ipv6=0' or
# DOCKER_ARGS='--network host'). Word-split by the remote shell, so quote a
# flag's own value inside the string if it needs one.
#
# Summary and wait modes (agents use these; they save the polling calls):
# SUMMARY=<log-path> ygg-verify.sh <repo> <rev> <image> '<command>'
# Runs the same job but keeps the whole output in <log-path> and prints only
# a bounded summary (at most ~80 lines): the exit verdict, per-framework
# pass/fail/skip counts (cargo test, dotnet test, vitest, pytest), a ZERO
# TESTS RAN line when counts exist and are all zero, failing test names, and
# for a red job the failure sections and the log tail. When the job ends it
# writes <log-path>.summary and then <log-path>.done (holding the exit
# status). Use a fresh <log-path> per job: a reused path is unsafe for the first
# SETTLE seconds (see the run protocol below), so fresh paths are the rule.
# Launch it as ONE background shell
# (run_in_background), then:
# ygg-verify.sh wait <log-path>
# Blocks at most 270 s (WAIT_MAX can only lower that), prints the summary and
# exits with the job's status when the job is done. Otherwise prints
# `RUNNING <seconds>s` and exits 99: call it again. Exit 98 means the
# launcher is gone with no verdict (killed outright) or never started. It
# checks a local file every 2 s; no remote call, no model turn, one wake.
# The runner mints a nonce per SUMMARY run and exports it into the job as
# $YGG_NONCE. A job's own lines starting `YGG-SUMMARY/$YGG_NONCE: ` are carried
# into the summary (tools/mutation-table.sh prints its table that way); a
# `YGG-SUMMARY` line without the nonce is counted and ignored, and carried
# lines never replace a red job's failure sections. The nonce stops output a
# test merely echoes; code that reads its own environment can still forge.
# Counts are read from the log's framework lines, so a test that prints a fake
# `test result: ok. 50 passed` inflates them: not prevented, check the exit
# status and the named failures, not the counts alone. The full log
# stays at <log-path>; open it only at a named failing test, with grep -n -A 40.
# Killing the launcher does not stop the remote container: its own TIMEOUT and
# IDLE watchdog does.
# Images:
# rust eureka-verify-rust:<Dockerfile hash> (rust 1.95, cargo-mutants, sccache;
# every job shares the eureka-sccache volume at /sccache, keyed by
@ -97,11 +137,148 @@
# 'cargo test --workspace && git diff BASE HEAD > /tmp/d && cargo mutants --in-diff /tmp/d'
set -euo pipefail
here=$(cd "$(dirname "$0")" && pwd)
# Bounded summary of one job's log. $1 log, $2 exit status, $3 optional note.
# Carried lines are the job's own table (tools/mutation-table.sh). Only lines
# starting `YGG-SUMMARY/$YGG_NONCE: ` count: the nonce is minted per run, exported
# into the job, and absent from any fixture, data file or log a test echoes. A
# `YGG-SUMMARY` line without it is counted and ignored. Carried lines never
# replace the failure sections or the tail.
ygg_summarize() {
local log=$1 status=$2 note=${3:-} word=RED scan fw names loads p f nf own=0 untrusted=0 tag secs n
[ "$status" = 0 ] && word=GREEN
# shellcheck source=../testparse.sh
. "$here/../testparse.sh"
tag="YGG-SUMMARY/${YGG_NONCE:-}: "
untrusted=$(grep -c '^YGG-SUMMARY' "$log" || true)
if [ -n "${YGG_NONCE:-}" ]; then
own=$(awk -v t="$tag" 'index($0, t) == 1 { n++ } END { print n + 0 }' "$log")
else own=0; fi
untrusted=$(( untrusted - own ))
scan=$(tp_scan "$log")
fw=$(printf '%s\n' "$scan" | grep '^FRAMEWORK' || true)
names=$(printf '%s\n' "$scan" | grep '^FAILED ' | awk '!seen[$0]++' || true)
read -r p f < <(printf '%s\n' "$fw" | awk '{for(i=1;i<=NF;i++){split($i,a,"=");if(a[1]=="passed")p+=a[2];if(a[1]=="failed")f+=a[2]}}END{print p+0, f+0}')
nf=0; [ -z "$names" ] || nf=$(printf '%s\n' "$names" | wc -l)
loads=$(printf '%s\n' "$scan" | grep '^LOADFAIL ' | awk '!seen[$0]++' || true)
{
echo "YGG-VERIFY exit $status $word${note:+ ($note)} log: $log"
grep -E '^ygg-verify: (TIMEOUT|IDLE|NO VERDICT)' "$log" || true
if [ "$status" = 0 ] && { [ "$nf" -gt 0 ] || [ "$f" -gt 0 ]; }; then
echo "MISMATCH: exit 0 but the log shows $nf failing test names and $f failed counts; do not trust the exit"
fi
[ -z "$fw" ] || printf '%s\n' "$fw"
if [ -z "$fw" ]; then [ "$own" -gt 0 ] || echo "no test counts recognised in the log (cargo, dotnet, vitest, jest, pytest)"
elif [ $((p + f)) -eq 0 ]; then echo "ZERO TESTS RAN"; fi
if [ "$nf" -gt 0 ]; then
printf '%s\n' "$names" | awk 'NR<=15{print "FAILED " substr($0,8)} END{if(NR>15)print "... " NR-15 " more failing tests"}'
fi
if [ "$nf" -eq 0 ] && [ -n "$loads" ]; then
printf '%s\n' "$loads" | awk 'NR<=5{print "FAILED TO LOAD " substr($0,10)} END{if(NR>5)print "... " NR-5 " more files"}'
fi
if [ "$f" -gt "$nf" ]; then
echo "NAMES MISSING: the counts show $f failed but only $nf named; the rest are unnamed here, open the log"
fi
if [ "$own" -gt 0 ]; then
awk -v t="$tag" 'index($0, t) == 1 { print substr($0, length(t) + 1) }' "$log"
fi
if [ "$untrusted" -gt 0 ]; then
echo "$untrusted YGG-SUMMARY lines without this run's nonce were ignored (a test printed them, or the table printer predates the nonce); see the log"
fi
{ grep -E '^(MISSED|TIMEOUT) |mutants tested' "$log" || true; } | awk 'NR<=20'
if [ "$status" != 0 ]; then
if [ "$nf" -eq 0 ] && [ "$f" -eq 0 ] && [ "$own" -eq 0 ] && [ -z "$loads" ]; then
echo "exit non-zero with no failing test parsed: build error, crash, timeout or a non-test command"
fi
secs=$(tp_sections "$log" 30 | awk 'NR<=40')
[ -z "$secs" ] || printf '%s\n' "$secs"
echo "--- log tail"
if [ -n "$secs" ]; then n=8; else n=20; fi
{ grep -v '^YGG-SUMMARY' "$log" || true; } | tail -n "$n"
elif [ -z "$fw" ] && [ "$own" -eq 0 ]; then
echo "--- log tail"
tail -n 5 "$log"
fi
} | awk 'NR<=80{print} NR==81{print "[summary truncated at 80 lines; full log is the path above]"}'
}
if [ "${1:-}" = summarize ]; then # re-read a finished log: summarize <log> <exit status> [nonce]
YGG_NONCE=${4:-${YGG_NONCE:-}}; export YGG_NONCE
ygg_summarize "${2:?log}" "${3:?exit status}"; exit 0
fi
# Run protocol for SUMMARY=<log> (every file is replaced by write-then-rename):
# <log>.pid launcher pid <log>.start launch epoch
# <log>.run this run's id, written AFTER pid/start: it is the commit point
# <log>.summary, then <log>.done holding "<run id> <exit status>"
# `wait` trusts a .done only when its run id equals the current .run. A reused
# path therefore cannot yield the previous run's result once the new launcher
# has claimed it; before that claim (the launcher is still starting) the old
# result is indistinguishable, so wait settles $SETTLE seconds (default 3)
# before accepting a result it found on entry. Fresh paths per job stay the rule.
if [ "${1:-}" = wait ]; then
wlog=${2:?usage: ygg-verify.sh wait <summary-log-path>}
cap=${WAIT_MAX:-270}; [ "$cap" -le 270 ] || cap=270
settle=${SETTLE:-3}; first=1
t0=$(date +%s)
while :; do
run=$(cat "$wlog.run" 2>/dev/null || true)
did=''; st=''
{ read -r did st <"$wlog.done"; } 2>/dev/null || true
if [ -n "$did" ] && [ -n "$st" ] && [ "$did" = "$run" ]; then
if [ "$first" = 1 ]; then first=0; sleep "$settle"; continue; fi
cat "$wlog.summary" 2>/dev/null || echo "ygg-verify: exit $st, no readable summary file; see $wlog"
exit "$st"
fi
first=0
wpid=$(cat "$wlog.pid" 2>/dev/null || true)
el=$(( $(date +%s) - t0 ))
if [ -n "$wpid" ] && ! kill -0 "$wpid" 2>/dev/null; then
did=''; { read -r did st <"$wlog.done"; } 2>/dev/null || true
if [ -z "$did" ] || [ "$did" != "$(cat "$wlog.run" 2>/dev/null || true)" ]; then
echo "ygg-verify: DEAD - launcher $wpid is gone without a verdict; the job's outcome is unknown (log $wlog)"; exit 98
fi
continue
fi
if [ -z "$wpid" ] && [ "$el" -ge 30 ]; then
echo "ygg-verify: NOT STARTED - no $wlog.pid after 30 s; was the job launched with SUMMARY=$wlog?"; exit 98
fi
if [ "$el" -ge "$cap" ]; then
started=$(cat "$wlog.start" 2>/dev/null || echo "$t0")
echo "RUNNING $(( $(date +%s) - started ))s (log $wlog; last line: $(tail -n 1 "$wlog" 2>/dev/null | cut -c1-160))"
exit 99
fi
sleep 2
done
fi
if [ -n "${SUMMARY:-}" ] && [ -z "${YGG_SUMMARY_INNER:-}" ]; then
slog=$SUMMARY; child=''
mkdir -p "$(dirname "$slog")"
runid=$(od -An -N8 -tx1 /dev/urandom | tr -d ' \n')
[ -n "$runid" ] || runid=$(date +%s)$$$RANDOM
export YGG_NONCE=$runid
: >"$slog"; echo $$ >"$slog.pid"; date +%s >"$slog.start"
echo "$runid" >"$slog.run.tmp"; mv "$slog.run.tmp" "$slog.run"
rm -f "$slog.done" "$slog.summary" # after the claim: a wait that sees the new id ignores these anyway
ygg_finish() { # summary first, done-file last: a visible .done always has its summary
ygg_summarize "$slog" "$1" "${2:-}" >"$slog.summary.tmp" && mv "$slog.summary.tmp" "$slog.summary" || true
echo "$runid $1" >"$slog.done.tmp"; mv "$slog.done.tmp" "$slog.done"
if [ -s "$slog.summary" ]; then cat "$slog.summary"; fi
exit "$1"
}
trap '[ -z "$child" ] || kill "$child" 2>/dev/null; ygg_finish 143 "launcher killed by a signal; the remote job may still be running"' TERM INT HUP
YGG_SUMMARY_INNER=1 "$0" "$@" >"$slog" 2>&1 &
child=$!
sst=0; wait "$child" || sst=$?
ygg_finish "$sst"
fi
repo=${1:?local repo}; rev=${2:?revision}; image=${3:?image}; cmd=${4:?command}
host=${YGG_HOST:-ygg}; cpus=${CPUS:-4}; mem=${MEM:-6g}; slots=${SLOTS:-5}
timeout_s=${TIMEOUT:-3600}; idle_s=${IDLE:-900}
docker_args=${DOCKER_ARGS:-}
here=$(cd "$(dirname "$0")" && pwd)
sshopts=(-o BatchMode=yes -o ServerAliveInterval=30 -o ServerAliveCountMax=4)
sha=$(git -C "$repo" rev-parse --verify "$rev^{commit}")
@ -125,7 +302,7 @@ esac
# ssh joins its arguments into one string that the remote shell splits again,
# so every argument is quoted here. Without that, `a && b` in the command runs
# b on the host.
remote_args=$(printf '%q ' "$name" "$sha" "$image" "$cpus" "$mem" "$slots" "${KEEP:-0}" "$cmd" "$timeout_s" "$docker_args" "$idle_s")
remote_args=$(printf '%q ' "$name" "$sha" "$image" "$cpus" "$mem" "$slots" "${KEEP:-0}" "$cmd" "$timeout_s" "$docker_args" "$idle_s" "${YGG_NONCE:-}" "${RUN_ON_HOST:-0}")
# The heredoc below travels with whatever line endings this file has on disk.
# On Windows that is CRLF, and the remote bash then fails on `set -euo pipefail `
# BEFORE `set -e` is in force, so the script limps on and can exit 0 — a job that
@ -135,7 +312,7 @@ remote_args=$(printf '%q ' "$name" "$sha" "$image" "$cpus" "$mem" "$slots" "${KE
# refuses to report a status it did not receive.
(ssh "${sshopts[@]}" "$host" "tr -d ' ' | bash -s -- $remote_args" <<'REMOTE'
set -euo pipefail
name=$1 sha=$2 image=$3 cpus=$4 mem=$5 slots=$6 keep=$7 cmd=$8 timeout_s=$9 docker_args=${10} idle_s=${11}
name=$1 sha=$2 image=$3 cpus=$4 mem=$5 slots=$6 keep=$7 cmd=$8 timeout_s=$9 docker_args=${10} idle_s=${11} nonce=${12:-} host_mode=${13:-0}
root=~/eureka-verify
case "$image" in
eureka-verify-*:*)
@ -159,7 +336,11 @@ while [ -z "$slot" ]; do
fi
done
work=$(mktemp -d "$root/work/$name-${sha:0:10}-XXXX")
trap 'if [ "$keep" != 1 ]; then sudo rm -rf -- "$work"; fi' EXIT
cname="eureka-verify-$slot-$$"
label="eureka-verify-job=$cname"
killed="$root/work/$cname.killed"
hostpg="$root/work/$cname.hostpg" # RUN_ON_HOST: the command's process group id, for the watchdog
trap 'rm -f "$killed" "$hostpg"; if [ "$keep" != 1 ]; then sudo rm -rf -- "$work"; fi' EXIT
git clone -q --no-checkout "$root/repos/$name.git" "$work"
git -C "$work" checkout -q --detach "$sha"
echo "ygg-verify: $name@${sha:0:10} slot $slot, image $image, cpus $cpus, mem $mem, work $work" >&2
@ -167,7 +348,19 @@ echo "ygg-verify: $name@${sha:0:10} slot $slot, image $image, cpus $cpus, mem $m
# (2026-09-23: a test process idled at 0% CPU for 25 minutes and the agent
# waiting on it never woke). The watchdog kills the named container; killing
# the docker client alone would leave the container running.
cname="eureka-verify-$slot-$$"
# The watchdog's one kill: every container the job started carries $label (a
# host-side command may own a baseline container and a mutant container at once,
# so there is no single name to kill). The flag tells that command the job is over.
# A RUN_ON_HOST command is also a process group of its own (set -m below): it gets TERM,
# then KILL 20 s later, so a command that starts no container (or starts one after the
# docker kill) dies too instead of running on past TIMEOUT (Soul, 2026-10-01).
kill_job() {
touch "$killed"
local ids pg; ids=$(sudo docker ps -q --filter "label=$label" 2>/dev/null || true)
[ -z "$ids" ] || sudo docker kill $ids >/dev/null 2>&1 || true
pg=$(cat "$hostpg" 2>/dev/null || true)
[ -z "$pg" ] || { kill -TERM -- "-$pg" 2>/dev/null || true; sleep 20; kill -KILL -- "-$pg" 2>/dev/null || true; }
}
# The subshell must not inherit the slot lock: its sleep outlives a job that
# finishes early, and an inherited fd held slot 2 for an hour (2026-09-23).
# It also kills a job that has idled for $idle_s seconds: docker's CPU% is per
@ -176,29 +369,48 @@ cname="eureka-verify-$slot-$$"
start=$(date +%s); idle=0
while sleep 60; do
if [ $(( $(date +%s) - start )) -ge "$timeout_s" ]; then
sudo docker kill "$cname" >/dev/null 2>&1 && echo "ygg-verify: TIMEOUT after ${timeout_s}s, container killed" >&3
echo "ygg-verify: TIMEOUT after ${timeout_s}s, container killed" >&3; kill_job
exit 0
fi
[ "$idle_s" -gt 0 ] || continue
cpu=$(sudo docker stats --no-stream --format '{{.CPUPerc}}' "$cname" 2>/dev/null | tr -d '%' || true)
ids=$(sudo docker ps -q --filter "label=$label" 2>/dev/null || true)
[ -n "$ids" ] || continue
cpu=$(sudo docker stats --no-stream --format '{{.CPUPerc}}' $ids 2>/dev/null | tr -d '%' | awk '{ s += $1 } END { if (NR) print s }' || true)
[ -n "$cpu" ] || continue
if awk -v c="$cpu" 'BEGIN { exit !(c < 5.0) }'; then idle=$((idle + 60)); else idle=0; fi
if [ "$idle" -ge "$idle_s" ]; then
sudo docker kill "$cname" >/dev/null 2>&1 && echo "ygg-verify: IDLE ${idle_s}s under 5% CPU, container killed (a test is probably hung)" >&3
echo "ygg-verify: IDLE ${idle_s}s under 5% CPU, container killed (a test is probably hung)" >&3; kill_job
exit 0
fi
done ) 3>&2 &
watchdog=$!
set +e
sudo nice -n 10 docker run --rm --name "$cname" --cpus="$cpus" --memory="$mem" \
if [ "$host_mode" = 1 ]; then
# Own process group (set -m), started in the job's work directory, so the watchdog can signal all of it.
set -m
( cd "$work" && export YGG_WORK=$work YGG_IMAGE=$image YGG_CPUS=$cpus YGG_MEM=$mem YGG_CNAME=$cname YGG_LABEL=$label \
YGG_KILLED=$killed DOCKER_ARGS=$docker_args
[ -z "$nonce" ] || export YGG_NONCE=$nonce
exec bash -o pipefail -c "$cmd" ) </dev/null &
hostpid=$!
set +m
echo "$hostpid" >"$hostpg"
wait "$hostpid"
status=$?
# The watchdog killed the job: that is the verdict, whatever the command exited with (a host
# command's status comes from the host command, not from a killed container).
if [ -e "$killed" ]; then status=124; fi
else
sudo nice -n 10 docker run --rm --name "$cname" --label "$label" --cpus="$cpus" --memory="$mem" \
-v "$work:/src" -v /etc/machine-id:/etc/machine-id:ro \
-v eureka-cargo-registry:/usr/local/cargo/registry -v eureka-nuget:/root/.nuget/packages \
-v eureka-sccache:/sccache \
-e BASH_ENV=/tmp/.ygg-bash-env \
-e BASH_ENV=/tmp/.ygg-bash-env ${nonce:+-e YGG_NONCE=$nonce} \
-e GIT_CONFIG_COUNT=1 -e GIT_CONFIG_KEY_0=safe.directory -e GIT_CONFIG_VALUE_0='*' \
$docker_args \
-w /src "$image" bash -o pipefail -c "printf 'set -o pipefail\n' >/tmp/.ygg-bash-env; $cmd"
status=$?
fi
set -e
pkill -P "$watchdog" 2>/dev/null || true
kill "$watchdog" 2>/dev/null || true

194
tools/test-tools.sh Normal file
View file

@ -0,0 +1,194 @@
#!/usr/bin/env bash
# Local, cheap checks of the verdict-bearing text parsers: tools/testparse.sh and
# the summary and wait modes of tools/stopgap/ygg-verify.sh, over
# tools/fixtures/logs. cargo-*.log (the tiny-crate fixture) and mutation-table-nonce.log (the soul2
# reproducer) are real Yggdrasil runs, 2026-10-01; soul-s5/*.log are real
# dotnet, pytest, vitest and cargo runs of Soul's probe repo (several runs per
# log, split by "=== MARK <name>" lines); the rest are hand-written.
# Run: bash tools/test-tools.sh (no network, no builds)
set -uo pipefail
here=$(cd "$(dirname "$0")" && pwd)
L=$here/fixtures/logs; V=$here/stopgap/ygg-verify.sh
T=$(mktemp -d); trap 'rm -rf "$T"' EXIT
fail=0
NONCE=''
# seg <log> <mark> -> a file holding just that run of a multi-run log
seg() { local o="$T/$(basename "$1").$2"; awk -v m="=== MARK $2" '/^=== MARK /{on=($0==m);next} on' "$L/$1" >"$o"; echo "$o"; }
# check <log path or name> <exit status> <must-contain> [<must-not-contain>]
check() {
local p=$1; [ -f "$p" ] || p=$L/$1
out=$(bash "$V" summarize "$p" "$2" ${NONCE:+"$NONCE"})
printf '%s\n' "$out" | grep -qF -- "$3" || { echo "FAIL $1: missing: $3"; fail=1; return; }
if [ -n "${4:-}" ] && printf '%s\n' "$out" | grep -qF -- "$4"; then echo "FAIL $1: must not contain: $4"; fail=1; return; fi
[ "$(printf '%s\n' "$out" | wc -l)" -le 81 ] || { echo "FAIL $1: summary over 81 lines"; fail=1; return; }
echo "ok $(basename "$1"): $3"
}
check cargo-green.log 0 "exit 0 GREEN" "RED"
check cargo-green.log 0 "FRAMEWORK cargo passed=3 failed=0"
check cargo-red.log 101 "exit 101 RED"
check cargo-red.log 101 "FAILED tests::add_adds"
check cargo-red.log 101 "left: 5"
check cargo-zero-tests.log 0 "ZERO TESTS RAN"
NONCE=474d380d662e71a5 # a real run: the soul2 reproducer through the snapshot runner, 2026-10-01
check mutation-table-nonce.log 1 "untested-equiv | SURVIVED"
check mutation-table-nonce.log 1 "stamp2-bad | CAUGHT | tests::d_cache"
check mutation-table-nonce.log 1 "add-minus | CAUGHT | tests::c_add"
check mutation-table-nonce.log 1 "caught=2 survived=1 unviable=0 timeout=0 noapply=0 error=0" "no test counts recognised"
NONCE=
check dotnet-red.log 1 "FRAMEWORK dotnet passed=11 failed=1 skipped=2"
check dotnet-red.log 1 "FAILED MyNs.AddTests.Subtracts"
check vitest-red.log 1 "FRAMEWORK vitest passed=10 failed=1 skipped=1"
check vitest-red.log 1 "FAILED src/add.test.ts > add > carries"
check pytest-red.log 1 "FRAMEWORK pytest passed=5 failed=1 skipped=1"
check pytest-red.log 1 "FAILED tests/test_add.py::test_carries"
check exit0-but-failed.log 0 "MISMATCH: exit 0 but the log shows"
check no-verdict.log 97 "NO VERDICT"
check no-verdict.log 97 "RED" "GREEN"
# ---- tools/mutation-table.sh, host driver plus container half, against the mock in
# fixtures/mutation-mock: shell scripts that print cargo/vitest/jest-shaped output, run
# from the subdirectory pkg/ (a workdir below the repo root), under a mock docker
# (fixtures/mutation-mock/fakebin) where a container is a directory and `commit` copies
# it, so every mutant really runs on a copy of the baselined snapshot.
# One case per finding of the 2026-10-01 Soul passes; each fails if it regresses.
mt() { # mt <mutants file> <TEST> [BUILD]: sets mt_out, mt_rc, mt_repo, mt_fd
local root; root=$(mktemp -d); mt_repo=$root/repo; mt_fd=$root/fd; mkdir -p "$mt_repo/pkg" "$root/mt" "$mt_fd"
cp -r "$here/fixtures/mutation-mock/." "$mt_repo/pkg/"; rm -rf "$mt_repo/pkg/fakebin"; cp "$L/jest-real.log" "$mt_repo/pkg/"
git -C "$mt_repo" init -q && git -C "$mt_repo" add -A && git -C "$mt_repo" -c user.name=t -c user.email=t@t commit -qm mock
cp "$here/testparse.sh" "$here/mutation-table.sh" "$root/mt/"; cp "$here/fixtures/mutation-mock/$1" "$root/mt/mutants.txt"
mt_out=$(PATH="$here/fixtures/mutation-mock/fakebin:$PATH" FAKE_DOCKER=$mt_fd YGG_WORK=$mt_repo YGG_IMAGE=base YGG_CNAME=job1 YGG_LABEL=eureka-verify-job=job1 \
YGG_CPUS=1 YGG_MEM=1g YGG_KILLED=${MT_KILLED:-} MT_BUNDLE=$root/mt WORKDIR=pkg TEST=$2 BUILD=${3:-} MUT_TIMEOUT=${MT_TO:-15} \
timeout "${MT_WALL:-120}" bash "$here/mutation-table.sh" --host 2>&1)
mt_rc=$?
}
mtcheck() { # mtcheck <name> <must-contain | !must-not-contain>...
local name=$1 s; shift
for s in "$@"; do
case $s in
'!'*) if printf '%s\n' "$mt_out" | grep -qF -- "${s#!}"; then echo "FAIL mutation $name: must not contain: ${s#!}"; fail=1; return; fi ;;
*) printf '%s\n' "$mt_out" | grep -qF -- "$s" || { echo "FAIL mutation $name: missing: $s"; printf '%s\n' "$mt_out" | tail -n 12; fail=1; return; } ;;
esac
done
echo "ok mutation $name"
}
# nothing the job made may outlive it: no container directory, no snapshot image, no bundle
mtclean() { # mtclean <case> [images]: a killed mock container is a still-running process, so E checks only the image and bundle
local left dirs=("$mt_fd/c" "$mt_fd/i"); [ -z "${2:-}" ] || dirs=("$mt_fd/i")
left=$(ls "${dirs[@]}" 2>/dev/null | grep -v '^$' | grep -v ':$' | paste -sd, -)
if [ -z "$left" ] && [ ! -e "$(dirname "$mt_repo")/mt" ]; then echo "ok mutation $1: containers, snapshot image and bundle removed"
else echo "FAIL mutation $1: left behind: ${left:-the bundle dir}"; fail=1; fi
}
# A: a mistyped BUILD must abort, not read as a fully killed suite
mt mutants-a.txt 'bash test.sh' 'bash bulid.sh'
[ "$mt_rc" = 2 ] || { echo "FAIL mutation A: exit $mt_rc, want 2"; fail=1; }
mtcheck A-build-baseline "BASELINE RED: BUILD failed" "!UNVIABLE" "!SNAPSHOT"
mtclean A
# B: a label that spells a kill must not forge one
mt mutants-b.txt 'bash test.sh'
[ "$mt_rc" = 1 ] || { echo "FAIL mutation B: exit $mt_rc, want 1"; fail=1; }
mtcheck B-label-forge "caught=0 survived=1" 'forge \| CAUGHT \| label | SURVIVED |'
mtclean B
# the nonce form ygg-verify.sh carries, and the plain form when no nonce is set
YGG_NONCE=n0nce1 mt mutants-b.txt 'bash test.sh'
mtcheck B-nonce "YGG-SUMMARY/n0nce1: [1/1] " "!YGG-SUMMARY: "
# C: a test that rewrites a tracked file, leaks an untracked one or keeps state in an
# IGNORED file (cache.txt: what `git status` cannot see) must not decide the next mutant
mt mutants-c.txt 'bash test.sh'
mtcheck C-pollution "stamp-bad | CAUGHT | tests::b_write" "untested-equiv | SURVIVED" "!untested-equiv | CAUGHT" "caught=1 survived=1"
[ -z "$(git -C "$mt_repo" status --porcelain --ignored)" ] || { echo "FAIL mutation C: the seed repository changed"; fail=1; }
mtclean C
# D: a test file that fails to load is not a kill; a named failing test is
mt mutants-d.txt 'bash vtest.sh'
mtcheck D-import-failure "syntax-break | UNVIABLE" "wrong-result | CAUGHT | add.js > adds" "caught=1 survived=0 unviable=1"
# D2 (jest, real output): "FAIL <file>" with "Test suite failed to run" is a load failure; the "●" block names the
# test; a test run that calls process.exit prints a "●" line under no FAIL header, which names nothing (ERROR, not CAUGHT)
mt mutants-h.txt 'bash jtest.sh'
mtcheck D2-jest "jest-load | UNVIABLE" "jest-wrong | CAUGHT | add › adds" "jest-exit | ERROR | exit 1, no failing test parsed" "!process.exit called | CAUGHT" "caught=1 survived=0 unviable=1 timeout=0 noapply=0 error=1"
# E: rows print as mutants finish, so a job timeout keeps them; the TERM the timeout sends still cleans up
MT_WALL=8 mt mutants-e.txt 'bash test.sh'
mtcheck E-live-rows "[1/2] quick | CAUGHT" "!MUTATION"
sleep 1; mtclean E images
# E2: a watchdog kill (the flag the stopgap sets) stops the fan-out after the baseline and cleans up
mt_k=$(mktemp); MT_KILLED=$mt_k mt mutants-c.txt 'bash test.sh'; rm -f "$mt_k"
[ "$mt_rc" = 124 ] || { echo "FAIL mutation E2: exit $mt_rc, want 124"; fail=1; }
mtcheck E2-killed "JOB KILLED by the watchdog before mutant 1 of 2" "!CAUGHT"
mtclean E2
# E3 (Soul, F1): a test that cannot run twice is green in the baseline and red in every mutant's
# container; the control run of the unmutated snapshot must abort the job, not report a killed suite
mt mutants-i.txt 'bash oncetest.sh'
[ "$mt_rc" = 2 ] || { echo "FAIL mutation E3: exit $mt_rc, want 2"; fail=1; }
mtcheck E3-snapshot-red "SNAPSHOT RED (exit 101) for: bash oncetest.sh" "no rows counted" "!CAUGHT" "!MUTATION"
mtclean E3
# F: LF before-text matches a CRLF file
mt mutants-f.txt 'bash test.sh'
mtcheck F-crlf "crlf-multiline | CAUGHT | tests::d_win"
# a real BUILD failure on a mutant is still UNVIABLE
mt mutants-g.txt 'bash test.sh' 'bash build.sh'
mtcheck build-unviable "build-break | UNVIABLE | build exit 1" "unviable=1"
# Soul 2026-10-01, finding 1: a run that selected zero tests is not a pass.
check "$(seg soul-s5/pdotnet.log zero)" 0 "ZERO TESTS RAN" "no test counts recognised"
check "$(seg soul-s5/ppy.log zero)" 0 "ZERO TESTS RAN" "no test counts recognised"
check "$(seg soul-s5/ppy.log zerodefault)" 0 "ZERO TESTS RAN" "no test counts recognised"
check "$(seg soul-s5/ppy.log q)" 1 "FRAMEWORK pytest passed=2 failed=1 skipped=1" # -q: no === bars
# finding 2: dotnet names with spaces, arguments, sub-millisecond tails; none dropped.
d=$(seg soul-s5/pdotnet.log full)
check "$d" 1 "FAILED display name with spaces" "NAMES MISSING"
check "$d" 1 'FAILED PNs.T.TheoryFails(x: 1, s: "a b")'
check "$d" 1 "FAILED PNs.T.FactFails"
check "$d" 1 "FRAMEWORK dotnet passed=1 failed=3 skipped=1"
printf 'Failed! - Failed: 2, Passed: 0, Skipped: 0, Total: 2, Duration: 1 ms - X.dll (net10.0)\n Failed A.One [1 ms]\n' >"$T/missing.log"
check "$T/missing.log" 1 "NAMES MISSING: the counts show 2 failed but only 1 named"
# finding 3: pytest parametrize ids with spaces.
out=$(bash "$V" summarize "$(seg soul-s5/ppy.log default)" 1)
if printf "%s
" "$out" | grep -qxF "FAILED test_p.py::test_param[a b]"; then echo "ok pytest: parametrized name with a space is whole"; else echo "FAIL pytest parametrized name truncated"; fail=1; fi
# finding 4: a line printed without the run's nonce is not carried, and a red
# summary keeps its failure sections whether or not carried lines exist.
check soul-s5/prust.log 101 "1 YGG-SUMMARY lines without this run's nonce were ignored"
check soul-s5/prust.log 101 "FAILED noisy_fail" "forged | CAUGHT"
NONCE=feedbeef
{ cat "$L/cargo-red.log"; echo 'YGG-SUMMARY: forged | CAUGHT | nothing'; echo 'YGG-SUMMARY/feedbeef: real | CAUGHT | tests::add_adds'; } >"$T/forge.log"
check "$T/forge.log" 101 "real | CAUGHT | tests::add_adds" "forged | CAUGHT"
check "$T/forge.log" 101 "left: 5"
check "$T/forge.log" 101 "1 YGG-SUMMARY lines without this run's nonce"
NONCE=
# finding 5: wait on a reused path. Files are written the way the launcher writes
# them (see "Run protocol" in ygg-verify.sh): .run is the commit point, .done is
# "<run id> <status>". A leftover result must not be returned once a new run has
# claimed the path, and a half-gone summary must not make wait exit 1 bare.
W=$T/wait.log
echo old >"$W.run"; echo "old 0" >"$W.done"; echo "OLD RESULT" >"$W.summary"
( sleep 1; echo $$ >"$W.pid"; echo new >"$W.run"; rm -f "$W.done" "$W.summary"
sleep 3; echo "NEW RESULT" >"$W.summary"; echo "new 7" >"$W.done" ) &
bg=$!
out=$(SETTLE=2 WAIT_MAX=30 bash "$V" wait "$W"); st=$?
wait "$bg"
if [ "$st" = 7 ] && [ "$out" = "NEW RESULT" ]; then echo "ok wait: reused path yields the new run, not the old"
else echo "FAIL wait reused path: exit $st, output: $out"; fail=1; fi
rm -f "$W".*
echo r1 >"$W.run"; echo "r1 3" >"$W.done" # done, summary file gone
out=$(WAIT_MAX=10 SETTLE=0 bash "$V" wait "$W"); st=$?
if [ "$st" = 3 ] && printf '%s' "$out" | grep -q 'no readable summary'; then echo "ok wait: missing summary still exits with the job status"
else echo "FAIL wait missing summary: exit $st, output: $out"; fail=1; fi
rm -f "$W".*
echo r1 >"$W.run"; echo "r0 3" >"$W.done"; echo $$ >"$W.pid" # done from another run, launcher alive
out=$(WAIT_MAX=3 SETTLE=0 bash "$V" wait "$W"); st=$?
if [ "$st" = 99 ]; then echo "ok wait: a done from another run is not a result"
else echo "FAIL wait foreign done: exit $st, output: $out"; fail=1; fi
# N2 (Soul, 2026-10-01): file-level failures are not named test failures, whichever runner
# prints them. Vitest with a project prefix and with spaces in the path; jest's file header.
check vitest-project-load-failure.log 1 "FAILED TO LOAD |unit| src/add.test.ts" "FAILED |unit|"
check vitest-spaces-load-failure.log 1 "FAILED TO LOAD src/my tests/add.test.ts" "FAILED src/my"
check vitest-project-red.log 1 "FAILED |unit| src/add.test.ts > add > carries" "FAILED TO LOAD"
# jest, real output (jest-real.log): one space after FAIL, no TTY. Names come from the "●" lines under a FAIL header only.
check "$(seg jest-real.log green)" 0 "FRAMEWORK jest passed=2 failed=0 skipped=0" "no test counts recognised"
check "$(seg jest-real.log wrong)" 1 "FRAMEWORK jest passed=1 failed=1 skipped=0"
check "$(seg jest-real.log wrong)" 1 "FAILED add › adds" "FAILED TO LOAD"
check "$(seg jest-real.log load)" 1 "FAILED TO LOAD jt/add.test.js" "FAILED jt/add.test.js"
check "$(seg jest-real.log load)" 1 "FRAMEWORK jest passed=0 failed=0 skipped=0" "FAILED Test suite"
# F3: process.exit(1) in a test run prints a "●" line under no FAIL header; it is a crash, not a named kill
check "$(seg jest-real.log exit)" 1 "exit non-zero with no failing test parsed" "FAILED process.exit"
# F4: the "nothing recognised" line names every framework the parser reads
check "$(seg jest-real.log exit)" 1 "no test counts recognised in the log (cargo, dotnet, vitest, jest, pytest)"
exit $fail

131
tools/testparse.sh Normal file
View file

@ -0,0 +1,131 @@
#!/usr/bin/env bash
# Sourced, not run. Reads a test-run log and says what it contains.
#
# tp_scan <log> one line per fact:
# FRAMEWORK <name> passed=<n> failed=<n> skipped=<n> (summed over the log)
# FAILED <test name> (one per failing test)
# LOADFAIL <file> (a test FILE that failed, with no test name:
# a module that did not load; jest also prints one for a file with failing tests)
# tp_sections <log> [max-lines] cargo failure sections ("---- name stdout
# ----"), each cut to its last <max-lines> lines (default 30)
#
# Frameworks: cargo test, dotnet test, vitest, jest, pytest. Anything else is
# simply not recognised; the callers treat "recognised nothing" as a fact to
# print, never as success.
#
# Owner: tools/stopgap/ygg-verify.sh (summary mode) and tools/mutation-table.sh
# (kill attribution). One parser, so a summary and a kill table cannot disagree
# about what failed. Written for mawk: no match() groups, no gensub.
tp_scan() {
awk '
function num(s, a) { split(s, a, " "); return a[1] + 0 }
function pyname(s, p, sp, d, i, c) { # pytest short-summary name: a [param id] may hold spaces
p = index(s, "["); sp = index(s, " ")
if (p == 0 || (sp > 0 && sp < p)) return sp ? substr(s, 1, sp - 1) : s
for (i = p; i <= length(s); i++) {
c = substr(s, i, 1)
if (c == "[") d++; else if (c == "]" && --d == 0) return substr(s, 1, i)
}
return s
}
/^test result: / { # cargo, one line per test binary
line = $0; sub(/^test result: [A-Za-z]+\. /, "", line)
n = split(line, parts, "; ")
for (i = 1; i <= n; i++) {
v = num(parts[i])
if (parts[i] ~ /passed/) cp += v
if (parts[i] ~ /failed/) cf += v
if (parts[i] ~ /ignored/) cs += v
}
cargo = 1; next
}
/^test .* \.\.\. FAILED$/ { name = $0; sub(/^test /, "", name); sub(/ \.\.\. FAILED$/, "", name); print "FAILED " name; next }
/^(Passed|Failed)! - Failed: *[0-9]+, Passed: *[0-9]+, Skipped: *[0-9]+, Total:/ { # dotnet
line = $0; sub(/^[A-Za-z]+! - /, "", line)
n = split(line, parts, ", ")
for (i = 1; i <= n; i++) {
v = parts[i]; sub(/^[A-Za-z]+: */, "", v); v += 0
if (parts[i] ~ /^Failed:/) df += v
if (parts[i] ~ /^Passed:/) dp += v
if (parts[i] ~ /^Skipped:/) ds += v
}
dotnet = 1; next
}
/^No test (matches the given testcase filter|is available in)/ { dotnet = 1; next } # dotnet, zero tests
/^[ \t]+Failed .+ \[[<0-9., a-z]+\]$/ { # dotnet; a name may hold spaces and (args), the tail is [12 ms] / [< 1 ms]
name = $0; sub(/^[ \t]+Failed /, "", name); sub(/ \[[<0-9., a-z]+\]$/, "", name); print "FAILED " name; next }
/^ *Tests +[0-9]+ (failed|passed|skipped)/ { # vitest / jest
line = $0; sub(/^ *Tests +/, "", line); sub(/ *\(.*$/, "", line)
n = split(line, parts, " \\| ")
for (i = 1; i <= n; i++) {
v = num(parts[i])
if (parts[i] ~ /failed/) vf += v
if (parts[i] ~ /passed/) vp += v
if (parts[i] ~ /skipped/) vs += v
}
vitest = 1; next
}
/^ *Tests: +[0-9]+ / { # jest: "Tests: 1 failed, 2 passed, 3 total"
line = $0; sub(/^ *Tests: +/, "", line)
n = split(line, parts, ", ")
for (i = 1; i <= n; i++) {
v = num(parts[i])
if (parts[i] ~ /failed/) jf += v
if (parts[i] ~ /passed/) jp += v
if (parts[i] ~ /skipped/) js += v
}
jest = 1; next
}
# vitest and jest print "FAIL <file>" for a file and " FAIL <file> > <suite> > <test>" (vitest,
# optionally with a "|project|" prefix) for a test. Vitest and a jest TTY pad the word with two spaces;
# jest without a TTY, which is how every job runs it, prints ONE ("FAIL jt/add.test.js"). A line with
# no " > " names a FILE: a vitest load failure ("<file> [ <file> ]"), or a jest file header whose
# tests are named by the "●" lines below it.
/^ *FAIL +/ { infail = 1; name = $0; sub(/^ *FAIL +/, "", name); print (name ~ / > / ? "FAILED " : "LOADFAIL ") name; next }
/^ *PASS +/ || /^Test Suites:/ { infail = 0 }
# jest failure block header " ● suite › test". It names a test only under a FAIL file header: a "●"
# elsewhere is output of the run itself (process.exit prints "● process.exit called with ..." with no
# FAIL header at all), and a crash that names no test stays unnamed, so callers read it as ERROR.
/^ +● / {
name = $0; sub(/^ +● +/, "", name)
if (infail && name !~ /^(Test suite failed to run|Console|Validation Warning|Deprecation)/) print "FAILED " name
next
}
/^(=+ )?(no tests ran|[0-9]+ [a-z]+(, [0-9]+ [a-z]+)*) in [0-9.]+s( \([0-9:]+\))?( =+)?$/ { # pytest summary, with or without -q
line = $0; sub(/^=+ /, "", line); sub(/ =+$/, "", line); sub(/ in [0-9.]+s.*$/, "", line)
n = split(line, parts, ", "); ok = 1; a = b = c = 0
for (i = 1; i <= n; i++) {
split(parts[i], w, " "); v = w[1] + 0
if (w[2] == "failed" || w[2] == "error" || w[2] == "errors") b += v
else if (w[2] == "passed" || w[2] == "xpassed") a += v
else if (w[2] == "skipped" || w[2] == "deselected" || w[2] == "xfailed") c += v
else if (w[2] != "warning" && w[2] != "warnings") ok = 0
}
if (line == "no tests ran") ok = 1
if (ok) { pp += a; pf += b; ps += c; pytest = 1 }
next
}
/^FAILED [^ ]+::/ { print "FAILED " pyname(substr($0, 8)); next }
END {
if (cargo) printf "FRAMEWORK cargo passed=%d failed=%d skipped=%d\n", cp, cf, cs
if (dotnet) printf "FRAMEWORK dotnet passed=%d failed=%d skipped=%d\n", dp, df, ds
if (vitest) printf "FRAMEWORK vitest passed=%d failed=%d skipped=%d\n", vp, vf, vs
if (jest) printf "FRAMEWORK jest passed=%d failed=%d skipped=%d\n", jp, jf, js
if (pytest) printf "FRAMEWORK pytest passed=%d failed=%d skipped=%d\n", pp, pf, ps
}' "$1"
}
tp_sections() {
awk -v max="${2:-30}" '
function flush( i, start) {
if (!n) return
start = n > max ? n - max + 1 : 1
for (i = start; i <= n; i++) print buf[i]
n = 0
}
/^---- .* stdout ----$/ { flush(); n = 0; buf[++n] = $0; inside = 1; next }
inside && (/^failures:$/ || /^test result: /) { flush(); inside = 0; next }
inside { buf[++n] = $0 }
END { flush() }' "$1"
}