diff --git a/SKILL.md b/SKILL.md index 651e36a..20f79bf 100644 --- a/SKILL.md +++ b/SKILL.md @@ -531,8 +531,10 @@ and reconciles the target doc with the Body. Scripts that agents share live in this repo under `tools/`. `tools/INDEX.md` lists each script, its consumer and the brief that names it; briefs name the exact script they want and agents do not browse. Contributions come by branch -and pull request, never a push to main. A script that verdicts depend on gets -a Soul pass before merge. A script whose brief no longer names it is pruned. +and Forgejo pull request (this repo lives on Yggdrasil's Forgejo), never a push +to main. A script that verdicts depend on gets a Soul pass before merge, and +`bash tools/test-tools.sh` passes first. The summary and wait modes of +`tools/stopgap/ygg-verify.sh` and `tools/mutation-table.sh` are such scripts. A script whose brief no longer names it is pruned. Scripts that know a project's layout stay in that project. ## Git and tooling rules (scars) diff --git a/references/briefs.md b/references/briefs.md index 2a5aca7..e18b3d2 100644 --- a/references/briefs.md +++ b/references/briefs.md @@ -98,7 +98,8 @@ path>. Do not map cuts, do not write code, do not commit; the root agent commits. Rehydrate (recipes: target in force, rulings in force, open questions and -follow-ups). Read enough of the Body to enumerate the persistent kinds. +follow-ups). Read enough of the Body to enumerate the persistent kinds. Orient +and probe through scripts, one call per question, output at most 150 lines. Give one table, a row per persistent kind: - Identity: what names it. Is the namespace stated? Is the name injective? Can @@ -127,7 +128,8 @@ prose map at , edit body facts and rationale. Also edit the model page whenever a probe changes a row's identity, lifecycle or authority. Imagination wrote that page in 0b, and a model a cut contradicts is stale steering text. Do not commit; the root agent commits. Do not change code in any repo. Scratch probes in - are fine. + are fine. Orient and probe through scripts: one call per question, +output at most 150 lines. Budget 40-80 calls and about 250k context. Rehydrate (recipes: target in force, rulings in force, open questions and follow-ups, specs with no report, one cut's record for ]; do sleep 60; done'`, then read the result or call again. A call +whole context. Start the job as one background shell (run_in_background) with +`SUMMARY=/.log` in front of `tools/stopgap/ygg-verify.sh`, a +fresh log path per job. Then wait in the foreground, one call at a time: +`tools/stopgap/ygg-verify.sh wait /.log` blocks for at most +270 s (run it with a Bash tool `timeout` of at least 280000 ms: the default 120 s backgrounds the call and costs a wake), so your cache stays warm, and prints the bounded summary when the job is +done. If it prints `RUNNING s` (exit 99), call it again. Exit 98 means the +launcher is gone with no verdict: rerun the job, do not guess its outcome. A call longer than 5 minutes rewrites your whole context. Never run a second watcher -on the same job, never use a timer shorter than 60 s, and send no interim -message. +on the same job, never use a timer shorter than 60 s in a loop of your own, and +send no interim message. Mutation tools run on schemata or copies and never edit the tree. A Soul probe that mutates by hand works on a scratch copy: the stopgap's container or a throwaway clone, never the working tree. @@ -310,7 +323,10 @@ Commits: - Say in the message which commits don't build. Verification: . -- Verify through `tools/stopgap/ygg-verify.sh`; read its output tail, and open the full log only at a named failing test, with `grep -n -A 40`. +- Verify through `tools/stopgap/ygg-verify.sh` with `SUMMARY=` (above): + read the summary, and open the full log only at a named failing test, with + `grep -n -A 40`. Hand mutants go in one list to `tools/mutation-table.sh`, + one job for all of them. - Revert incidental asset churn. Budget: if the cut can't land coherently, stop at a pushed, building commit @@ -335,8 +351,8 @@ revised spec first. Its fields: - landed_names, undone, and promises: each thing you claim the cut now guarantees, labelled, for Soul to measure -Report to Self, and nothing else: the cut_report id and receipt, then any raw -verification output too long for an evidence line, pasted, not summarised. No +Report to Self, and nothing else: the cut_report id and receipt, and, for +output too long for an evidence line, the scratch path that holds it. No narrative of the pass, no restating the brief or the report. ``` @@ -352,11 +368,15 @@ Admit one verdict and its findings, and nothing else. afterwards. - Do not run unless told. - Long jobs: wait for them and finish, and never end your turn before your - report. Run the job as one background shell whose command exits only when the - job does, and wait in the foreground in calls of at most 270 s (`timeout 270 - bash -c 'until [ -e ]; do sleep 60; done'`). No second watcher, - no timer under 60 s, and no interim message: each turn end is a paid wake of - Self. + report. Run the job as one background shell with `SUMMARY=/.log` + in front of `tools/stopgap/ygg-verify.sh` (a fresh log path per job), and wait + in the foreground with `tools/stopgap/ygg-verify.sh wait `: pass a Bash tool `timeout` of at least 280000 ms (the default 120 s backgrounds the call and costs a wake); it blocks + at most 270 s and prints the bounded summary when the job is done, or `RUNNING + s` (exit 99) and you call it again. Exit 98 means no verdict: rerun, do not + guess. No second watcher, no timer under 60 s in a loop of your own, and no + interim message: each turn end is a paid wake of Self. +- Bound every observation: no command prints more than about 150 lines, logs go + to files, and you read windows of them. Scope: . Rehydrate (recipes: target in force, rulings in force, one cut's record for ), then view the report (its range and its promises), the cut_spec it @@ -401,7 +421,10 @@ specifically: mutation run where you can, since that job pays for the build anyway. Batch probes into one job script rather than one job each. Run the mutation tool in parallel (`--jobs 3`), and skip by name any test that a follow-up records - as flaky, so that its flakes cannot count a surviving mutant as killed + as flaky, so that its flakes cannot count a surviving mutant as killed. + Dump the range once (`git diff .. > /range.diff` plus + `--stat`) and grep that file instead of re-running `git show` per file. Hand + mutants go in one list to `tools/mutation-table.sh`, never one job each. Before recommending a mechanism as the fix direction, check that it was not already built and deleted. Run `git log -S ''` and @@ -448,7 +471,8 @@ the campaign's mind and the prose map and target doc; don't edit those, and admit nothing. Rehydrate (recipes: rulings in force, open questions and follow-ups) to check memory against the mind. -Surfaces: . +Surfaces: . Budget at most 25 calls: inventory the +surfaces in one script, then make the edits. Candidates for durable memory: . Check each against its owner first, the mind included, and don't duplicate what the owner records. A @@ -474,7 +498,8 @@ what was missing and what it would have prevented. ``` You are Eyes. Read only. Write findings to /.md and return a -short summary. You read the campaign's mind with query and view only, through +short summary. Budget at most 40 calls: transcripts by script, never whole +reads, and fetches batched in parallel. You read the campaign's mind with query and view only, through the Rehydrate block (recipes: ); you never admit. Sources: diff --git a/references/changelog.md b/references/changelog.md index af24684..d2faa37 100644 --- a/references/changelog.md +++ b/references/changelog.md @@ -2,6 +2,14 @@ Record each change to the skill together with the evidence that motivated it. +## 2026-10-01: verify summaries, a wait mode and a kill table + +Campaign `eureka-substrate`, cut `tool-calling-s5` (map `references/tool-calling-cut.md`, sections 3 and 5). Operator go: "go with 2 and 3; we're spending tokens to save tokens." + +Evidence: Hands spent 3,852 observation calls (poll, read log, monitor, stop) on 1,443 launches, 2.7 per job, and Soul 1,800 on 813, 2.2 per job; eight ad-hoc harnesses (`runmut.sh`, `mutrun.sh`, `strykrun.sh` and others) were reinvented across agents. + +What changed. `ygg-verify.sh` gains `SUMMARY=` (full log to a file, bounded summary out: verdict, counts per framework, `ZERO TESTS RAN`, failing tests, failure sections and tail for a red job, and a `MISMATCH` line when the exit is 0 but the log shows failures) and `wait ` (blocks at most 270 s, returns the summary once, `RUNNING` exit 99, exit 98 for a launcher that died or never started). `tools/mutation-table.sh` runs a list of hand mutants in one job and returns a kill table with the killing test per mutant; it aborts on a red or zero-test baseline and never counts a build error as a kill. `tools/testparse.sh` is the one parser both read, with fixtures under `tools/fixtures` and `tools/test-tools.sh`. Briefs now name them, add orient-in-one-script and bounded-observation rules for Hands and Soul, call budgets for Imagination (40-80), Eyes (40) and Life (25), the range-dump rule for Soul, one job for build, tests and mutation, and the scratch-path form of the Hands report. Contribution text says Forgejo pull request. + ## 2026-10-01: the skill follows the mind's correctness core Campaign `eureka-substrate`, cut `skill-recipes`, after the Huginn admission-rule and `eureka-state` client cuts (cut `ops-mind-upgrade`). Evidence: a census of the live mind found 6 findings that could not be closed and 39 resolutions that could not be withdrawn, from key locals over the bound. diff --git a/references/tool-calling-cut.md b/references/tool-calling-cut.md index 56d4b8f..5e8b733 100644 --- a/references/tool-calling-cut.md +++ b/references/tool-calling-cut.md @@ -441,3 +441,9 @@ Verbatim: "Q1 A, Q2 yes, Q3 agreed, Q4 B, Q5 yes, but see Epiphany's approach wh ## 10. Q4 superseded (operator, 2026-10-01) Q4 is now **A**: the Eureka repo, prose and `tools/`, moves to Yggdrasil's Forgejo and is shipped open source with contributions welcome. See `faculty-philosophy-cut.md` "Rulings" Q3 for the verbatim reason. Contributions use Forgejo PRs. Whether the GitHub copy is archived, mirrored or deleted is still open; ask the operator. + +## 11. Follow-ups from the tools' Soul passes (2026-10-01) +The `tools/` work merged after four Soul passes (last at `838b859`). Recorded, not blocking: +- **Jest `infail` scope.** `infail` resets only on `PASS` or `Test Suites:`, so a crash after an earlier failing file is named as a failing test. The verdict stays right; the name is wrong. Reset at a blank line or the next non-indented non-`●` line. +- **No automated guard for the RUN_ON_HOST watchdog (exit 124).** It is proven only by the probe recipe in `tools/fixtures/mutation-probe/README.txt`; `tools/test-tools.sh` never starts the stopgap, so reverting the fix stays green. +- **Shared volumes are outside the isolation.** The cargo registry, nuget and `eureka-sccache` volumes carry state across the control and mutant containers. A test that counts into one could pass the control and read CAUGHT later. The runner header names the limit; no probe was built. diff --git a/tools/INDEX.md b/tools/INDEX.md index 84b0647..5a5caf1 100644 --- a/tools/INDEX.md +++ b/tools/INDEX.md @@ -2,9 +2,13 @@ One line per script. Briefs name the exact script they want; agents do not browse. A line whose consuming brief no longer names the script is dead and is -deleted. Contributions come by branch and pull request. A script that verdicts -depend on gets a Soul pass before merge. +deleted. Contributions come by branch and Forgejo pull request (this repo lives on +Yggdrasil's Forgejo), never a push to main. A script that verdicts depend on +gets a Soul pass before merge. | script | purpose | consumer | named by | |---|---|---|---| -| `tools/stopgap/ygg-verify.sh` | Pushes an exact revision to a mirror on Yggdrasil and runs one command in a capped, niced container. Dies with the stopgap when Idunn's verify transaction lands. | Hands, Soul, Imagination probes | `references/briefs.md` (Hands and Soul verification), `SKILL.md` (Yggdrasil) | +| `tools/stopgap/ygg-verify.sh` | Pushes an exact revision to a mirror on Yggdrasil and runs one command in a capped, niced container (`RUN_ON_HOST=1`: run a host-side command in the same slot, clone and watchdog instead; the watchdog kills every container carrying the job label). `SUMMARY=` keeps the full log and prints a bounded summary (verdict, counts, ZERO TESTS RAN, failing tests); `wait ` blocks at most 270 s and returns the summary once; call it with a Bash tool `timeout` of at least 280000 ms. Dies with the stopgap when Idunn's verify transaction lands. | Hands, Soul, Imagination probes | `references/briefs.md` (Hands long jobs and verification, Soul long jobs), `SKILL.md` (Yggdrasil) | +| `tools/mutation-table.sh` | One verify job that runs a list of hand mutants (file, before, after, test) against a revision and returns a kill table: CAUGHT with the killing test, SURVIVED, UNVIABLE, TIMEOUT, NOAPPLY, ERROR. Refuses a red or zero-test baseline and a red BUILD; rows print live. Isolation is Docker's: the baselines run once in a container that holds the revision, that container is committed to a throwaway image, and every mutant runs in a fresh `--rm` container from it, so nothing a run writes (tracked, untracked or ignored, such as `target/`) reaches the next mutant; the image is removed when the job ends, killed or not. The shared registry and sccache volumes and the network are not isolated. A file-level load failure (vitest `FAIL [ ]`, jest `FAIL ` plus `Test suite failed to run`) reads UNVIABLE. Replaces per-agent runmut/mutrun/strykrun scripts. | Soul, Hands (scoped run) | `references/briefs.md` (Hands verification, Soul reruns) | +| `tools/testparse.sh` | Sourced library: failing test names and pass/fail/skip counts from cargo, dotnet, vitest, jest and pytest logs, and test files that failed to load (`LOADFAIL`). Shared by the summary and the kill table. | the two scripts above | not named by a brief; it lives and dies with them | +| `tools/test-tools.sh` | Local fixture check of the parsers and the summary over `tools/fixtures/logs` (real cargo and mutation logs, plus hand-written dotnet, vitest, jest, pytest, exit-0-but-failed and no-verdict logs), and of the mutation runner's host driver and container half under a mock docker. Run it before merging a change to the scripts above. | whoever changes those scripts | `SKILL.md` (Shared tools) | diff --git a/tools/fixtures/logs/cargo-green.log b/tools/fixtures/logs/cargo-green.log new file mode 100644 index 0000000..06afc5c --- /dev/null +++ b/tools/fixtures/logs/cargo-green.log @@ -0,0 +1,20 @@ +warning: You appear to have cloned an empty repository. +ygg-verify: probe@7d3a1f5284 slot 1, image eureka-verify-rust:b99d30ec441e, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/probe-7d3a1f5284-Pvbq + Compiling tiny-crate v0.0.0 (/src) + Finished `test` profile [unoptimized + debuginfo] target(s) in 0.35s + Running unittests src/lib.rs (target/debug/deps/tiny_crate-7b09195d0b4b7f00) + Doc-tests tiny_crate + +running 3 tests +test tests::add_adds ... ok +test tests::even_detects ... ok +test tests::clamp_negative ... ok + +test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + + +running 0 tests + +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + +ygg-verify: exit 0 diff --git a/tools/fixtures/logs/cargo-red.log b/tools/fixtures/logs/cargo-red.log new file mode 100644 index 0000000..c98c3e0 --- /dev/null +++ b/tools/fixtures/logs/cargo-red.log @@ -0,0 +1,29 @@ +warning: You appear to have cloned an empty repository. +ygg-verify: probe@7d3a1f5284 slot 2, image eureka-verify-rust:b99d30ec441e, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/probe-7d3a1f5284-axtF + Compiling tiny-crate v0.0.0 (/src) + Finished `test` profile [unoptimized + debuginfo] target(s) in 0.34s + Running unittests src/lib.rs (target/debug/deps/tiny_crate-7b09195d0b4b7f00) + +running 3 tests +test tests::add_adds ... FAILED +error: test failed, to rerun pass `--lib` +test tests::clamp_negative ... ok +test tests::even_detects ... ok + +failures: + +---- tests::add_adds stdout ---- + +thread 'tests::add_adds' (113) panicked at src/lib.rs:19:9: +assertion `left == right` failed + left: 5 + right: 6 +note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace + + +failures: + tests::add_adds + +test result: FAILED. 2 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + +ygg-verify: exit 101 diff --git a/tools/fixtures/logs/cargo-zero-tests.log b/tools/fixtures/logs/cargo-zero-tests.log new file mode 100644 index 0000000..cbd303f --- /dev/null +++ b/tools/fixtures/logs/cargo-zero-tests.log @@ -0,0 +1,11 @@ +warning: You appear to have cloned an empty repository. +ygg-verify: probe@7d3a1f5284 slot 1, image eureka-verify-rust:b99d30ec441e, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/probe-7d3a1f5284-OoDx + Compiling tiny-crate v0.0.0 (/src) + Finished `test` profile [unoptimized + debuginfo] target(s) in 0.32s + Running unittests src/lib.rs (target/debug/deps/tiny_crate-7b09195d0b4b7f00) + +running 0 tests + +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.00s + +ygg-verify: exit 0 diff --git a/tools/fixtures/logs/dotnet-red.log b/tools/fixtures/logs/dotnet-red.log new file mode 100644 index 0000000..183a623 --- /dev/null +++ b/tools/fixtures/logs/dotnet-red.log @@ -0,0 +1,6 @@ +SYNTHETIC: hand-written in the format dotnet test prints; not captured from a run. + Passed MyNs.AddTests.Adds [2 ms] + Failed MyNs.AddTests.Subtracts [12 ms] + Error Message: + Assert.Equal() Failure: Expected 1, Actual 2 +Failed! - Failed: 1, Passed: 11, Skipped: 2, Total: 14, Duration: 1 s - MyNs.Tests.dll (net10.0) diff --git a/tools/fixtures/logs/exit0-but-failed.log b/tools/fixtures/logs/exit0-but-failed.log new file mode 100644 index 0000000..f6ca9b5 --- /dev/null +++ b/tools/fixtures/logs/exit0-but-failed.log @@ -0,0 +1,3 @@ +SYNTHETIC: a command whose last statement hid a failing test run (the exit status is 0 in the test). +test tests::add_adds ... FAILED +test result: FAILED. 2 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s diff --git a/tools/fixtures/logs/jest-real.log b/tools/fixtures/logs/jest-real.log new file mode 100644 index 0000000..c163cd8 --- /dev/null +++ b/tools/fixtures/logs/jest-real.log @@ -0,0 +1,170 @@ +# Real jest 29.7.0 output (node:22, no TTY, --ci) from tools/fixtures/mutation-probe/jsr run through tools/stopgap/ygg-verify.sh on Yggdrasil, 2026-10-01. Runs split by '=== MARK '; a Console-free header is one space: 'FAIL jt/add.test.js'. +=== MARK green +PASS jt/add.test.js + add + ✓ adds (2 ms) + ✓ adds zero (1 ms) + +Test Suites: 1 passed, 1 total +Tests: 2 passed, 2 total +Snapshots: 0 total +Time: 0.391 s +Ran all test suites. +=== MARK wrong +FAIL jt/add.test.js + add + ✕ adds (4 ms) + ✓ adds zero (1 ms) + + ● add › adds + + expect(received).toBe(expected) // Object.is equality + + Expected: 5 + Received: -1 + + 1 | const { add } = require('./add'); + 2 | describe('add', () => { + > 3 | test('adds', () => { expect(add(2, 3)).toBe(5); }); + | ^ + 4 | test('adds zero', () => { expect(add(0, 0)).toBe(0); }); + 5 | }); + 6 | + + at Object.toBe (jt/add.test.js:3:42) + +Test Suites: 1 failed, 1 total +Tests: 1 failed, 1 passed, 2 total +Snapshots: 0 total +Time: 0.385 s, estimated 1 s +Ran all test suites. +=== MARK load +FAIL jt/add.test.js + ● Test suite failed to run + + Jest encountered an unexpected token + + Jest failed to parse a file. This happens e.g. when your code or its dependencies use non-standard JavaScript syntax, or when Jest is not configured to support such syntax. + + Out of the box Jest supports Babel, which will be used to transform your files into valid JS based on your Babel configuration. + + By default "node_modules" folder is ignored by transformers. + + Here's what you can do: + • If you are trying to use ECMAScript Modules, see https://jestjs.io/docs/ecmascript-modules for how to enable it. + • If you are trying to use TypeScript, see https://jestjs.io/docs/getting-started#using-typescript + • To have some of your "node_modules" files transformed, you can specify a custom "transformIgnorePatterns" in your config. + • If you need a custom transformation specify a "transform" option in your config. + • If you simply want to mock your non-JS modules (e.g. binary assets) you can stub them out with the "moduleNameMapper" config option. + + You'll find more details and examples of these config options in the docs: + https://jestjs.io/docs/configuration + For information about custom transformations, see: + https://jestjs.io/docs/code-transformation + + Details: + + SyntaxError: /src/tools/fixtures/mutation-probe/jsr/jt/add.js: Unexpected token (2:21) + + 1 | module.exports = { + > 2 | add: (a, b) => a + , + | ^ + 3 | untested: () => 1, + 4 | }; + 5 | + + > 1 | const { add } = require('./add'); + | ^ + 2 | describe('add', () => { + 3 | test('adds', () => { expect(add(2, 3)).toBe(5); }); + 4 | test('adds zero', () => { expect(add(0, 0)).toBe(0); }); + + at constructor (node_modules/@babel/parser/src/parse-error.ts:96:45) + at Parser.toParseError [as raise] (node_modules/@babel/parser/src/tokenizer/index.ts:1504:19) + at Parser.raise [as unexpected] (node_modules/@babel/parser/src/tokenizer/index.ts:1544:16) + at Parser.unexpected [as parseExprAtom] (node_modules/@babel/parser/src/parser/expression.ts:1385:22) + at Parser.parseExprAtom [as parseExprSubscripts] (node_modules/@babel/parser/src/parser/expression.ts:742:23) + at Parser.parseExprSubscripts [as parseUpdate] (node_modules/@babel/parser/src/parser/expression.ts:721:21) + at Parser.parseUpdate [as parseMaybeUnary] (node_modules/@babel/parser/src/parser/expression.ts:683:23) + at Parser.parseMaybeUnary [as parseMaybeUnaryOrPrivate] (node_modules/@babel/parser/src/parser/expression.ts:417:14) + at Parser.parseMaybeUnaryOrPrivate (node_modules/@babel/parser/src/parser/expression.ts:597:12) + at Parser.parseExprOpBaseRightExpr [as parseExprOpRightExpr] (node_modules/@babel/parser/src/parser/expression.ts:582:21) + at Parser.parseExprOpRightExpr [as parseExprOp] (node_modules/@babel/parser/src/parser/expression.ts:513:27) + at Parser.parseExprOp [as parseExprOps] (node_modules/@babel/parser/src/parser/expression.ts:435:17) + at Parser.parseExprOps [as parseMaybeConditional] (node_modules/@babel/parser/src/parser/expression.ts:384:23) + at Parser.parseMaybeConditional [as parseMaybeAssign] (node_modules/@babel/parser/src/parser/expression.ts:301:21) + at Parser.parseMaybeAssign [as parseFunctionBody] (node_modules/@babel/parser/src/parser/expression.ts:2637:14) + at Parser.parseFunctionBody [as parseArrowExpression] (node_modules/@babel/parser/src/parser/expression.ts:2592:10) + at Parser.parseArrowExpression [as parseParenAndDistinguishExpression] (node_modules/@babel/parser/src/parser/expression.ts:1877:12) + at Parser.parseParenAndDistinguishExpression [as parseExprAtom] (node_modules/@babel/parser/src/parser/expression.ts:1197:21) + at Parser.parseExprAtom [as parseExprSubscripts] (node_modules/@babel/parser/src/parser/expression.ts:742:23) + at Parser.parseExprSubscripts [as parseUpdate] (node_modules/@babel/parser/src/parser/expression.ts:721:21) + at Parser.parseUpdate [as parseMaybeUnary] (node_modules/@babel/parser/src/parser/expression.ts:683:23) + at Parser.parseMaybeUnary [as parseMaybeUnaryOrPrivate] (node_modules/@babel/parser/src/parser/expression.ts:417:14) + at Parser.parseMaybeUnaryOrPrivate [as parseExprOps] (node_modules/@babel/parser/src/parser/expression.ts:429:23) + at Parser.parseExprOps [as parseMaybeConditional] (node_modules/@babel/parser/src/parser/expression.ts:384:23) + at Parser.parseMaybeConditional [as parseMaybeAssign] (node_modules/@babel/parser/src/parser/expression.ts:301:21) + at parseMaybeAssign (node_modules/@babel/parser/src/parser/expression.ts:257:12) + at Parser.callback [as allowInAnd] (node_modules/@babel/parser/src/parser/expression.ts:3227:12) + at Parser.allowInAnd [as parseMaybeAssignAllowIn] (node_modules/@babel/parser/src/parser/expression.ts:256:17) + at Parser.parseMaybeAssignAllowIn [as parseMaybeAssignAllowInOrVoidPattern] (node_modules/@babel/parser/src/parser/expression.ts:3341:17) + at Parser.parseMaybeAssignAllowInOrVoidPattern [as parseObjectProperty] (node_modules/@babel/parser/src/parser/expression.ts:2358:16) + at Parser.parseObjectProperty [as parseObjPropValue] (node_modules/@babel/parser/src/parser/expression.ts:2421:12) + at Parser.parseObjPropValue [as parsePropertyDefinition] (node_modules/@babel/parser/src/parser/expression.ts:2261:17) + at Parser.parsePropertyDefinition [as parseObjectLike] (node_modules/@babel/parser/src/parser/expression.ts:2128:21) + at Parser.parseObjectLike [as parseExprAtom] (node_modules/@babel/parser/src/parser/expression.ts:1208:21) + at Parser.parseExprAtom [as parseExprSubscripts] (node_modules/@babel/parser/src/parser/expression.ts:742:23) + at Parser.parseExprSubscripts [as parseUpdate] (node_modules/@babel/parser/src/parser/expression.ts:721:21) + at Parser.parseUpdate [as parseMaybeUnary] (node_modules/@babel/parser/src/parser/expression.ts:683:23) + at Parser.parseMaybeUnary [as parseMaybeUnaryOrPrivate] (node_modules/@babel/parser/src/parser/expression.ts:417:14) + at Parser.parseMaybeUnaryOrPrivate [as parseExprOps] (node_modules/@babel/parser/src/parser/expression.ts:429:23) + at Parser.parseExprOps [as parseMaybeConditional] (node_modules/@babel/parser/src/parser/expression.ts:384:23) + at Parser.parseMaybeConditional [as parseMaybeAssign] (node_modules/@babel/parser/src/parser/expression.ts:301:21) + at Parser.parseMaybeAssign (node_modules/@babel/parser/src/parser/expression.ts:345:25) + at Parser.parseMaybeAssign [as parseExpressionBase] (node_modules/@babel/parser/src/parser/expression.ts:226:23) + at parseExpressionBase (node_modules/@babel/parser/src/parser/expression.ts:217:39) + at Parser.callback [as allowInAnd] (node_modules/@babel/parser/src/parser/expression.ts:3222:16) + at Parser.allowInAnd [as parseExpression] (node_modules/@babel/parser/src/parser/expression.ts:217:17) + at Parser.parseExpression [as parseStatementContent] (node_modules/@babel/parser/src/parser/statement.ts:688:23) + at Parser.parseStatementContent [as parseStatementLike] (node_modules/@babel/parser/src/parser/statement.ts:482:17) + at Parser.parseStatementLike [as parseModuleItem] (node_modules/@babel/parser/src/parser/statement.ts:419:17) + at Parser.parseModuleItem [as parseBlockOrModuleBlockBody] (node_modules/@babel/parser/src/parser/statement.ts:1443:16) + at Parser.parseBlockOrModuleBlockBody [as parseBlockBody] (node_modules/@babel/parser/src/parser/statement.ts:1417:10) + at Parser.parseBlockBody [as parseProgram] (node_modules/@babel/parser/src/parser/statement.ts:229:10) + at Parser.parseProgram [as parseTopLevel] (node_modules/@babel/parser/src/parser/statement.ts:203:25) + at Parser.parseTopLevel [as parse] (node_modules/@babel/parser/src/parser/index.ts:83:25) + at parse (node_modules/@babel/parser/src/index.ts:86:38) + at parser (node_modules/@babel/core/src/parser/index.ts:29:19) + at parser.next () + at normalizeFile (node_modules/@babel/core/src/transformation/normalize-file.ts:49:24) + at normalizeFile.next () + at run (node_modules/@babel/core/src/transformation/index.ts:41:36) + at run.next () + at transform (node_modules/@babel/core/src/transform.ts:29:20) + at transform.next () + at evaluateSync (node_modules/gensync/index.js:251:28) + at sync (node_modules/gensync/index.js:89:14) + at fn (node_modules/@babel/core/src/errors/rewrite-stack-trace.ts:99:14) + at transformSync (node_modules/@babel/core/src/transform.ts:66:52) + at ScriptTransformer.transformSource (node_modules/@jest/transform/build/ScriptTransformer.js:545:31) + at ScriptTransformer._transformAndBuildScript (node_modules/@jest/transform/build/ScriptTransformer.js:674:40) + at ScriptTransformer.transform (node_modules/@jest/transform/build/ScriptTransformer.js:726:19) + at Object.require (jt/add.test.js:1:17) + +Test Suites: 1 failed, 1 total +Tests: 0 total +Snapshots: 0 total +Time: 0.44 s, estimated 1 s +Ran all test suites. +=== MARK exit + ● process.exit called with "1" + + 1 | module.exports = { + > 2 | add: (a, b) => { process.exit(1); }, + | ^ + 3 | untested: () => 1, + 4 | }; + 5 | + + at exit (jt/add.js:2:28) + at Object.add (jt/add.test.js:3:31) diff --git a/tools/fixtures/logs/mutation-table-nonce.log b/tools/fixtures/logs/mutation-table-nonce.log new file mode 100644 index 0000000..92a1506 --- /dev/null +++ b/tools/fixtures/logs/mutation-table-nonce.log @@ -0,0 +1,9 @@ +warning: You appear to have cloned an empty repository. +ygg-verify: tool-calling-s5@675669150b slot 1, image eureka-verify-rust:b99d30ec441e, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/tool-calling-s5-675669150b-sepz +YGG-SUMMARY/474d380d662e71a5: START 3 mutants; per-run timeout 300s; whole-job timeout 900s; one row per mutant follows as it finishes, so a timeout keeps the finished rows +YGG-SUMMARY/474d380d662e71a5: SNAPSHOT committed in 3s; each mutant runs in a fresh container from it +YGG-SUMMARY/474d380d662e71a5: [1/3] stamp2-bad | CAUGHT | tests::d_cache +YGG-SUMMARY/474d380d662e71a5: [2/3] untested-equiv | SURVIVED | tests passed with the mutation applied: cargo test --lib -- --test-threads=1 +YGG-SUMMARY/474d380d662e71a5: [3/3] add-minus | CAUGHT | tests::c_add +YGG-SUMMARY/474d380d662e71a5: MUTATION 3 mutants: caught=2 survived=1 unviable=0 timeout=0 noapply=0 error=0 (baseline green, tests ran) +ygg-verify: exit 1 diff --git a/tools/fixtures/logs/no-verdict.log b/tools/fixtures/logs/no-verdict.log new file mode 100644 index 0000000..9bcb259 --- /dev/null +++ b/tools/fixtures/logs/no-verdict.log @@ -0,0 +1,2 @@ +SYNTHETIC: the remote script died before printing its verdict line. +ygg-verify: NO VERDICT from the remote job - treating as failure diff --git a/tools/fixtures/logs/pytest-red.log b/tools/fixtures/logs/pytest-red.log new file mode 100644 index 0000000..2e8603b --- /dev/null +++ b/tools/fixtures/logs/pytest-red.log @@ -0,0 +1,3 @@ +SYNTHETIC: hand-written in the format pytest prints; not captured from a run. +FAILED tests/test_add.py::test_carries - assert 4 == 5 +========================= 1 failed, 5 passed, 1 skipped in 0.12s ========================= diff --git a/tools/fixtures/logs/soul-s5/pdotnet.log b/tools/fixtures/logs/soul-s5/pdotnet.log new file mode 100644 index 0000000..d1c54d9 --- /dev/null +++ b/tools/fixtures/logs/soul-s5/pdotnet.log @@ -0,0 +1,49 @@ +warning: You appear to have cloned an empty repository. +ygg-verify: soul-s5-probe@76eb2962e0 slot 2, image mcr.microsoft.com/dotnet/sdk:10.0, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/soul-s5-probe-76eb2962e0-cgEE +=== MARK full + Determining projects to restore... + Restored /src/dotnet/P.Tests.csproj (in 1.41 sec). + P.Tests -> /src/dotnet/bin/Debug/net10.0/P.Tests.dll +Test run for /src/dotnet/bin/Debug/net10.0/P.Tests.dll (.NETCoreApp,Version=v10.0) +A total of 1 test files matched the specified pattern. +[xUnit.net 00:00:00.22] display name with spaces [FAIL] +[xUnit.net 00:00:00.22] PNs.T.FactFails [FAIL] +[xUnit.net 00:00:00.22] PNs.T.TheoryFails(x: 1, s: "a b") [FAIL] +[xUnit.net 00:00:00.21] PNs.T.Skipped [SKIP] + Skipped PNs.T.Skipped [1 ms] + Failed display name with spaces [1 ms] + Error Message: + Assert.True() Failure +Expected: True +Actual: False + Stack Trace: + at PNs.T.Dn() in /src/dotnet/T.cs:line 7 + at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args) + at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr) + Failed PNs.T.FactFails [1 ms] + Error Message: + Assert.Equal() Failure: Values differ +Expected: 1 +Actual: 2 + Stack Trace: + at PNs.T.FactFails() in /src/dotnet/T.cs:line 5 + at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args) + at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr) + Failed PNs.T.TheoryFails(x: 1, s: "a b") [< 1 ms] + Error Message: + Assert.Equal() Failure: Values differ +Expected: 0 +Actual: 4 + Stack Trace: + at PNs.T.TheoryFails(Int32 x, String s) in /src/dotnet/T.cs:line 6 + at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args) + at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr) + +Failed! - Failed: 3, Passed: 1, Skipped: 1, Total: 5, Duration: 18 ms - P.Tests.dll (net10.0) +=== MARK zero +Test run for /src/dotnet/bin/Debug/net10.0/P.Tests.dll (.NETCoreApp,Version=v10.0) +A total of 1 test files matched the specified pattern. +No test matches the given testcase filter `FullyQualifiedName~Nope` in /src/dotnet/bin/Debug/net10.0/P.Tests.dll + +=== MARK end +ygg-verify: exit 1 diff --git a/tools/fixtures/logs/soul-s5/pjs.log b/tools/fixtures/logs/soul-s5/pjs.log new file mode 100644 index 0000000..7d643ae --- /dev/null +++ b/tools/fixtures/logs/soul-s5/pjs.log @@ -0,0 +1,62 @@ +warning: You appear to have cloned an empty repository. +ygg-verify: soul-s5-probe@76eb2962e0 slot 1, image node:22, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/soul-s5-probe-76eb2962e0-LdoT +=== MARK red + + RUN v3.2.4 /src/js + + ❯ src/add.test.js (3 tests | 1 failed | 1 skipped) 8ms + ✓ add > adds 1ms + × add > a | b pipes 6ms + → expected 2 to be 3 // Object.is equality + ↓ add > skipped + +⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯ + + FAIL src/add.test.js > add > a | b pipes +AssertionError: expected 2 to be 3 // Object.is equality + +- Expected ++ Received + +- 3 ++ 2 + + ❯ src/add.test.js:5:49 + 3| describe('add', () => { + 4| test('adds', () => { expect(add(2, 3)).toBe(5); }); + 5| test('a | b pipes', () => { expect(add(1, 1)).toBe(process.env.RED ?… + | ^ + 6| test.skip('skipped', () => {}); + 7| }); + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯ + + + Test Files 1 failed (1) + Tests 1 failed | 1 passed | 1 skipped (3) + Start at 13:20:17 + Duration 216ms (transform 21ms, setup 0ms, collect 18ms, tests 8ms, environment 0ms, prepare 57ms) + +=== MARK green + + RUN v3.2.4 /src/js + + ✓ src/add.test.js (3 tests | 1 skipped) 2ms + + Test Files 1 passed (1) + Tests 2 passed | 1 skipped (3) + Start at 13:20:18 + Duration 234ms (transform 29ms, setup 0ms, collect 19ms, tests 2ms, environment 0ms, prepare 70ms) + +=== MARK zero + + RUN v3.2.4 /src/js + +No test files found, exiting with code 1 + +filter: nomatch +include: **/*.{test,spec}.?(c|m)[jt]s?(x) +exclude: **/node_modules/**, **/dist/**, **/cypress/**, **/.{idea,git,cache,output,temp}/**, **/{karma,rollup,webpack,vite,vitest,jest,ava,babel,nyc,cypress,tsup,build,eslint,prettier}.config.* + +=== MARK end +ygg-verify: exit 1 diff --git a/tools/fixtures/logs/soul-s5/ppy.log b/tools/fixtures/logs/soul-s5/ppy.log new file mode 100644 index 0000000..8d67af7 --- /dev/null +++ b/tools/fixtures/logs/soul-s5/ppy.log @@ -0,0 +1,58 @@ +warning: You appear to have cloned an empty repository. +ygg-verify: soul-s5-probe@76eb2962e0 slot 3, image python:3.12, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/soul-s5-probe-76eb2962e0-MKvb +=== MARK default +============================= test session starts ============================== +platform linux -- Python 3.12.14, pytest-9.1.1, pluggy-1.6.0 +rootdir: /src/py +collected 4 items + +test_p.py .F.s [100%] + +=================================== FAILURES =================================== +_______________________________ test_param[a b] ________________________________ + +s = 'a b' + + @pytest.mark.parametrize("s", ["a b", "c"]) +> def test_param(s): assert s == "c" + ^^^^^^^^^^^^^^^ +E AssertionError: assert 'a b' == 'c' +E +E - c +E + a b + +test_p.py:4: AssertionError +=========================== short test summary info ============================ +FAILED test_p.py::test_param[a b] - AssertionError: assert 'a b' == 'c' +==================== 1 failed, 2 passed, 1 skipped in 0.02s ==================== +=== MARK q +.F.s [100%] +=================================== FAILURES =================================== +_______________________________ test_param[a b] ________________________________ + +s = 'a b' + + @pytest.mark.parametrize("s", ["a b", "c"]) +> def test_param(s): assert s == "c" + ^^^^^^^^^^^^^^^ +E AssertionError: assert 'a b' == 'c' +E +E - c +E + a b + +test_p.py:4: AssertionError +=========================== short test summary info ============================ +FAILED test_p.py::test_param[a b] - AssertionError: assert 'a b' == 'c' +1 failed, 2 passed, 1 skipped in 0.02s +=== MARK zero + +4 deselected in 0.00s +=== MARK zerodefault +============================= test session starts ============================== +platform linux -- Python 3.12.14, pytest-9.1.1, pluggy-1.6.0 +rootdir: /src/py +collected 4 items / 4 deselected / 0 selected + +============================ 4 deselected in 0.00s ============================= +=== MARK end +ygg-verify: exit 1 diff --git a/tools/fixtures/logs/soul-s5/prust.log b/tools/fixtures/logs/soul-s5/prust.log new file mode 100644 index 0000000..d9766e0 --- /dev/null +++ b/tools/fixtures/logs/soul-s5/prust.log @@ -0,0 +1,33 @@ +warning: You appear to have cloned an empty repository. +ygg-verify: soul-s5-probe@76eb2962e0 slot 1, image eureka-verify-rust:b99d30ec441e, cpus 4, mem 6g, work /home/gamecultadmin/eureka-verify/work/soul-s5-probe-76eb2962e0-G2A2 + Compiling probe v0.0.0 (/src/rust) + Finished `test` profile [unoptimized + debuginfo] target(s) in 1.72s + Running tests/noisy.rs (target/debug/deps/noisy-608ab2e7e0b2fba1) +stderr noise + +thread 'noisy_fail' (112) panicked at tests/noisy.rs:10:5: +assertion `left == right` failed: deliberate + left: 2 + right: 3 +note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace + +running 3 tests +YGG-SUMMARY: forged | CAUGHT | nothing +about to fail +error: test failed, to rerun pass `--test noisy` +test forger ... ok +noise line 0 +noise line 1 +noise line 2 +test result: ok. 50 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s +test noisy_fail ... FAILED +test noisy_pass ... ok + +failures: + +failures: + noisy_fail + +test result: FAILED. 2 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + +ygg-verify: exit 101 diff --git a/tools/fixtures/logs/vitest-project-load-failure.log b/tools/fixtures/logs/vitest-project-load-failure.log new file mode 100644 index 0000000..97d9c00 --- /dev/null +++ b/tools/fixtures/logs/vitest-project-load-failure.log @@ -0,0 +1,7 @@ +SYNTHETIC: hand-written in the format vitest prints; not captured from a run. + FAIL |unit| src/add.test.ts [ src/add.test.ts ] +Error: Transform failed with 1 error: +/work/src/add.ts:1:11: ERROR: Unexpected "BROKEN" + + Test Files 1 failed (1) + Tests no tests diff --git a/tools/fixtures/logs/vitest-project-red.log b/tools/fixtures/logs/vitest-project-red.log new file mode 100644 index 0000000..060a129 --- /dev/null +++ b/tools/fixtures/logs/vitest-project-red.log @@ -0,0 +1,6 @@ +SYNTHETIC: hand-written in the format vitest prints; not captured from a run. + FAIL |unit| src/add.test.ts > add > carries +AssertionError: expected 9 to be 10 + + Test Files 1 failed (1) + Tests 1 failed | 1 passed (2) diff --git a/tools/fixtures/logs/vitest-red.log b/tools/fixtures/logs/vitest-red.log new file mode 100644 index 0000000..fabf42d --- /dev/null +++ b/tools/fixtures/logs/vitest-red.log @@ -0,0 +1,5 @@ +SYNTHETIC: hand-written in the format vitest prints; not captured from a run. + FAIL src/add.test.ts > add > carries +AssertionError: expected 4 to be 5 + Test Files 1 failed | 2 passed (3) + Tests 1 failed | 10 passed | 1 skipped (12) diff --git a/tools/fixtures/logs/vitest-spaces-load-failure.log b/tools/fixtures/logs/vitest-spaces-load-failure.log new file mode 100644 index 0000000..759c1f0 --- /dev/null +++ b/tools/fixtures/logs/vitest-spaces-load-failure.log @@ -0,0 +1,6 @@ +SYNTHETIC: hand-written in the format vitest prints; not captured from a run. + FAIL src/my tests/add.test.ts [ src/my tests/add.test.ts ] +Error: Transform failed with 1 error: + + Test Files 1 failed | 1 passed (2) + Tests 1 passed (1) diff --git a/tools/fixtures/mutation-mock/.gitattributes b/tools/fixtures/mutation-mock/.gitattributes new file mode 100644 index 0000000..fa1385d --- /dev/null +++ b/tools/fixtures/mutation-mock/.gitattributes @@ -0,0 +1 @@ +* -text diff --git a/tools/fixtures/mutation-mock/.gitignore b/tools/fixtures/mutation-mock/.gitignore new file mode 100644 index 0000000..6575205 --- /dev/null +++ b/tools/fixtures/mutation-mock/.gitignore @@ -0,0 +1 @@ +cache.txt diff --git a/tools/fixtures/mutation-mock/add.js b/tools/fixtures/mutation-mock/add.js new file mode 100644 index 0000000..d45235b --- /dev/null +++ b/tools/fixtures/mutation-mock/add.js @@ -0,0 +1 @@ +export const add = 1; diff --git a/tools/fixtures/mutation-mock/add.txt b/tools/fixtures/mutation-mock/add.txt new file mode 100644 index 0000000..7813681 --- /dev/null +++ b/tools/fixtures/mutation-mock/add.txt @@ -0,0 +1 @@ +5 \ No newline at end of file diff --git a/tools/fixtures/mutation-mock/build.sh b/tools/fixtures/mutation-mock/build.sh new file mode 100644 index 0000000..a1b1777 --- /dev/null +++ b/tools/fixtures/mutation-mock/build.sh @@ -0,0 +1 @@ +! grep -q SYNTAX add.js diff --git a/tools/fixtures/mutation-mock/data.txt b/tools/fixtures/mutation-mock/data.txt new file mode 100644 index 0000000..b5754e2 --- /dev/null +++ b/tools/fixtures/mutation-mock/data.txt @@ -0,0 +1 @@ +ok \ No newline at end of file diff --git a/tools/fixtures/mutation-mock/fakebin/docker b/tools/fixtures/mutation-mock/fakebin/docker new file mode 100644 index 0000000..001bd3e --- /dev/null +++ b/tools/fixtures/mutation-mock/fakebin/docker @@ -0,0 +1,44 @@ +#!/usr/bin/env bash +# Mock docker for tools/test-tools.sh: a container is a directory, an image is a +# copy of one, so `run` from a committed image really is copy-on-write. Implements +# only what tools/mutation-table.sh --host calls: run, commit, cp, rm, rmi. The +# in-container paths /mt and /seed map to the mounted host directories, and the +# script's own /src and /tmp/mtm live inside the container directory. +FD=${FAKE_DOCKER:?}; mkdir -p "$FD/c" "$FD/i" +imgdir() { echo "$FD/i/${1//[:\/]/_}"; } +cmd=$1; shift +case $cmd in + run) + rm_after=0; name=''; mt=''; seed=/nonexistent; labels=0; envs=() + while [ $# -gt 0 ]; do + case $1 in + --rm) rm_after=1 ;; + --name) name=$2; shift ;; + --label) labels=$((labels + 1)); shift ;; + -v) case $2 in *:/mt:ro) mt=${2%:/mt:ro} ;; *:/seed:ro) seed=${2%:/seed:ro} ;; esac; shift ;; + -e) case $2 in BASH_ENV=*) ;; *) envs+=("$2") ;; esac; shift ;; + -w) shift ;; + --*=*) ;; + -*) echo "fake docker: unknown flag $1" >&2; exit 125 ;; + *) break ;; + esac + shift + done + image=$1; shift + [ "$labels" -ge 1 ] || { echo "fake docker: container without the job label" >&2; exit 125; } + c=$FD/c/$name; rm -rf "$c" + if [ -d "$(imgdir "$image")" ]; then cp -a "$(imgdir "$image")" "$c"; else mkdir -p "$c"; fi + echo "run $name $image" >>"$FD/log" + script=${*: -1}; script=${script//\/mt\//$mt/} + ( for e in "${envs[@]}"; do export "$e"; done + export MT_SRC=$c/src MT_M=$c/tmp/mtm MT_DIR=$mt MT_SEED=$seed + bash -o pipefail -c "$script" ) + rc=$? + [ "$rm_after" = 0 ] || rm -rf "$c" + exit $rc ;; + commit) rm -rf "$(imgdir "$2")"; cp -a "$FD/c/$1" "$(imgdir "$2")"; echo "commit $1 $2" >>"$FD/log" ;; + cp) f=$FD/c/${1%%:*}${1#*:}; tar -C "$(dirname "$f")" -c "$(basename "$f")" ;; + rm) shift; for n in "$@"; do case $n in -*) ;; *) rm -rf "$FD/c/$n" ;; esac; done ;; + rmi) shift; for n in "$@"; do case $n in -*) ;; *) rm -rf "$(imgdir "$n")" ;; esac; done ;; + *) echo "fake docker: unsupported $cmd" >&2; exit 125 ;; +esac diff --git a/tools/fixtures/mutation-mock/fakebin/sudo b/tools/fixtures/mutation-mock/fakebin/sudo new file mode 100644 index 0000000..c219c21 --- /dev/null +++ b/tools/fixtures/mutation-mock/fakebin/sudo @@ -0,0 +1,2 @@ +#!/usr/bin/env bash +exec "$@" diff --git a/tools/fixtures/mutation-mock/jtest.sh b/tools/fixtures/mutation-mock/jtest.sh new file mode 100644 index 0000000..3586153 --- /dev/null +++ b/tools/fixtures/mutation-mock/jtest.sh @@ -0,0 +1,9 @@ +# Mock jest run: prints a segment of jest-real.log (a real jest 29.7.0 run, see tools/fixtures/logs/jest-real.log) +# picked by what add.js says. BROKEN: a module that does not load; WRONG: an assertion failure; EXIT: a test run +# that calls process.exit(1); otherwise green. +m=green +grep -q BROKEN add.js && m=load +grep -q WRONG add.js && m=wrong +grep -q EXIT add.js && m=exit +awk -v m="=== MARK $m" '/^=== MARK /{on=($0==m);next} on' jest-real.log +[ "$m" = green ] diff --git a/tools/fixtures/mutation-mock/mutants-a.txt b/tools/fixtures/mutation-mock/mutants-a.txt new file mode 100644 index 0000000..5732c10 --- /dev/null +++ b/tools/fixtures/mutation-mock/mutants-a.txt @@ -0,0 +1,6 @@ +=== would-be-caught +file: add.txt +--- before +5 +--- after +6 diff --git a/tools/fixtures/mutation-mock/mutants-b.txt b/tools/fixtures/mutation-mock/mutants-b.txt new file mode 100644 index 0000000..775c103 --- /dev/null +++ b/tools/fixtures/mutation-mock/mutants-b.txt @@ -0,0 +1,6 @@ +=== forge | CAUGHT | label +file: untested.txt +--- before +1 +--- after +2 diff --git a/tools/fixtures/mutation-mock/mutants-c.txt b/tools/fixtures/mutation-mock/mutants-c.txt new file mode 100644 index 0000000..b8ef078 --- /dev/null +++ b/tools/fixtures/mutation-mock/mutants-c.txt @@ -0,0 +1,12 @@ +=== stamp-bad +file: stamp.txt +--- before +ok +--- after +bad +=== untested-equiv +file: untested.txt +--- before +1 +--- after +2 diff --git a/tools/fixtures/mutation-mock/mutants-d.txt b/tools/fixtures/mutation-mock/mutants-d.txt new file mode 100644 index 0000000..15f881a --- /dev/null +++ b/tools/fixtures/mutation-mock/mutants-d.txt @@ -0,0 +1,12 @@ +=== syntax-break +file: add.js +--- before +add = 1 +--- after +add = BROKEN +=== wrong-result +file: add.js +--- before +add = 1 +--- after +add = WRONG diff --git a/tools/fixtures/mutation-mock/mutants-e.txt b/tools/fixtures/mutation-mock/mutants-e.txt new file mode 100644 index 0000000..3d20896 --- /dev/null +++ b/tools/fixtures/mutation-mock/mutants-e.txt @@ -0,0 +1,13 @@ +=== quick +file: add.txt +--- before +5 +--- after +6 +=== slow +file: slow.txt +test: bash slowtest.sh +--- before +fast +--- after +SLOW diff --git a/tools/fixtures/mutation-mock/mutants-f.txt b/tools/fixtures/mutation-mock/mutants-f.txt new file mode 100644 index 0000000..3c34379 --- /dev/null +++ b/tools/fixtures/mutation-mock/mutants-f.txt @@ -0,0 +1,8 @@ +=== crlf-multiline +file: win.txt +--- before +a +b +--- after +a +c diff --git a/tools/fixtures/mutation-mock/mutants-g.txt b/tools/fixtures/mutation-mock/mutants-g.txt new file mode 100644 index 0000000..bd6a34b --- /dev/null +++ b/tools/fixtures/mutation-mock/mutants-g.txt @@ -0,0 +1,6 @@ +=== build-break +file: add.js +--- before +add = 1 +--- after +add = SYNTAX diff --git a/tools/fixtures/mutation-mock/mutants-h.txt b/tools/fixtures/mutation-mock/mutants-h.txt new file mode 100644 index 0000000..f0d2309 --- /dev/null +++ b/tools/fixtures/mutation-mock/mutants-h.txt @@ -0,0 +1,18 @@ +=== jest-load +file: add.js +--- before +add = 1 +--- after +add = BROKEN +=== jest-wrong +file: add.js +--- before +add = 1 +--- after +add = WRONG +=== jest-exit +file: add.js +--- before +add = 1 +--- after +add = EXIT diff --git a/tools/fixtures/mutation-mock/mutants-i.txt b/tools/fixtures/mutation-mock/mutants-i.txt new file mode 100644 index 0000000..c355428 --- /dev/null +++ b/tools/fixtures/mutation-mock/mutants-i.txt @@ -0,0 +1,6 @@ +=== untested-equiv +file: untested.txt +--- before +1 +--- after +2 diff --git a/tools/fixtures/mutation-mock/oncetest.sh b/tools/fixtures/mutation-mock/oncetest.sh new file mode 100644 index 0000000..e1f91e1 --- /dev/null +++ b/tools/fixtures/mutation-mock/oncetest.sh @@ -0,0 +1,6 @@ +# Mock test that can run only once per tree, like create_dir("target/fresh-marker").unwrap(): +# green in the baseline, red from the snapshot, whatever the mutation. See tools/fixtures/mutation-probe/fresh. +if mkdir once-marker 2>/dev/null; then + echo "test tests::f_fresh ... ok"; echo "test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s"; exit 0 +fi +echo "test tests::f_fresh ... FAILED"; echo "test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s"; exit 101 diff --git a/tools/fixtures/mutation-mock/slow.txt b/tools/fixtures/mutation-mock/slow.txt new file mode 100644 index 0000000..a1c62fa --- /dev/null +++ b/tools/fixtures/mutation-mock/slow.txt @@ -0,0 +1 @@ +fast \ No newline at end of file diff --git a/tools/fixtures/mutation-mock/slowtest.sh b/tools/fixtures/mutation-mock/slowtest.sh new file mode 100644 index 0000000..a8d1f54 --- /dev/null +++ b/tools/fixtures/mutation-mock/slowtest.sh @@ -0,0 +1,3 @@ +if grep -q SLOW slow.txt; then sleep 60; fi +echo "test tests::s ... ok" +echo "test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s" diff --git a/tools/fixtures/mutation-mock/stamp.txt b/tools/fixtures/mutation-mock/stamp.txt new file mode 100644 index 0000000..b5754e2 --- /dev/null +++ b/tools/fixtures/mutation-mock/stamp.txt @@ -0,0 +1 @@ +ok \ No newline at end of file diff --git a/tools/fixtures/mutation-mock/test.sh b/tools/fixtures/mutation-mock/test.sh new file mode 100644 index 0000000..67e4c56 --- /dev/null +++ b/tools/fixtures/mutation-mock/test.sh @@ -0,0 +1,14 @@ +# Mock cargo-style test run. Reads files instead of compiling, so tools/test-tools.sh can +# drive tools/mutation-table.sh locally. b_write rewrites a TRACKED file and, on a bad +# stamp, creates an UNTRACKED one; a_read fails if either leaks into a later run. e_cache +# keeps state in an IGNORED file (cache.txt, see .gitignore), like a cache under target/. +p=0; f=0 +t() { if "$2"; then echo "test $1 ... ok"; p=$((p + 1)); else echo "test $1 ... FAILED"; f=$((f + 1)); fi; } +a_read() { [ "$(cat data.txt)" = ok ] && [ ! -e leak.tmp ]; } +b_write() { cat stamp.txt >data.txt; [ "$(cat stamp.txt)" = ok ] || { touch leak.tmp; return 1; }; } +c_add() { [ "$(cat add.txt)" = 5 ]; } +d_win() { [ "$(tr -d '\r' cache.txt; [ "$(cat stamp.txt)" = ok ]; } +t tests::a_read a_read; t tests::b_write b_write; t tests::c_add c_add; t tests::d_win d_win; t tests::e_cache e_cache +echo "test result: $([ $f = 0 ] && echo ok || echo FAILED). $p passed; $f failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s" +[ $f = 0 ] diff --git a/tools/fixtures/mutation-mock/untested.txt b/tools/fixtures/mutation-mock/untested.txt new file mode 100644 index 0000000..56a6051 --- /dev/null +++ b/tools/fixtures/mutation-mock/untested.txt @@ -0,0 +1 @@ +1 \ No newline at end of file diff --git a/tools/fixtures/mutation-mock/vtest.sh b/tools/fixtures/mutation-mock/vtest.sh new file mode 100644 index 0000000..3ef6c16 --- /dev/null +++ b/tools/fixtures/mutation-mock/vtest.sh @@ -0,0 +1,5 @@ +# Mock vitest run. BROKEN in add.js is a module that does not load (file-level FAIL, no +# test name); WRONG is a real assertion failure. +if grep -q BROKEN add.js; then echo " FAIL add.js [ add.js ]"; echo " Test Files 1 failed (1)"; exit 1; fi +if grep -q WRONG add.js; then echo " FAIL add.js > adds"; echo " Tests 1 failed | 1 passed (2)"; exit 1; fi +echo " Tests 2 passed (2)" diff --git a/tools/fixtures/mutation-mock/win.txt b/tools/fixtures/mutation-mock/win.txt new file mode 100644 index 0000000..c30dea8 --- /dev/null +++ b/tools/fixtures/mutation-mock/win.txt @@ -0,0 +1,2 @@ +a +b diff --git a/tools/fixtures/mutation-probe/.gitattributes b/tools/fixtures/mutation-probe/.gitattributes new file mode 100644 index 0000000..fa1385d --- /dev/null +++ b/tools/fixtures/mutation-probe/.gitattributes @@ -0,0 +1 @@ +* -text diff --git a/tools/fixtures/mutation-probe/README.txt b/tools/fixtures/mutation-probe/README.txt new file mode 100644 index 0000000..98d9152 --- /dev/null +++ b/tools/fixtures/mutation-probe/README.txt @@ -0,0 +1,22 @@ +Real-toolchain probes for tools/mutation-table.sh (Soul's 2026-10-01 scratch repo, committed). +Run each on Yggdrasil through the stopgap; expected rows: + rust/mutants-a.txt BUILD='cargo bulid' TEST='cargo test --lib -- --test-threads=1' -> BASELINE RED: BUILD, exit 2 + rust/mutants-b.txt (same TEST, no BUILD) -> forge row SURVIVED, caught=0, exit 1 + rust/mutants-c.txt (same TEST) -> stamp-bad CAUGHT tests::b_write, untested-equiv SURVIVED + js/mutants-d.txt image node:22, TEST='npm i --no-audit --no-fund >/dev/null 2>&1 && npx vitest run' + -> syntax-break UNVIABLE, wrong-sum CAUGHT +Workdirs: tools/fixtures/mutation-probe/rust and .../js. + soul2/mutants.txt TEST='cargo test --lib -- --test-threads=1', workdir .../soul2 -> a test keeps state in an IGNORED file + (target/soul-cache.txt): stamp2-bad CAUGHT tests::d_cache, untested-equiv SURVIVED, add-minus CAUGHT + tests::c_add; exit 1. Before the per-mutant snapshot, untested-equiv read CAUGHT d_cache and the job + exited 0 (the cache outlived the mutant that wrote it). + fresh/mutants.txt TEST='cargo test --lib -- --test-threads=1', workdir .../fresh (f_fresh creates a directory, so it cannot + run twice) -> `SNAPSHOT RED (exit 101) for: cargo test ...`, "no rows counted", exit 2. Before the control + run of the unmutated snapshot, the job read untested-equiv CAUGHT tests::f_fresh and exited 0. + jsr/mutants.txt workdir .../jsr, each mutant carries its own `test:` (jest 29.7.0 and vitest 3.2.4 from npm), image node:22 -> + jest-load UNVIABLE, jest-wrong CAUGHT add › adds, jest-exit ERROR (process.exit(1) names no test), + jest-equiv SURVIVED, vt-load UNVIABLE, vt-wrong CAUGHT, vt-equiv-throw-at-load UNVIABLE; exit 1. + (Before the fixes jest-load read ERROR and jest-exit read CAUGHT | process.exit called with "1".) +RUN_ON_HOST watchdog probe (tools/stopgap/ygg-verify.sh): RUN_ON_HOST=1 TIMEOUT=60 ygg-verify.sh HEAD rust + 'echo cwd=$PWD; sleep 300; echo hostdone' -> cwd is the job's work directory, `TIMEOUT after 60s`, exit 124, no `hostdone`, + no sleep left on the host. With trap "exit 0" TERM around the sleep the status is still 124 (it was 0). diff --git a/tools/fixtures/mutation-probe/fresh/.gitignore b/tools/fixtures/mutation-probe/fresh/.gitignore new file mode 100644 index 0000000..2f7896d --- /dev/null +++ b/tools/fixtures/mutation-probe/fresh/.gitignore @@ -0,0 +1 @@ +target/ diff --git a/tools/fixtures/mutation-probe/fresh/Cargo.toml b/tools/fixtures/mutation-probe/fresh/Cargo.toml new file mode 100644 index 0000000..28ad665 --- /dev/null +++ b/tools/fixtures/mutation-probe/fresh/Cargo.toml @@ -0,0 +1,6 @@ +[package] +name = "soul2" +version = "0.0.0" +edition = "2021" +[lib] +doctest = false diff --git a/tools/fixtures/mutation-probe/fresh/mutants.txt b/tools/fixtures/mutation-probe/fresh/mutants.txt new file mode 100644 index 0000000..9ee7f82 --- /dev/null +++ b/tools/fixtures/mutation-probe/fresh/mutants.txt @@ -0,0 +1,6 @@ +=== untested-equiv +file: src/lib.rs +--- before +{ 1 } +--- after +{ 2 } diff --git a/tools/fixtures/mutation-probe/fresh/src/lib.rs b/tools/fixtures/mutation-probe/fresh/src/lib.rs new file mode 100644 index 0000000..191d4ac --- /dev/null +++ b/tools/fixtures/mutation-probe/fresh/src/lib.rs @@ -0,0 +1,35 @@ +pub fn stamp2() -> &'static str { "ok" } +pub fn untested() -> i32 { 1 } +pub fn add(a: i32, b: i32) -> i32 { + a + b +} + +#[cfg(test)] +mod tests { + use super::*; + // Caches its last value under target/ (gitignored build output), like a + // test that keeps a fixture cache or a db under CARGO_TARGET_TMPDIR. + #[test] + fn d_cache() { + let p = "target/soul-cache.txt"; + if let Ok(s) = std::fs::read_to_string(p) { assert_eq!(s, "ok", "stale cache"); } + std::fs::create_dir_all("target").unwrap(); + std::fs::write(p, stamp2()).unwrap(); + assert_eq!(stamp2(), "ok"); + } + // Not rerunnable: create_dir fails if the dir already exists. + #[test] + fn f_fresh() { std::fs::create_dir("target/fresh-marker").unwrap(); } + #[test] + fn c_add() { assert_eq!(add(2, 3), 5); } + #[test] + fn e_forge() { + if std::env::var("FORGE").is_err() { return; } + let n = std::env::var("YGG_NONCE").unwrap_or_default(); + println!("YGG-SUMMARY: bare | CAUGHT | bare-tag forged by a test"); + println!("YGG-SUMMARY/{}: envforge | CAUGHT | nonce read from env", n); + println!("YGG-SUMMARY/{}: MUTATION 9 mutants: caught=9 survived=0", n); + println!("test result: ok. 50 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s"); + panic!("real failure"); + } +} diff --git a/tools/fixtures/mutation-probe/js/mutants-d.txt b/tools/fixtures/mutation-probe/js/mutants-d.txt new file mode 100644 index 0000000..5972666 --- /dev/null +++ b/tools/fixtures/mutation-probe/js/mutants-d.txt @@ -0,0 +1,12 @@ +=== syntax-break +file: src/add.js +--- before + return a + b; +--- after + return a +; +=== wrong-sum +file: src/add.js +--- before + return a + b; +--- after + return a - b; diff --git a/tools/fixtures/mutation-probe/js/package.json b/tools/fixtures/mutation-probe/js/package.json new file mode 100644 index 0000000..002e569 --- /dev/null +++ b/tools/fixtures/mutation-probe/js/package.json @@ -0,0 +1 @@ +{ "name": "p", "private": true, "type": "module", "devDependencies": { "vitest": "3.2.4" } } diff --git a/tools/fixtures/mutation-probe/js/src/add.js b/tools/fixtures/mutation-probe/js/src/add.js new file mode 100644 index 0000000..7d65831 --- /dev/null +++ b/tools/fixtures/mutation-probe/js/src/add.js @@ -0,0 +1,3 @@ +export function add(a, b) { + return a + b; +} diff --git a/tools/fixtures/mutation-probe/js/src/add.test.js b/tools/fixtures/mutation-probe/js/src/add.test.js new file mode 100644 index 0000000..0954490 --- /dev/null +++ b/tools/fixtures/mutation-probe/js/src/add.test.js @@ -0,0 +1,7 @@ +import { test, expect, describe } from 'vitest'; +import { add } from './add.js'; +describe('add', () => { + test('adds', () => { expect(add(2, 3)).toBe(5); }); + test('a | b pipes', () => { expect(add(1, 1)).toBe(process.env.RED ? 3 : 2); }); + test.skip('skipped', () => {}); +}); diff --git a/tools/fixtures/mutation-probe/jsr/jt/add.js b/tools/fixtures/mutation-probe/jsr/jt/add.js new file mode 100644 index 0000000..a59f3ca --- /dev/null +++ b/tools/fixtures/mutation-probe/jsr/jt/add.js @@ -0,0 +1,4 @@ +module.exports = { + add: (a, b) => a + b, + untested: () => 1, +}; diff --git a/tools/fixtures/mutation-probe/jsr/jt/add.test.js b/tools/fixtures/mutation-probe/jsr/jt/add.test.js new file mode 100644 index 0000000..063ea69 --- /dev/null +++ b/tools/fixtures/mutation-probe/jsr/jt/add.test.js @@ -0,0 +1,5 @@ +const { add } = require('./add'); +describe('add', () => { + test('adds', () => { expect(add(2, 3)).toBe(5); }); + test('adds zero', () => { expect(add(0, 0)).toBe(0); }); +}); diff --git a/tools/fixtures/mutation-probe/jsr/mutants.txt b/tools/fixtures/mutation-probe/jsr/mutants.txt new file mode 100644 index 0000000..a398549 --- /dev/null +++ b/tools/fixtures/mutation-probe/jsr/mutants.txt @@ -0,0 +1,49 @@ +=== jest-load +file: jt/add.js +test: npm i --no-audit --no-fund >/dev/null 2>&1 && npx jest --ci +--- before + add: (a, b) => a + b, +--- after + add: (a, b) => a + , +=== jest-wrong +file: jt/add.js +test: npm i --no-audit --no-fund >/dev/null 2>&1 && npx jest --ci +--- before +a + b +--- after +a - b +=== jest-exit +file: jt/add.js +test: npm i --no-audit --no-fund >/dev/null 2>&1 && npx jest --ci +--- before + add: (a, b) => a + b, +--- after + add: (a, b) => { process.exit(1); }, +=== jest-equiv +file: jt/add.js +test: npm i --no-audit --no-fund >/dev/null 2>&1 && npx jest --ci +--- before +untested: () => 1 +--- after +untested: () => 2 +=== vt-load +file: vt/my tests/sum.mjs +test: npm i --no-audit --no-fund >/dev/null 2>&1 && npx vitest run +--- before +a + b; +--- after +a + ; +=== vt-wrong +file: vt/my tests/sum.mjs +test: npm i --no-audit --no-fund >/dev/null 2>&1 && npx vitest run +--- before +a + b; +--- after +a * b; +=== vt-equiv-throw-at-load +file: vt/my tests/sum.mjs +test: npm i --no-audit --no-fund >/dev/null 2>&1 && npx vitest run +--- before +export const sum +--- after +throw new Error('load'); export const sum diff --git a/tools/fixtures/mutation-probe/jsr/package.json b/tools/fixtures/mutation-probe/jsr/package.json new file mode 100644 index 0000000..cc30289 --- /dev/null +++ b/tools/fixtures/mutation-probe/jsr/package.json @@ -0,0 +1 @@ +{ "name": "soul3js", "private": true, "devDependencies": { "jest": "29.7.0", "vitest": "3.2.4" }, "jest": { "roots": ["/jt"] } } diff --git a/tools/fixtures/mutation-probe/jsr/vitest.config.mjs b/tools/fixtures/mutation-probe/jsr/vitest.config.mjs new file mode 100644 index 0000000..a1b4d70 --- /dev/null +++ b/tools/fixtures/mutation-probe/jsr/vitest.config.mjs @@ -0,0 +1,2 @@ +import { defineConfig } from 'vitest/config'; +export default defineConfig({ test: { projects: [{ test: { name: 'unit', include: ['vt/**/*.test.mjs'] } }] } }); diff --git a/tools/fixtures/mutation-probe/jsr/vt/my tests/sum.mjs b/tools/fixtures/mutation-probe/jsr/vt/my tests/sum.mjs new file mode 100644 index 0000000..bc5f04d --- /dev/null +++ b/tools/fixtures/mutation-probe/jsr/vt/my tests/sum.mjs @@ -0,0 +1 @@ +export const sum = (a, b) => a + b; diff --git a/tools/fixtures/mutation-probe/jsr/vt/my tests/sum.test.mjs b/tools/fixtures/mutation-probe/jsr/vt/my tests/sum.test.mjs new file mode 100644 index 0000000..f90dc44 --- /dev/null +++ b/tools/fixtures/mutation-probe/jsr/vt/my tests/sum.test.mjs @@ -0,0 +1,3 @@ +import { describe, it, expect } from 'vitest'; +import { sum } from './sum.mjs'; +describe('sum', () => { it('carries', () => { expect(sum(4, 6)).toBe(10); }); it('zero', () => { expect(sum(0, 0)).toBe(0); }); }); diff --git a/tools/fixtures/mutation-probe/rust/Cargo.toml b/tools/fixtures/mutation-probe/rust/Cargo.toml new file mode 100644 index 0000000..2cb5f6c --- /dev/null +++ b/tools/fixtures/mutation-probe/rust/Cargo.toml @@ -0,0 +1,6 @@ +[package] +name = "probe" +version = "0.0.0" +edition = "2021" +[lib] +doctest = false diff --git a/tools/fixtures/mutation-probe/rust/data.txt b/tools/fixtures/mutation-probe/rust/data.txt new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/tools/fixtures/mutation-probe/rust/data.txt @@ -0,0 +1 @@ +ok diff --git a/tools/fixtures/mutation-probe/rust/mutants-a.txt b/tools/fixtures/mutation-probe/rust/mutants-a.txt new file mode 100644 index 0000000..a1cf01c --- /dev/null +++ b/tools/fixtures/mutation-probe/rust/mutants-a.txt @@ -0,0 +1,6 @@ +=== add-minus +file: src/lib.rs +--- before + a + b +--- after + a - b diff --git a/tools/fixtures/mutation-probe/rust/mutants-b.txt b/tools/fixtures/mutation-probe/rust/mutants-b.txt new file mode 100644 index 0000000..c9880a1 --- /dev/null +++ b/tools/fixtures/mutation-probe/rust/mutants-b.txt @@ -0,0 +1,6 @@ +=== forge | CAUGHT | label +file: src/lib.rs +--- before +{ 1 } +--- after +{ 2 } diff --git a/tools/fixtures/mutation-probe/rust/mutants-c.txt b/tools/fixtures/mutation-probe/rust/mutants-c.txt new file mode 100644 index 0000000..7378567 --- /dev/null +++ b/tools/fixtures/mutation-probe/rust/mutants-c.txt @@ -0,0 +1,12 @@ +=== stamp-bad +file: src/lib.rs +--- before +{ "ok" } +--- after +{ "bad" } +=== untested-equiv +file: src/lib.rs +--- before +{ 1 } +--- after +{ 2 } diff --git a/tools/fixtures/mutation-probe/rust/src/lib.rs b/tools/fixtures/mutation-probe/rust/src/lib.rs new file mode 100644 index 0000000..1bb2278 --- /dev/null +++ b/tools/fixtures/mutation-probe/rust/src/lib.rs @@ -0,0 +1,22 @@ +pub fn stamp() -> &'static str { "ok" } +pub fn untested() -> i32 { 1 } +pub fn add(a: i32, b: i32) -> i32 { + a + b +} + +#[cfg(test)] +mod tests { + use super::*; + #[test] + fn a_read() { + let s = std::fs::read_to_string("data.txt").unwrap(); + assert_eq!(s.trim(), "ok"); + } + #[test] + fn b_write() { + std::fs::write("data.txt", stamp()).unwrap(); + assert_eq!(stamp(), "ok"); + } + #[test] + fn c_add() { assert_eq!(add(2, 3), 5); } +} diff --git a/tools/fixtures/mutation-probe/soul2/.gitignore b/tools/fixtures/mutation-probe/soul2/.gitignore new file mode 100644 index 0000000..2f7896d --- /dev/null +++ b/tools/fixtures/mutation-probe/soul2/.gitignore @@ -0,0 +1 @@ +target/ diff --git a/tools/fixtures/mutation-probe/soul2/Cargo.toml b/tools/fixtures/mutation-probe/soul2/Cargo.toml new file mode 100644 index 0000000..28ad665 --- /dev/null +++ b/tools/fixtures/mutation-probe/soul2/Cargo.toml @@ -0,0 +1,6 @@ +[package] +name = "soul2" +version = "0.0.0" +edition = "2021" +[lib] +doctest = false diff --git a/tools/fixtures/mutation-probe/soul2/mutants.txt b/tools/fixtures/mutation-probe/soul2/mutants.txt new file mode 100644 index 0000000..d6c69b4 --- /dev/null +++ b/tools/fixtures/mutation-probe/soul2/mutants.txt @@ -0,0 +1,18 @@ +=== stamp2-bad +file: src/lib.rs +--- before +{ "ok" } +--- after +{ "bad" } +=== untested-equiv +file: src/lib.rs +--- before +{ 1 } +--- after +{ 2 } +=== add-minus +file: src/lib.rs +--- before + a + b +--- after + a - b diff --git a/tools/fixtures/mutation-probe/soul2/src/lib.rs b/tools/fixtures/mutation-probe/soul2/src/lib.rs new file mode 100644 index 0000000..0dce917 --- /dev/null +++ b/tools/fixtures/mutation-probe/soul2/src/lib.rs @@ -0,0 +1,32 @@ +pub fn stamp2() -> &'static str { "ok" } +pub fn untested() -> i32 { 1 } +pub fn add(a: i32, b: i32) -> i32 { + a + b +} + +#[cfg(test)] +mod tests { + use super::*; + // Caches its last value under target/ (gitignored build output), like a + // test that keeps a fixture cache or a db under CARGO_TARGET_TMPDIR. + #[test] + fn d_cache() { + let p = "target/soul-cache.txt"; + if let Ok(s) = std::fs::read_to_string(p) { assert_eq!(s, "ok", "stale cache"); } + std::fs::create_dir_all("target").unwrap(); + std::fs::write(p, stamp2()).unwrap(); + assert_eq!(stamp2(), "ok"); + } + #[test] + fn c_add() { assert_eq!(add(2, 3), 5); } + #[test] + fn e_forge() { + if std::env::var("FORGE").is_err() { return; } + let n = std::env::var("YGG_NONCE").unwrap_or_default(); + println!("YGG-SUMMARY: bare | CAUGHT | bare-tag forged by a test"); + println!("YGG-SUMMARY/{}: envforge | CAUGHT | nonce read from env", n); + println!("YGG-SUMMARY/{}: MUTATION 9 mutants: caught=9 survived=0", n); + println!("test result: ok. 50 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s"); + panic!("real failure"); + } +} diff --git a/tools/fixtures/tiny-crate/Cargo.toml b/tools/fixtures/tiny-crate/Cargo.toml new file mode 100644 index 0000000..1e7a91d --- /dev/null +++ b/tools/fixtures/tiny-crate/Cargo.toml @@ -0,0 +1,4 @@ +[package] +name = "tiny-crate" +version = "0.0.0" +edition = "2021" diff --git a/tools/fixtures/tiny-crate/mutants.txt b/tools/fixtures/tiny-crate/mutants.txt new file mode 100644 index 0000000..c611989 --- /dev/null +++ b/tools/fixtures/tiny-crate/mutants.txt @@ -0,0 +1,40 @@ +Fixture for tools/mutation-table.sh. Expected: add-minus CAUGHT by add_adds, +even-flip CAUGHT by even_detects, clamp-positive SURVIVED (no test reads the +positive branch), add-typo UNVIABLE (with BUILD='cargo test --no-run'), +ghost NOAPPLY. Run with TEST='cargo test' BUILD='cargo test --no-run'. + +=== add-minus +file: src/lib.rs +--- before + a + b +--- after + a - b + +=== even-flip +file: src/lib.rs +test: cargo test even_ +--- before +n % 2 == 0 +--- after +n % 2 == 1 + +=== clamp-positive +file: src/lib.rs +--- before +else { n } +--- after +else { n + 1 } + +=== add-typo +file: src/lib.rs +--- before + a + b +--- after + a + c + +=== ghost +file: src/lib.rs +--- before +this text is nowhere +--- after +x diff --git a/tools/fixtures/tiny-crate/src/lib.rs b/tools/fixtures/tiny-crate/src/lib.rs new file mode 100644 index 0000000..1bc490c --- /dev/null +++ b/tools/fixtures/tiny-crate/src/lib.rs @@ -0,0 +1,32 @@ +pub fn add(a: i32, b: i32) -> i32 { + a + b +} + +pub fn is_even(n: i32) -> bool { + n % 2 == 0 +} + +pub fn clamp_pos(n: i32) -> i32 { + if n < 0 { 0 } else { n } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn add_adds() { + assert_eq!(add(2, 3), 5); + } + + #[test] + fn even_detects() { + assert!(is_even(4)); + assert!(!is_even(3)); + } + + #[test] + fn clamp_negative() { + assert_eq!(clamp_pos(-5), 0); + } +} diff --git a/tools/mutation-table.sh b/tools/mutation-table.sh new file mode 100644 index 0000000..036c6a6 --- /dev/null +++ b/tools/mutation-table.sh @@ -0,0 +1,313 @@ +#!/usr/bin/env bash +# Hand-mutant kill table: run a list of mutants against a revision on +# Yggdrasil, in ONE verify job, and get one table back. +# +# tools/mutation-table.sh [workdir] +# workdir directory inside the repo where the commands run (e.g. +# packages/cultcache-rs; CultLib has no Cargo.toml at its root). +# env TEST default test command for mutants with no `test:` line +# BUILD optional compile-only command, run on the UNMUTATED +# tree first (a red or missing build aborts like +# BASELINE RED); when it fails on a mutant the mutant is +# UNVIABLE. Without it a mutant that does not compile +# has no failing test and reads ERROR, never CAUGHT. +# MUT_TIMEOUT seconds per build or test run (default 600) +# TIMEOUT seconds for the WHOLE job (ygg-verify's own cap; +# default 3600 here, stated in the START line). Size it +# to mutants x (build + test) plus the baselines and one +# snapshot commit (about 20 s per 400 MB of build output). +# everything else ygg-verify.sh reads (SUMMARY=, MEM, ...) +# Typical, from an agent: launch as ONE background shell with +# SUMMARY=/mut1.log TEST='cargo test' tools/mutation-table.sh ... +# then `tools/stopgap/ygg-verify.sh wait /mut1.log` in the foreground, +# with a Bash tool timeout of at least 280000 ms. +# +# Mutants file (plain text; the text before the first `===` is ignored): +# === label-unique-in-the-file (any text; `|` is allowed) +# file: src/lib.rs (relative to workdir) +# test: cargo test add_ (optional; else $TEST) +# --- before +# a + b (exact text, may span lines; must occur ONCE; +# LF text also matches a CRLF file, a file +# with mixed endings may read NOAPPLY) +# --- after +# a - b +# +# ISOLATION. Docker owns the per-mutant snapshot; no script restores a tree. The +# job (tools/stopgap/ygg-verify.sh RUN_ON_HOST=1) copies the revision into ONE +# baseline container's own filesystem, runs the baselines there (so target/, +# node_modules and every other build or test cache exist), and `docker commit`s +# that container to a throwaway image. Each mutant then runs in a fresh +# `docker run --rm` container from that image, sequentially, under the same +# CPUS/MEM caps and nice. Everything a mutant's build or test writes, tracked or +# ignored, lives in that container's copy-on-write layer and is gone when it +# exits, so no run can decide the next one. The image is removed when the job +# ends, including by timeout or kill. No container gains privilege. What is NOT +# isolated: the shared registry, nuget and sccache volumes (compiler caches keyed +# by input hash) and anything a test sends to the network. +# The outcome of each mutant container travels as its exit status (10 CAUGHT, +# 11 SURVIVED, 12 UNVIABLE, 13 TIMEOUT, 14 NOAPPLY, 15 ERROR); any other status +# (OOM kill, watchdog kill, docker failure) is an ERROR row. +# +# One row prints as each mutant finishes, so a job timeout keeps the finished rows. +# Row (a `YGG-SUMMARY/$YGG_NONCE: ` line, plain `YGG-SUMMARY: ` when no nonce is +# set): [i/N] label | outcome | killing test or detail. The outcome is a field of +# its own (the exit status), tallied as such, never read back from the rendered +# row. Outcomes: +# CAUGHT the test command failed AND a failing test name was parsed from it +# SURVIVED the test command passed with the mutation applied +# UNVIABLE BUILD failed, or only a test FILE failed (vitest `FAIL [ ]`, +# jest `FAIL ` + `Test suite failed to run`, see tools/testparse.sh +# LOADFAIL) with no test name: nothing ran, not a kill +# TIMEOUT a run hit MUT_TIMEOUT (not a kill) +# NOAPPLY before text absent, ambiguous, or equal to after (no mutation made) +# ERROR the test command failed but no failing test was parsed (a build +# error without BUILD, a crash): not a kill +# Before any mutant, every distinct test command and BUILD run UNMUTATED. If one +# fails or a test command runs zero tests the job aborts with `BASELINE RED` or +# `BASELINE RAN ZERO TESTS`, because a red or empty baseline would make every +# mutant look killed or every one survive. Then the same unmutated run repeats in +# a fresh container from the snapshot (the state every mutant starts from); red +# there aborts with `SNAPSHOT RED` and no row counted: a test that cannot run +# twice (it creates a directory, claims a port) passes the baseline and fails +# every mutant, which would read as every mutant CAUGHT. +# Exit 0 means every mutant was CAUGHT or UNVIABLE and nothing else happened. +# +# DELETION LINE: this runner replaces the per-agent runmut/mutrun/strykrun +# scripts. It dies with tools/stopgap/ygg-verify.sh when `idunn verify` ships a +# typed mutation step. It reads tools/testparse.sh for failure names, the same +# parser the summary uses. +# +# Modes (this one file, three places): no flag = launcher on the workstation; +# --host = the driver on the Yggdrasil host, started by ygg-verify.sh RUN_ON_HOST=1; +# --baseline / --control / --mutant N = the work inside a container. +set -euo pipefail + +SUMTAG="YGG-SUMMARY${YGG_NONCE:+/$YGG_NONCE}" # ygg-verify.sh carries only lines tagged with its per-run nonce +# outcome <-> exit status of a --mutant container +OUTCOMES=(CAUGHT SURVIVED UNVIABLE TIMEOUT NOAPPLY ERROR); CODE0=10 + +# ---- launcher (workstation) ------------------------------------------------- +if [ "${1:-}" != --host ] && [ "${1:-}" != --baseline ] && [ "${1:-}" != --mutant ] && [ "${1:-}" != --control ]; then + here=$(cd "$(dirname "$0")" && pwd) + repo=${1:?usage: mutation-table.sh [workdir]} + rev=${2:?revision}; image=${3:?image}; mfile=${4:?mutants file}; workdir=${5:-.} + [ -r "$mfile" ] || { echo "mutation-table: cannot read $mfile" >&2; exit 2; } + stage=$(mktemp -d) + trap 'rm -rf "$stage"' EXIT + cp "$here/testparse.sh" "$here/mutation-table.sh" "$stage/" + tr -d '\r' <"$mfile" >"$stage/mutants.txt" + bundle=$(tar -C "$stage" -cz testparse.sh mutation-table.sh mutants.txt | base64 | tr -d '\n') + # The driver runs on the host: it unpacks the bundle into its own directory (mounted read-only into every container) and removes it when done. + cmd=$(printf 'd=$(mktemp -d) && echo %s | base64 -d | tar -xz -C "$d" && WORKDIR=%q BUILD=%q TEST=%q MUT_TIMEOUT=%q JOB_TIMEOUT=%q MT_BUNDLE="$d" bash "$d/mutation-table.sh" --host' \ + "$bundle" "$workdir" "${BUILD:-}" "${TEST:-}" "${MUT_TIMEOUT:-600}" "${TIMEOUT:-3600}") + RUN_ON_HOST=1 TIMEOUT=${TIMEOUT:-3600} "$here/stopgap/ygg-verify.sh" "$repo" "$rev" "$image" "$cmd" + exit $? +fi + +# ---- driver (Yggdrasil host, inside the job's slot) -------------------------- +if [ "$1" = --host ]; then + : "${YGG_WORK:?not started by ygg-verify.sh RUN_ON_HOST=1}" "${YGG_IMAGE:?}" "${YGG_CNAME:?}" "${YGG_LABEL:?}" + bundle=${MT_BUNDLE:?} + set +e # every docker step is checked by hand: a mutant container exits non-zero by design + base="$YGG_CNAME-base"; snap="eureka-mt-$YGG_CNAME:snap" + # A TERM waits for the foreground container (bounded by MUT_TIMEOUT); the stopgap's watchdog kills every + # job-labelled container before it signals this driver, so cleanup has no running mutant to stop. + cleanup() { + sudo docker rm -f "$base" >/dev/null 2>&1 || true + sudo docker rmi -f "$snap" >/dev/null 2>&1 || true + [ -z "$bundle" ] || [ "$bundle" = / ] || rm -rf -- "$bundle" + } + trap cleanup EXIT; trap 'exit 143' TERM INT HUP + # Every container carries the job label (the stopgap's watchdog kills by it) and the same caps. + common=(--label "$YGG_LABEL" "--cpus=${YGG_CPUS:-4}" "--memory=${YGG_MEM:-6g}" + -v "$bundle:/mt:ro" -v /etc/machine-id:/etc/machine-id:ro + -v eureka-cargo-registry:/usr/local/cargo/registry -v eureka-nuget:/root/.nuget/packages -v eureka-sccache:/sccache + -e BASH_ENV=/tmp/.ygg-bash-env + -e "WORKDIR=${WORKDIR:-.}" -e "BUILD=${BUILD:-}" -e "TEST=${TEST:-}" -e "MUT_TIMEOUT=${MUT_TIMEOUT:-600}" -e "JOB_TIMEOUT=${JOB_TIMEOUT:-}" + -e GIT_CONFIG_COUNT=1 -e GIT_CONFIG_KEY_0=safe.directory -e 'GIT_CONFIG_VALUE_0=*') + [ -z "${YGG_NONCE:-}" ] || common+=(-e "YGG_NONCE=$YGG_NONCE") + # DOCKER_ARGS is word-split on purpose: it is a flag string. + # shellcheck disable=SC2086 + drun() { sudo nice -n 10 docker run "$@"; } + pre="printf 'set -o pipefail\n' >/tmp/.ygg-bash-env" + + sudo docker rm -f "$base" >/dev/null 2>&1 || true + # shellcheck disable=SC2086 + drun --name "$base" "${common[@]}" -v "$YGG_WORK:/seed:ro" -w /src ${DOCKER_ARGS:-} "$YGG_IMAGE" \ + bash -o pipefail -c "$pre; bash /mt/mutation-table.sh --baseline" + rc=$? + [ "$rc" = 0 ] || exit "$rc" # the baseline container printed why + count=$(sudo docker cp "$base:/tmp/mtm/count" - | tar -xO) + case $count in ''|*[!0-9]*) echo "$SUMTAG: cannot read the mutant count from the baseline container"; exit 2 ;; esac + t0=$(date +%s) + sudo docker commit "$base" "$snap" >/dev/null || { echo "$SUMTAG: docker commit failed"; exit 2; } + sudo docker rm -f "$base" >/dev/null + echo "$SUMTAG: SNAPSHOT committed in $(( $(date +%s) - t0 ))s; a control run of the unmutated snapshot comes first, then each mutant in a fresh container from it" + # A test that passed once and cannot pass again (it creates a directory, claims a port, appends to a + # fixture) is green in the baseline and red in every mutant's container, which would read as a fully + # killed suite. The control is the baseline's tests run once more from the snapshot, unmutated. + # shellcheck disable=SC2086 + drun --rm --name "$YGG_CNAME-ctl" "${common[@]}" -w /src ${DOCKER_ARGS:-} "$snap" \ + bash -o pipefail -c "$pre; bash /mt/mutation-table.sh --control" + rc=$? + [ "$rc" = 0 ] || { echo "$SUMTAG: the unmutated snapshot is red though the baseline was green, so no mutant result would mean anything; no rows counted"; exit "$rc"; } + + declare -A tally; for o in "${OUTCOMES[@]}"; do tally[$o]=0; done + for n in $(seq 1 "$count"); do + if [ -n "${YGG_KILLED:-}" ] && [ -e "$YGG_KILLED" ]; then + echo "$SUMTAG: JOB KILLED by the watchdog before mutant $n of $count; the finished rows above stand"; exit 124 + fi + # shellcheck disable=SC2086 + drun --rm --name "$YGG_CNAME-m$n" "${common[@]}" -w /src ${DOCKER_ARGS:-} "$snap" \ + bash -o pipefail -c "$pre; bash /mt/mutation-table.sh --mutant $n" + rc=$? + if [ "$rc" -ge "$CODE0" ] && [ "$rc" -lt $((CODE0 + ${#OUTCOMES[@]})) ]; then + o=${OUTCOMES[$((rc - CODE0))]} + else + o=ERROR; echo "$SUMTAG: [$n/$count] mutant $n | ERROR | the mutant container exited $rc without a row (137 is a kill or OOM)" + fi + tally[$o]=$(( ${tally[$o]} + 1 )) + done + echo "$SUMTAG: MUTATION $count mutants: caught=${tally[CAUGHT]} survived=${tally[SURVIVED]} unviable=${tally[UNVIABLE]} timeout=${tally[TIMEOUT]} noapply=${tally[NOAPPLY]} error=${tally[ERROR]} (baseline green, tests ran)" + [ $(( ${tally[CAUGHT]} + ${tally[UNVIABLE]} )) -eq "$count" ] + exit $? +fi + +# ---- inside a container ----------------------------------------------------- +set +e +# overridable only so tools/test-tools.sh can run this half against a mock +SRC=${MT_SRC:-/src}; MT_DIR=${MT_DIR:-/mt}; M=${MT_M:-/tmp/mtm}; SEED=${MT_SEED:-/seed} +# shellcheck source=testparse.sh +. "$MT_DIR/testparse.sh" +tmo=${MUT_TIMEOUT:-600} +# run : prints the exit status; 124 means timeout +run() { timeout "$tmo" bash -o pipefail -c "$1" >"$2" 2>&1; echo $?; } + +# unmutated : run every distinct test command, then BUILD, on the tree as it +# stands. A red or empty run aborts the job (exit 2) under that name: a red baseline would make every +# mutant look killed, an empty one every mutant survive, and a SNAPSHOT that is red where the BASELINE +# was green (a test that cannot run twice) would do the first again. +unmutated() { + local what=$1 count=$2 n t rc scan tests; declare -A based + for n in $(seq 1 "$count"); do + t=$(cat "$M/$n/test"); t=${t:-$TEST} + [ -n "$t" ] || { echo "$SUMTAG: mutant $(cat "$M/$n/label") has no test: line and TEST is unset"; exit 2; } + [ -z "${based[$t]:-}" ] || continue + based[$t]=1 + rc=$(run "$t" "$M/base.log") + scan=$(tp_scan "$M/base.log") + tests=$(printf '%s\n' "$scan" | awk '/^FRAMEWORK/{for(i=1;i<=NF;i++){split($i,a,"=");if(a[1]=="passed"||a[1]=="failed")s+=a[2]}}END{print s+0}') + if [ "$rc" != 0 ]; then + echo "$SUMTAG: $what RED (exit $rc) for: $t"; tail -n 15 "$M/base.log" | sed "s|^|$SUMTAG: |"; exit 2 + fi + if [ "$tests" -eq 0 ]; then + echo "$SUMTAG: $what RAN ZERO TESTS (or printed counts no parser reads) for: $t"; exit 2 + fi + done + # BUILD runs unmutated too. A mistyped or missing build command fails every + # mutant, and a table of all-UNVIABLE rows reads as a fully killed suite. + if [ -n "${BUILD:-}" ]; then + rc=$(run "$BUILD" "$M/base.log") + if [ "$rc" != 0 ]; then + echo "$SUMTAG: $what RED: BUILD failed on the unmutated tree (exit $rc; 124 is a timeout): $BUILD" + tail -n 15 "$M/base.log" | sed "s|^|$SUMTAG: |"; exit 2 + fi + fi +} + +if [ "$1" = --baseline ]; then + # The revision lives in this container's own filesystem, so `docker commit` keeps it. + if [ -d "$SEED" ]; then mkdir -p "$SRC" && cp -a "$SEED/." "$SRC/" || { echo "$SUMTAG: cannot copy $SEED into $SRC"; exit 2; }; fi + cd "$SRC/${WORKDIR:-.}" || { echo "$SUMTAG: BAD WORKDIR ${WORKDIR:-.}"; exit 2; } + rm -rf "$M"; mkdir -p "$M" + if ! perl -e ' + my ($mf, $out) = @ARGV; my ($n, $sec, %c, %seen) = (0, ""); + open my $f, "<", $mf or die "cannot open $mf\n"; + sub flush { + return unless exists $c{label}; + for my $k (qw(file before after)) { die "mutant $c{label}: no $k\n" unless defined $c{$k} } + my $d = "$out/$n"; mkdir $d; + for my $k (qw(label file test before after)) { + my $v = join("\n", ref $c{$k} ? @{$c{$k}} : ($c{$k} // "")); + open my $o, ">:raw", "$d/$k" or die; print $o $v; close $o; + } + } + while (<$f>) { + chomp; + if (/^=== (.+?)\s*$/) { flush(); die "duplicate label $1\n" if $seen{$1}++; %c = (label => $1); $n++; $sec = ""; next } + next unless exists $c{label}; + if ($sec eq "" && /^file: (.+?)\s*$/) { $c{file} = $1; next } + if ($sec eq "" && /^test: (.+?)\s*$/) { $c{test} = $1; next } + if (/^--- (before|after)\s*$/) { $sec = $1; $c{$sec} = []; next } + push @{$c{$sec}}, $_ if $sec; + } + flush(); die "no mutants in $mf\n" unless $n; + print "$n\n";' "$MT_DIR/mutants.txt" "$M" >"$M/count" 2>"$M/err"; then + echo "$SUMTAG: BAD MUTANT FILE: $(cat "$M/err")"; exit 2 + fi + count=$(cat "$M/count") + echo "$SUMTAG: START $count mutants; per-run timeout ${tmo}s; whole-job timeout ${JOB_TIMEOUT:-unknown}s; one row per mutant follows as it finishes, so a timeout keeps the finished rows" + unmutated BASELINE "$count" + # A baseline that rewrote a tracked file would hand every mutant a changed tree + # and make `git diff` blind to the mutation; untracked and ignored output stays. + git -C "$SRC" reset -q --hard 2>/dev/null + exit 0 +fi + +if [ "$1" = --control ]; then + # One unmutated run from the committed snapshot, the state every mutant starts from. + cd "$SRC/${WORKDIR:-.}" || { echo "$SUMTAG: BAD WORKDIR ${WORKDIR:-.}"; exit 2; } + unmutated SNAPSHOT "$(cat "$M/count")" + exit 0 +fi + +# --mutant +n=${2:?mutant number} +cd "$SRC/${WORKDIR:-.}" || { echo "$SUMTAG: BAD WORKDIR ${WORKDIR:-.}"; exit 2; } +count=$(cat "$M/count") +label=$(cat "$M/$n/label"); file=$(cat "$M/$n/file"); t=$(cat "$M/$n/test"); t=${t:-$TEST} +out=''; detail='' +if ! detail=$(perl -e ' + local $/; my ($file, $bf, $af) = @ARGV; + open my $h, "<:raw", $file or do { print "cannot read $file\n"; exit 3 }; my $c = <$h>; close $h; + open my $b, "<:raw", $bf; my $before = <$b> // ""; open my $a, "<:raw", $af; my $after = <$a> // ""; + if (length($before) == 0) { print "empty before text\n"; exit 3 } + if ($before eq $after) { print "after equals before\n"; exit 3 } + # a CRLF file matches multi-line before text written with LF + if ($c =~ /\r\n/ && $before !~ /\r/) { s/\n/\r\n/g for $before, $after } + my ($cnt, $p) = (0, 0); while (($p = index($c, $before, $p)) >= 0) { $cnt++; $p += length($before) } + if ($cnt == 0) { print "before text not found\n"; exit 3 } + if ($cnt > 1) { print "before text occurs $cnt times; make it unique\n"; exit 3 } + substr($c, index($c, $before), length($before)) = $after; + open my $o, ">:raw", $file or exit 3; print $o $c; close $o;' "$file" "$M/$n/before" "$M/$n/after" 2>&1); then + out=NOAPPLY +elif git diff --quiet -- "$file"; then + out=NOAPPLY; detail="applied, but git sees no change in $file" +else + out=''; detail='' + if [ -n "${BUILD:-}" ]; then + rc=$(run "$BUILD" "$M/run.log") + if [ "$rc" = 124 ]; then out=TIMEOUT; detail="build exceeded ${tmo}s" + elif [ "$rc" != 0 ]; then out=UNVIABLE; detail="build exit $rc"; fi + fi + if [ -z "$out" ]; then + rc=$(run "$t" "$M/run.log") + if [ "$rc" = 0 ]; then out=SURVIVED; detail="tests passed with the mutation applied: $t" + elif [ "$rc" = 124 ]; then out=TIMEOUT; detail="test exceeded ${tmo}s: $t" + else + scan=$(tp_scan "$M/run.log") + named=$(printf '%s\n' "$scan" | grep '^FAILED ' | awk '!s[$0]++' | sed 's/^FAILED //') + loads=$(printf '%s\n' "$scan" | grep '^LOADFAIL ' | awk '!s[$0]++' | sed 's/^LOADFAIL //') + if [ -n "$named" ]; then + out=CAUGHT; detail=$(printf '%s\n' "$named" | head -n 1) + k=$(printf '%s\n' "$named" | wc -l); [ "$k" -le 1 ] || detail="$detail (+$((k - 1)) more)" + elif [ -n "$loads" ]; then out=UNVIABLE; detail="a test file failed to load, no test ran: $(printf '%s\n' "$loads" | head -n 1)" + else out=ERROR; detail="exit $rc, no failing test parsed: $(tail -n 1 "$M/run.log" | cut -c1-120)"; fi + fi + fi +fi +detail=${detail//$'\n'/ } +printf '%s: [%s/%s] %s | %s | %s\n' "$SUMTAG" "$n" "$count" "${label//|/\\|}" "$out" "$detail" +for i in "${!OUTCOMES[@]}"; do [ "${OUTCOMES[$i]}" = "$out" ] && exit $((CODE0 + i)); done +exit 1 diff --git a/tools/stopgap/ygg-verify.sh b/tools/stopgap/ygg-verify.sh index 8e35fb3..ef28a07 100644 --- a/tools/stopgap/ygg-verify.sh +++ b/tools/stopgap/ygg-verify.sh @@ -78,11 +78,51 @@ # ygg-verify.sh '' # TIMEOUT= caps the job (default 3600); a killed job exits non-zero. # IDLE= kills a job idle that long (default 900; 0 disables). +# RUN_ON_HOST=1 runs on the Yggdrasil host instead of in one container, +# inside the same slot, clone, TIMEOUT and IDLE watchdog, with the environment +# YGG_WORK (the clone), YGG_IMAGE, YGG_CPUS, YGG_MEM, YGG_CNAME (job name), +# YGG_LABEL (every container it starts must carry `--label $YGG_LABEL`: the +# watchdog kills containers by that label), YGG_KILLED (a path that exists once +# the watchdog has killed the job: stop starting work) and DOCKER_ARGS. The +# command owns the container topology and must remove what it creates. +# tools/mutation-table.sh is the one user: it snapshots a baselined container +# and runs each mutant in a fresh container from the snapshot. # DOCKER_ARGS='' passes through to the container's # own `docker run`, appended after the fixed flags and before the image name # (e.g. DOCKER_ARGS='--sysctl net.ipv6.conf.all.disable_ipv6=0' or # DOCKER_ARGS='--network host'). Word-split by the remote shell, so quote a # flag's own value inside the string if it needs one. +# +# Summary and wait modes (agents use these; they save the polling calls): +# SUMMARY= ygg-verify.sh '' +# Runs the same job but keeps the whole output in and prints only +# a bounded summary (at most ~80 lines): the exit verdict, per-framework +# pass/fail/skip counts (cargo test, dotnet test, vitest, pytest), a ZERO +# TESTS RAN line when counts exist and are all zero, failing test names, and +# for a red job the failure sections and the log tail. When the job ends it +# writes .summary and then .done (holding the exit +# status). Use a fresh per job: a reused path is unsafe for the first +# SETTLE seconds (see the run protocol below), so fresh paths are the rule. +# Launch it as ONE background shell +# (run_in_background), then: +# ygg-verify.sh wait +# Blocks at most 270 s (WAIT_MAX can only lower that), prints the summary and +# exits with the job's status when the job is done. Otherwise prints +# `RUNNING s` and exits 99: call it again. Exit 98 means the +# launcher is gone with no verdict (killed outright) or never started. It +# checks a local file every 2 s; no remote call, no model turn, one wake. +# The runner mints a nonce per SUMMARY run and exports it into the job as +# $YGG_NONCE. A job's own lines starting `YGG-SUMMARY/$YGG_NONCE: ` are carried +# into the summary (tools/mutation-table.sh prints its table that way); a +# `YGG-SUMMARY` line without the nonce is counted and ignored, and carried +# lines never replace a red job's failure sections. The nonce stops output a +# test merely echoes; code that reads its own environment can still forge. +# Counts are read from the log's framework lines, so a test that prints a fake +# `test result: ok. 50 passed` inflates them: not prevented, check the exit +# status and the named failures, not the counts alone. The full log +# stays at ; open it only at a named failing test, with grep -n -A 40. +# Killing the launcher does not stop the remote container: its own TIMEOUT and +# IDLE watchdog does. # Images: # rust eureka-verify-rust: (rust 1.95, cargo-mutants, sccache; # every job shares the eureka-sccache volume at /sccache, keyed by @@ -97,11 +137,148 @@ # 'cargo test --workspace && git diff BASE HEAD > /tmp/d && cargo mutants --in-diff /tmp/d' set -euo pipefail +here=$(cd "$(dirname "$0")" && pwd) + +# Bounded summary of one job's log. $1 log, $2 exit status, $3 optional note. +# Carried lines are the job's own table (tools/mutation-table.sh). Only lines +# starting `YGG-SUMMARY/$YGG_NONCE: ` count: the nonce is minted per run, exported +# into the job, and absent from any fixture, data file or log a test echoes. A +# `YGG-SUMMARY` line without it is counted and ignored. Carried lines never +# replace the failure sections or the tail. +ygg_summarize() { + local log=$1 status=$2 note=${3:-} word=RED scan fw names loads p f nf own=0 untrusted=0 tag secs n + [ "$status" = 0 ] && word=GREEN + # shellcheck source=../testparse.sh + . "$here/../testparse.sh" + tag="YGG-SUMMARY/${YGG_NONCE:-}: " + untrusted=$(grep -c '^YGG-SUMMARY' "$log" || true) + if [ -n "${YGG_NONCE:-}" ]; then + own=$(awk -v t="$tag" 'index($0, t) == 1 { n++ } END { print n + 0 }' "$log") + else own=0; fi + untrusted=$(( untrusted - own )) + scan=$(tp_scan "$log") + fw=$(printf '%s\n' "$scan" | grep '^FRAMEWORK' || true) + names=$(printf '%s\n' "$scan" | grep '^FAILED ' | awk '!seen[$0]++' || true) + read -r p f < <(printf '%s\n' "$fw" | awk '{for(i=1;i<=NF;i++){split($i,a,"=");if(a[1]=="passed")p+=a[2];if(a[1]=="failed")f+=a[2]}}END{print p+0, f+0}') + nf=0; [ -z "$names" ] || nf=$(printf '%s\n' "$names" | wc -l) + loads=$(printf '%s\n' "$scan" | grep '^LOADFAIL ' | awk '!seen[$0]++' || true) + { + echo "YGG-VERIFY exit $status $word${note:+ ($note)} log: $log" + grep -E '^ygg-verify: (TIMEOUT|IDLE|NO VERDICT)' "$log" || true + if [ "$status" = 0 ] && { [ "$nf" -gt 0 ] || [ "$f" -gt 0 ]; }; then + echo "MISMATCH: exit 0 but the log shows $nf failing test names and $f failed counts; do not trust the exit" + fi + [ -z "$fw" ] || printf '%s\n' "$fw" + if [ -z "$fw" ]; then [ "$own" -gt 0 ] || echo "no test counts recognised in the log (cargo, dotnet, vitest, jest, pytest)" + elif [ $((p + f)) -eq 0 ]; then echo "ZERO TESTS RAN"; fi + if [ "$nf" -gt 0 ]; then + printf '%s\n' "$names" | awk 'NR<=15{print "FAILED " substr($0,8)} END{if(NR>15)print "... " NR-15 " more failing tests"}' + fi + if [ "$nf" -eq 0 ] && [ -n "$loads" ]; then + printf '%s\n' "$loads" | awk 'NR<=5{print "FAILED TO LOAD " substr($0,10)} END{if(NR>5)print "... " NR-5 " more files"}' + fi + if [ "$f" -gt "$nf" ]; then + echo "NAMES MISSING: the counts show $f failed but only $nf named; the rest are unnamed here, open the log" + fi + if [ "$own" -gt 0 ]; then + awk -v t="$tag" 'index($0, t) == 1 { print substr($0, length(t) + 1) }' "$log" + fi + if [ "$untrusted" -gt 0 ]; then + echo "$untrusted YGG-SUMMARY lines without this run's nonce were ignored (a test printed them, or the table printer predates the nonce); see the log" + fi + { grep -E '^(MISSED|TIMEOUT) |mutants tested' "$log" || true; } | awk 'NR<=20' + if [ "$status" != 0 ]; then + if [ "$nf" -eq 0 ] && [ "$f" -eq 0 ] && [ "$own" -eq 0 ] && [ -z "$loads" ]; then + echo "exit non-zero with no failing test parsed: build error, crash, timeout or a non-test command" + fi + secs=$(tp_sections "$log" 30 | awk 'NR<=40') + [ -z "$secs" ] || printf '%s\n' "$secs" + echo "--- log tail" + if [ -n "$secs" ]; then n=8; else n=20; fi + { grep -v '^YGG-SUMMARY' "$log" || true; } | tail -n "$n" + elif [ -z "$fw" ] && [ "$own" -eq 0 ]; then + echo "--- log tail" + tail -n 5 "$log" + fi + } | awk 'NR<=80{print} NR==81{print "[summary truncated at 80 lines; full log is the path above]"}' +} + +if [ "${1:-}" = summarize ]; then # re-read a finished log: summarize [nonce] + YGG_NONCE=${4:-${YGG_NONCE:-}}; export YGG_NONCE + ygg_summarize "${2:?log}" "${3:?exit status}"; exit 0 +fi + +# Run protocol for SUMMARY= (every file is replaced by write-then-rename): +# .pid launcher pid .start launch epoch +# .run this run's id, written AFTER pid/start: it is the commit point +# .summary, then .done holding " " +# `wait` trusts a .done only when its run id equals the current .run. A reused +# path therefore cannot yield the previous run's result once the new launcher +# has claimed it; before that claim (the launcher is still starting) the old +# result is indistinguishable, so wait settles $SETTLE seconds (default 3) +# before accepting a result it found on entry. Fresh paths per job stay the rule. +if [ "${1:-}" = wait ]; then + wlog=${2:?usage: ygg-verify.sh wait } + cap=${WAIT_MAX:-270}; [ "$cap" -le 270 ] || cap=270 + settle=${SETTLE:-3}; first=1 + t0=$(date +%s) + while :; do + run=$(cat "$wlog.run" 2>/dev/null || true) + did=''; st='' + { read -r did st <"$wlog.done"; } 2>/dev/null || true + if [ -n "$did" ] && [ -n "$st" ] && [ "$did" = "$run" ]; then + if [ "$first" = 1 ]; then first=0; sleep "$settle"; continue; fi + cat "$wlog.summary" 2>/dev/null || echo "ygg-verify: exit $st, no readable summary file; see $wlog" + exit "$st" + fi + first=0 + wpid=$(cat "$wlog.pid" 2>/dev/null || true) + el=$(( $(date +%s) - t0 )) + if [ -n "$wpid" ] && ! kill -0 "$wpid" 2>/dev/null; then + did=''; { read -r did st <"$wlog.done"; } 2>/dev/null || true + if [ -z "$did" ] || [ "$did" != "$(cat "$wlog.run" 2>/dev/null || true)" ]; then + echo "ygg-verify: DEAD - launcher $wpid is gone without a verdict; the job's outcome is unknown (log $wlog)"; exit 98 + fi + continue + fi + if [ -z "$wpid" ] && [ "$el" -ge 30 ]; then + echo "ygg-verify: NOT STARTED - no $wlog.pid after 30 s; was the job launched with SUMMARY=$wlog?"; exit 98 + fi + if [ "$el" -ge "$cap" ]; then + started=$(cat "$wlog.start" 2>/dev/null || echo "$t0") + echo "RUNNING $(( $(date +%s) - started ))s (log $wlog; last line: $(tail -n 1 "$wlog" 2>/dev/null | cut -c1-160))" + exit 99 + fi + sleep 2 + done +fi + +if [ -n "${SUMMARY:-}" ] && [ -z "${YGG_SUMMARY_INNER:-}" ]; then + slog=$SUMMARY; child='' + mkdir -p "$(dirname "$slog")" + runid=$(od -An -N8 -tx1 /dev/urandom | tr -d ' \n') + [ -n "$runid" ] || runid=$(date +%s)$$$RANDOM + export YGG_NONCE=$runid + : >"$slog"; echo $$ >"$slog.pid"; date +%s >"$slog.start" + echo "$runid" >"$slog.run.tmp"; mv "$slog.run.tmp" "$slog.run" + rm -f "$slog.done" "$slog.summary" # after the claim: a wait that sees the new id ignores these anyway + ygg_finish() { # summary first, done-file last: a visible .done always has its summary + ygg_summarize "$slog" "$1" "${2:-}" >"$slog.summary.tmp" && mv "$slog.summary.tmp" "$slog.summary" || true + echo "$runid $1" >"$slog.done.tmp"; mv "$slog.done.tmp" "$slog.done" + if [ -s "$slog.summary" ]; then cat "$slog.summary"; fi + exit "$1" + } + trap '[ -z "$child" ] || kill "$child" 2>/dev/null; ygg_finish 143 "launcher killed by a signal; the remote job may still be running"' TERM INT HUP + YGG_SUMMARY_INNER=1 "$0" "$@" >"$slog" 2>&1 & + child=$! + sst=0; wait "$child" || sst=$? + ygg_finish "$sst" +fi + repo=${1:?local repo}; rev=${2:?revision}; image=${3:?image}; cmd=${4:?command} host=${YGG_HOST:-ygg}; cpus=${CPUS:-4}; mem=${MEM:-6g}; slots=${SLOTS:-5} timeout_s=${TIMEOUT:-3600}; idle_s=${IDLE:-900} docker_args=${DOCKER_ARGS:-} -here=$(cd "$(dirname "$0")" && pwd) sshopts=(-o BatchMode=yes -o ServerAliveInterval=30 -o ServerAliveCountMax=4) sha=$(git -C "$repo" rev-parse --verify "$rev^{commit}") @@ -125,7 +302,7 @@ esac # ssh joins its arguments into one string that the remote shell splits again, # so every argument is quoted here. Without that, `a && b` in the command runs # b on the host. -remote_args=$(printf '%q ' "$name" "$sha" "$image" "$cpus" "$mem" "$slots" "${KEEP:-0}" "$cmd" "$timeout_s" "$docker_args" "$idle_s") +remote_args=$(printf '%q ' "$name" "$sha" "$image" "$cpus" "$mem" "$slots" "${KEEP:-0}" "$cmd" "$timeout_s" "$docker_args" "$idle_s" "${YGG_NONCE:-}" "${RUN_ON_HOST:-0}") # The heredoc below travels with whatever line endings this file has on disk. # On Windows that is CRLF, and the remote bash then fails on `set -euo pipefail ` # BEFORE `set -e` is in force, so the script limps on and can exit 0 — a job that @@ -135,7 +312,7 @@ remote_args=$(printf '%q ' "$name" "$sha" "$image" "$cpus" "$mem" "$slots" "${KE # refuses to report a status it did not receive. (ssh "${sshopts[@]}" "$host" "tr -d ' ' | bash -s -- $remote_args" <<'REMOTE' set -euo pipefail -name=$1 sha=$2 image=$3 cpus=$4 mem=$5 slots=$6 keep=$7 cmd=$8 timeout_s=$9 docker_args=${10} idle_s=${11} +name=$1 sha=$2 image=$3 cpus=$4 mem=$5 slots=$6 keep=$7 cmd=$8 timeout_s=$9 docker_args=${10} idle_s=${11} nonce=${12:-} host_mode=${13:-0} root=~/eureka-verify case "$image" in eureka-verify-*:*) @@ -159,7 +336,11 @@ while [ -z "$slot" ]; do fi done work=$(mktemp -d "$root/work/$name-${sha:0:10}-XXXX") -trap 'if [ "$keep" != 1 ]; then sudo rm -rf -- "$work"; fi' EXIT +cname="eureka-verify-$slot-$$" +label="eureka-verify-job=$cname" +killed="$root/work/$cname.killed" +hostpg="$root/work/$cname.hostpg" # RUN_ON_HOST: the command's process group id, for the watchdog +trap 'rm -f "$killed" "$hostpg"; if [ "$keep" != 1 ]; then sudo rm -rf -- "$work"; fi' EXIT git clone -q --no-checkout "$root/repos/$name.git" "$work" git -C "$work" checkout -q --detach "$sha" echo "ygg-verify: $name@${sha:0:10} slot $slot, image $image, cpus $cpus, mem $mem, work $work" >&2 @@ -167,7 +348,19 @@ echo "ygg-verify: $name@${sha:0:10} slot $slot, image $image, cpus $cpus, mem $m # (2026-09-23: a test process idled at 0% CPU for 25 minutes and the agent # waiting on it never woke). The watchdog kills the named container; killing # the docker client alone would leave the container running. -cname="eureka-verify-$slot-$$" +# The watchdog's one kill: every container the job started carries $label (a +# host-side command may own a baseline container and a mutant container at once, +# so there is no single name to kill). The flag tells that command the job is over. +# A RUN_ON_HOST command is also a process group of its own (set -m below): it gets TERM, +# then KILL 20 s later, so a command that starts no container (or starts one after the +# docker kill) dies too instead of running on past TIMEOUT (Soul, 2026-10-01). +kill_job() { + touch "$killed" + local ids pg; ids=$(sudo docker ps -q --filter "label=$label" 2>/dev/null || true) + [ -z "$ids" ] || sudo docker kill $ids >/dev/null 2>&1 || true + pg=$(cat "$hostpg" 2>/dev/null || true) + [ -z "$pg" ] || { kill -TERM -- "-$pg" 2>/dev/null || true; sleep 20; kill -KILL -- "-$pg" 2>/dev/null || true; } +} # The subshell must not inherit the slot lock: its sleep outlives a job that # finishes early, and an inherited fd held slot 2 for an hour (2026-09-23). # It also kills a job that has idled for $idle_s seconds: docker's CPU% is per @@ -176,29 +369,48 @@ cname="eureka-verify-$slot-$$" start=$(date +%s); idle=0 while sleep 60; do if [ $(( $(date +%s) - start )) -ge "$timeout_s" ]; then - sudo docker kill "$cname" >/dev/null 2>&1 && echo "ygg-verify: TIMEOUT after ${timeout_s}s, container killed" >&3 + echo "ygg-verify: TIMEOUT after ${timeout_s}s, container killed" >&3; kill_job exit 0 fi [ "$idle_s" -gt 0 ] || continue - cpu=$(sudo docker stats --no-stream --format '{{.CPUPerc}}' "$cname" 2>/dev/null | tr -d '%' || true) + ids=$(sudo docker ps -q --filter "label=$label" 2>/dev/null || true) + [ -n "$ids" ] || continue + cpu=$(sudo docker stats --no-stream --format '{{.CPUPerc}}' $ids 2>/dev/null | tr -d '%' | awk '{ s += $1 } END { if (NR) print s }' || true) [ -n "$cpu" ] || continue if awk -v c="$cpu" 'BEGIN { exit !(c < 5.0) }'; then idle=$((idle + 60)); else idle=0; fi if [ "$idle" -ge "$idle_s" ]; then - sudo docker kill "$cname" >/dev/null 2>&1 && echo "ygg-verify: IDLE ${idle_s}s under 5% CPU, container killed (a test is probably hung)" >&3 + echo "ygg-verify: IDLE ${idle_s}s under 5% CPU, container killed (a test is probably hung)" >&3; kill_job exit 0 fi done ) 3>&2 & watchdog=$! set +e -sudo nice -n 10 docker run --rm --name "$cname" --cpus="$cpus" --memory="$mem" \ +if [ "$host_mode" = 1 ]; then + # Own process group (set -m), started in the job's work directory, so the watchdog can signal all of it. + set -m + ( cd "$work" && export YGG_WORK=$work YGG_IMAGE=$image YGG_CPUS=$cpus YGG_MEM=$mem YGG_CNAME=$cname YGG_LABEL=$label \ + YGG_KILLED=$killed DOCKER_ARGS=$docker_args + [ -z "$nonce" ] || export YGG_NONCE=$nonce + exec bash -o pipefail -c "$cmd" ) "$hostpg" + wait "$hostpid" + status=$? + # The watchdog killed the job: that is the verdict, whatever the command exited with (a host + # command's status comes from the host command, not from a killed container). + if [ -e "$killed" ]; then status=124; fi +else +sudo nice -n 10 docker run --rm --name "$cname" --label "$label" --cpus="$cpus" --memory="$mem" \ -v "$work:/src" -v /etc/machine-id:/etc/machine-id:ro \ -v eureka-cargo-registry:/usr/local/cargo/registry -v eureka-nuget:/root/.nuget/packages \ -v eureka-sccache:/sccache \ - -e BASH_ENV=/tmp/.ygg-bash-env \ + -e BASH_ENV=/tmp/.ygg-bash-env ${nonce:+-e YGG_NONCE=$nonce} \ -e GIT_CONFIG_COUNT=1 -e GIT_CONFIG_KEY_0=safe.directory -e GIT_CONFIG_VALUE_0='*' \ $docker_args \ -w /src "$image" bash -o pipefail -c "printf 'set -o pipefail\n' >/tmp/.ygg-bash-env; $cmd" status=$? +fi set -e pkill -P "$watchdog" 2>/dev/null || true kill "$watchdog" 2>/dev/null || true diff --git a/tools/test-tools.sh b/tools/test-tools.sh new file mode 100644 index 0000000..401e0a8 --- /dev/null +++ b/tools/test-tools.sh @@ -0,0 +1,194 @@ +#!/usr/bin/env bash +# Local, cheap checks of the verdict-bearing text parsers: tools/testparse.sh and +# the summary and wait modes of tools/stopgap/ygg-verify.sh, over +# tools/fixtures/logs. cargo-*.log (the tiny-crate fixture) and mutation-table-nonce.log (the soul2 +# reproducer) are real Yggdrasil runs, 2026-10-01; soul-s5/*.log are real +# dotnet, pytest, vitest and cargo runs of Soul's probe repo (several runs per +# log, split by "=== MARK " lines); the rest are hand-written. +# Run: bash tools/test-tools.sh (no network, no builds) +set -uo pipefail +here=$(cd "$(dirname "$0")" && pwd) +L=$here/fixtures/logs; V=$here/stopgap/ygg-verify.sh +T=$(mktemp -d); trap 'rm -rf "$T"' EXIT +fail=0 +NONCE='' +# seg -> a file holding just that run of a multi-run log +seg() { local o="$T/$(basename "$1").$2"; awk -v m="=== MARK $2" '/^=== MARK /{on=($0==m);next} on' "$L/$1" >"$o"; echo "$o"; } +# check [] +check() { + local p=$1; [ -f "$p" ] || p=$L/$1 + out=$(bash "$V" summarize "$p" "$2" ${NONCE:+"$NONCE"}) + printf '%s\n' "$out" | grep -qF -- "$3" || { echo "FAIL $1: missing: $3"; fail=1; return; } + if [ -n "${4:-}" ] && printf '%s\n' "$out" | grep -qF -- "$4"; then echo "FAIL $1: must not contain: $4"; fail=1; return; fi + [ "$(printf '%s\n' "$out" | wc -l)" -le 81 ] || { echo "FAIL $1: summary over 81 lines"; fail=1; return; } + echo "ok $(basename "$1"): $3" +} +check cargo-green.log 0 "exit 0 GREEN" "RED" +check cargo-green.log 0 "FRAMEWORK cargo passed=3 failed=0" +check cargo-red.log 101 "exit 101 RED" +check cargo-red.log 101 "FAILED tests::add_adds" +check cargo-red.log 101 "left: 5" +check cargo-zero-tests.log 0 "ZERO TESTS RAN" +NONCE=474d380d662e71a5 # a real run: the soul2 reproducer through the snapshot runner, 2026-10-01 +check mutation-table-nonce.log 1 "untested-equiv | SURVIVED" +check mutation-table-nonce.log 1 "stamp2-bad | CAUGHT | tests::d_cache" +check mutation-table-nonce.log 1 "add-minus | CAUGHT | tests::c_add" +check mutation-table-nonce.log 1 "caught=2 survived=1 unviable=0 timeout=0 noapply=0 error=0" "no test counts recognised" +NONCE= +check dotnet-red.log 1 "FRAMEWORK dotnet passed=11 failed=1 skipped=2" +check dotnet-red.log 1 "FAILED MyNs.AddTests.Subtracts" +check vitest-red.log 1 "FRAMEWORK vitest passed=10 failed=1 skipped=1" +check vitest-red.log 1 "FAILED src/add.test.ts > add > carries" +check pytest-red.log 1 "FRAMEWORK pytest passed=5 failed=1 skipped=1" +check pytest-red.log 1 "FAILED tests/test_add.py::test_carries" +check exit0-but-failed.log 0 "MISMATCH: exit 0 but the log shows" +check no-verdict.log 97 "NO VERDICT" +check no-verdict.log 97 "RED" "GREEN" + +# ---- tools/mutation-table.sh, host driver plus container half, against the mock in +# fixtures/mutation-mock: shell scripts that print cargo/vitest/jest-shaped output, run +# from the subdirectory pkg/ (a workdir below the repo root), under a mock docker +# (fixtures/mutation-mock/fakebin) where a container is a directory and `commit` copies +# it, so every mutant really runs on a copy of the baselined snapshot. +# One case per finding of the 2026-10-01 Soul passes; each fails if it regresses. +mt() { # mt [BUILD]: sets mt_out, mt_rc, mt_repo, mt_fd + local root; root=$(mktemp -d); mt_repo=$root/repo; mt_fd=$root/fd; mkdir -p "$mt_repo/pkg" "$root/mt" "$mt_fd" + cp -r "$here/fixtures/mutation-mock/." "$mt_repo/pkg/"; rm -rf "$mt_repo/pkg/fakebin"; cp "$L/jest-real.log" "$mt_repo/pkg/" + git -C "$mt_repo" init -q && git -C "$mt_repo" add -A && git -C "$mt_repo" -c user.name=t -c user.email=t@t commit -qm mock + cp "$here/testparse.sh" "$here/mutation-table.sh" "$root/mt/"; cp "$here/fixtures/mutation-mock/$1" "$root/mt/mutants.txt" + mt_out=$(PATH="$here/fixtures/mutation-mock/fakebin:$PATH" FAKE_DOCKER=$mt_fd YGG_WORK=$mt_repo YGG_IMAGE=base YGG_CNAME=job1 YGG_LABEL=eureka-verify-job=job1 \ + YGG_CPUS=1 YGG_MEM=1g YGG_KILLED=${MT_KILLED:-} MT_BUNDLE=$root/mt WORKDIR=pkg TEST=$2 BUILD=${3:-} MUT_TIMEOUT=${MT_TO:-15} \ + timeout "${MT_WALL:-120}" bash "$here/mutation-table.sh" --host 2>&1) + mt_rc=$? +} +mtcheck() { # mtcheck ... + local name=$1 s; shift + for s in "$@"; do + case $s in + '!'*) if printf '%s\n' "$mt_out" | grep -qF -- "${s#!}"; then echo "FAIL mutation $name: must not contain: ${s#!}"; fail=1; return; fi ;; + *) printf '%s\n' "$mt_out" | grep -qF -- "$s" || { echo "FAIL mutation $name: missing: $s"; printf '%s\n' "$mt_out" | tail -n 12; fail=1; return; } ;; + esac + done + echo "ok mutation $name" +} +# nothing the job made may outlive it: no container directory, no snapshot image, no bundle +mtclean() { # mtclean [images]: a killed mock container is a still-running process, so E checks only the image and bundle + local left dirs=("$mt_fd/c" "$mt_fd/i"); [ -z "${2:-}" ] || dirs=("$mt_fd/i") + left=$(ls "${dirs[@]}" 2>/dev/null | grep -v '^$' | grep -v ':$' | paste -sd, -) + if [ -z "$left" ] && [ ! -e "$(dirname "$mt_repo")/mt" ]; then echo "ok mutation $1: containers, snapshot image and bundle removed" + else echo "FAIL mutation $1: left behind: ${left:-the bundle dir}"; fail=1; fi +} +# A: a mistyped BUILD must abort, not read as a fully killed suite +mt mutants-a.txt 'bash test.sh' 'bash bulid.sh' +[ "$mt_rc" = 2 ] || { echo "FAIL mutation A: exit $mt_rc, want 2"; fail=1; } +mtcheck A-build-baseline "BASELINE RED: BUILD failed" "!UNVIABLE" "!SNAPSHOT" +mtclean A +# B: a label that spells a kill must not forge one +mt mutants-b.txt 'bash test.sh' +[ "$mt_rc" = 1 ] || { echo "FAIL mutation B: exit $mt_rc, want 1"; fail=1; } +mtcheck B-label-forge "caught=0 survived=1" 'forge \| CAUGHT \| label | SURVIVED |' +mtclean B +# the nonce form ygg-verify.sh carries, and the plain form when no nonce is set +YGG_NONCE=n0nce1 mt mutants-b.txt 'bash test.sh' +mtcheck B-nonce "YGG-SUMMARY/n0nce1: [1/1] " "!YGG-SUMMARY: " +# C: a test that rewrites a tracked file, leaks an untracked one or keeps state in an +# IGNORED file (cache.txt: what `git status` cannot see) must not decide the next mutant +mt mutants-c.txt 'bash test.sh' +mtcheck C-pollution "stamp-bad | CAUGHT | tests::b_write" "untested-equiv | SURVIVED" "!untested-equiv | CAUGHT" "caught=1 survived=1" +[ -z "$(git -C "$mt_repo" status --porcelain --ignored)" ] || { echo "FAIL mutation C: the seed repository changed"; fail=1; } +mtclean C +# D: a test file that fails to load is not a kill; a named failing test is +mt mutants-d.txt 'bash vtest.sh' +mtcheck D-import-failure "syntax-break | UNVIABLE" "wrong-result | CAUGHT | add.js > adds" "caught=1 survived=0 unviable=1" +# D2 (jest, real output): "FAIL " with "Test suite failed to run" is a load failure; the "●" block names the +# test; a test run that calls process.exit prints a "●" line under no FAIL header, which names nothing (ERROR, not CAUGHT) +mt mutants-h.txt 'bash jtest.sh' +mtcheck D2-jest "jest-load | UNVIABLE" "jest-wrong | CAUGHT | add › adds" "jest-exit | ERROR | exit 1, no failing test parsed" "!process.exit called | CAUGHT" "caught=1 survived=0 unviable=1 timeout=0 noapply=0 error=1" +# E: rows print as mutants finish, so a job timeout keeps them; the TERM the timeout sends still cleans up +MT_WALL=8 mt mutants-e.txt 'bash test.sh' +mtcheck E-live-rows "[1/2] quick | CAUGHT" "!MUTATION" +sleep 1; mtclean E images +# E2: a watchdog kill (the flag the stopgap sets) stops the fan-out after the baseline and cleans up +mt_k=$(mktemp); MT_KILLED=$mt_k mt mutants-c.txt 'bash test.sh'; rm -f "$mt_k" +[ "$mt_rc" = 124 ] || { echo "FAIL mutation E2: exit $mt_rc, want 124"; fail=1; } +mtcheck E2-killed "JOB KILLED by the watchdog before mutant 1 of 2" "!CAUGHT" +mtclean E2 +# E3 (Soul, F1): a test that cannot run twice is green in the baseline and red in every mutant's +# container; the control run of the unmutated snapshot must abort the job, not report a killed suite +mt mutants-i.txt 'bash oncetest.sh' +[ "$mt_rc" = 2 ] || { echo "FAIL mutation E3: exit $mt_rc, want 2"; fail=1; } +mtcheck E3-snapshot-red "SNAPSHOT RED (exit 101) for: bash oncetest.sh" "no rows counted" "!CAUGHT" "!MUTATION" +mtclean E3 +# F: LF before-text matches a CRLF file +mt mutants-f.txt 'bash test.sh' +mtcheck F-crlf "crlf-multiline | CAUGHT | tests::d_win" +# a real BUILD failure on a mutant is still UNVIABLE +mt mutants-g.txt 'bash test.sh' 'bash build.sh' +mtcheck build-unviable "build-break | UNVIABLE | build exit 1" "unviable=1" +# Soul 2026-10-01, finding 1: a run that selected zero tests is not a pass. +check "$(seg soul-s5/pdotnet.log zero)" 0 "ZERO TESTS RAN" "no test counts recognised" +check "$(seg soul-s5/ppy.log zero)" 0 "ZERO TESTS RAN" "no test counts recognised" +check "$(seg soul-s5/ppy.log zerodefault)" 0 "ZERO TESTS RAN" "no test counts recognised" +check "$(seg soul-s5/ppy.log q)" 1 "FRAMEWORK pytest passed=2 failed=1 skipped=1" # -q: no === bars +# finding 2: dotnet names with spaces, arguments, sub-millisecond tails; none dropped. +d=$(seg soul-s5/pdotnet.log full) +check "$d" 1 "FAILED display name with spaces" "NAMES MISSING" +check "$d" 1 'FAILED PNs.T.TheoryFails(x: 1, s: "a b")' +check "$d" 1 "FAILED PNs.T.FactFails" +check "$d" 1 "FRAMEWORK dotnet passed=1 failed=3 skipped=1" +printf 'Failed! - Failed: 2, Passed: 0, Skipped: 0, Total: 2, Duration: 1 ms - X.dll (net10.0)\n Failed A.One [1 ms]\n' >"$T/missing.log" +check "$T/missing.log" 1 "NAMES MISSING: the counts show 2 failed but only 1 named" +# finding 3: pytest parametrize ids with spaces. +out=$(bash "$V" summarize "$(seg soul-s5/ppy.log default)" 1) +if printf "%s +" "$out" | grep -qxF "FAILED test_p.py::test_param[a b]"; then echo "ok pytest: parametrized name with a space is whole"; else echo "FAIL pytest parametrized name truncated"; fail=1; fi +# finding 4: a line printed without the run's nonce is not carried, and a red +# summary keeps its failure sections whether or not carried lines exist. +check soul-s5/prust.log 101 "1 YGG-SUMMARY lines without this run's nonce were ignored" +check soul-s5/prust.log 101 "FAILED noisy_fail" "forged | CAUGHT" +NONCE=feedbeef +{ cat "$L/cargo-red.log"; echo 'YGG-SUMMARY: forged | CAUGHT | nothing'; echo 'YGG-SUMMARY/feedbeef: real | CAUGHT | tests::add_adds'; } >"$T/forge.log" +check "$T/forge.log" 101 "real | CAUGHT | tests::add_adds" "forged | CAUGHT" +check "$T/forge.log" 101 "left: 5" +check "$T/forge.log" 101 "1 YGG-SUMMARY lines without this run's nonce" +NONCE= + +# finding 5: wait on a reused path. Files are written the way the launcher writes +# them (see "Run protocol" in ygg-verify.sh): .run is the commit point, .done is +# " ". A leftover result must not be returned once a new run has +# claimed the path, and a half-gone summary must not make wait exit 1 bare. +W=$T/wait.log +echo old >"$W.run"; echo "old 0" >"$W.done"; echo "OLD RESULT" >"$W.summary" +( sleep 1; echo $$ >"$W.pid"; echo new >"$W.run"; rm -f "$W.done" "$W.summary" + sleep 3; echo "NEW RESULT" >"$W.summary"; echo "new 7" >"$W.done" ) & +bg=$! +out=$(SETTLE=2 WAIT_MAX=30 bash "$V" wait "$W"); st=$? +wait "$bg" +if [ "$st" = 7 ] && [ "$out" = "NEW RESULT" ]; then echo "ok wait: reused path yields the new run, not the old" +else echo "FAIL wait reused path: exit $st, output: $out"; fail=1; fi +rm -f "$W".* +echo r1 >"$W.run"; echo "r1 3" >"$W.done" # done, summary file gone +out=$(WAIT_MAX=10 SETTLE=0 bash "$V" wait "$W"); st=$? +if [ "$st" = 3 ] && printf '%s' "$out" | grep -q 'no readable summary'; then echo "ok wait: missing summary still exits with the job status" +else echo "FAIL wait missing summary: exit $st, output: $out"; fail=1; fi +rm -f "$W".* +echo r1 >"$W.run"; echo "r0 3" >"$W.done"; echo $$ >"$W.pid" # done from another run, launcher alive +out=$(WAIT_MAX=3 SETTLE=0 bash "$V" wait "$W"); st=$? +if [ "$st" = 99 ]; then echo "ok wait: a done from another run is not a result" +else echo "FAIL wait foreign done: exit $st, output: $out"; fail=1; fi +# N2 (Soul, 2026-10-01): file-level failures are not named test failures, whichever runner +# prints them. Vitest with a project prefix and with spaces in the path; jest's file header. +check vitest-project-load-failure.log 1 "FAILED TO LOAD |unit| src/add.test.ts" "FAILED |unit|" +check vitest-spaces-load-failure.log 1 "FAILED TO LOAD src/my tests/add.test.ts" "FAILED src/my" +check vitest-project-red.log 1 "FAILED |unit| src/add.test.ts > add > carries" "FAILED TO LOAD" +# jest, real output (jest-real.log): one space after FAIL, no TTY. Names come from the "●" lines under a FAIL header only. +check "$(seg jest-real.log green)" 0 "FRAMEWORK jest passed=2 failed=0 skipped=0" "no test counts recognised" +check "$(seg jest-real.log wrong)" 1 "FRAMEWORK jest passed=1 failed=1 skipped=0" +check "$(seg jest-real.log wrong)" 1 "FAILED add › adds" "FAILED TO LOAD" +check "$(seg jest-real.log load)" 1 "FAILED TO LOAD jt/add.test.js" "FAILED jt/add.test.js" +check "$(seg jest-real.log load)" 1 "FRAMEWORK jest passed=0 failed=0 skipped=0" "FAILED Test suite" +# F3: process.exit(1) in a test run prints a "●" line under no FAIL header; it is a crash, not a named kill +check "$(seg jest-real.log exit)" 1 "exit non-zero with no failing test parsed" "FAILED process.exit" +# F4: the "nothing recognised" line names every framework the parser reads +check "$(seg jest-real.log exit)" 1 "no test counts recognised in the log (cargo, dotnet, vitest, jest, pytest)" +exit $fail diff --git a/tools/testparse.sh b/tools/testparse.sh new file mode 100644 index 0000000..4bae433 --- /dev/null +++ b/tools/testparse.sh @@ -0,0 +1,131 @@ +#!/usr/bin/env bash +# Sourced, not run. Reads a test-run log and says what it contains. +# +# tp_scan one line per fact: +# FRAMEWORK passed= failed= skipped= (summed over the log) +# FAILED (one per failing test) +# LOADFAIL (a test FILE that failed, with no test name: +# a module that did not load; jest also prints one for a file with failing tests) +# tp_sections [max-lines] cargo failure sections ("---- name stdout +# ----"), each cut to its last lines (default 30) +# +# Frameworks: cargo test, dotnet test, vitest, jest, pytest. Anything else is +# simply not recognised; the callers treat "recognised nothing" as a fact to +# print, never as success. +# +# Owner: tools/stopgap/ygg-verify.sh (summary mode) and tools/mutation-table.sh +# (kill attribution). One parser, so a summary and a kill table cannot disagree +# about what failed. Written for mawk: no match() groups, no gensub. + +tp_scan() { + awk ' + function num(s, a) { split(s, a, " "); return a[1] + 0 } + function pyname(s, p, sp, d, i, c) { # pytest short-summary name: a [param id] may hold spaces + p = index(s, "["); sp = index(s, " ") + if (p == 0 || (sp > 0 && sp < p)) return sp ? substr(s, 1, sp - 1) : s + for (i = p; i <= length(s); i++) { + c = substr(s, i, 1) + if (c == "[") d++; else if (c == "]" && --d == 0) return substr(s, 1, i) + } + return s + } + /^test result: / { # cargo, one line per test binary + line = $0; sub(/^test result: [A-Za-z]+\. /, "", line) + n = split(line, parts, "; ") + for (i = 1; i <= n; i++) { + v = num(parts[i]) + if (parts[i] ~ /passed/) cp += v + if (parts[i] ~ /failed/) cf += v + if (parts[i] ~ /ignored/) cs += v + } + cargo = 1; next + } + /^test .* \.\.\. FAILED$/ { name = $0; sub(/^test /, "", name); sub(/ \.\.\. FAILED$/, "", name); print "FAILED " name; next } + /^(Passed|Failed)! - Failed: *[0-9]+, Passed: *[0-9]+, Skipped: *[0-9]+, Total:/ { # dotnet + line = $0; sub(/^[A-Za-z]+! - /, "", line) + n = split(line, parts, ", ") + for (i = 1; i <= n; i++) { + v = parts[i]; sub(/^[A-Za-z]+: */, "", v); v += 0 + if (parts[i] ~ /^Failed:/) df += v + if (parts[i] ~ /^Passed:/) dp += v + if (parts[i] ~ /^Skipped:/) ds += v + } + dotnet = 1; next + } + /^No test (matches the given testcase filter|is available in)/ { dotnet = 1; next } # dotnet, zero tests + /^[ \t]+Failed .+ \[[<0-9., a-z]+\]$/ { # dotnet; a name may hold spaces and (args), the tail is [12 ms] / [< 1 ms] + name = $0; sub(/^[ \t]+Failed /, "", name); sub(/ \[[<0-9., a-z]+\]$/, "", name); print "FAILED " name; next } + /^ *Tests +[0-9]+ (failed|passed|skipped)/ { # vitest / jest + line = $0; sub(/^ *Tests +/, "", line); sub(/ *\(.*$/, "", line) + n = split(line, parts, " \\| ") + for (i = 1; i <= n; i++) { + v = num(parts[i]) + if (parts[i] ~ /failed/) vf += v + if (parts[i] ~ /passed/) vp += v + if (parts[i] ~ /skipped/) vs += v + } + vitest = 1; next + } + /^ *Tests: +[0-9]+ / { # jest: "Tests: 1 failed, 2 passed, 3 total" + line = $0; sub(/^ *Tests: +/, "", line) + n = split(line, parts, ", ") + for (i = 1; i <= n; i++) { + v = num(parts[i]) + if (parts[i] ~ /failed/) jf += v + if (parts[i] ~ /passed/) jp += v + if (parts[i] ~ /skipped/) js += v + } + jest = 1; next + } + # vitest and jest print "FAIL " for a file and " FAIL > > " (vitest, + # optionally with a "|project|" prefix) for a test. Vitest and a jest TTY pad the word with two spaces; + # jest without a TTY, which is how every job runs it, prints ONE ("FAIL jt/add.test.js"). A line with + # no " > " names a FILE: a vitest load failure (" [ ]"), or a jest file header whose + # tests are named by the "●" lines below it. + /^ *FAIL +/ { infail = 1; name = $0; sub(/^ *FAIL +/, "", name); print (name ~ / > / ? "FAILED " : "LOADFAIL ") name; next } + /^ *PASS +/ || /^Test Suites:/ { infail = 0 } + # jest failure block header " ● suite › test". It names a test only under a FAIL file header: a "●" + # elsewhere is output of the run itself (process.exit prints "● process.exit called with ..." with no + # FAIL header at all), and a crash that names no test stays unnamed, so callers read it as ERROR. + /^ +● / { + name = $0; sub(/^ +● +/, "", name) + if (infail && name !~ /^(Test suite failed to run|Console|Validation Warning|Deprecation)/) print "FAILED " name + next + } + /^(=+ )?(no tests ran|[0-9]+ [a-z]+(, [0-9]+ [a-z]+)*) in [0-9.]+s( \([0-9:]+\))?( =+)?$/ { # pytest summary, with or without -q + line = $0; sub(/^=+ /, "", line); sub(/ =+$/, "", line); sub(/ in [0-9.]+s.*$/, "", line) + n = split(line, parts, ", "); ok = 1; a = b = c = 0 + for (i = 1; i <= n; i++) { + split(parts[i], w, " "); v = w[1] + 0 + if (w[2] == "failed" || w[2] == "error" || w[2] == "errors") b += v + else if (w[2] == "passed" || w[2] == "xpassed") a += v + else if (w[2] == "skipped" || w[2] == "deselected" || w[2] == "xfailed") c += v + else if (w[2] != "warning" && w[2] != "warnings") ok = 0 + } + if (line == "no tests ran") ok = 1 + if (ok) { pp += a; pf += b; ps += c; pytest = 1 } + next + } + /^FAILED [^ ]+::/ { print "FAILED " pyname(substr($0, 8)); next } + END { + if (cargo) printf "FRAMEWORK cargo passed=%d failed=%d skipped=%d\n", cp, cf, cs + if (dotnet) printf "FRAMEWORK dotnet passed=%d failed=%d skipped=%d\n", dp, df, ds + if (vitest) printf "FRAMEWORK vitest passed=%d failed=%d skipped=%d\n", vp, vf, vs + if (jest) printf "FRAMEWORK jest passed=%d failed=%d skipped=%d\n", jp, jf, js + if (pytest) printf "FRAMEWORK pytest passed=%d failed=%d skipped=%d\n", pp, pf, ps + }' "$1" +} + +tp_sections() { + awk -v max="${2:-30}" ' + function flush( i, start) { + if (!n) return + start = n > max ? n - max + 1 : 1 + for (i = start; i <= n; i++) print buf[i] + n = 0 + } + /^---- .* stdout ----$/ { flush(); n = 0; buf[++n] = $0; inside = 1; next } + inside && (/^failures:$/ || /^test result: /) { flush(); inside = 0; next } + inside { buf[++n] = $0 } + END { flush() }' "$1" +}