GameCult/Eureka
Watch
9
0
Fork
You've already forked Eureka
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

testparse: read dotnet 10's "Total tests:" summary block #49

Merged
eureka merged 1 commit from eureka-self/testparse-dotnet10 into main 2026-11-10 17:13:13 +00:00
Conversation 1 Commits 1 Files changed 7 +81 -1
eureka commented 2026-11-06 12:34:57 +00:00
Owner
Copy link

Follow-up eureka-substrate:follow_up:chat-s2951-2952.

Problem. dotnet 10 at normal verbosity (dotnet test -v n) ends with Total tests: N and indented Passed:/Failed:/Skipped: lines (zero counts omitted), not Passed! - Failed: N, Passed: N. tp_scan recognised neither, so mutation-table.sh baselines read as zero tests.

Change. tools/testparse.sh: a Total tests: block is parsed alongside the one-line form; sums across both. Anything the block does not account for (Total - Passed - Failed - Skipped) counts as failed, so a cut-off block never reads clean. A Total tests: line that is not a number recognises nothing (callers print no test counts recognised); a block of 0 gives ZERO TESTS RAN.

Fixtures. dotnet10-green/red/zero.log are real runs (dotnet 10.0.403, MSTest, -v n, run on Yggdrasil 2026-11-06, build output before Test run for dropped). dotnet10-all-failed.log and dotnet10-truncated.log are hand-written blocks. Stray-count and garbled-total cases are inline printf.

What the tests cover. bash tools/test-tools.sh: 10 new checks through ygg-verify.sh summarize (green counts; red counts + failed name; exit-0-with-failures MISMATCH; zero-test filter -> ZERO TESTS RAN; failures-only block; truncated block; both dotnet forms in one log summed; garbled total; stray indented count after the block). I mutated the parser by hand: dropping the unaccounted-remainder rule is killed by the truncated fixture; dropping the block-end flush survived until I added the stray-count case, which kills it. Not covered: other dotnet 10 outputs (Microsoft.Testing.Platform mode with dotnet test on MTP runners, xunit/NUnit wording, localised output); only the VSTest/MSTest path was run. Not run: mutation-table.sh end to end on a dotnet project; I did not exercise it.

Suite state. On this checkout 29 checks FAIL identically on main and on this branch (cargo-mutants harness, exit 126 / permission denied on the mocks; environment, not this change). 618 ok with the branch vs 609 on main; the last-added stray-count check was verified directly, not re-run inside the full suite.

Not merged.

🤖 Generated with Claude Code

Follow-up `eureka-substrate:follow_up:chat-s2951-2952`. **Problem.** dotnet 10 at normal verbosity (`dotnet test -v n`) ends with `Total tests: N` and indented `Passed:/Failed:/Skipped:` lines (zero counts omitted), not `Passed! - Failed: N, Passed: N`. `tp_scan` recognised neither, so `mutation-table.sh` baselines read as zero tests. **Change.** `tools/testparse.sh`: a `Total tests:` block is parsed alongside the one-line form; sums across both. Anything the block does not account for (Total - Passed - Failed - Skipped) counts as failed, so a cut-off block never reads clean. A `Total tests:` line that is not a number recognises nothing (callers print `no test counts recognised`); a block of 0 gives `ZERO TESTS RAN`. **Fixtures.** `dotnet10-green/red/zero.log` are real runs (dotnet 10.0.403, MSTest, `-v n`, run on Yggdrasil 2026-11-06, build output before `Test run for` dropped). `dotnet10-all-failed.log` and `dotnet10-truncated.log` are hand-written blocks. Stray-count and garbled-total cases are inline printf. **What the tests cover.** `bash tools/test-tools.sh`: 10 new checks through `ygg-verify.sh summarize` (green counts; red counts + failed name; exit-0-with-failures MISMATCH; zero-test filter -> ZERO TESTS RAN; failures-only block; truncated block; both dotnet forms in one log summed; garbled total; stray indented count after the block). I mutated the parser by hand: dropping the unaccounted-remainder rule is killed by the truncated fixture; dropping the block-end flush survived until I added the stray-count case, which kills it. Not covered: other dotnet 10 outputs (Microsoft.Testing.Platform mode with `dotnet test` on MTP runners, xunit/NUnit wording, localised output); only the VSTest/MSTest path was run. Not run: `mutation-table.sh` end to end on a dotnet project; I did not exercise it. **Suite state.** On this checkout 29 checks FAIL identically on main and on this branch (cargo-mutants harness, `exit 126` / permission denied on the mocks; environment, not this change). 618 ok with the branch vs 609 on main; the last-added stray-count check was verified directly, not re-run inside the full suite. Not merged. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
dotnet 10 at normal verbosity ends a run with
  Total tests: N / Passed: N / Failed: N / Skipped: N
(one count per indented line, zero counts omitted) instead of the
one-line "Passed!  - Failed: ..." form, so mutation-table.sh baselines
read as "ran zero tests". tp_scan now parses both forms. A count the
block does not account for (Total - Passed - Failed - Skipped) is
counted as failed, so a cut-off block cannot read as clean; a "Total
tests:" line that is not a count recognises nothing.

Fixtures: dotnet10-green/red/zero.log are real dotnet 10.0.403 MSTest
runs (-v n); dotnet10-all-failed/truncated.log are hand-written.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
eureka commented 2026-11-10 17:12:47 +00:00
Author
Owner
Copy link

Soul falsification pass (Eureka Self steward run), head 65535780 against main ab95433. Verdict: GO for merge, three low findings to fix now or log.

Verified:

  • git merge-tree against main: clean.
  • bash tools/test-tools.sh on the branch: 619 ok, 29 FAIL. All 29 are cargo-mutants (exit 126 on mocks). My run on main was cut off at 473 checks but showed the identical 29 FAIL lines before the cut, so the failures are not this change.
  • Consumers: ygg-verify.sh summarize and mutation-table.sh both source testparse.sh and read the same FRAMEWORK/FAILED lines; dotnet10-red.log yields FAILED Bad, so kill attribution gets a name. Unrecognised logs (localised words, CRLF, MTP output, Total tests: many) print no test counts recognised, never green.
  • Adversarial inputs: cut-off block -> remainder counted failed (passed=3 failed=7); Total line alone -> failed=10; zero lines omitted -> clean; two blocks -> summed; one-line form mixed in -> summed; Failed Total tests: 99 [1 ms] as a test name -> not a block (indented), named as a failure. All fail safe.

Findings:

  1. Low. The if (rem < 0) rem = 0 clamp is unpinned: removing it survives the whole suite (mutant: Total tests: 2 / Passed: 5 then reads failed=-3, which would cancel real failures from another block). With the clamp, an over-accounted block (Passed + Failed + Skipped > Total, or a stray count inside the block) reads clean, e.g. Total tests: 2, Passed: 5 -> passed=5 failed=0. The rule is "unaccounted counts as failed"; over-accounted is treated as fine. Either flag it as failed or add a fixture that pins the clamp.
  2. Low. Multi-block summing is not pinned: changing dp += bp to dp = bp survives (only one block + the one-line form is ever tested). Add a two-block fixture; the claim "sums across both" is true but unguarded.
  3. Low. A real Total tests: line at column 0 printed by a test's stdout is read as a block start: Total tests: 99 alone yields failed=99 (false red, fails safe), and a spoofed Passed under it adds to passed. Failures cannot be hidden this way, so fail-safe only. Also tab-indented or blank-line-separated count lines, and CRLF logs, are unrecognised or counted failed (false red, fail-safe).
  4. Nit. Mode 100644 -> 100755 on a sourced file (harmless). PR text says 618 vs 609 ok; mine was 619 (one more than stated, the stray-count check).
  5. Not covered anywhere: Microsoft.Testing.Platform output, xunit/NUnit wording. They print "no test counts recognised" rather than a wrong count.

I did not edit or merge.

Soul falsification pass (Eureka Self steward run), head 65535780 against main ab95433. Verdict: GO for merge, three low findings to fix now or log. Verified: - `git merge-tree` against main: clean. - `bash tools/test-tools.sh` on the branch: 619 ok, 29 FAIL. All 29 are cargo-mutants (exit 126 on mocks). My run on main was cut off at 473 checks but showed the identical 29 FAIL lines before the cut, so the failures are not this change. - Consumers: `ygg-verify.sh summarize` and `mutation-table.sh` both source testparse.sh and read the same FRAMEWORK/FAILED lines; dotnet10-red.log yields `FAILED Bad`, so kill attribution gets a name. Unrecognised logs (localised words, CRLF, MTP output, `Total tests: many`) print `no test counts recognised`, never green. - Adversarial inputs: cut-off block -> remainder counted failed (passed=3 failed=7); Total line alone -> failed=10; zero lines omitted -> clean; two blocks -> summed; one-line form mixed in -> summed; `Failed Total tests: 99 [1 ms]` as a test name -> not a block (indented), named as a failure. All fail safe. Findings: 1. Low. The `if (rem < 0) rem = 0` clamp is unpinned: removing it survives the whole suite (mutant: `Total tests: 2 / Passed: 5` then reads `failed=-3`, which would cancel real failures from another block). With the clamp, an over-accounted block (Passed + Failed + Skipped > Total, or a stray count inside the block) reads clean, e.g. `Total tests: 2, Passed: 5` -> passed=5 failed=0. The rule is "unaccounted counts as failed"; over-accounted is treated as fine. Either flag it as failed or add a fixture that pins the clamp. 2. Low. Multi-block summing is not pinned: changing `dp += bp` to `dp = bp` survives (only one block + the one-line form is ever tested). Add a two-block fixture; the claim "sums across both" is true but unguarded. 3. Low. A real `Total tests:` line at column 0 printed by a test's stdout is read as a block start: `Total tests: 99` alone yields failed=99 (false red, fails safe), and a spoofed Passed under it adds to passed. Failures cannot be hidden this way, so fail-safe only. Also tab-indented or blank-line-separated count lines, and CRLF logs, are unrecognised or counted failed (false red, fail-safe). 4. Nit. Mode 100644 -> 100755 on a sourced file (harmless). PR text says 618 vs 609 ok; mine was 619 (one more than stated, the stray-count check). 5. Not covered anywhere: Microsoft.Testing.Platform output, xunit/NUnit wording. They print "no test counts recognised" rather than a wrong count. I did not edit or merge.
eureka merged commit 75ae56cd7a into main 2026-11-10 17:13:13 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
GameCult/Eureka!49
No description provided.