Mutation Testing

CultLib measures its C# test suites with Stryker.NET, pinned as a repo-local tool in dotnet-tools.json. Coverage is added project by project; a project with no stryker-config.json next to its test project has not been scoped yet (see "Scope and follow-ups" below for what's outstanding).

Running it

Restore the tool once per checkout:

dotnet tool restore

Run a full mutation pass from a covered test project's directory, for example CultMath:

cd packages/cultmath/tests/CultMath.Tests
dotnet stryker -t mtp

-t mtp selects Stryker's Microsoft.Testing.Platform runner. It is required for test projects on xunit.v3 (CultMath.Tests is one): Stryker's default vstest runner's coverage capture does not work against xunit.v3's native test host and the run stalls retrying test sessions indefinitely. Projects still on xunit v2 with xunit.runner.visualstudio can use the default vstest runner.

For a single cut, scope the run to what changed instead of the whole project:

dotnet stryker -t mtp --since:<base-commit-or-branch>

Use a full commit SHA for --since, not an abbreviation: git rev-parse <short-sha> first, then pass the full result. Stryker resolves --since through the same git plumbing a short SHA can resolve ambiguously (or fail to resolve at all) depending on the checkout's object database, and a misresolved base silently changes what "since" means instead of erroring.

In a container, also set git config --global --add safe.directory '*' before running Stryker (or any other command that shells out to git against this checkout) - git refuses to operate on a repository owned by a different user by default, which is the common case for a bind-mounted checkout in a container.

Reports land in StrykerOutput/ next to the test project (git-ignored). Open the reports/mutation-report.html file for a browsable view, or read reports/mutation-report.json for a scriptable one.

Triage rules

The mutation score is not a gate (break threshold is 0). Every surviving mutant is a finding about the tests, not a build failure, and gets triaged into one of:

missing behavioural test — name the test that should exist and what branch, edge case, or component it would exercise.
degenerate fixture — the code path is exercised, but the fixture's values make two different mutants produce the same observable result (a zero component that erases a +/- distinction, an already-normalized input that erases a *// distinction, all-equal or all-same-sign inputs that erase a boundary or &&/|| distinction). Fix the fixture, don't invent a new one that only kills that one mutant.
equivalent — the mutant cannot change behaviour. Float-threshold boundary flips (> to >=) are equivalent by default. A reasoned equivalence claim (for example, >> and >>> on an already-unsigned operand) needs its one-line reason recorded, not assumed.

Do not write a hand-rolled mutation suite or anchor test as a substitute. Where the tool can't reach (native/shader/editor code), the defence is a behavioural test at the layer the rule is decided, not a committed mutation harness.

Known tool caveat

Stryker 5.0.0's mtp runner is documented as preview. Its per-test coverage capture was found to report false "Survived" verdicts specifically for mutations inside static readonly field initializers that a test does cover (confirmed by hand-mutating bool3.@false and bool4.@true and running the full suite directly with dotnet test, both of which fail against the mutation that Stryker reported as surviving). Treat a survivor inside a static readonly field initializer as unproven until confirmed with a direct dotnet test run against the same mutation; do not triage it as a missing test without that check.

A survivor that looks like it must be covered is unproven until confirmed with a direct dotnet test run against the same hand-applied mutation - but confirm what the mutant actually is before drawing a tooling conclusion from it. An earlier pass here reported three survivors in CultGeometrySurfaceNets.TryGetOtherEndpoint's per-axis if (x + 1 >= sizeX) return false; guards and initially triaged them as a vstest coverage false-survivor (the same shape as the mtp/static readonly caveat above). That triage was wrong: the actual mutant Stryker reports there is return false -> return true inside the guard, which is equivalent, not a coverage miss. Returning true from TryGetOtherEndpoint still leaves the out other coordinate equal to the current coordinate (see the method: other is cloned from coord and only one component is incremented when the method returns true), so the caller compares a sample against itself, the inside/outside flags always match, and the crossing is skipped exactly as it would have been had the method correctly returned false. No test can distinguish the two branches because there is nothing to distinguish - it is a legitimate equivalent mutant, not evidence of a vstest false survivor.

CultGeometrySurfaceNets.Extract's if (inside == (OrientationSign[axis] < 0)) (the sole authority for quad winding after the geometric cross-product fallback was deleted) has three mutants on that line: == -> !=, and < 0 negated to > 0 or widened to <= 0. The first two, inside != (OrientationSign[axis] < 0) and OrientationSign[axis] > 0, are both reported Killed. The third, OrientationSign[axis] <= 0, survives, but it is an equivalent mutant, not a coverage gap: OrientationSign is the fixed table { 1, -1, 1 } and never holds 0, so < 0 and <= 0 agree on every value the table can actually produce. There is nothing for a test to distinguish, the same shape as the TryGetOtherEndpoint equivalent mutant above.

No vstest coverage-based false survivor has actually been confirmed for GameCult.Geometry. Every survivor triaged here so far has turned out to be a genuine equivalent mutant once hand-mutated and checked directly against dotnet test; treat any survivor that looks like it must obviously be covered as unproven until confirmed the same way, but do not cite this project as precedent for the vstest caveat itself - that caveat is mtp-runner-specific and documented above for static readonly field initializers, and GameCult.Geometry has never actually hit it.

Scope and follow-ups

CultMath.Tests, GameCult.Geometry.Tests and GameCult.Caching.Tests are the projects currently scoped. GameCult.Geometry's stryker-config.json has no test-runner entry: the project is NUnit 4 with NUnit3TestAdapter and Microsoft.NET.Test.Sdk, so Stryker's default vstest runner applies directly (no mtp workaround needed, unlike CultMath.Tests's xunit.v3 suite). GameCult.Caching.Tests is the same shape (NUnit, default vstest) and its stryker-config.json is driven the same way, by the repo-local dotnet-stryker tool: dotnet tool restore once, then dotnet tool run dotnet-stryker from tests/GameCult.Caching.Tests.

GameCult.Geometry has only ever been run --since:-scoped to the cut that added surface nets; a full-project baseline (running mutation against all of CultGeometryIsoSurface, CultGeometryDocuments, and CultGeometryPrimitives, not just the diff) is a recorded follow-up, not something either cut did.

That first --since pass did not actually stay diff-scoped: adding stryker-config.json itself in the same cut that first used it made Stryker log "391 mutants will be tested because: Non-CSharp files in test project were changed" and mutate the whole mutate glob (all four existing GameCult.Geometry source files), not just the new surface-nets files. A cut that changes a non-C# file inside the test project (a fixture, this config file, anything under Fixtures/) loses --since line-level scoping for that run; a future cut whose diff is C#-only, against a base commit that already has stryker-config.json, should scope correctly.

GameCult.Caching.Tests

tests/GameCult.Caching.Tests/stryker-config.json mutates all of src/GameCult.Caching (the project is GameCult.Caching.csproj; Standard mutation level, break 0, the same thresholds as Geometry). Caching was scoped before Networking because its suite is deterministic and quick (1415 tests, about 40 s) over about 7.8k lines of source, where GameCult.Networking is about 20k lines and its tests lean on socket timing, so a mutant timing out there says little about the test. A full Caching run is about 2,900 candidate mutants (3,602 created, minus 730 that fail to compile) at 1.25 to 1.6 s each, so about 1 to 1.3 hours; run it detached and read the JSON report, never as one attached session.

The first measurement was a narrow pass over the three files that carry the store-replace and element-id machinery, from tests/GameCult.Caching.Tests:

dotnet tool run dotnet-stryker \
  -m '**/src/GameCult.Caching/StoreReplace.cs' \
  -m '**/src/GameCult.Caching/CultStoreLinks.cs' \
  -m '**/src/GameCult.Caching/CultElementIds.cs'

A clean single run of that command tests 243 to 265 mutants (the count moves with the revision) and scored 85.28% on 568a60fa, before any test of this scope was added; 86.79% on d38b5728, with the first batch of tests; and 87.17% on 76d9e7d1, with the throwing-enumerator case, in about 10 minutes. The first measurement of it overlapped two Stryker runs, so its numbers (and the 114 survivors it showed for CultElementIds.cs, where tests that kill a mutant by hand reported Survived or Timeout) are not measurements and are not quoted here. Never run two Stryker runs against one checkout at once, and check ps before starting one. A clean single run on CultElementIds.cs alone, after the tests below, tested 228 mutants: 208 Killed, 8 Timeout, 12 Survived, a score of 94.74%. The 12th survivor was 321, which the throwing-enumerator case in ListsOfValuesHoldNoIds now kills; the other 11 are equivalent mutants. With that case a clean single run on CultElementIds.cs is 209 Killed, 8 Timeout, 11 Survived, 95.18%.

Real gaps that run exposed, each now pinned by ElementIdWalkTests (and each failing when its mutant is applied by hand): the order of the id path (the codec refuses at the first type MessagePack cannot allow, so order is what an error names first; Append to Prepend at 163, 167 and 180); the union-cycle guard in ConcreteTypes (248; the mutant is a stack overflow, so the red is a crashed test host: a depth guard would need production code in ConcreteTypes, since the recursion is private and a stack overflow cannot be caught by the test, so the regression test is left to crash loudly); a list of values reporting that it holds an id (Holds, 322); [Key] read through an override (KeyOf, 583); and the IsWrittenWhole and IsKeyed predicates (629 to 631 and 636: formatter, [MessagePackObject] and KeyAsPropertyName inherited from a base, and the [DataContract] without [MessagePackObject] and || arms).

Equivalent survivors in CultElementIds.cs (11), each hand-checked against the full suite; none can be told apart by any test:

142, 251, 636 (second arm), 506 (IsDefined(UnionAttribute, false) to true): UnionAttribute is Inherited = false, so its inherit argument changes nothing. Likewise 630's DataContractAttribute inherit argument (DataContractAttribute is Inherited = false).
264, 542: group.First() to FirstOrDefault() over GroupBy groups, which are never empty.
491: new StringBuilder(bytes.Length * 2) to / 2 changes only the initial capacity.
578: removing the Nullable.GetUnderlyingType unwrap in WalksAsOwner; Classify unwraps nullables itself, so the unwrap is dead. A cut candidate, not a test gap.
645, 657: Append(type) to Prepend(type) in ElementTypeOf and MapTypes; type matches only when it is itself IEnumerable<>, IDictionary<,> or IReadOnlyDictionary<,>, and none of those has a second matching interface, so the position of type never decides the answer.

The Windows-only P/Invoke group is exercised only on Windows: its mutants are NoCoverage on Linux and are a target for a Windows-leg mutation run, not a Linux gap or a finding. StoreReplace.cs line 45 is a different case: its negated IsOSPlatform(Windows) mutant is Survived on Linux, because the negated branch calls ReplaceWindows, whose DllNotFound fallback gives the same result.

GameCult.Mesh (+ .Quic, .Quic.Native), GameCult.Networking (+ .WebSockets), and the non-.NET runtimes (cultcache-rs/py/ts, cultmesh-*, cultnet-*) each still need their own stryker-config.json (or ecosystem-equivalent tool: cargo-mutants for Rust, StrykerJS for TypeScript, mutmut for Python) and a baseline pass. None of that is wired into CI yet.

13 KiB Raw Blame History