238 lines
13 KiB
Markdown
238 lines
13 KiB
Markdown
# Mutation Testing
|
|
|
|
CultLib measures its C# test suites with Stryker.NET, pinned as a repo-local
|
|
tool in `dotnet-tools.json`. Coverage is added project by project; a project
|
|
with no `stryker-config.json` next to its test project has not been scoped
|
|
yet (see "Scope and follow-ups" below for what's outstanding).
|
|
|
|
## Running it
|
|
|
|
Restore the tool once per checkout:
|
|
|
|
```powershell
|
|
dotnet tool restore
|
|
```
|
|
|
|
Run a full mutation pass from a covered test project's directory, for
|
|
example CultMath:
|
|
|
|
```powershell
|
|
cd packages/cultmath/tests/CultMath.Tests
|
|
dotnet stryker -t mtp
|
|
```
|
|
|
|
`-t mtp` selects Stryker's Microsoft.Testing.Platform runner. It is required
|
|
for test projects on `xunit.v3` (CultMath.Tests is one): Stryker's default
|
|
`vstest` runner's coverage capture does not work against xunit.v3's native
|
|
test host and the run stalls retrying test sessions indefinitely. Projects
|
|
still on `xunit` v2 with `xunit.runner.visualstudio` can use the default
|
|
`vstest` runner.
|
|
|
|
For a single cut, scope the run to what changed instead of the whole
|
|
project:
|
|
|
|
```powershell
|
|
dotnet stryker -t mtp --since:<base-commit-or-branch>
|
|
```
|
|
|
|
Use a full commit SHA for `--since`, not an abbreviation: `git rev-parse
|
|
<short-sha>` first, then pass the full result. Stryker resolves `--since`
|
|
through the same git plumbing a short SHA can resolve ambiguously (or fail to
|
|
resolve at all) depending on the checkout's object database, and a
|
|
misresolved base silently changes what "since" means instead of erroring.
|
|
|
|
In a container, also set `git config --global --add safe.directory '*'`
|
|
before running Stryker (or any other command that shells out to `git`
|
|
against this checkout) - git refuses to operate on a repository owned by a
|
|
different user by default, which is the common case for a bind-mounted
|
|
checkout in a container.
|
|
|
|
Reports land in `StrykerOutput/` next to the test project (git-ignored).
|
|
Open the `reports/mutation-report.html` file for a browsable view, or read
|
|
`reports/mutation-report.json` for a scriptable one.
|
|
|
|
## Triage rules
|
|
|
|
The mutation score is not a gate (`break` threshold is 0). Every surviving
|
|
mutant is a finding about the tests, not a build failure, and gets triaged
|
|
into one of:
|
|
|
|
- **missing behavioural test** — name the test that should exist and what
|
|
branch, edge case, or component it would exercise.
|
|
- **degenerate fixture** — the code path is exercised, but the fixture's
|
|
values make two different mutants produce the same observable result (a
|
|
zero component that erases a `+`/`-` distinction, an already-normalized
|
|
input that erases a `*`/`/` distinction, all-equal or all-same-sign inputs
|
|
that erase a boundary or `&&`/`||` distinction). Fix the fixture, don't
|
|
invent a new one that only kills that one mutant.
|
|
- **equivalent** — the mutant cannot change behaviour. Float-threshold
|
|
boundary flips (`>` to `>=`) are equivalent by default. A reasoned
|
|
equivalence claim (for example, `>>` and `>>>` on an already-unsigned
|
|
operand) needs its one-line reason recorded, not assumed.
|
|
|
|
Do not write a hand-rolled mutation suite or anchor test as a substitute.
|
|
Where the tool can't reach (native/shader/editor code), the defence is a
|
|
behavioural test at the layer the rule is decided, not a committed mutation
|
|
harness.
|
|
|
|
## Known tool caveat
|
|
|
|
Stryker 5.0.0's `mtp` runner is documented as preview. Its per-test coverage
|
|
capture was found to report false "Survived" verdicts specifically for
|
|
mutations inside `static readonly` field initializers that a test does
|
|
cover (confirmed by hand-mutating `bool3.@false` and `bool4.@true` and
|
|
running the full suite directly with `dotnet test`, both of which fail
|
|
against the mutation that Stryker reported as surviving). Treat a survivor
|
|
inside a `static readonly` field initializer as unproven until confirmed
|
|
with a direct `dotnet test` run against the same mutation; do not triage it
|
|
as a missing test without that check.
|
|
|
|
A survivor that looks like it must be covered is unproven until confirmed
|
|
with a direct `dotnet test` run against the same hand-applied mutation - but
|
|
confirm what the mutant actually is before drawing a tooling conclusion from
|
|
it. An earlier pass here reported three survivors in
|
|
`CultGeometrySurfaceNets.TryGetOtherEndpoint`'s per-axis `if (x + 1 >=
|
|
sizeX) return false;` guards and initially triaged them as a `vstest`
|
|
coverage false-survivor (the same shape as the `mtp`/`static readonly`
|
|
caveat above). That triage was wrong: the actual mutant Stryker reports
|
|
there is `return false` -> `return true` inside the guard, which is
|
|
equivalent, not a coverage miss. Returning `true` from
|
|
`TryGetOtherEndpoint` still leaves the out `other` coordinate equal to the
|
|
current coordinate (see the method: `other` is cloned from `coord` and only
|
|
one component is incremented when the method returns `true`), so the caller
|
|
compares a sample against itself, the inside/outside flags always match,
|
|
and the crossing is skipped exactly as it would have been had the method
|
|
correctly returned `false`. No test can distinguish the two branches because
|
|
there is nothing to distinguish - it is a legitimate equivalent mutant, not
|
|
evidence of a `vstest` false survivor.
|
|
|
|
`CultGeometrySurfaceNets.Extract`'s `if (inside == (OrientationSign[axis] <
|
|
0))` (the sole authority for quad winding after the geometric cross-product
|
|
fallback was deleted) has three mutants on that line: `==` -> `!=`, and `< 0`
|
|
negated to `> 0` or widened to `<= 0`. The first two, `inside != (OrientationSign[axis]
|
|
< 0)` and `OrientationSign[axis] > 0`, are both reported `Killed`. The third,
|
|
`OrientationSign[axis] <= 0`, survives, but it is an equivalent mutant, not a
|
|
coverage gap: `OrientationSign` is the fixed table `{ 1, -1, 1 }` and never
|
|
holds `0`, so `< 0` and `<= 0` agree on every value the table can actually
|
|
produce. There is nothing for a test to distinguish, the same shape as the
|
|
`TryGetOtherEndpoint` equivalent mutant above.
|
|
|
|
No `vstest` coverage-based false survivor has actually been confirmed for
|
|
GameCult.Geometry. Every survivor triaged here so far has turned out to be a
|
|
genuine equivalent mutant once hand-mutated and checked directly against
|
|
`dotnet test`; treat any survivor that looks like it must obviously be
|
|
covered as unproven until confirmed the same way, but do not cite this
|
|
project as precedent for the `vstest` caveat itself - that caveat is
|
|
`mtp`-runner-specific and documented above for `static readonly` field
|
|
initializers, and GameCult.Geometry has never actually hit it.
|
|
|
|
## Scope and follow-ups
|
|
|
|
CultMath.Tests, GameCult.Geometry.Tests and GameCult.Caching.Tests are the
|
|
projects currently scoped. GameCult.Geometry's `stryker-config.json` has no
|
|
`test-runner` entry: the project is NUnit 4 with `NUnit3TestAdapter` and
|
|
`Microsoft.NET.Test.Sdk`, so Stryker's default `vstest` runner applies
|
|
directly (no `mtp` workaround needed, unlike CultMath.Tests's xunit.v3
|
|
suite). GameCult.Caching.Tests is the same shape (NUnit, default `vstest`)
|
|
and its `stryker-config.json` is driven the same way, by the repo-local
|
|
`dotnet-stryker` tool: `dotnet tool restore` once, then `dotnet tool run
|
|
dotnet-stryker` from `tests/GameCult.Caching.Tests`.
|
|
|
|
GameCult.Geometry has only ever been run `--since:`-scoped to the cut that
|
|
added surface nets; a full-project baseline (running mutation against all of
|
|
`CultGeometryIsoSurface`, `CultGeometryDocuments`, and
|
|
`CultGeometryPrimitives`, not just the diff) is a recorded follow-up, not
|
|
something either cut did.
|
|
|
|
That first `--since` pass did not actually stay diff-scoped: adding
|
|
`stryker-config.json` itself in the same cut that first used it made Stryker
|
|
log "391 mutants will be tested because: Non-CSharp files in test project
|
|
were changed" and mutate the whole `mutate` glob (all four existing
|
|
GameCult.Geometry source files), not just the new surface-nets files. A cut
|
|
that changes a non-C# file inside the test project (a fixture, this config
|
|
file, anything under `Fixtures/`) loses `--since` line-level scoping for that
|
|
run; a future cut whose diff is C#-only, against a base commit that already
|
|
has `stryker-config.json`, should scope correctly.
|
|
|
|
### GameCult.Caching.Tests
|
|
|
|
`tests/GameCult.Caching.Tests/stryker-config.json` mutates all of
|
|
`src/GameCult.Caching` (the project is `GameCult.Caching.csproj`; Standard
|
|
mutation level, `break` 0, the same thresholds as Geometry). Caching was
|
|
scoped before Networking because its suite is deterministic and quick (1415
|
|
tests, about 40 s) over about 7.8k lines of source, where GameCult.Networking
|
|
is about 20k lines and its tests lean on socket timing, so a mutant timing out
|
|
there says little about the test. A full Caching run is about 2,900 candidate
|
|
mutants (3,602 created, minus 730 that fail to compile) at 1.25 to 1.6 s
|
|
each, so about 1 to 1.3 hours; run it detached and read the JSON report,
|
|
never as one attached session.
|
|
|
|
The first measurement was a narrow pass over the three files that carry the
|
|
store-replace and element-id machinery, from `tests/GameCult.Caching.Tests`:
|
|
|
|
```bash
|
|
dotnet tool run dotnet-stryker \
|
|
-m '**/src/GameCult.Caching/StoreReplace.cs' \
|
|
-m '**/src/GameCult.Caching/CultStoreLinks.cs' \
|
|
-m '**/src/GameCult.Caching/CultElementIds.cs'
|
|
```
|
|
|
|
A clean single run of that command tests 243 to 265 mutants (the count moves with the revision) and scored 85.28% on
|
|
568a60fa, before any test of this scope was added; 86.79% on d38b5728, with
|
|
the first batch of tests; and 87.17% on 76d9e7d1, with the throwing-enumerator
|
|
case, in about 10 minutes. The first measurement of it overlapped two Stryker
|
|
runs, so its numbers (and the 114 survivors it showed for `CultElementIds.cs`,
|
|
where tests that kill a mutant by hand reported `Survived` or `Timeout`) are
|
|
not measurements and are not quoted here. Never run two Stryker runs against
|
|
one checkout at once, and check `ps` before starting one. A clean single run
|
|
on `CultElementIds.cs` alone, after the tests below, tested 228 mutants: 208
|
|
`Killed`, 8 `Timeout`, 12 `Survived`, a score of 94.74%. The 12th survivor
|
|
was 321, which the throwing-enumerator case in `ListsOfValuesHoldNoIds` now
|
|
kills; the other 11 are equivalent mutants. With that case a clean single run
|
|
on `CultElementIds.cs` is 209 `Killed`, 8 `Timeout`, 11 `Survived`, 95.18%.
|
|
|
|
Real gaps that run exposed, each now pinned by `ElementIdWalkTests` (and each
|
|
failing when its mutant is applied by hand): the order of the id path (the
|
|
codec refuses at the first type MessagePack cannot allow, so order is what an
|
|
error names first; `Append` to `Prepend` at 163, 167 and 180); the
|
|
union-cycle guard in `ConcreteTypes` (248; the mutant is a stack overflow, so
|
|
the red is a crashed test host: a depth guard would need production code in
|
|
`ConcreteTypes`, since the recursion is private and a stack overflow cannot be
|
|
caught by the test, so the regression test is left to crash loudly); a list of values reporting that it holds an
|
|
id (`Holds`, 322); `[Key]` read through an override (`KeyOf`, 583); and the
|
|
`IsWrittenWhole` and `IsKeyed` predicates (629 to 631 and 636: formatter,
|
|
`[MessagePackObject]` and `KeyAsPropertyName` inherited from a base, and the
|
|
`[DataContract]` without `[MessagePackObject]` and `||` arms).
|
|
|
|
Equivalent survivors in `CultElementIds.cs` (11), each hand-checked against the full
|
|
suite; none can be told apart by any test:
|
|
|
|
- 142, 251, 636 (second arm), 506 (`IsDefined(UnionAttribute, false)` to
|
|
`true`): `UnionAttribute` is `Inherited = false`, so its `inherit` argument
|
|
changes nothing. Likewise 630's `DataContractAttribute` `inherit` argument
|
|
(`DataContractAttribute` is `Inherited = false`).
|
|
- 264, 542: `group.First()` to `FirstOrDefault()` over `GroupBy` groups, which
|
|
are never empty.
|
|
- 491: `new StringBuilder(bytes.Length * 2)` to `/ 2` changes only the initial
|
|
capacity.
|
|
- 578: removing the `Nullable.GetUnderlyingType` unwrap in `WalksAsOwner`;
|
|
`Classify` unwraps nullables itself, so the unwrap is dead. A cut
|
|
candidate, not a test gap.
|
|
- 645, 657: `Append(type)` to `Prepend(type)` in `ElementTypeOf` and
|
|
`MapTypes`; `type` matches only when it is itself `IEnumerable<>`,
|
|
`IDictionary<,>` or `IReadOnlyDictionary<,>`, and none of those has a second
|
|
matching interface, so the position of `type` never decides the answer.
|
|
|
|
The Windows-only P/Invoke group is exercised only on Windows: its mutants are
|
|
`NoCoverage` on Linux and are a target for a Windows-leg mutation run, not a
|
|
Linux gap or a finding. `StoreReplace.cs` line 45 is a different case: its
|
|
negated `IsOSPlatform(Windows)` mutant is `Survived` on Linux, because the
|
|
negated branch calls `ReplaceWindows`, whose `DllNotFound` fallback gives the
|
|
same result.
|
|
|
|
`GameCult.Mesh` (+ `.Quic`, `.Quic.Native`),
|
|
`GameCult.Networking` (+ `.WebSockets`), and the non-.NET runtimes
|
|
(`cultcache-rs/py/ts`, `cultmesh-*`, `cultnet-*`) each still need their own
|
|
`stryker-config.json` (or ecosystem-equivalent tool: `cargo-mutants` for
|
|
Rust, `StrykerJS` for TypeScript, `mutmut` for Python) and a baseline pass.
|
|
None of that is wired into CI yet.
|