diff --git a/docs/mutation-testing.md b/docs/mutation-testing.md index 8db67be5..a8671a40 100644 --- a/docs/mutation-testing.md +++ b/docs/mutation-testing.md @@ -3,7 +3,7 @@ CultLib measures its C# test suites with Stryker.NET, pinned as a repo-local tool in `dotnet-tools.json`. Coverage is added project by project; a project with no `stryker-config.json` next to its test project has not been scoped -yet (see "Follow-ups" below for what's outstanding). +yet (see "Scope and follow-ups" below for what's outstanding). ## Running it @@ -126,14 +126,17 @@ project as precedent for the `vstest` caveat itself - that caveat is `mtp`-runner-specific and documented above for `static readonly` field initializers, and GameCult.Geometry has never actually hit it. -## Follow-ups +## Scope and follow-ups -CultMath.Tests and GameCult.Geometry.Tests are the only projects currently -scoped. GameCult.Geometry's `stryker-config.json` has no `test-runner` -entry: the project is NUnit 4 with `NUnit3TestAdapter` and +CultMath.Tests, GameCult.Geometry.Tests and GameCult.Caching.Tests are the +projects currently scoped. GameCult.Geometry's `stryker-config.json` has no +`test-runner` entry: the project is NUnit 4 with `NUnit3TestAdapter` and `Microsoft.NET.Test.Sdk`, so Stryker's default `vstest` runner applies directly (no `mtp` workaround needed, unlike CultMath.Tests's xunit.v3 -suite). +suite). GameCult.Caching.Tests is the same shape (NUnit, default `vstest`) +and its `stryker-config.json` is driven the same way, by the repo-local +`dotnet-stryker` tool: `dotnet tool restore` once, then `dotnet tool run +dotnet-stryker` from `tests/GameCult.Caching.Tests`. GameCult.Geometry has only ever been run `--since:`-scoped to the cut that added surface nets; a full-project baseline (running mutation against all of @@ -151,7 +154,76 @@ file, anything under `Fixtures/`) loses `--since` line-level scoping for that run; a future cut whose diff is C#-only, against a base commit that already has `stryker-config.json`, should scope correctly. -`GameCult.Caching`, `GameCult.Mesh` (+ `.Quic`, `.Quic.Native`), +### GameCult.Caching.Tests + +`tests/GameCult.Caching.Tests/stryker-config.json` mutates all of +`src/GameCult.Caching` (the project is `GameCult.Caching.csproj`; Standard +mutation level, `break` 0, the same thresholds as Geometry). Caching was +scoped before Networking because its suite is deterministic and quick (1415 +tests, about 40 s) over about 7.8k lines of source, where GameCult.Networking +is about 20k lines and its tests lean on socket timing, so a mutant timing out +there says little about the test. A full Caching run is about 2,200 mutants at +roughly 1.6 s each, about an hour; run it detached and read the JSON report, +never as one attached session. + +The first measurement was a narrow pass over the three files that carry the +store-replace and element-id machinery, from `tests/GameCult.Caching.Tests`: + +```bash +dotnet tool run dotnet-stryker \ + -m '**/src/GameCult.Caching/StoreReplace.cs' \ + -m '**/src/GameCult.Caching/CultStoreLinks.cs' \ + -m '**/src/GameCult.Caching/CultElementIds.cs' +``` + +It tested 243 mutants and scored 84.91% in about 8.5 minutes. Two Stryker +runs overlapped during that pass, and it showed: the same tests that kill a +mutant by hand reported it `Survived` or `Timeout` (for example the `Holds` +guards at `CultElementIds.cs` 311 and 316), which inflated that file's survivor +list to 114. Never run two Stryker runs against one checkout at once, and check +`ps` before starting one. Re-run on `CultElementIds.cs` alone, after the tests +below, it tested 228 mutants: 208 `Killed`, 8 `Timeout`, 12 `Survived`, a score +of 94.74%, and every one of the 12 is an equivalent mutant. + +Real gaps that run exposed, each now pinned by `ElementIdWalkTests` (and each +failing when its mutant is applied by hand): the order of the id path (the +codec refuses at the first type MessagePack cannot allow, so order is what an +error names first; `Append` to `Prepend` at 163, 167 and 180); the +union-cycle guard in `ConcreteTypes` (248; the mutant is a stack overflow, so +the red is a crashed test host); a list of values reporting that it holds an +id (`Holds`, 322); `[Key]` read through an override (`KeyOf`, 583); and the +`IsWrittenWhole` and `IsKeyed` predicates (629 to 631 and 636: formatter, +`[MessagePackObject]` and `KeyAsPropertyName` inherited from a base, and the +`[DataContract]` without `[MessagePackObject]` and `||` arms). + +Equivalent survivors in `CultElementIds.cs`, each hand-checked against the full +suite; none can be told apart by any test: + +- 142, 251, 636 (second arm), 506 (`IsDefined(UnionAttribute, false)` to + `true`): `UnionAttribute` is `Inherited = false`, so its `inherit` argument + changes nothing. Likewise 630's `DataContractAttribute` `inherit` argument + (`DataContractAttribute` is `Inherited = false`). +- 264, 542: `group.First()` to `FirstOrDefault()` over `GroupBy` groups, which + are never empty. +- 491: `new StringBuilder(bytes.Length * 2)` to `/ 2` changes only the initial + capacity. +- 321: `!elements` to `elements` in `Holds`; `elements` implies the inner + type classifies as an owner, so `elements && Leaf` is never true and the + early return is a pure shortcut (the loop below returns the same `false`). +- 578: removing the `Nullable.GetUnderlyingType` unwrap in `WalksAsOwner`; + `Classify` unwraps nullables itself, so the unwrap is dead. A cut + candidate, not a test gap. +- 645, 657: `Append(type)` to `Prepend(type)` in `ElementTypeOf` and + `MapTypes`; `type` matches only when it is itself `IEnumerable<>`, + `IDictionary<,>` or `IReadOnlyDictionary<,>`, and none of those has a second + matching interface, so the position of `type` never decides the answer. + +The Windows-only P/Invoke group (the native-library and `RuntimeInformation` +paths, such as `StoreReplace.cs` line 45) can only be exercised on Windows. It +is a target for a Windows-leg mutation run, not a Linux coverage gap, and its +`NoCoverage` mutants on Linux are not findings. + +`GameCult.Mesh` (+ `.Quic`, `.Quic.Native`), `GameCult.Networking` (+ `.WebSockets`), and the non-.NET runtimes (`cultcache-rs/py/ts`, `cultmesh-*`, `cultnet-*`) each still need their own `stryker-config.json` (or ecosystem-equivalent tool: `cargo-mutants` for diff --git a/tests/GameCult.Caching.Tests/stryker-config.json b/tests/GameCult.Caching.Tests/stryker-config.json new file mode 100644 index 00000000..ac366203 --- /dev/null +++ b/tests/GameCult.Caching.Tests/stryker-config.json @@ -0,0 +1,11 @@ +{ + "stryker-config": { + "project": "GameCult.Caching.csproj", + "mutate": [ + "**/src/GameCult.Caching/**/*.cs" + ], + "mutation-level": "Standard", + "reporters": [ "progress", "html", "json" ], + "thresholds": { "high": 80, "low": 60, "break": 0 } + } +}