From 76d9e7d180eba773015c79c5a018e59d7b2a0b67 Mon Sep 17 00:00:00 2001 From: Libby Date: Fri, 13 Nov 2026 09:23:36 +0000 Subject: [PATCH] Mutation testing doc and tests: kill 321 with a throwing enumerator, restate narrow-pass and full-run numbers, fix StoreReplace line 45 grouping (Soul s1) Co-Authored-By: Claude Sonnet 5.5 --- docs/mutation-testing.md | 43 +++++++++++-------- .../ElementIdWalkTests.cs | 7 +++ 2 files changed, 31 insertions(+), 19 deletions(-) diff --git a/docs/mutation-testing.md b/docs/mutation-testing.md index a8671a40..c6a0f5bd 100644 --- a/docs/mutation-testing.md +++ b/docs/mutation-testing.md @@ -162,8 +162,9 @@ mutation level, `break` 0, the same thresholds as Geometry). Caching was scoped before Networking because its suite is deterministic and quick (1415 tests, about 40 s) over about 7.8k lines of source, where GameCult.Networking is about 20k lines and its tests lean on socket timing, so a mutant timing out -there says little about the test. A full Caching run is about 2,200 mutants at -roughly 1.6 s each, about an hour; run it detached and read the JSON report, +there says little about the test. A full Caching run is about 2,900 candidate +mutants (3,602 created, minus 730 that fail to compile) at about 2 s each, +about 1.5 hours; run it detached and read the JSON report, never as one attached session. The first measurement was a narrow pass over the three files that carry the @@ -176,27 +177,32 @@ dotnet tool run dotnet-stryker \ -m '**/src/GameCult.Caching/CultElementIds.cs' ``` -It tested 243 mutants and scored 84.91% in about 8.5 minutes. Two Stryker -runs overlapped during that pass, and it showed: the same tests that kill a -mutant by hand reported it `Survived` or `Timeout` (for example the `Holds` -guards at `CultElementIds.cs` 311 and 316), which inflated that file's survivor -list to 114. Never run two Stryker runs against one checkout at once, and check -`ps` before starting one. Re-run on `CultElementIds.cs` alone, after the tests -below, it tested 228 mutants: 208 `Killed`, 8 `Timeout`, 12 `Survived`, a score -of 94.74%, and every one of the 12 is an equivalent mutant. +A clean single run of that command tested 243 mutants and scored 86.79% in +9 minutes 55 seconds. The first measurement of it overlapped two Stryker +runs, so its numbers (and the 114 survivors it showed for `CultElementIds.cs`, +where tests that kill a mutant by hand reported `Survived` or `Timeout`) are +not measurements and are not quoted here. Never run two Stryker runs against +one checkout at once, and check `ps` before starting one. A clean single run +on `CultElementIds.cs` alone, after the tests below, tested 228 mutants: 208 +`Killed`, 8 `Timeout`, 12 `Survived`, a score of 94.74%. The 12th survivor +was 321, which the throwing-enumerator case in `ListsOfValuesHoldNoIds` now +kills; the other 11 are equivalent mutants. With that case a clean single run +on `CultElementIds.cs` is 209 `Killed`, 8 `Timeout`, 11 `Survived`, 95.18%. Real gaps that run exposed, each now pinned by `ElementIdWalkTests` (and each failing when its mutant is applied by hand): the order of the id path (the codec refuses at the first type MessagePack cannot allow, so order is what an error names first; `Append` to `Prepend` at 163, 167 and 180); the union-cycle guard in `ConcreteTypes` (248; the mutant is a stack overflow, so -the red is a crashed test host); a list of values reporting that it holds an +the red is a crashed test host: a depth guard would need production code in +`ConcreteTypes`, since the recursion is private and a stack overflow cannot be +caught by the test, so the regression test is left to crash loudly); a list of values reporting that it holds an id (`Holds`, 322); `[Key]` read through an override (`KeyOf`, 583); and the `IsWrittenWhole` and `IsKeyed` predicates (629 to 631 and 636: formatter, `[MessagePackObject]` and `KeyAsPropertyName` inherited from a base, and the `[DataContract]` without `[MessagePackObject]` and `||` arms). -Equivalent survivors in `CultElementIds.cs`, each hand-checked against the full +Equivalent survivors in `CultElementIds.cs` (11), each hand-checked against the full suite; none can be told apart by any test: - 142, 251, 636 (second arm), 506 (`IsDefined(UnionAttribute, false)` to @@ -207,9 +213,6 @@ suite; none can be told apart by any test: are never empty. - 491: `new StringBuilder(bytes.Length * 2)` to `/ 2` changes only the initial capacity. -- 321: `!elements` to `elements` in `Holds`; `elements` implies the inner - type classifies as an owner, so `elements && Leaf` is never true and the - early return is a pure shortcut (the loop below returns the same `false`). - 578: removing the `Nullable.GetUnderlyingType` unwrap in `WalksAsOwner`; `Classify` unwraps nullables itself, so the unwrap is dead. A cut candidate, not a test gap. @@ -218,10 +221,12 @@ suite; none can be told apart by any test: `IDictionary<,>` or `IReadOnlyDictionary<,>`, and none of those has a second matching interface, so the position of `type` never decides the answer. -The Windows-only P/Invoke group (the native-library and `RuntimeInformation` -paths, such as `StoreReplace.cs` line 45) can only be exercised on Windows. It -is a target for a Windows-leg mutation run, not a Linux coverage gap, and its -`NoCoverage` mutants on Linux are not findings. +The Windows-only P/Invoke group is exercised only on Windows: its mutants are +`NoCoverage` on Linux and are a target for a Windows-leg mutation run, not a +Linux gap or a finding. `StoreReplace.cs` line 45 is a different case: its +negated `IsOSPlatform(Windows)` mutant is `Survived` on Linux, because the +negated branch calls `ReplaceWindows`, whose `DllNotFound` fallback gives the +same result. `GameCult.Mesh` (+ `.Quic`, `.Quic.Native`), `GameCult.Networking` (+ `.WebSockets`), and the non-.NET runtimes diff --git a/tests/GameCult.Caching.Tests/ElementIdWalkTests.cs b/tests/GameCult.Caching.Tests/ElementIdWalkTests.cs index cd6b3be3..bd6cb903 100644 --- a/tests/GameCult.Caching.Tests/ElementIdWalkTests.cs +++ b/tests/GameCult.Caching.Tests/ElementIdWalkTests.cs @@ -48,6 +48,7 @@ namespace GameCult.Caching.Tests Assert.That(CultElementIds.Holds(typeof(List), new List { 1, 2 }), Is.False); Assert.That(CultElementIds.Holds(typeof(List>), new List> { new() { 1 } }), Is.False); Assert.That(CultElementIds.Holds(typeof(List), new List { "a" }), Is.False); + Assert.That(CultElementIds.Holds(typeof(MsBoom), new MsBoom()), Is.False, "a sequence of values is not enumerated"); } // A nullable struct is walked as the struct it wraps; a nullable value is not walked at all. @@ -107,6 +108,12 @@ namespace GameCult.Caching.Tests [Key(3)] public Dictionary> Map { get; set; } = new(); } + public sealed class MsBoom : IEnumerable + { + public IEnumerator GetEnumerator() => throw new InvalidOperationException("enumerated"); + System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator() => GetEnumerator(); + } + [MessagePackObject] public struct MsKeyedStruct { [Key(0)] public int X; } [MessagePackObject]