Mutation testing doc and tests: kill 321 with a throwing enumerator, restate narrow-pass and full-run numbers, fix StoreReplace line 45 grouping (Soul s1)
All checks were successful
CultMesh portability and performance / ubuntu-latest reactive semantics and leases (pull_request) Successful in 24m54s

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Libby 2026-11-13 09:23:36 +00:00
commit 76d9e7d180
2 changed files with 31 additions and 19 deletions

View file

@ -162,8 +162,9 @@ mutation level, `break` 0, the same thresholds as Geometry). Caching was
scoped before Networking because its suite is deterministic and quick (1415
tests, about 40 s) over about 7.8k lines of source, where GameCult.Networking
is about 20k lines and its tests lean on socket timing, so a mutant timing out
there says little about the test. A full Caching run is about 2,200 mutants at
roughly 1.6 s each, about an hour; run it detached and read the JSON report,
there says little about the test. A full Caching run is about 2,900 candidate
mutants (3,602 created, minus 730 that fail to compile) at about 2 s each,
about 1.5 hours; run it detached and read the JSON report,
never as one attached session.
The first measurement was a narrow pass over the three files that carry the
@ -176,27 +177,32 @@ dotnet tool run dotnet-stryker \
-m '**/src/GameCult.Caching/CultElementIds.cs'
```
It tested 243 mutants and scored 84.91% in about 8.5 minutes. Two Stryker
runs overlapped during that pass, and it showed: the same tests that kill a
mutant by hand reported it `Survived` or `Timeout` (for example the `Holds`
guards at `CultElementIds.cs` 311 and 316), which inflated that file's survivor
list to 114. Never run two Stryker runs against one checkout at once, and check
`ps` before starting one. Re-run on `CultElementIds.cs` alone, after the tests
below, it tested 228 mutants: 208 `Killed`, 8 `Timeout`, 12 `Survived`, a score
of 94.74%, and every one of the 12 is an equivalent mutant.
A clean single run of that command tested 243 mutants and scored 86.79% in
9 minutes 55 seconds. The first measurement of it overlapped two Stryker
runs, so its numbers (and the 114 survivors it showed for `CultElementIds.cs`,
where tests that kill a mutant by hand reported `Survived` or `Timeout`) are
not measurements and are not quoted here. Never run two Stryker runs against
one checkout at once, and check `ps` before starting one. A clean single run
on `CultElementIds.cs` alone, after the tests below, tested 228 mutants: 208
`Killed`, 8 `Timeout`, 12 `Survived`, a score of 94.74%. The 12th survivor
was 321, which the throwing-enumerator case in `ListsOfValuesHoldNoIds` now
kills; the other 11 are equivalent mutants. With that case a clean single run
on `CultElementIds.cs` is 209 `Killed`, 8 `Timeout`, 11 `Survived`, 95.18%.
Real gaps that run exposed, each now pinned by `ElementIdWalkTests` (and each
failing when its mutant is applied by hand): the order of the id path (the
codec refuses at the first type MessagePack cannot allow, so order is what an
error names first; `Append` to `Prepend` at 163, 167 and 180); the
union-cycle guard in `ConcreteTypes` (248; the mutant is a stack overflow, so
the red is a crashed test host); a list of values reporting that it holds an
the red is a crashed test host: a depth guard would need production code in
`ConcreteTypes`, since the recursion is private and a stack overflow cannot be
caught by the test, so the regression test is left to crash loudly); a list of values reporting that it holds an
id (`Holds`, 322); `[Key]` read through an override (`KeyOf`, 583); and the
`IsWrittenWhole` and `IsKeyed` predicates (629 to 631 and 636: formatter,
`[MessagePackObject]` and `KeyAsPropertyName` inherited from a base, and the
`[DataContract]` without `[MessagePackObject]` and `||` arms).
Equivalent survivors in `CultElementIds.cs`, each hand-checked against the full
Equivalent survivors in `CultElementIds.cs` (11), each hand-checked against the full
suite; none can be told apart by any test:
- 142, 251, 636 (second arm), 506 (`IsDefined(UnionAttribute, false)` to
@ -207,9 +213,6 @@ suite; none can be told apart by any test:
are never empty.
- 491: `new StringBuilder(bytes.Length * 2)` to `/ 2` changes only the initial
capacity.
- 321: `!elements` to `elements` in `Holds`; `elements` implies the inner
type classifies as an owner, so `elements && Leaf` is never true and the
early return is a pure shortcut (the loop below returns the same `false`).
- 578: removing the `Nullable.GetUnderlyingType` unwrap in `WalksAsOwner`;
`Classify` unwraps nullables itself, so the unwrap is dead. A cut
candidate, not a test gap.
@ -218,10 +221,12 @@ suite; none can be told apart by any test:
`IDictionary<,>` or `IReadOnlyDictionary<,>`, and none of those has a second
matching interface, so the position of `type` never decides the answer.
The Windows-only P/Invoke group (the native-library and `RuntimeInformation`
paths, such as `StoreReplace.cs` line 45) can only be exercised on Windows. It
is a target for a Windows-leg mutation run, not a Linux coverage gap, and its
`NoCoverage` mutants on Linux are not findings.
The Windows-only P/Invoke group is exercised only on Windows: its mutants are
`NoCoverage` on Linux and are a target for a Windows-leg mutation run, not a
Linux gap or a finding. `StoreReplace.cs` line 45 is a different case: its
negated `IsOSPlatform(Windows)` mutant is `Survived` on Linux, because the
negated branch calls `ReplaceWindows`, whose `DllNotFound` fallback gives the
same result.
`GameCult.Mesh` (+ `.Quic`, `.Quic.Native`),
`GameCult.Networking` (+ `.WebSockets`), and the non-.NET runtimes

View file

@ -48,6 +48,7 @@ namespace GameCult.Caching.Tests
Assert.That(CultElementIds.Holds(typeof(List<int>), new List<int> { 1, 2 }), Is.False);
Assert.That(CultElementIds.Holds(typeof(List<List<int>>), new List<List<int>> { new() { 1 } }), Is.False);
Assert.That(CultElementIds.Holds(typeof(List<string>), new List<string> { "a" }), Is.False);
Assert.That(CultElementIds.Holds(typeof(MsBoom), new MsBoom()), Is.False, "a sequence of values is not enumerated");
}
// A nullable struct is walked as the struct it wraps; a nullable value is not walked at all.
@ -107,6 +108,12 @@ namespace GameCult.Caching.Tests
[Key(3)] public Dictionary<string, List<int>> Map { get; set; } = new();
}
public sealed class MsBoom : IEnumerable<int>
{
public IEnumerator<int> GetEnumerator() => throw new InvalidOperationException("enumerated");
System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator() => GetEnumerator();
}
[MessagePackObject] public struct MsKeyedStruct { [Key(0)] public int X; }
[MessagePackObject]