Mutation testing doc and tests: kill 321 with a throwing enumerator, restate narrow-pass and full-run numbers, fix StoreReplace line 45 grouping (Soul s1)
All checks were successful
CultMesh portability and performance / ubuntu-latest reactive semantics and leases (pull_request) Successful in 24m54s
All checks were successful
CultMesh portability and performance / ubuntu-latest reactive semantics and leases (pull_request) Successful in 24m54s
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
d38b57284e
commit
76d9e7d180
2 changed files with 31 additions and 19 deletions
|
|
@ -162,8 +162,9 @@ mutation level, `break` 0, the same thresholds as Geometry). Caching was
|
|||
scoped before Networking because its suite is deterministic and quick (1415
|
||||
tests, about 40 s) over about 7.8k lines of source, where GameCult.Networking
|
||||
is about 20k lines and its tests lean on socket timing, so a mutant timing out
|
||||
there says little about the test. A full Caching run is about 2,200 mutants at
|
||||
roughly 1.6 s each, about an hour; run it detached and read the JSON report,
|
||||
there says little about the test. A full Caching run is about 2,900 candidate
|
||||
mutants (3,602 created, minus 730 that fail to compile) at about 2 s each,
|
||||
about 1.5 hours; run it detached and read the JSON report,
|
||||
never as one attached session.
|
||||
|
||||
The first measurement was a narrow pass over the three files that carry the
|
||||
|
|
@ -176,27 +177,32 @@ dotnet tool run dotnet-stryker \
|
|||
-m '**/src/GameCult.Caching/CultElementIds.cs'
|
||||
```
|
||||
|
||||
It tested 243 mutants and scored 84.91% in about 8.5 minutes. Two Stryker
|
||||
runs overlapped during that pass, and it showed: the same tests that kill a
|
||||
mutant by hand reported it `Survived` or `Timeout` (for example the `Holds`
|
||||
guards at `CultElementIds.cs` 311 and 316), which inflated that file's survivor
|
||||
list to 114. Never run two Stryker runs against one checkout at once, and check
|
||||
`ps` before starting one. Re-run on `CultElementIds.cs` alone, after the tests
|
||||
below, it tested 228 mutants: 208 `Killed`, 8 `Timeout`, 12 `Survived`, a score
|
||||
of 94.74%, and every one of the 12 is an equivalent mutant.
|
||||
A clean single run of that command tested 243 mutants and scored 86.79% in
|
||||
9 minutes 55 seconds. The first measurement of it overlapped two Stryker
|
||||
runs, so its numbers (and the 114 survivors it showed for `CultElementIds.cs`,
|
||||
where tests that kill a mutant by hand reported `Survived` or `Timeout`) are
|
||||
not measurements and are not quoted here. Never run two Stryker runs against
|
||||
one checkout at once, and check `ps` before starting one. A clean single run
|
||||
on `CultElementIds.cs` alone, after the tests below, tested 228 mutants: 208
|
||||
`Killed`, 8 `Timeout`, 12 `Survived`, a score of 94.74%. The 12th survivor
|
||||
was 321, which the throwing-enumerator case in `ListsOfValuesHoldNoIds` now
|
||||
kills; the other 11 are equivalent mutants. With that case a clean single run
|
||||
on `CultElementIds.cs` is 209 `Killed`, 8 `Timeout`, 11 `Survived`, 95.18%.
|
||||
|
||||
Real gaps that run exposed, each now pinned by `ElementIdWalkTests` (and each
|
||||
failing when its mutant is applied by hand): the order of the id path (the
|
||||
codec refuses at the first type MessagePack cannot allow, so order is what an
|
||||
error names first; `Append` to `Prepend` at 163, 167 and 180); the
|
||||
union-cycle guard in `ConcreteTypes` (248; the mutant is a stack overflow, so
|
||||
the red is a crashed test host); a list of values reporting that it holds an
|
||||
the red is a crashed test host: a depth guard would need production code in
|
||||
`ConcreteTypes`, since the recursion is private and a stack overflow cannot be
|
||||
caught by the test, so the regression test is left to crash loudly); a list of values reporting that it holds an
|
||||
id (`Holds`, 322); `[Key]` read through an override (`KeyOf`, 583); and the
|
||||
`IsWrittenWhole` and `IsKeyed` predicates (629 to 631 and 636: formatter,
|
||||
`[MessagePackObject]` and `KeyAsPropertyName` inherited from a base, and the
|
||||
`[DataContract]` without `[MessagePackObject]` and `||` arms).
|
||||
|
||||
Equivalent survivors in `CultElementIds.cs`, each hand-checked against the full
|
||||
Equivalent survivors in `CultElementIds.cs` (11), each hand-checked against the full
|
||||
suite; none can be told apart by any test:
|
||||
|
||||
- 142, 251, 636 (second arm), 506 (`IsDefined(UnionAttribute, false)` to
|
||||
|
|
@ -207,9 +213,6 @@ suite; none can be told apart by any test:
|
|||
are never empty.
|
||||
- 491: `new StringBuilder(bytes.Length * 2)` to `/ 2` changes only the initial
|
||||
capacity.
|
||||
- 321: `!elements` to `elements` in `Holds`; `elements` implies the inner
|
||||
type classifies as an owner, so `elements && Leaf` is never true and the
|
||||
early return is a pure shortcut (the loop below returns the same `false`).
|
||||
- 578: removing the `Nullable.GetUnderlyingType` unwrap in `WalksAsOwner`;
|
||||
`Classify` unwraps nullables itself, so the unwrap is dead. A cut
|
||||
candidate, not a test gap.
|
||||
|
|
@ -218,10 +221,12 @@ suite; none can be told apart by any test:
|
|||
`IDictionary<,>` or `IReadOnlyDictionary<,>`, and none of those has a second
|
||||
matching interface, so the position of `type` never decides the answer.
|
||||
|
||||
The Windows-only P/Invoke group (the native-library and `RuntimeInformation`
|
||||
paths, such as `StoreReplace.cs` line 45) can only be exercised on Windows. It
|
||||
is a target for a Windows-leg mutation run, not a Linux coverage gap, and its
|
||||
`NoCoverage` mutants on Linux are not findings.
|
||||
The Windows-only P/Invoke group is exercised only on Windows: its mutants are
|
||||
`NoCoverage` on Linux and are a target for a Windows-leg mutation run, not a
|
||||
Linux gap or a finding. `StoreReplace.cs` line 45 is a different case: its
|
||||
negated `IsOSPlatform(Windows)` mutant is `Survived` on Linux, because the
|
||||
negated branch calls `ReplaceWindows`, whose `DllNotFound` fallback gives the
|
||||
same result.
|
||||
|
||||
`GameCult.Mesh` (+ `.Quic`, `.Quic.Native`),
|
||||
`GameCult.Networking` (+ `.WebSockets`), and the non-.NET runtimes
|
||||
|
|
|
|||
|
|
@ -48,6 +48,7 @@ namespace GameCult.Caching.Tests
|
|||
Assert.That(CultElementIds.Holds(typeof(List<int>), new List<int> { 1, 2 }), Is.False);
|
||||
Assert.That(CultElementIds.Holds(typeof(List<List<int>>), new List<List<int>> { new() { 1 } }), Is.False);
|
||||
Assert.That(CultElementIds.Holds(typeof(List<string>), new List<string> { "a" }), Is.False);
|
||||
Assert.That(CultElementIds.Holds(typeof(MsBoom), new MsBoom()), Is.False, "a sequence of values is not enumerated");
|
||||
}
|
||||
|
||||
// A nullable struct is walked as the struct it wraps; a nullable value is not walked at all.
|
||||
|
|
@ -107,6 +108,12 @@ namespace GameCult.Caching.Tests
|
|||
[Key(3)] public Dictionary<string, List<int>> Map { get; set; } = new();
|
||||
}
|
||||
|
||||
public sealed class MsBoom : IEnumerable<int>
|
||||
{
|
||||
public IEnumerator<int> GetEnumerator() => throw new InvalidOperationException("enumerated");
|
||||
System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator() => GetEnumerator();
|
||||
}
|
||||
|
||||
[MessagePackObject] public struct MsKeyedStruct { [Key(0)] public int X; }
|
||||
|
||||
[MessagePackObject]
|
||||
|
|
|
|||
Loading…
Reference in a new issue