Mutation testing: scope GameCult.Caching.Tests; record the narrow pass, CultElementIds survivors and equivalents
Some checks failed
CultMesh portability and performance / ubuntu-latest reactive semantics and leases (pull_request) Has been cancelled
Some checks failed
CultMesh portability and performance / ubuntu-latest reactive semantics and leases (pull_request) Has been cancelled
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
4c99673658
commit
d38b57284e
2 changed files with 90 additions and 7 deletions
|
|
@ -3,7 +3,7 @@
|
|||
CultLib measures its C# test suites with Stryker.NET, pinned as a repo-local
|
||||
tool in `dotnet-tools.json`. Coverage is added project by project; a project
|
||||
with no `stryker-config.json` next to its test project has not been scoped
|
||||
yet (see "Follow-ups" below for what's outstanding).
|
||||
yet (see "Scope and follow-ups" below for what's outstanding).
|
||||
|
||||
## Running it
|
||||
|
||||
|
|
@ -126,14 +126,17 @@ project as precedent for the `vstest` caveat itself - that caveat is
|
|||
`mtp`-runner-specific and documented above for `static readonly` field
|
||||
initializers, and GameCult.Geometry has never actually hit it.
|
||||
|
||||
## Follow-ups
|
||||
## Scope and follow-ups
|
||||
|
||||
CultMath.Tests and GameCult.Geometry.Tests are the only projects currently
|
||||
scoped. GameCult.Geometry's `stryker-config.json` has no `test-runner`
|
||||
entry: the project is NUnit 4 with `NUnit3TestAdapter` and
|
||||
CultMath.Tests, GameCult.Geometry.Tests and GameCult.Caching.Tests are the
|
||||
projects currently scoped. GameCult.Geometry's `stryker-config.json` has no
|
||||
`test-runner` entry: the project is NUnit 4 with `NUnit3TestAdapter` and
|
||||
`Microsoft.NET.Test.Sdk`, so Stryker's default `vstest` runner applies
|
||||
directly (no `mtp` workaround needed, unlike CultMath.Tests's xunit.v3
|
||||
suite).
|
||||
suite). GameCult.Caching.Tests is the same shape (NUnit, default `vstest`)
|
||||
and its `stryker-config.json` is driven the same way, by the repo-local
|
||||
`dotnet-stryker` tool: `dotnet tool restore` once, then `dotnet tool run
|
||||
dotnet-stryker` from `tests/GameCult.Caching.Tests`.
|
||||
|
||||
GameCult.Geometry has only ever been run `--since:`-scoped to the cut that
|
||||
added surface nets; a full-project baseline (running mutation against all of
|
||||
|
|
@ -151,7 +154,76 @@ file, anything under `Fixtures/`) loses `--since` line-level scoping for that
|
|||
run; a future cut whose diff is C#-only, against a base commit that already
|
||||
has `stryker-config.json`, should scope correctly.
|
||||
|
||||
`GameCult.Caching`, `GameCult.Mesh` (+ `.Quic`, `.Quic.Native`),
|
||||
### GameCult.Caching.Tests
|
||||
|
||||
`tests/GameCult.Caching.Tests/stryker-config.json` mutates all of
|
||||
`src/GameCult.Caching` (the project is `GameCult.Caching.csproj`; Standard
|
||||
mutation level, `break` 0, the same thresholds as Geometry). Caching was
|
||||
scoped before Networking because its suite is deterministic and quick (1415
|
||||
tests, about 40 s) over about 7.8k lines of source, where GameCult.Networking
|
||||
is about 20k lines and its tests lean on socket timing, so a mutant timing out
|
||||
there says little about the test. A full Caching run is about 2,200 mutants at
|
||||
roughly 1.6 s each, about an hour; run it detached and read the JSON report,
|
||||
never as one attached session.
|
||||
|
||||
The first measurement was a narrow pass over the three files that carry the
|
||||
store-replace and element-id machinery, from `tests/GameCult.Caching.Tests`:
|
||||
|
||||
```bash
|
||||
dotnet tool run dotnet-stryker \
|
||||
-m '**/src/GameCult.Caching/StoreReplace.cs' \
|
||||
-m '**/src/GameCult.Caching/CultStoreLinks.cs' \
|
||||
-m '**/src/GameCult.Caching/CultElementIds.cs'
|
||||
```
|
||||
|
||||
It tested 243 mutants and scored 84.91% in about 8.5 minutes. Two Stryker
|
||||
runs overlapped during that pass, and it showed: the same tests that kill a
|
||||
mutant by hand reported it `Survived` or `Timeout` (for example the `Holds`
|
||||
guards at `CultElementIds.cs` 311 and 316), which inflated that file's survivor
|
||||
list to 114. Never run two Stryker runs against one checkout at once, and check
|
||||
`ps` before starting one. Re-run on `CultElementIds.cs` alone, after the tests
|
||||
below, it tested 228 mutants: 208 `Killed`, 8 `Timeout`, 12 `Survived`, a score
|
||||
of 94.74%, and every one of the 12 is an equivalent mutant.
|
||||
|
||||
Real gaps that run exposed, each now pinned by `ElementIdWalkTests` (and each
|
||||
failing when its mutant is applied by hand): the order of the id path (the
|
||||
codec refuses at the first type MessagePack cannot allow, so order is what an
|
||||
error names first; `Append` to `Prepend` at 163, 167 and 180); the
|
||||
union-cycle guard in `ConcreteTypes` (248; the mutant is a stack overflow, so
|
||||
the red is a crashed test host); a list of values reporting that it holds an
|
||||
id (`Holds`, 322); `[Key]` read through an override (`KeyOf`, 583); and the
|
||||
`IsWrittenWhole` and `IsKeyed` predicates (629 to 631 and 636: formatter,
|
||||
`[MessagePackObject]` and `KeyAsPropertyName` inherited from a base, and the
|
||||
`[DataContract]` without `[MessagePackObject]` and `||` arms).
|
||||
|
||||
Equivalent survivors in `CultElementIds.cs`, each hand-checked against the full
|
||||
suite; none can be told apart by any test:
|
||||
|
||||
- 142, 251, 636 (second arm), 506 (`IsDefined(UnionAttribute, false)` to
|
||||
`true`): `UnionAttribute` is `Inherited = false`, so its `inherit` argument
|
||||
changes nothing. Likewise 630's `DataContractAttribute` `inherit` argument
|
||||
(`DataContractAttribute` is `Inherited = false`).
|
||||
- 264, 542: `group.First()` to `FirstOrDefault()` over `GroupBy` groups, which
|
||||
are never empty.
|
||||
- 491: `new StringBuilder(bytes.Length * 2)` to `/ 2` changes only the initial
|
||||
capacity.
|
||||
- 321: `!elements` to `elements` in `Holds`; `elements` implies the inner
|
||||
type classifies as an owner, so `elements && Leaf` is never true and the
|
||||
early return is a pure shortcut (the loop below returns the same `false`).
|
||||
- 578: removing the `Nullable.GetUnderlyingType` unwrap in `WalksAsOwner`;
|
||||
`Classify` unwraps nullables itself, so the unwrap is dead. A cut
|
||||
candidate, not a test gap.
|
||||
- 645, 657: `Append(type)` to `Prepend(type)` in `ElementTypeOf` and
|
||||
`MapTypes`; `type` matches only when it is itself `IEnumerable<>`,
|
||||
`IDictionary<,>` or `IReadOnlyDictionary<,>`, and none of those has a second
|
||||
matching interface, so the position of `type` never decides the answer.
|
||||
|
||||
The Windows-only P/Invoke group (the native-library and `RuntimeInformation`
|
||||
paths, such as `StoreReplace.cs` line 45) can only be exercised on Windows. It
|
||||
is a target for a Windows-leg mutation run, not a Linux coverage gap, and its
|
||||
`NoCoverage` mutants on Linux are not findings.
|
||||
|
||||
`GameCult.Mesh` (+ `.Quic`, `.Quic.Native`),
|
||||
`GameCult.Networking` (+ `.WebSockets`), and the non-.NET runtimes
|
||||
(`cultcache-rs/py/ts`, `cultmesh-*`, `cultnet-*`) each still need their own
|
||||
`stryker-config.json` (or ecosystem-equivalent tool: `cargo-mutants` for
|
||||
|
|
|
|||
11
tests/GameCult.Caching.Tests/stryker-config.json
Normal file
11
tests/GameCult.Caching.Tests/stryker-config.json
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
{
|
||||
"stryker-config": {
|
||||
"project": "GameCult.Caching.csproj",
|
||||
"mutate": [
|
||||
"**/src/GameCult.Caching/**/*.cs"
|
||||
],
|
||||
"mutation-level": "Standard",
|
||||
"reporters": [ "progress", "html", "json" ],
|
||||
"thresholds": { "high": 80, "low": 60, "break": 0 }
|
||||
}
|
||||
}
|
||||
Loading…
Reference in a new issue