diff --git a/docs/mutation-testing.md b/docs/mutation-testing.md index 8db67be5..c6a0f5bd 100644 --- a/docs/mutation-testing.md +++ b/docs/mutation-testing.md @@ -3,7 +3,7 @@ CultLib measures its C# test suites with Stryker.NET, pinned as a repo-local tool in `dotnet-tools.json`. Coverage is added project by project; a project with no `stryker-config.json` next to its test project has not been scoped -yet (see "Follow-ups" below for what's outstanding). +yet (see "Scope and follow-ups" below for what's outstanding). ## Running it @@ -126,14 +126,17 @@ project as precedent for the `vstest` caveat itself - that caveat is `mtp`-runner-specific and documented above for `static readonly` field initializers, and GameCult.Geometry has never actually hit it. -## Follow-ups +## Scope and follow-ups -CultMath.Tests and GameCult.Geometry.Tests are the only projects currently -scoped. GameCult.Geometry's `stryker-config.json` has no `test-runner` -entry: the project is NUnit 4 with `NUnit3TestAdapter` and +CultMath.Tests, GameCult.Geometry.Tests and GameCult.Caching.Tests are the +projects currently scoped. GameCult.Geometry's `stryker-config.json` has no +`test-runner` entry: the project is NUnit 4 with `NUnit3TestAdapter` and `Microsoft.NET.Test.Sdk`, so Stryker's default `vstest` runner applies directly (no `mtp` workaround needed, unlike CultMath.Tests's xunit.v3 -suite). +suite). GameCult.Caching.Tests is the same shape (NUnit, default `vstest`) +and its `stryker-config.json` is driven the same way, by the repo-local +`dotnet-stryker` tool: `dotnet tool restore` once, then `dotnet tool run +dotnet-stryker` from `tests/GameCult.Caching.Tests`. GameCult.Geometry has only ever been run `--since:`-scoped to the cut that added surface nets; a full-project baseline (running mutation against all of @@ -151,7 +154,81 @@ file, anything under `Fixtures/`) loses `--since` line-level scoping for that run; a future cut whose diff is C#-only, against a base commit that already has `stryker-config.json`, should scope correctly. -`GameCult.Caching`, `GameCult.Mesh` (+ `.Quic`, `.Quic.Native`), +### GameCult.Caching.Tests + +`tests/GameCult.Caching.Tests/stryker-config.json` mutates all of +`src/GameCult.Caching` (the project is `GameCult.Caching.csproj`; Standard +mutation level, `break` 0, the same thresholds as Geometry). Caching was +scoped before Networking because its suite is deterministic and quick (1415 +tests, about 40 s) over about 7.8k lines of source, where GameCult.Networking +is about 20k lines and its tests lean on socket timing, so a mutant timing out +there says little about the test. A full Caching run is about 2,900 candidate +mutants (3,602 created, minus 730 that fail to compile) at about 2 s each, +about 1.5 hours; run it detached and read the JSON report, +never as one attached session. + +The first measurement was a narrow pass over the three files that carry the +store-replace and element-id machinery, from `tests/GameCult.Caching.Tests`: + +```bash +dotnet tool run dotnet-stryker \ + -m '**/src/GameCult.Caching/StoreReplace.cs' \ + -m '**/src/GameCult.Caching/CultStoreLinks.cs' \ + -m '**/src/GameCult.Caching/CultElementIds.cs' +``` + +A clean single run of that command tested 243 mutants and scored 86.79% in +9 minutes 55 seconds. The first measurement of it overlapped two Stryker +runs, so its numbers (and the 114 survivors it showed for `CultElementIds.cs`, +where tests that kill a mutant by hand reported `Survived` or `Timeout`) are +not measurements and are not quoted here. Never run two Stryker runs against +one checkout at once, and check `ps` before starting one. A clean single run +on `CultElementIds.cs` alone, after the tests below, tested 228 mutants: 208 +`Killed`, 8 `Timeout`, 12 `Survived`, a score of 94.74%. The 12th survivor +was 321, which the throwing-enumerator case in `ListsOfValuesHoldNoIds` now +kills; the other 11 are equivalent mutants. With that case a clean single run +on `CultElementIds.cs` is 209 `Killed`, 8 `Timeout`, 11 `Survived`, 95.18%. + +Real gaps that run exposed, each now pinned by `ElementIdWalkTests` (and each +failing when its mutant is applied by hand): the order of the id path (the +codec refuses at the first type MessagePack cannot allow, so order is what an +error names first; `Append` to `Prepend` at 163, 167 and 180); the +union-cycle guard in `ConcreteTypes` (248; the mutant is a stack overflow, so +the red is a crashed test host: a depth guard would need production code in +`ConcreteTypes`, since the recursion is private and a stack overflow cannot be +caught by the test, so the regression test is left to crash loudly); a list of values reporting that it holds an +id (`Holds`, 322); `[Key]` read through an override (`KeyOf`, 583); and the +`IsWrittenWhole` and `IsKeyed` predicates (629 to 631 and 636: formatter, +`[MessagePackObject]` and `KeyAsPropertyName` inherited from a base, and the +`[DataContract]` without `[MessagePackObject]` and `||` arms). + +Equivalent survivors in `CultElementIds.cs` (11), each hand-checked against the full +suite; none can be told apart by any test: + +- 142, 251, 636 (second arm), 506 (`IsDefined(UnionAttribute, false)` to + `true`): `UnionAttribute` is `Inherited = false`, so its `inherit` argument + changes nothing. Likewise 630's `DataContractAttribute` `inherit` argument + (`DataContractAttribute` is `Inherited = false`). +- 264, 542: `group.First()` to `FirstOrDefault()` over `GroupBy` groups, which + are never empty. +- 491: `new StringBuilder(bytes.Length * 2)` to `/ 2` changes only the initial + capacity. +- 578: removing the `Nullable.GetUnderlyingType` unwrap in `WalksAsOwner`; + `Classify` unwraps nullables itself, so the unwrap is dead. A cut + candidate, not a test gap. +- 645, 657: `Append(type)` to `Prepend(type)` in `ElementTypeOf` and + `MapTypes`; `type` matches only when it is itself `IEnumerable<>`, + `IDictionary<,>` or `IReadOnlyDictionary<,>`, and none of those has a second + matching interface, so the position of `type` never decides the answer. + +The Windows-only P/Invoke group is exercised only on Windows: its mutants are +`NoCoverage` on Linux and are a target for a Windows-leg mutation run, not a +Linux gap or a finding. `StoreReplace.cs` line 45 is a different case: its +negated `IsOSPlatform(Windows)` mutant is `Survived` on Linux, because the +negated branch calls `ReplaceWindows`, whose `DllNotFound` fallback gives the +same result. + +`GameCult.Mesh` (+ `.Quic`, `.Quic.Native`), `GameCult.Networking` (+ `.WebSockets`), and the non-.NET runtimes (`cultcache-rs/py/ts`, `cultmesh-*`, `cultnet-*`) each still need their own `stryker-config.json` (or ecosystem-equivalent tool: `cargo-mutants` for diff --git a/tests/GameCult.Caching.Tests/ElementIdWalkTests.cs b/tests/GameCult.Caching.Tests/ElementIdWalkTests.cs new file mode 100644 index 00000000..bd6cb903 --- /dev/null +++ b/tests/GameCult.Caching.Tests/ElementIdWalkTests.cs @@ -0,0 +1,151 @@ +#nullable enable +using System; +using System.Collections.Generic; +using System.Linq; +using System.Runtime.Serialization; +using GameCult.Caching; +using MessagePack; +using NUnit.Framework; + +namespace GameCult.Caching.Tests +{ + // The id walk's own contracts, one mutant-sized fact at a time: the order of the id path, a union that names itself, a list of + // values never counting as an id, and the attribute predicates that decide whether a type is walked or written whole. + public class ElementIdWalkTests + { + // The codec asks MessagePack about the id path in order and refuses at the first type it cannot allow, so the order is what + // an error names first: the document, then the collection types from the outermost in, then the element and its arms. + [Test] + public void TheIdPathListsCollectionsFromTheOutermostIn() + { + Assert.That(CultElementIds.Inspect(typeof(MsMapMap)).IdPath, Is.EqualTo(new[] + { + typeof(MsMapMap), typeof(Dictionary>>), + typeof(Dictionary>), typeof(List), typeof(ElementIdEngineTests.EIded) + }), "maps nested in maps"); + Assert.That(CultElementIds.Inspect(typeof(MsListDeep)).IdPath, Is.EqualTo(new[] + { + typeof(MsListDeep), typeof(List>>), typeof(List>), + typeof(List), typeof(ElementIdEngineTests.EIded) + }), "lists nested in lists"); + } + + // ConcreteTypes stops at a type it has already expanded, so a union that lists its own base is finished, not a stack overflow. + [Test] + public void AUnionThatNamesItselfIsWalkedOnce() + { + var survey = CultElementIds.Inspect(typeof(MsSelfHolder)); + Assert.That(survey.Problems, Is.Empty); + Assert.That(survey.IdPath, Is.EquivalentTo(new[] { typeof(MsSelfHolder), typeof(List), typeof(MsSelfBase), typeof(MsSelfArm) })); + } + + // Value lists hold no ids however they are nested, and a root that is itself a value list holds none either. + [Test] + public void ListsOfValuesHoldNoIds() + { + var values = new MsValues { Numbers = { 1 }, Tags = { "a" }, Grid = { new List { 1 } }, Map = { ["k"] = new List { 2 } } }; + Assert.That(CultElementIds.Holds(typeof(MsValues), values), Is.False); + Assert.That(CultElementIds.Holds(typeof(List), new List { 1, 2 }), Is.False); + Assert.That(CultElementIds.Holds(typeof(List>), new List> { new() { 1 } }), Is.False); + Assert.That(CultElementIds.Holds(typeof(List), new List { "a" }), Is.False); + Assert.That(CultElementIds.Holds(typeof(MsBoom), new MsBoom()), Is.False, "a sequence of values is not enumerated"); + } + + // A nullable struct is walked as the struct it wraps; a nullable value is not walked at all. + [Test] + public void ANullableKeyedStructWalksAsItsOwner() + { + Assert.That(CultElementIds.WalksAsOwner(typeof(MsKeyedStruct)), Is.True); + Assert.That(CultElementIds.WalksAsOwner(typeof(MsKeyedStruct?)), Is.True); + Assert.That(CultElementIds.WalksAsOwner(typeof(int?)), Is.False); + Assert.That(CultElementIds.WalksAsOwner(typeof(int)), Is.False); + } + + // [Key] is read through an override, so an element whose id property overrides a keyed base member is still an id with a key. + [Test] + public void AKeyOnTheBaseOfAnOverrideCountsForTheOverride() + { + Assert.That(CultElementIds.Inspect(typeof(MsOverrideHolder)).Problems, Is.Empty); + Assert.That(CultElementIds.Inspect(typeof(MsOverrideHolder)).IdPath, Does.Contain(typeof(MsOverrideEl))); + } + + // A type is walked by key when it is [MessagePackObject] or [Union], read through its bases for the former; it is one whole + // value when it declares its own formatter, is [DataContract] without being [MessagePackObject], or is keyed by name. + [Test] + public void KeyedTypesWalkAndWholeTypesDoNot() + { + Assert.That(CultElementIds.WalksAsOwner(typeof(MsMpBase)), Is.True, "[MessagePackObject]"); + Assert.That(CultElementIds.WalksAsOwner(typeof(MsUBase)), Is.True, "[Union]"); + Assert.That(CultElementIds.WalksAsOwner(typeof(MsMpDerived)), Is.True, "[MessagePackObject] inherited"); + Assert.That(CultElementIds.WalksAsOwner(typeof(MsPlain)), Is.False, "[Key] members without [MessagePackObject] or [Union]"); + Assert.That(CultElementIds.WalksAsOwner(typeof(MsUArm)), Is.True, "an arm that is itself [MessagePackObject]"); + + Assert.That(CultElementIds.WalksAsOwner(typeof(MsDc)), Is.False, "[DataContract] alone"); + Assert.That(CultElementIds.WalksAsOwner(typeof(MsDcMp)), Is.True, "[DataContract] beside [MessagePackObject]"); + Assert.That(CultElementIds.WalksAsOwner(typeof(MsDcOverMpBase)), Is.True, "[DataContract] over an inherited [MessagePackObject]"); + + Assert.That(CultElementIds.WalksAsOwner(typeof(MsPropName)), Is.False, "KeyAsPropertyName"); + Assert.That(CultElementIds.WalksAsOwner(typeof(MsPropNameDerived)), Is.False, "KeyAsPropertyName inherited"); + Assert.That(CultElementIds.WalksAsOwner(typeof(MsStringKey)), Is.False, "a string [Key]"); + + Assert.That(CultElementIds.WalksAsOwner(typeof(MsFormattedBase)), Is.False, "[MessagePackFormatter]"); + Assert.That(CultElementIds.WalksAsOwner(typeof(MsFormattedDerived)), Is.False, "[MessagePackFormatter] inherited"); + } + + [MessagePackObject] public class MsMapMap { [Key(0)] public Dictionary>> M { get; set; } = new(); } + [MessagePackObject] public class MsListDeep { [Key(0)] public List>> G { get; set; } = new(); } + + [Union(0, typeof(MsSelfBase)), Union(1, typeof(MsSelfArm))] public abstract class MsSelfBase { } + [MessagePackObject] public class MsSelfArm : MsSelfBase { [Key(0)] [CultElementId] public string Id { get; set; } = ""; } + [MessagePackObject] public class MsSelfHolder { [Key(0)] public List Items { get; set; } = new(); } + + [MessagePackObject] + public class MsValues + { + [Key(0)] public List Numbers { get; set; } = new(); + [Key(1)] public List Tags { get; set; } = new(); + [Key(2)] public List> Grid { get; set; } = new(); + [Key(3)] public Dictionary> Map { get; set; } = new(); + } + + public sealed class MsBoom : IEnumerable + { + public IEnumerator GetEnumerator() => throw new InvalidOperationException("enumerated"); + System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator() => GetEnumerator(); + } + + [MessagePackObject] public struct MsKeyedStruct { [Key(0)] public int X; } + + [MessagePackObject] + public class MsOverrideBase + { + [Key(0)] [CultElementId] public virtual string Id { get; set; } = ""; + } + + [MessagePackObject] + public class MsOverrideEl : MsOverrideBase + { + public override string Id { get; set; } = ""; + } + + [MessagePackObject] public class MsOverrideHolder { [Key(0)] public List Items { get; set; } = new(); } + + [MessagePackObject] public class MsMpBase { [Key(0)] public int A { get; set; } } + public class MsMpDerived : MsMpBase { } + public class MsPlain { [Key(0)] public int A { get; set; } } + [Union(0, typeof(MsUArm))] public abstract class MsUBase { } + [MessagePackObject] public class MsUArm : MsUBase { [Key(0)] public int A { get; set; } } + + [DataContract] public class MsDc { [DataMember] public int A { get; set; } } + [DataContract] [MessagePackObject] public class MsDcMp { [Key(0)] public int A { get; set; } } + [DataContract] public class MsDcOverMpBase : MsMpBase { } + + [MessagePackObject(keyAsPropertyName: true)] public class MsPropName { public int A { get; set; } } + public class MsPropNameDerived : MsPropName { } + [MessagePackObject] public class MsStringKey { [Key("a")] public int A { get; set; } } + + public sealed class MsNoFormatter { } + [MessagePackFormatter(typeof(MsNoFormatter))] [MessagePackObject] public class MsFormattedBase { [Key(0)] public int A { get; set; } } + [MessagePackObject] public class MsFormattedDerived : MsFormattedBase { [Key(1)] public int B { get; set; } } + } +} diff --git a/tests/GameCult.Caching.Tests/stryker-config.json b/tests/GameCult.Caching.Tests/stryker-config.json new file mode 100644 index 00000000..ac366203 --- /dev/null +++ b/tests/GameCult.Caching.Tests/stryker-config.json @@ -0,0 +1,11 @@ +{ + "stryker-config": { + "project": "GameCult.Caching.csproj", + "mutate": [ + "**/src/GameCult.Caching/**/*.cs" + ], + "mutation-level": "Standard", + "reporters": [ "progress", "html", "json" ], + "thresholds": { "high": 80, "low": 60, "break": 0 } + } +}