Mesh single-file store: tests for dangling link, directory, zero byte, lock, concurrent replace and the older layout; contract sentence

Behavioural tests through Mesh and CultCache for each rule the cut owns, and the persistence
contract's single-readability paragraph states that one runtime has one reader, lock and replace
and that only nothing at the path is missing. Not yet run: verification follows in one job.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

src/GameCult.Caching/Contracts/cultcache-persistence-format.md

@@ -298,8 +298,11 @@ array (a second store appended included), is not an array, has fewer or more tha
 (a header alone included), whose first slot is not a header it reads, or whose catalog or
 records it cannot decode (a record with more slots than its header allows, a variant record
 it does not read). A runtime never skips a slot it does not understand. It refuses on open
-and on rewrite alike, because the rewrite asks the same full reader as the open. A missing
-file and an empty array are an empty store; a zero-byte file is not a store and is refused as unreadable. A flush or a commit replaces a file only when the
+and on rewrite alike, because the rewrite asks the same full reader as the open. Only nothing at the path is a missing
+file, and a missing file and an empty array are an empty store; a zero-byte file is not a store and is refused as unreadable.
+In C#, a link whose target is gone and a directory at the path are I/O errors, never an empty store. Within a runtime, every
+reader and writer of a single-file store shares that one reader, lock and replace (in C#, CultMesh's single-file document
+helpers included), and a catalog entry has one layout. A flush or a commit replaces a file only when the
 runtime's own reader opens it, so one verdict covers open, flush and commit, and a file the
 runtime cannot read is left as it is. Each runtime refuses with one typed error:
 `CultStoreUnreadableException` (C#), `CultCacheStoreUnreadable{path, kind}` (Rust, its kind

tests/GameCult.Caching.Tests/StoreReadabilityTests.cs

@@ -230,6 +230,27 @@ namespace GameCult.Caching.Tests
             Assert.That(File.Exists(target), Is.False, "a write went through the link");
         }
 
+        // A directory at the path is something at the path: it is an I/O error on open and on commit, never an empty store, and
+        // it is left as it was.
+        [Test]
+        public void ADirectoryAtTheStorePathIsAnIoErrorOnOpenAndOnCommitNotAnEmptyStore()
+        {
+            var path = Path.Combine(_directory, "directory.cc");
+            Directory.CreateDirectory(path);
+
+            Assert.That(() => CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = Registry }),
+                Throws.TypeOf<IOException>().With.Message.Contains("directory"));
+
+            var later = Path.Combine(_directory, "later.cc");
+            using var cache = CultCacheMessagePack.Create(later, new CultCacheOpenOptions { Registry = Registry });
+            Directory.CreateDirectory(later);
+            Assert.That(() => cache.Commit(batch => batch.Upsert(typeof(IdDeck), new IdDeck { Name = "e" }, new CultRecordKey("e"))),
+                Throws.TypeOf<IOException>().With.Message.Contains("directory"));
+
+            Assert.That(Directory.GetFileSystemEntries(path), Is.Empty);
+            Assert.That(Directory.GetFileSystemEntries(later), Is.Empty);
+        }
+
         // A link whose target is gone. A directory junction needs no privilege on Windows, where a file symlink needs one this
         // workstation lacks; elsewhere a symlink to a missing directory is the same thing.
         private static void CreateDanglingDirectoryLink(string path, string target)

tests/GameCult.Mesh.Tests/CultMeshSingleFileStoreTests.cs

@@ -0,0 +1,305 @@
+using System;
+using System.Collections.Concurrent;
+using System.Collections.Generic;
+using System.Diagnostics;
+using System.IO;
+using System.Linq;
+using System.Threading;
+using System.Threading.Tasks;
+using FluentAssertions;
+using GameCult.Caching;
+using GameCult.Caching.MessagePack;
+using NUnit.Framework;
+
+#nullable enable
+
+namespace GameCult.Mesh.Tests;
+
+// CultMesh's single-file helpers reach the store file through the one CultCache single-file store: what is at the path, the lock
+// and the atomic replace are decided there, once, for a Mesh read or write and a cache open, flush or commit alike.
+public sealed class CultMeshSingleFileStoreTests
+{
+    private static readonly CultRecordKey Key = new("publication");
+    private static readonly CultMeshSingleFileDocumentSchema Raw = new("raw:schema", "RawSchema", "1");
+    private static readonly byte[] EmptyArray = { 0x90 };
+    private static readonly CultCacheOpenOptions ReadOnlyOpen = new() { FlushOnDispose = false, StoreFlushOnDispose = false };
+    private string _root = null!;
+
+    [SetUp]
+    public void SetUp()
+    {
+        _root = Path.Combine(Path.GetTempPath(), "cultmesh-single-file-store", Guid.NewGuid().ToString("N"));
+        Directory.CreateDirectory(_root);
+    }
+
+    [TearDown]
+    public void TearDown()
+    {
+        if (!Directory.Exists(_root)) return;
+        // A recursive delete cannot remove a link whose target is gone; take the links out by themselves first.
+        foreach (var entry in Directory.EnumerateFileSystemEntries(_root).Where(IsALink))
+        {
+            if ((new FileInfo(entry).Attributes & FileAttributes.Directory) != 0) Directory.Delete(entry, false);
+            else File.Delete(entry);
+        }
+
+        Directory.Delete(_root, true);
+    }
+
+    private static CultMeshBodyPublicationDocument Publication(string producer = "aetheria") => new()
+    {
+        BodyId = "aetheria:entities",
+        ProducerId = producer,
+        SchemaId = "eve.entity_soa.v1"
+    };
+
+    private static string SchemaId() => CultDocumentRegistry.Shared.GetRequired<CultMeshBodyPublicationDocument>().SchemaId;
+
+    private static bool IsALink(string path)
+    {
+        var attributes = new FileInfo(path).Attributes;
+        return (int)attributes != -1 && (attributes & FileAttributes.ReparsePoint) != 0;
+    }
+
+    private static int RecordsSeenByACache(string path)
+    {
+        using var cache = CultCacheMessagePack.Create(path, ReadOnlyOpen);
+        return cache.GetAll<CultMeshBodyPublicationDocument>().Count();
+    }
+
+    private IEnumerable<string> TempFiles() =>
+        Directory.GetFileSystemEntries(_root).Where(entry => entry.EndsWith(".tmp", StringComparison.Ordinal));
+
+    // A link whose target is gone: a symlink where one can be made, a directory junction on Windows, which needs no privilege.
+    private static void CreateDanglingLink(string path, string target)
+    {
+        if (!OperatingSystem.IsWindows())
+        {
+            File.CreateSymbolicLink(path, target);
+            return;
+        }
+
+        Directory.CreateDirectory(target);
+        var mklink = Process.Start(new ProcessStartInfo("cmd.exe", $"/c mklink /J \"{path}\" \"{target}\"")
+        {
+            CreateNoWindow = true,
+            RedirectStandardOutput = true,
+            RedirectStandardError = true,
+        })!;
+        mklink.WaitForExit();
+        Assert.That(mklink.ExitCode, Is.Zero, "mklink /J failed");
+        Directory.Delete(target);
+    }
+
+    private static void CreateLiveLinkOrIgnore(string path, string target)
+    {
+        try
+        {
+            File.CreateSymbolicLink(path, target);
+        }
+        catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
+        {
+            Assert.Ignore("A file symlink needs a privilege this account lacks: " + ex.Message);
+        }
+    }
+
+    [Test]
+    public void ADanglingLinkIsAnIoErrorOnTypedWriteRawWriteAndReadNeverAMissingFile()
+    {
+        var path = Path.Combine(_root, "dangling.cc");
+        var target = Path.Combine(_root, "unmounted", "store.cc");
+        CreateDanglingLink(path, target);
+
+        var operations = new (string Name, Action Run)[]
+        {
+            ("typed write", () => CultMesh.WriteSingleFileDocument(path, Key, Publication())),
+            ("raw write", () => CultMesh.WriteSingleFileDocumentPayload(path, Key, Raw, null, EmptyArray)),
+            ("payload read", () => CultMesh.ReadSingleFileDocumentPayload(path, Key, "raw:schema")),
+            ("keyless payload read", () => CultMesh.ReadSingleFileDocumentPayload(path, "raw:schema")),
+            ("typed read", () => CultMesh.ReadSingleFileDocument<CultMeshBodyPublicationDocument>(path, Key)),
+        };
+        foreach (var (name, run) in operations)
+        {
+            var error = Assert.Catch<IOException>(() => run(), name)!;
+            error.Should().NotBeOfType<FileNotFoundException>(name + " reported a dangling link as a missing file");
+        }
+
+        IsALink(path).Should().BeTrue("the link was replaced");
+        Directory.Exists(Path.GetDirectoryName(target)).Should().BeFalse("a write went through the link");
+        TempFiles().Should().BeEmpty();
+    }
+
+    [Test]
+    public void OnlyNothingAtThePathIsAMissingFile()
+    {
+        var path = Path.Combine(_root, "missing.cc");
+        var missing = Assert.Throws<FileNotFoundException>(() => CultMesh.ReadSingleFileDocumentPayload(path, Key, "raw:schema"))!;
+        missing.FileName.Should().Be(path);
+        Assert.Throws<FileNotFoundException>(() => CultMesh.ReadSingleFileDocumentPayload(path, "raw:schema"));
+        Assert.Throws<FileNotFoundException>(() => CultMesh.ReadSingleFileDocument<CultMeshBodyPublicationDocument>(path, Key));
+        File.Exists(path).Should().BeFalse("a read created the file");
+
+        // A write to nothing at the path creates the store, and the reader reads it back.
+        CultMesh.WriteSingleFileDocumentPayload(path, Key, Raw, null, EmptyArray);
+        CultMesh.ReadSingleFileDocumentPayload(path, Key, "raw:schema").Should().Equal(EmptyArray);
+    }
+
+    [Test]
+    public void ADirectoryAtThePathIsAnIoErrorOnReadAndWriteAndIsLeftAlone()
+    {
+        var path = Path.Combine(_root, "directory.cc");
+        Directory.CreateDirectory(path);
+
+        var operations = new (string Name, Action Run)[]
+        {
+            ("typed write", () => CultMesh.WriteSingleFileDocument(path, Key, Publication())),
+            ("raw write", () => CultMesh.WriteSingleFileDocumentPayload(path, Key, Raw, null, EmptyArray)),
+            ("payload read", () => CultMesh.ReadSingleFileDocumentPayload(path, Key, "raw:schema")),
+            ("keyless payload read", () => CultMesh.ReadSingleFileDocumentPayload(path, "raw:schema")),
+        };
+        foreach (var (name, run) in operations)
+        {
+            var error = Assert.Catch<IOException>(() => run(), name)!;
+            error.Should().NotBeOfType<FileNotFoundException>(name);
+            error.Message.Should().Contain("directory", name);
+            error.Message.Should().NotContain(_root, name + " echoed the path in its refusal text");
+        }
+
+        Directory.Exists(path).Should().BeTrue();
+        Directory.GetFileSystemEntries(path).Should().BeEmpty();
+        TempFiles().Should().BeEmpty();
+    }
+
+    [Test]
+    public void AZeroByteFileIsRefusedOnTypedAndRawWriteAndLeftAlone()
+    {
+        var path = Path.Combine(_root, "zero.cc");
+        File.WriteAllBytes(path, Array.Empty<byte>());
+
+        Assert.Throws<CultStoreUnreadableException>(() => CultMesh.WriteSingleFileDocument(path, Key, Publication()));
+        Assert.Throws<CultStoreUnreadableException>(() => CultMesh.WriteSingleFileDocumentPayload(path, Key, Raw, null, EmptyArray));
+        Assert.Throws<CultStoreUnreadableException>(() => CultMesh.ReadSingleFileDocument<CultMeshBodyPublicationDocument>(path, Key));
+        new FileInfo(path).Length.Should().Be(0);
+    }
+
+    [Test]
+    public void AMeshWriteWaitsForTheStoreLock()
+    {
+        var path = Path.Combine(_root, "locked.cc");
+        FileStream? held = new FileStream(path + ".lock", FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.None);
+        try
+        {
+            var write = Task.Run(() => CultMesh.WriteSingleFileDocumentPayload(path, Key, Raw, null, EmptyArray));
+
+            write.Wait(TimeSpan.FromMilliseconds(300)).Should().BeFalse("a Mesh write went ahead of the store lock");
+            File.Exists(path).Should().BeFalse("a Mesh write landed while the lock was held");
+
+            held.Dispose();
+            held = null;
+            write.Wait(TimeSpan.FromSeconds(10)).Should().BeTrue("the write never completed after the lock was released");
+            write.IsCompletedSuccessfully.Should().BeTrue();
+        }
+        finally
+        {
+            held?.Dispose();
+        }
+
+        CultMesh.ReadSingleFileDocumentPayload(path, Key, "raw:schema").Should().Equal(EmptyArray);
+    }
+
+    [Test]
+    public void ACacheOpeningDuringMeshWritesNeverSeesAnEmptyStore()
+    {
+        var path = Path.Combine(_root, "race.cc");
+        CultMesh.WriteSingleFileDocument(path, Key, Publication());
+        const int writes = 500;
+        var stop = 0;
+        var errors = new ConcurrentQueue<string>();
+        var writer = Task.Run(() =>
+        {
+            for (var index = 0; index < writes; index++)
+            {
+                try { CultMesh.WriteSingleFileDocument(path, Key, Publication()); }
+                catch (Exception ex) { errors.Enqueue("write: " + ex.GetType().Name + ": " + ex.Message); }
+            }
+
+            Volatile.Write(ref stop, 1);
+        });
+
+        int opens = 0, empty = 0, reads = 0;
+        var clock = Stopwatch.StartNew();
+        while (Volatile.Read(ref stop) == 0 && clock.Elapsed < TimeSpan.FromSeconds(60))
+        {
+            try
+            {
+                opens++;
+                if (RecordsSeenByACache(path) == 0) empty++;
+                CultMesh.ReadSingleFileDocumentPayload(path, Key, SchemaId());
+                reads++;
+            }
+            catch (Exception ex)
+            {
+                errors.Enqueue("read: " + ex.GetType().Name + ": " + ex.Message);
+            }
+        }
+
+        writer.Wait(TimeSpan.FromSeconds(60)).Should().BeTrue();
+        TestContext.Out.WriteLine($"mesh writes={writes} cache opens={opens} empty stores={empty} mesh reads ok={reads} errors={errors.Count}");
+        errors.Take(3).Should().BeEmpty();
+        empty.Should().Be(0, "a cache opened in the window where nothing was at the path");
+        opens.Should().BeGreaterThan(0);
+    }
+
+    [Test]
+    public void ConcurrentMeshWritersNeverErrorAndLeaveAReadableFile()
+    {
+        var path = Path.Combine(_root, "writers.cc");
+        const int writers = 4, each = 100;
+        var errors = new ConcurrentQueue<string>();
+        var tasks = Enumerable.Range(0, writers).Select(writer => Task.Run(() =>
+        {
+            for (var index = 0; index < each; index++)
+            {
+                try { CultMesh.WriteSingleFileDocument(path, Key, Publication("writer-" + writer)); }
+                catch (Exception ex) { errors.Enqueue(ex.GetType().Name + ": " + ex.Message); }
+            }
+        })).ToArray();
+
+        Task.WaitAll(tasks, TimeSpan.FromSeconds(120)).Should().BeTrue();
+        TestContext.Out.WriteLine($"mesh writers={writers} writes each={each} errors={errors.Count}");
+        errors.Take(3).Should().BeEmpty();
+        File.Exists(path).Should().BeTrue("concurrent writers deleted the store");
+        CultMesh.ReadSingleFileDocument<CultMeshBodyPublicationDocument>(path, Key).ProducerId.Should().StartWith("writer-");
+        RecordsSeenByACache(path).Should().Be(1);
+        TempFiles().Should().BeEmpty();
+    }
+
+    // The store's replace decides what a write leaves where a live link was, and a Mesh write leaves what a cache commit leaves:
+    // one replace, whatever the platform does with a link.
+    [Test]
+    public void AMeshWriteOverALiveLinkLeavesWhatACacheCommitLeaves()
+    {
+        string Outcome(string name, Action<string> write)
+        {
+            var target = Path.Combine(_root, name + "-target.cc");
+            var link = Path.Combine(_root, name + "-link.cc");
+            CultMesh.WriteSingleFileDocument(target, Key, Publication("first"));
+            CreateLiveLinkOrIgnore(link, target);
+            var before = File.ReadAllBytes(target);
+
+            write(link);
+
+            CultMesh.ReadSingleFileDocument<CultMeshBodyPublicationDocument>(link, Key).ProducerId.Should().Be("second", name);
+            return $"linkKept={IsALink(link)} targetChanged={!File.ReadAllBytes(target).SequenceEqual(before)}";
+        }
+
+        var viaMesh = Outcome("mesh", link => CultMesh.WriteSingleFileDocument(link, Key, Publication("second")));
+        var viaCache = Outcome("cache", link =>
+        {
+            using var cache = CultCacheMessagePack.Create(link, ReadOnlyOpen);
+            cache.Commit(batch => batch.Upsert(typeof(CultMeshBodyPublicationDocument), Publication("second"), Key));
+        });
+
+        viaMesh.Should().Be(viaCache);
+    }
+}

tests/GameCult.Mesh.Tests/CultMeshVariantRefusalTests.cs

@@ -2,6 +2,7 @@ using System;
 using System.IO;
 using FluentAssertions;
 using GameCult.Caching;
+using GameCult.Caching.MessagePack;
 using NUnit.Framework;
 
 #nullable enable
@@ -38,6 +39,38 @@ public sealed class CultMeshVariantRefusalTests
         message.Should().Contain("item:anvil").And.Contain(ItemSchemaId);
     }
 
+    // The older catalog layout (the content hash at entry slot 5) is a store no C# reader reads: CultMesh refuses it as CultCache
+    // does, on a read and on a raw write, and a refused write leaves the bytes as they were.
+    [TestCase("legacy-catalog-plain.msgpack")]
+    [TestCase("legacy-catalog-extra-slot.msgpack")]
+    [TestCase("legacy-catalog-trailing.msgpack")]
+    public void TheOlderCatalogLayoutIsRefusedByMeshAndCultCacheAlike(string vector)
+    {
+        Refusal(vector).Path.Should().Be(VectorPath(vector));
+
+        var directory = Path.Combine(Path.GetTempPath(), "cultmesh-legacy-layout", Guid.NewGuid().ToString("N"));
+        Directory.CreateDirectory(directory);
+        try
+        {
+            var path = Path.Combine(directory, vector);
+            File.Copy(VectorPath(vector), path);
+            var bytes = File.ReadAllBytes(path);
+            var raw = new CultMeshSingleFileDocumentSchema("raw:schema", "RawSchema", "1");
+
+            Assert.Throws<CultStoreUnreadableException>(() =>
+                CultMesh.WriteSingleFileDocumentPayload(path, new CultRecordKey("item:bellows"), raw, null, new byte[] { 0x90 }))!
+                .Path.Should().Be(path);
+            File.ReadAllBytes(path).Should().Equal(bytes);
+
+            Assert.Throws<CultStoreUnreadableException>(() => CultCacheMessagePack.Create(path, new CultCacheOpenOptions { FlushOnDispose = false, StoreFlushOnDispose = false }))!
+                .Path.Should().Be(path);
+        }
+        finally
+        {
+            Directory.Delete(directory, true);
+        }
+    }
+
     // A store in the current layout with one catalog slot malformed is refused as CultCache refuses it.
     [TestCase("current-catalog-bad-canonical.msgpack")]
     [TestCase("current-catalog-bad-members.msgpack")]

tests/vectors/document-variants-c0/generate.py

@@ -6,18 +6,16 @@ at test time. Run: python generate.py  (needs msgpack).
   extra-slot-full-payload.msgpack v1 header, record item:anvil gains slot 4, payload stays full
   legacy-catalog-extra-slot.msgpack
                                   v1 header, catalog entry in the older layout (contentHash at slot
-                                  5), so CultMesh's legacy snapshot reader takes it, plus the
-                                  extra-slot record
-  legacy-catalog-plain.msgpack    v1 header, the older-layout catalog entry, the plain v1 records: a store CultMesh's
-                                  legacy reader accepts
-  legacy-catalog-trailing.msgpack legacy-catalog-plain plus three bytes after the array: one array is
-                                  required of the legacy layout too
+                                  5), plus the extra-slot record; no C# reader reads the older layout
+  legacy-catalog-plain.msgpack    v1 header, the older-layout catalog entry, the plain v1 records: refused by
+                                  every C# reader, CultMesh included
+  legacy-catalog-trailing.msgpack legacy-catalog-plain plus three bytes after the array
   current-catalog-bad-canonical.msgpack
   current-catalog-bad-members.msgpack
   current-catalog-bad-compat.msgpack
                                   v1 header, plain records, and the first catalog entry of the current layout with one slot
-                                  malformed (canonical schema json an int; members an int; a compatible id an int). Not the
-                                  older layout, so CultMesh refuses them as CultCache does
+                                  malformed (canonical schema json an int; members an int; a compatible id an int). Refused
+                                  by every C# reader, CultMesh included
   current-catalog-bad-slot5-nil.msgpack
   current-catalog-bad-slot5-int.msgpack
                                   the same, with slot 5 (the compatible ids) nil or an integer: neither is a string, so neither is