C# write set: tests and contract text for the refusal that reloads and the header-only staleness rule

The merge tests become the refuse, reload, retry rule; new tests pin copied records that
cannot be decoded, the later storedAt over a replaced record, a batch that removes a staged
key, and a Mesh write into a variant store. The persistence and composition contracts state
the rule for the parity cuts.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Meta Vens 2026-10-03 05:11:17 +02:00
commit cc8604ae94
6 changed files with 283 additions and 35 deletions

View file

@ -58,15 +58,37 @@ applies only what it staged (writes and removals) and, for a commit, its batch,
and replaces the store with the result. Every other record, and every catalog
entry the result references, is copied forward as stored: key, schema id,
storedAt, payload and variant, and a catalog entry's exact bytes, unknown
fields included. No type interprets a copied record. A plain flush and an
unconditional commit write the same bytes and are last-writer-wins only per
staged key; a document changed in place and not staged is not written. An
fields included. No type interprets a copied record: a write reads other
records' headers (key, schema id, storedAt, variant slot), never their payloads.
A record written over one the store holds gets a storedAt later than the one it
replaces. A document changed in place and not staged is not written.
**In a store that holds no variant**, a plain flush and an unconditional commit
write the same bytes and are last-writer-wins only per staged key. An
unconditional commit also persists writes staged earlier and leaves the store
clean. Conditional commit is how a writer learns that another writer moved the
keys it depends on. A writer refuses a file whose header it cannot read
(truncated, or a format it does not read) and leaves the file as it was. C#
applies this to single-file stores, directory stores and CultMesh's single-file
document writes; Rust, TypeScript and Python rewrite the store from their own copy until their
clean. Conditional commit is how such a writer learns that another writer moved
the keys it depends on.
**A store that holds a variant, or would after the write, is written only from
what its writer last read.** A variant resolves against other records (its base
chain, and the index values it shares the store with). So before writing, the
writer compares every record header in the store with the headers it last read
or wrote. If any record moved, appeared or disappeared, the whole write is
refused with the typed write conflict, naming the keys it would have written and
the keys that moved, and nothing is written. A writer that has read nothing,
such as CultMesh's single-file document write, never writes into a store holding
a variant.
**A refused write reloads what it refused.** A refusal (a foreign record at a
key the write names, or a write conflict) reloads the durable view of every
refused and moved key into the cache, as a pull would, and only then forgets the
staged changes at those keys. The cache then serves what the store holds, and a
retry is judged against that. If the reload is itself refused (the store holds a
record this cache cannot load), nothing changes and the refusal carries that
cause. A writer refuses a file whose header it cannot read (truncated, or a
format it does not read) and leaves the file as it was. C# applies this to
single-file stores, directory stores and CultMesh's single-file document writes;
Rust, TypeScript and Python rewrite the store from their own copy until their
parity cuts land.
A store that fails to hydrate on open (corrupt bytes, unresolvable schema)
@ -160,7 +182,7 @@ A record that resolves to no local type is a foreign record, for example one of
have, or one renamed without declaring its old id. A store never destroys or relabels it: the store carries it
byte for byte under its own id and its own catalog entry, lists it as foreign (`CacheBackingStore.ForeignRecords`
in C#), and the cache never holds it. Writes of other records proceed and copy it forward as the file holds it. A write that would replace or remove it is refused with the typed schema conflict
naming its key and id, and nothing is written. A marked header stays marked while a store carries a record the writer cannot show to hold no element id. Declaring its id on a type claims it at the next load.
naming its key and id, and nothing is written. The refused keys are reloaded as the store holds them (a foreign record is not held), and their staged changes are forgotten. A marked header stays marked while a store carries a record the writer cannot show to hold no element id. Declaring its id on a type claims it at the next load.
A reader resolves a record's schema id to the entry that owns it, that entry's own `schemaId`, and
only when no entry owns the id to an entry that lists it as a compatible id (the first such entry
@ -525,10 +547,12 @@ The v1 concurrent single-file policy is:
2. Writers take an exclusive sidecar lock derived from the `.cc` path.
3. Writers re-read the current snapshot after taking the lock.
4. Writers using conditional commit evaluate their conditions against the
latest snapshot and write nothing if one fails. A plain flush and an
unconditional commit compare nothing: they apply their staged writes onto the
current snapshot, last-writer-wins per staged key; one whose header they
cannot read is refused untouched.
latest snapshot and write nothing if one fails. In a store with no variant, a
plain flush and an unconditional commit compare nothing: they apply their
staged writes onto the current snapshot, last-writer-wins per staged key. A
write into a store that holds or would hold a variant compares every record
header with what the writer last read, and is refused if any moved. One whose
header they cannot read is refused untouched.
5. Writers write a temp file, flush it, atomically replace the `.cc` file, and
release the lock.

View file

@ -248,8 +248,8 @@ replaces; writers bump a minted timestamp by one tick when it is not later.
match its content address, at most five times; a pull that has not settled
by then throws.
**A plain flush and an unconditional commit are last-writer-wins per staged
key, in both store kinds.** Both run under the same lock, so two writers never
**In a store holding no variant, a plain flush and an unconditional commit are
last-writer-wins per staged key, in both store kinds.** Both run under the same lock, so two writers never
interleave bytes, but they compare nothing. Only a conditional commit
(`Expect` or `ExpectUnchanged`) tells a writer another writer moved the keys it
depends on. A writer wins only over a file it can read: a single-file store
@ -258,6 +258,10 @@ refuses to replace a file whose header it cannot read, and leaves it as it was.
- A single-file store and a directory store apply this cache's staged writes and
removals onto what the store holds now: another writer's unrelated records
survive as stored, a key both wrote holds the last write.
- A single-file store holding a variant (the directory store refuses variants)
is written only from what its writer last read. A record that moved refuses
the whole write with the typed write conflict, and the refused and moved keys
are reloaded (cultcache-persistence-format.md).
- An unconditional commit writes exactly what a flush of the same staged state
plus the batch would write. It also persists any single writes staged earlier
in that store, and leaves the store clean.

View file

@ -665,11 +665,11 @@ namespace GameCult.Caching.Tests
Assert.That(reopened.Get(other), Is.Not.Null);
}
// A write lands on the file as it is now, so it is judged on the set the file will hold: the variants another writer left in it
// and this cache's own staged edits. Another writer adds a variant whose indexed value this cache's staged edit of its base
// now takes.
// A store holding a variant is written only from what its writer last read. Another writer adds a variant whose indexed value
// this cache's staged edit of its base now takes: the write is refused whole, the cache reloads what it refused and what moved,
// and the retry is judged by the cache's own rules.
[Test]
public void AWriteIsJudgedOnTheStagedEditsAndTheVariantsAnotherWriterLeftInTheFile([Values] bool viaCommit)
public void AWriteIntoAVariantStoreAnotherWriterChangedIsRefusedReloadedAndJudgedOnRetry([Values] bool viaCommit)
{
var path = PathOf(viaCommit ? "judged-commit.cc" : "judged-flush.cc");
using var cache = Open(path);
@ -679,18 +679,77 @@ namespace GameCult.Caching.Tests
var edited = Laser();
edited.Code = "laser big";
cache.UpsertAsync(typeof(VariantGear), edited, BaseKey).GetAwaiter().GetResult();
var before = File.ReadAllBytes(path);
var refusal = Refused(() =>
var conflict = Assert.Throws<CultWriteConflictException>(() =>
{
if (viaCommit)
cache.Commit(batch => batch.Upsert(typeof(VariantGear), Other("n"), new CultRecordKey("n")));
else
cache.FlushAllBackingStores();
});
})!;
Assert.That(refusal.Message, Does.Contain(BigKey.Value).And.Contain(BaseKey.Value));
using var check = Open(path);
Assert.That(check.Get<VariantGear>(BaseKey)!.Code, Is.EqualTo("l1"), "the refused edit was not written");
Assert.That(conflict.RecordKeys, Does.Contain(BaseKey.Value));
Assert.That(conflict.ChangedKeys, Does.Contain(BigKey.Value));
Assert.That(cache.Get<VariantGear>(BaseKey)!.Code, Is.EqualTo("l1"), "the refused edit is not served");
Assert.That(cache.Get(BigKey), Is.Not.Null, "the moved record is loaded");
Assert.That(cache.IsDirty, Is.False);
Assert.That(File.ReadAllBytes(path), Is.EqualTo(before), "nothing was written");
Assert.That(Refused(() => cache.UpsertAsync(typeof(VariantGear), edited, BaseKey).GetAwaiter().GetResult()).Message,
Does.Contain(BigKey.Value), "the retry is judged by the cache's own index rule");
}
// A refused base removal leaves the cache serving the store, writable, and judging the next attempt itself.
[Test]
public void ARefusedBaseRemovalReloadsAndLeavesTheCacheWritable()
{
var path = PathOf("gear.cc");
var n = new CultRecordKey("n");
using var a = Open(path);
SeedBase(a);
using (var b = Open(path))
SeedBig(b);
a.Remove(BaseKey);
var conflict = Assert.Throws<CultWriteConflictException>(() => a.FlushAllBackingStores())!;
Assert.That(conflict.RecordKeys, Is.EqualTo(new[] { BaseKey.Value }));
Assert.That(conflict.ChangedKeys, Is.EqualTo(new[] { BigKey.Value }));
Assert.That(a.Get(BaseKey), Is.Not.Null);
Assert.That(a.Get(BigKey), Is.Not.Null);
Assert.That(a.IsDirty, Is.False);
Assert.DoesNotThrow(() => a.FlushAllBackingStores());
Assert.DoesNotThrow(() => a.Commit(batch => batch.Upsert(typeof(VariantGear), Named("n", "nn"), n)));
Assert.That(Refused(() => a.Remove(BaseKey)).Message, Does.Contain(BigKey.Value), "a base with variants is refused by this cache's own rule");
using var reopened = Open(path);
Assert.That(reopened.Get(n), Is.Not.Null);
Assert.That(reopened.Get(BigKey), Is.Not.Null);
Assert.That(reopened.Get(BaseKey), Is.Not.Null);
}
// A held record another writer changed is not trusted: the write is refused although this cache never staged it.
[Test]
public void AWriteIsRefusedWhenAnotherWriterChangedAVariantThisCacheHolds()
{
var path = PathOf("gear.cc");
using var a = Open(path);
SeedBase(a);
SeedBig(a);
using (var b = Open(path))
b.Commit(batch => batch.UpsertVariant(BigKey, BaseKey, new[]
{
b.Override<VariantGear>(nameof(VariantGear.Name), "laser big"),
b.Override<VariantGear>(nameof(VariantGear.Power), 99),
b.Override<VariantGear>(nameof(VariantGear.Code), "laser big")
}));
Assert.That(a.Get<VariantGear>(BigKey)!.Power, Is.EqualTo(25));
var conflict = Assert.Throws<CultWriteConflictException>(() =>
a.Commit(batch => batch.Upsert(typeof(VariantGear), Named("y", "yy"), new CultRecordKey("y"))))!;
Assert.That(conflict.ChangedKeys, Is.EqualTo(new[] { BigKey.Value }));
Assert.That(a.Get<VariantGear>(BigKey)!.Power, Is.EqualTo(99), "the cache serves the other writer's override after the refusal");
}
// ---- a write copies what it did not stage, and re-encodes what it stages under the type's own id ----
@ -962,12 +1021,16 @@ namespace GameCult.Caching.Tests
a.Override<VariantGear>(nameof(VariantGear.Code), "shared")
}));
var refused = Refused(() => b.Commit(batch =>
void Write() => b.Commit(batch =>
{
batch.Expect(other, null);
batch.Upsert(typeof(VariantGear), Named("other", "shared"), other);
}));
Assert.That(refused.Message, Does.Contain(other.Value).And.Contain(BigKey.Value).And.Contain("code").And.Contain("shared"));
});
var conflict = Assert.Throws<CultWriteConflictException>(Write)!;
Assert.That(conflict.ChangedKeys, Does.Contain(BigKey.Value));
var refused = Refused(Write);
Assert.That(refused.Message, Does.Contain(other.Value).And.Contain(BigKey.Value).And.Contain("code").And.Contain("shared"), "the retry is judged by Resolve");
Assert.That(b.Get(other), Is.Null, "nothing landed in memory");
}
@ -989,12 +1052,16 @@ namespace GameCult.Caching.Tests
var seen = b.Get<VariantGear>(BaseKey)!;
SeedBig(a);
var refused = Refused(() => b.Commit(batch =>
void Write() => b.Commit(batch =>
{
batch.Expect(BaseKey, seen);
batch.Remove(BaseKey);
}));
Assert.That(refused.Message, Does.Contain(BaseKey.Value).And.Contain(BigKey.Value));
});
var conflict = Assert.Throws<CultWriteConflictException>(Write)!;
Assert.That(conflict.ChangedKeys, Does.Contain(BigKey.Value));
var refused = Refused(Write);
Assert.That(refused.Message, Does.Contain(BaseKey.Value).And.Contain(BigKey.Value), "the retry is judged by Resolve");
Assert.That(b.Get(BaseKey), Is.Not.Null, "nothing landed in memory");
}
@ -1004,7 +1071,7 @@ namespace GameCult.Caching.Tests
}
[Test]
public void AConditionalCommitThatIsSoundOnTheMergedSetStillLands()
public void AConditionalCommitThatIsSoundOntoAVariantStoreAnotherWriterChangedIsRefusedThenLandsOnRetry()
{
var path = PathOf("gear.cc");
var other = new CultRecordKey("other");
@ -1015,11 +1082,15 @@ namespace GameCult.Caching.Tests
using (var b = Open(path))
{
SeedBig(a);
Assert.That(b.Commit(batch =>
bool Write() => b.Commit(batch =>
{
batch.Expect(other, null);
batch.Upsert(typeof(VariantGear), Named("other", "fine"), other);
}), Is.True);
});
Assert.Throws<CultWriteConflictException>(() => Write());
Assert.That(b.Get(BigKey), Is.Not.Null, "the refusal reloaded the other writer's variant");
Assert.That(Write(), Is.True);
}
using var reopened = Open(path);
@ -1135,7 +1206,7 @@ namespace GameCult.Caching.Tests
}
[Test]
public void AVariantAnotherWriterLandedIsStillJudgedWhenThisCacheHoldsNoVariant()
public void AVariantAnotherWriterLandedIsRefusedAsChangedWhenThisCacheHoldsNoVariant()
{
var path = PathOf("gear.cc");
var other = new CultRecordKey("other");
@ -1146,7 +1217,7 @@ namespace GameCult.Caching.Tests
using (var b = Open(path))
{
VariantOnCode(a, "shared");
Refused(() => b.Commit(batch =>
Assert.Throws<CultWriteConflictException>(() => b.Commit(batch =>
{
batch.Expect(other, null);
batch.Upsert(typeof(VariantGear), Named("other", "shared"), other);

View file

@ -407,5 +407,110 @@ namespace GameCult.Caching.Tests
Assert.That(Read(path).FormatVersion, Is.EqualTo(CultPersistedStoreSnapshot.FormatV1));
}
// Another writer adds a record under this cache's schema id whose payload this cache's type cannot decode.
private static void AddUndecodableRecord(string path, string key)
{
var snapshot = Read(path);
snapshot.Records = snapshot.Records.Concat(new[]
{
new CultPersistedRecord { Key = key, SchemaId = ItemId, StoredAt = DateTimeOffset.UtcNow.ToString("O"), Payload = new byte[] { 0xC1 } }
}).OrderBy(record => record.Key, StringComparer.Ordinal).ToArray();
Write(path, snapshot);
}
// A flush decodes nothing it did not stage: a copied record this cache's type cannot decode is carried byte for byte.
[Test]
public void AFlushCopiesARecordThisCacheCannotDecode([Values] bool commit)
{
var path = PathOf("undecodable.cc");
Seed(path, false, "a");
using var a = Open(path, false);
AddUndecodableRecord(path, "r");
var carried = Stored(path, "r", false);
Land(a, commit, typeof(WsItem), new WsItem { Name = "name-y" }, new CultRecordKey("y"));
Assert.That(Keys(path), Is.EqualTo(new[] { "a", "r", "y" }));
Assert.That(Stored(path, "r", false), Is.EqualTo(carried));
}
// The same record in a store holding a variant cannot be judged without decoding it: the write is refused with the decode
// failure as its cause, and the disk, the cache and its staged change are as they were.
[Test]
public void AWriteIntoAVariantStoreWithARecordThisCacheCannotLoadChangesNothing([Values] bool commit)
{
var path = PathOf("undecodable-variant.cc");
Seed(path, false, "a");
using (var seed = Open(path, false))
seed.Commit(batch => batch.UpsertVariant(new CultRecordKey("v"), A, new[] { seed.Override<WsItem>(nameof(WsItem.Note), "variant") }));
var a = Open(path, false);
AddUndecodableRecord(path, "r");
var before = File.ReadAllBytes(path);
var y = new CultRecordKey("y");
var conflict = Assert.Throws<CultWriteConflictException>(() =>
{
if (commit)
a.Commit(batch => batch.Upsert(typeof(WsItem), new WsItem { Name = "name-y" }, y));
else
Land(a, false, typeof(WsItem), new WsItem { Name = "name-y" }, y);
})!;
Assert.That(conflict.InnerException, Is.Not.Null, "the decode failure is the cause");
Assert.That(conflict.ChangedKeys, Is.EqualTo(new[] { "r" }));
Assert.That(File.ReadAllBytes(path), Is.EqualTo(before));
Assert.That(a.Get(new CultRecordKey("r")), Is.Null);
if (!commit)
{
Assert.That(a.IsDirty, Is.True, "a reload that fails changes nothing, the staged write included");
Assert.That(a.Get<WsItem>(y), Is.Not.Null);
}
}
// A stages k, B writes k later, A flushes: A's bytes land at a storedAt later than B's, so a writer holding B's k is told it moved.
[Test]
public void AWriteOverAnotherWritersLaterRecordStoresALaterStoredAt()
{
var path = PathOf("later.cc");
Seed(path, false, "a");
var k = new CultRecordKey("k");
using var a = Open(path, false);
a.UpsertAsync(typeof(WsItem), new WsItem { Name = "name-k", Note = "by-a" }, k).GetAwaiter().GetResult();
System.Threading.Thread.Sleep(30);
using var b = Open(path, false);
b.Commit(batch => batch.Upsert(typeof(WsItem), new WsItem { Name = "name-k", Note = "by-b" }, k));
var seenByB = b.Get<WsItem>(k)!;
var bAt = Read(path).Records.Single(record => record.Key == "k").StoredAt;
a.FlushAllBackingStores();
var onDisk = Read(path).Records.Single(record => record.Key == "k");
Assert.That(DateTimeOffset.Parse(onDisk.StoredAt, null, System.Globalization.DateTimeStyles.RoundtripKind),
Is.GreaterThan(DateTimeOffset.Parse(bAt, null, System.Globalization.DateTimeStyles.RoundtripKind)));
Assert.That(a.Commit(batch => { batch.Expect(k, a.Get<WsItem>(k)!); batch.Upsert(typeof(WsItem), new WsItem { Name = "name-k", Note = "again" }, k); }),
Is.True, "A's own condition holds without a pull");
Assert.That(b.Commit(batch => { batch.Expect(k, seenByB); batch.Upsert(typeof(WsItem), new WsItem { Name = "name-k", Note = "stale" }, k); }),
Is.False, "B's copy of k is no longer what the store holds");
}
// A commit that removes a key this cache staged removes it everywhere: the staged write does not survive the batch.
[Test]
public void ACommitThatRemovesAStagedKeyRemovesIt()
{
var path = PathOf("staged-removed.cc");
Seed(path, false, "a");
var k = new CultRecordKey("k");
using (var cache = Open(path, false))
{
cache.UpsertAsync(typeof(WsItem), new WsItem { Name = "name-k" }, k).GetAwaiter().GetResult();
Assert.That(cache.Commit(batch => batch.Remove(k)), Is.True);
Assert.That(Keys(path), Is.EqualTo(new[] { "a" }));
Assert.That(cache.Get(k), Is.Null);
}
using var reopened = Open(path, false);
Assert.That(reopened.Get(k), Is.Null);
}
}
}

View file

@ -73,6 +73,13 @@ public sealed class CultMeshSingleFileReadabilityTests
if (reads)
{
var before = CultDocumentMessagePackSerialization.DeserializeSnapshot(bytes);
if (before.Records.Any(record => record.Variant != null))
{
Assert.That(() => write(path), Throws.TypeOf<CultWriteConflictException>(), name);
File.ReadAllBytes(path).Should().Equal(bytes, name + " wrote into a store holding a variant it cannot resolve");
continue;
}
write(path);
var after = CultDocumentMessagePackSerialization.DeserializeSnapshot(File.ReadAllBytes(path));
after.Records.Where(record => record.Key == "publication").Should().ContainSingle(name);

View file

@ -158,6 +158,43 @@ public sealed class CultMeshSingleFileStoreTests
again.Records.Single(record => record.Key == key).Payload.Should().Equal(before.Records.Single(record => record.Key == key).Payload);
}
// A Mesh write has read nothing and has no cache to resolve a variant against its base, so it never writes into a store that holds
// one: typed or raw, at the variant's key, its base's key or an unrelated key, it is refused and the file is left byte for byte.
[Test]
public void AMeshWriteIntoAStoreHoldingAVariantIsRefusedAndLeavesItByteIdentical(
[Values("variant", "base", "unrelated")] string target, [Values] bool typed)
{
var path = Path.Combine(_root, $"variant-{target}-{typed}.cc");
var baseKey = new CultRecordKey("a");
using (var cache = CultCacheMessagePack.Create(path, new CultCacheOpenOptions()))
{
cache.Commit(batch =>
{
batch.Upsert(typeof(CultMeshBodyPublicationDocument), PublicationOf("a"), baseKey);
batch.UpsertVariant(new CultRecordKey("v"), baseKey, new[]
{
cache.Override<CultMeshBodyPublicationDocument>(nameof(CultMeshBodyPublicationDocument.BodyId), "aetheria:v"),
cache.Override<CultMeshBodyPublicationDocument>(nameof(CultMeshBodyPublicationDocument.ProducerId), "variant-producer")
});
});
}
var bytes = File.ReadAllBytes(path);
Snapshot(path).Records.Should().Contain(record => record.Variant != null);
var key = new CultRecordKey(target switch { "variant" => "v", "base" => "a", _ => "z" });
var conflict = Assert.Throws<CultWriteConflictException>(() =>
{
if (typed)
CultMesh.WriteSingleFileDocument(path, key, PublicationOf("z"));
else
CultMesh.WriteSingleFileDocumentPayload(path, key, Raw, null, new byte[] { 0x90 });
})!;
conflict.Message.Should().Contain("CultCache");
File.ReadAllBytes(path).Should().Equal(bytes);
}
// The writer cannot see whether the raw payload or the records it copies hold element ids, so a marked file stays marked.
[Test]
public void AMeshRawWriteIntoAMarkedStoreKeepsItMarked()