GameCult/CultLib
Watch
9
0
Fork
You've already forked CultLib
 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

Write ownership by the written type: one per-key predicate, the registry leaves upserts

Browse source 
An upsert may replace the record at its key only when the type it writes there owns the record
(CultDocumentRegistry.WrittenOwns): the entry declares the stored id, or reads the stored entry whole
under its own schema name. Cache and cacheless writers, single-file and directory stores ask the same
per-key judge (CacheBackingStore.UndeclaredAt); a cache no longer replaces another registered type's
record. Registry.Owns is the removal/check rule only. Deleted: CatalogEntryLookup's registry half,
CultDocumentRegistry.Lookup, Owns<T>'s mayClaim, the per-batch union and the per-id memo.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Meta Vens 2026-10-28 02:36:17 +01:00
commit 340c9507e4
5 changed files with 173 additions and 159 deletions
Download patch file Download diff file Expand all files Collapse all files

24
src/GameCult.Caching.MessagePack/DirectoryMessagePackBackingStore.cs
View file

@ -185,7 +185,11 @@ public sealed class DirectoryMessagePackBackingStore : CacheBackingStore
manifest.Records.ToDictionary(record => record.Key, record => VersionOf(record.Payload), StringComparer.Ordinal), Entries.Values, ForeignRecords))
return CultCommitOutcome.Mismatch;
RefuseUndeclaredOnDisk(manifest, request.Upserts.Concat(request.Deletes).Select(entry => entry.Key.Value).Concat(request.Checks).Concat(StagedKeys));
RefuseUndeclaredOnDisk(
manifest,
request.Upserts.Concat(request.Deletes).Select(entry => entry.Key.Value).Concat(request.Checks).Concat(StagedKeys),
request.Upserts,
request.Deletes);
if (request.Upserts.Count == 0 && request.Deletes.Count == 0 && !IsDirty)
{
// Nothing to write: the batch only judged keys it does not hold.
@ -243,7 +247,7 @@ public sealed class DirectoryMessagePackBackingStore : CacheBackingStore
Directory.CreateDirectory(_manifestFile.DirectoryName!);
using var commitLease = AcquireCommitLease(wait: true, create: true);
var (manifest, catalogBytes) = ReadManifest();
RefuseUndeclaredOnDisk(manifest, StagedKeys);
RefuseUndeclaredOnDisk(manifest, StagedKeys, Array.Empty<CultStoredDocument>(), Array.Empty<CultStoredDocument>());
if (IsDirty)
WriteGeneration(manifest, catalogBytes);
else
@ -275,7 +279,7 @@ public sealed class DirectoryMessagePackBackingStore : CacheBackingStore
using var commitLease = AcquireCommitLease(wait, create: true);
if (commitLease == null)
return CultCommitOutcome.Contended;
RefuseUndeclaredOnDisk(ReadManifest().Snapshot, keys);
RefuseUndeclaredOnDisk(ReadManifest().Snapshot, keys, Array.Empty<CultStoredDocument>(), Array.Empty<CultStoredDocument>());
return CultCommitOutcome.Committed;
});
}
@ -286,14 +290,22 @@ public sealed class DirectoryMessagePackBackingStore : CacheBackingStore
// The manifest's other records stay as they are, foreign and read-only ones included; a write that would replace or remove a
// record the manifest holds that the writer does not own is refused before any page is written, by the one rule every
// store shares (CacheBackingStore.RefuseUndeclaredOnDisk). The refused records reload as a pull loads them.
private void RefuseUndeclaredOnDisk(CultPersistedStoreSnapshot currentManifest, IEnumerable<string> written)
// store shares (CacheBackingStore.UndeclaredAt, CacheBackingStore.RefuseUndeclaredOnDisk). The refused records reload as a
// pull loads them. A key is owned by the entry the write puts there: the batch's upsert, or the staged document. A key it
// removes (the batch's deletes, the staged deletes) or only checks goes to the registry's removal rule.
private void RefuseUndeclaredOnDisk(CultPersistedStoreSnapshot currentManifest, IEnumerable<string> written, IReadOnlyList<CultStoredDocument> upserts, IReadOnlyList<CultStoredDocument> deletes)
{
CultPersistedRecord? HeldAt(string key) =>
currentManifest.Records.FirstOrDefault(record => string.Equals(record.Key, key, StringComparison.Ordinal));
var batch = upserts.ToDictionary(entry => entry.Key.Value, entry => entry.Descriptor, StringComparer.Ordinal);
var removed = deletes.Select(entry => entry.Key.Value).ToHashSet(StringComparer.Ordinal);
CultSchemaCatalogEntry? WrittenAt(string key) =>
batch.TryGetValue(key, out var descriptor) ? descriptor.ToCatalogEntry()
: !removed.Contains(key) && _dirtyKeys.ContainsKey(key) && Entries.TryGetValue(key, out var staged) ? staged.Descriptor.ToCatalogEntry()
: null;
RefuseUndeclaredOnDisk(
written,
key => HeldAt(key) is { } held ? Undeclared(held, currentManifest.SchemaCatalog, Registry.Owns) : null,
UndeclaredAt(HeldAt, currentManifest.SchemaCatalog, WrittenAt, Registry.Owns),
key => HeldAt(key) is { } held && Foreign(held, currentManifest.SchemaCatalog, VersionOf(held.Payload)) == null
? LoadPage(held, currentManifest.SchemaCatalog, out _, out _)
: null,

20
src/GameCult.Caching/Contracts/cultcache-persistence-format.md
View file

@ -206,16 +206,16 @@ The rule is checked again where the write lands, under the store's lock: a write
whose durable record is not owned, because another writer rewrote it after this cache
loaded, is refused typed, and the record is reloaded as the store holds it, read-only when a type can
read it. This holds for every write through the store, the store's own `Push`, `Delete` and
`CommitBatch` included. A CultMesh single-file write (`WriteDocument`, `WriteRecord`, or `PushAll` on an unattached store) asks the same rule: it owns the ids its
written entries declare (their own ids and the ids they list compatible) and an id whose publishing entry its claimant reads whole (no persisted
slot unread). The claimant is picked by the one precedence over the written entries and the call's registry, a written entry being the only
description of its schema id (the registry's type for that id is never asked), and the record is owned only when the claimant is a written
entry. A registry adds claimants and declares nothing, so it only adds refusals; with none the written entries are the only claimants. Refused:
a record another registered type claims, under that type's current id or an old one it lists; a record whose stored entry lists another
registered type's id ahead of the written one's, whatever name the stored entry carries; a record two registered types list compatible or two
registered versions of the written schema could read; a record the picked written entry reads only by dropping persisted slots; and the
removal of any stored record by a call that writes no entry. A registry-less `WriteDocument` reads `CultDocumentRegistry.Shared`. It is
refused the same way otherwise.
`CommitBatch` included. Whether a write may replace the record at a key is decided per key by the type it writes there, for a cache
and for a CultMesh single-file write (`WriteDocument`, `WriteRecord`, or `PushAll` on an unattached store) alike, in a single-file store and
a directory store alike: the written type owns the stored record when its entry declares the stored id (its own id, or one it lists
compatible), or when it is the type that reads the stored record's published entry and reads it whole (no persisted slot unread), so an older
id of its own schema name that it reads whole is its own and is restamped. Anything else is refused, another registered type's record
included, and so is a record the written type would read only by dropping persisted slots. No registry enters the decision, `Shared` included:
the answer for a given written entry, stored id and catalog is the same lent a registry or not, and a cache's registered types do not make
each other's records writable. Retyping a key is a removal and an add, in separate commits. A removal or check of a key goes to the
registry's removal rule (a record a registered type declares, or reads whole, may be removed), and a cacheless call, which has no registry,
removes a record only when an entry it writes owns it; a call that writes no entry removes nothing.
A record that resolves to no local type is a foreign record, for example one of a type this build does not
have, or one renamed under a new schema name without declaring its old id. A store never destroys or relabels it: the store carries it

227
src/GameCult.Caching/CultCache.cs
View file

@ -524,6 +524,12 @@ namespace GameCult.Caching
// asked.
internal bool StoredSchemaReadWhole { get; set; }
// The catalog entry that publishes StoredSchemaId in the store this record was loaded from, with the id it was published for,
// so staging can ask WrittenOwns the question the store's judge asks, over the catalog the record came with. Null for a record
// this cache constructed or wrote, which is stored under Descriptor's own id and published by Descriptor's entry. A reader
// uses it only while the id still matches StoredSchemaId.
internal (string Id, CultSchemaCatalogEntry Entry)? StoredPublishedBy { get; set; }
// Names the bytes this record was read from or written as: the SHA-256 of its persisted encoding, set by the store that
// read or wrote them. A record no store has read or written has a version of its own that equals nothing else.
internal string StoredVersion { get; set; }
@ -593,45 +599,28 @@ namespace GameCult.Caching
public string TypeLabel { get; }
}
// The lookup of a cacheless writer: the catalog entries the call writes, and the lent registry's other types. A written entry is
// the only description of its schema id, so the registry's type for that id is never asked: a registry adds claimants but never
// re-describes the written type.
// The lookup a write's ownership asks: only the catalog entries the call writes. No registry enters it, so the answer is the
// written type's own (variants:ruling:write-ownership-by-written-type).
internal sealed class CatalogEntryLookup : IClaimantLookup<CultSchemaCatalogEntry>
{
private readonly IReadOnlyCollection<CultSchemaCatalogEntry> _entries;
private readonly IClaimantLookup<CultDocumentDescriptor>? _others;
public CatalogEntryLookup(IReadOnlyCollection<CultSchemaCatalogEntry> entries, CultDocumentRegistry? others)
public CatalogEntryLookup(IReadOnlyCollection<CultSchemaCatalogEntry> entries)
{
_entries = entries;
_others = others?.Lookup;
}
public CultSchemaCatalogEntry? Owner(string schemaId) =>
_entries.FirstOrDefault(entry => string.Equals(entry.SchemaId, schemaId, StringComparison.Ordinal))
?? Other(_others?.Owner(schemaId));
_entries.FirstOrDefault(entry => string.Equals(entry.SchemaId, schemaId, StringComparison.Ordinal));
public IReadOnlyList<CultSchemaCatalogEntry> Listing(string schemaId) =>
_entries.Where(entry => entry.CompatibleSchemaIds.Contains(schemaId, StringComparer.Ordinal))
.Concat(Others(_others?.Listing(schemaId)))
.ToArray();
_entries.Where(entry => entry.CompatibleSchemaIds.Contains(schemaId, StringComparer.Ordinal)).ToArray();
public IReadOnlyList<CultSchemaCatalogEntry> Named(string schemaName) =>
_entries.Where(entry => string.Equals(entry.SchemaName, schemaName, StringComparison.Ordinal))
.Concat(Others(_others?.Named(schemaName)))
.ToArray();
_entries.Where(entry => string.Equals(entry.SchemaName, schemaName, StringComparison.Ordinal)).ToArray();
public ClaimantFacts Describe(CultSchemaCatalogEntry claimant) =>
new(claimant.SchemaName, claimant.SchemaVersion, claimant.SchemaId);
// A registry type counts only when no written entry has its schema id.
private CultSchemaCatalogEntry? Other(CultDocumentDescriptor? descriptor) =>
descriptor == null || _entries.Any(entry => string.Equals(entry.SchemaId, descriptor.SchemaId, StringComparison.Ordinal))
? null
: descriptor.ToCatalogEntry();
private IEnumerable<CultSchemaCatalogEntry> Others(IEnumerable<CultDocumentDescriptor>? descriptors) =>
(descriptors ?? Array.Empty<CultDocumentDescriptor>()).Select(Other).OfType<CultSchemaCatalogEntry>();
}
public sealed class CultDocumentRegistry
@ -832,29 +821,42 @@ namespace GameCult.Caching
catalog.FirstOrDefault(entry => string.Equals(entry.SchemaId, schemaId, StringComparison.Ordinal))
?? catalog.FirstOrDefault(entry => entry.CompatibleSchemaIds.Contains(schemaId, StringComparer.Ordinal));
// The one answer to whether a writer may write, relabel or remove a record stored under this id
// (variants:ruling:drift-record-write-ownership): a registered type declares the id (owns it or lists it compatible), or the
// type that reads it reads it whole. Read live, so a type registered later that declares the id claims its records at their
// next write. A record read only by dropping slots is not owned: declaring its id compatible is the consent to the loss.
// Header only: the persisted catalog entry that publishes the id is compared with the claimant's entry, never a payload. A
// claimant that cannot be named (ambiguous, id missing from the catalog, shapes incompatible) owns nothing.
// Whether a removal, or a check of a key, may proceed over a record stored under this id, and the load-time marking of a
// record read-only: a registered type declares the id (owns it or lists it compatible), or the type that reads it reads it
// whole (variants:ruling:drift-record-write-ownership). It never decides an upsert: WrittenOwns does
// (variants:ruling:write-ownership-by-written-type). Read live, so a type registered later that declares the id claims its
// records at their next removal. A record read only by dropping slots is not owned: declaring its id compatible is the
// consent to the loss. Header only: the persisted catalog entry that publishes the id is compared with the claimant's
// entry, never a payload. A claimant that cannot be named (ambiguous, id missing from the catalog, shapes incompatible) owns
// nothing.
internal bool Owns(string schemaId, IReadOnlyCollection<CultSchemaCatalogEntry> catalog) =>
Owns(schemaId, catalog, Declared, _indexes, descriptor => descriptor.ToCatalogEntry(), _ => true);
Owns(schemaId, catalog, Declared, _indexes, descriptor => descriptor.ToCatalogEntry());
// The registry's lookup, for a cacheless writer that lends it only its other types.
internal IClaimantLookup<CultDocumentDescriptor> Lookup => _indexes;
// The one answer to whether an upsert of the written type may replace the record stored under this id at its key, for a cache
// and a cacheless writer, a single-file store and a directory store alike. The written entry owns the record when it declares
// the stored id (its own id, or one it lists compatible), or when it is the type that reads the record's published entry
// (the one Claimant precedence over this one entry) and reads it whole, so an older id of its own schema name that it reads
// whole is its own (variants:ruling:own-entry-reads-whole). Anything else, another registered type's record included, is
// refused: retyping a key is a removal and an add (variants:ruling:cache-refuses-retype). No registry enters it.
internal static bool WrittenOwns(
CultSchemaCatalogEntry written,
string storedId,
IReadOnlyCollection<CultSchemaCatalogEntry> catalog) =>
Owns(storedId, catalog, id => DeclaredBy(written, id), new CatalogEntryLookup(new[] { written }), entry => entry);
// The rule itself, for any writer: declared, or the claimant (picked by the one Claimant over a lookup) reads the record's
// published entry whole. The lookup is a registry's indexes, or the entries a registry-less cacheless writer writes. mayClaim
// is a condition on the claimant's answer, not a second lookup: a cache accepts any claimant, a cacheless writer only one of
// the entries the call writes, so it never owns a record another registered type reads.
// WrittenOwns' declared half: the entry's own id or one it lists compatible.
internal static bool DeclaredBy(CultSchemaCatalogEntry written, string storedId) =>
string.Equals(written.SchemaId, storedId, StringComparison.Ordinal) ||
written.CompatibleSchemaIds.Contains(storedId, StringComparer.Ordinal);
// The rule itself: declared, or the claimant (picked by the one Claimant over a lookup) reads the record's published entry
// whole. The lookup is a registry's indexes (the removal rule) or the one entry a write puts at the key (WrittenOwns).
internal static bool Owns<T>(
string schemaId,
IReadOnlyCollection<CultSchemaCatalogEntry> catalog,
Func<string, bool> declared,
IClaimantLookup<T> lookup,
Func<T, CultSchemaCatalogEntry> entryOf,
Func<CultSchemaCatalogEntry, bool> mayClaim)
Func<T, CultSchemaCatalogEntry> entryOf)
where T : class
{
if (declared(schemaId))
@ -864,8 +866,7 @@ namespace GameCult.Caching
var claimant = Claimant(schemaId, catalog, lookup, out var persisted);
if (persisted == null || claimant == null)
return false;
var claimantEntry = entryOf(claimant);
return mayClaim(claimantEntry) && ReadsWhole(persisted, claimantEntry);
return ReadsWhole(persisted, entryOf(claimant));
}
catch (InvalidOperationException)
{
@ -3105,14 +3106,14 @@ namespace GameCult.Caching
if (globals.TryGetValue(type, out var key) && key == stored.Key.Value)
globals.Remove(type);
if (source == null)
Claim(stored);
Claim(stored, upsert: false);
}
foreach (var stored in admitted)
{
var descriptor = stored.Descriptor;
if (source == null)
Claim(stored);
Claim(stored, upsert: true);
else if (Home(descriptor.DocumentType) is var home && home != source)
throw new InvalidOperationException(
$"{descriptor.SchemaName} record {stored.Key.Value} was loaded from {source} but its home is {home?.ToString() ?? "no store"}.");
@ -3140,7 +3141,7 @@ namespace GameCult.Caching
$"The batch spans {distinct[0]} and {distinct[1]}; a commit lands in one home store.");
return distinct.FirstOrDefault();
void Claim(CultStoredDocument stored)
void Claim(CultStoredDocument stored, bool upsert)
{
var home = Home(stored.Descriptor.DocumentType);
if (home == null)
@ -3153,7 +3154,7 @@ namespace GameCult.Caching
if (home.IsReadOnly)
throw new InvalidOperationException(
$"{stored.Descriptor.SchemaName} record {stored.Key.Value} routes to read-only backing store {home}.");
home.RefuseUndeclared(stored.Key.Value);
home.RefuseUndeclared(stored.Key.Value, upsert ? stored.Descriptor : null);
homes.Add(home);
}
}
@ -3944,6 +3945,7 @@ namespace GameCult.Caching
{
StoredSchemaId = record.SchemaId,
StoredSchemaReadWhole = readWhole,
StoredPublishedBy = PublishedByIn(record.SchemaId, catalog),
StoredVersion = version
};
}
@ -3968,10 +3970,14 @@ namespace GameCult.Caching
{
StoredSchemaId = record.SchemaId,
StoredSchemaReadWhole = readWhole,
StoredPublishedBy = PublishedByIn(record.SchemaId, catalog),
StoredVersion = version
};
}
private static (string Id, CultSchemaCatalogEntry Entry)? PublishedByIn(string schemaId, IReadOnlyCollection<CultSchemaCatalogEntry> catalog) =>
CultDocumentRegistry.PublishedBy(schemaId, catalog) is { } entry ? (schemaId, entry) : null;
protected void SetLastSchemaMigrationReports(IEnumerable<CultSchemaMigrationReport> reports)
{
_lastSchemaMigrationReports = reports?.ToArray() ?? Array.Empty<CultSchemaMigrationReport>();
@ -3991,19 +3997,34 @@ namespace GameCult.Caching
_foreignRecords = foreign.OrderBy(record => record.Key, StringComparer.Ordinal).ToArray();
}
// A cache never writes, relabels or removes a record it does not own: the staging half of rule 3. Refused when the store
// carries a foreign record at the key, or holds one that is not owned: its stored id is not declared by a registered type and
// the type that reads it drops persisted slots (read-only). A record read whole is owned; ToStoredDocument decided that from
// the same comparison Owns makes. Called for every record a cache write admits or evicts, before anything is staged.
protected internal void RefuseUndeclared(string key)
// A cache never writes, relabels or removes a record it does not own: the staging half of rule 3. A removal or check of a key
// is refused when the store carries a foreign record at the key, or holds one that is not owned: its stored id is not
// declared by a registered type and the type that reads it drops persisted slots (read-only). A record read whole is owned;
// ToStoredDocument decided that from the same comparison Owns makes.
protected internal void RefuseUndeclared(string key) => RefuseUndeclared(key, null);
// The same, for every record a cache write admits or evicts, before anything is staged: an upsert (written is its type) may
// replace only a record that type owns, WrittenOwns' rule (variants:ruling:write-ownership-by-written-type), asked over the
// catalog entry that published the held record's id when it was loaded, the question the store's judge asks of the file.
internal void RefuseUndeclared(string key, CultDocumentDescriptor? written)
{
var foreign = _foreignRecords.FirstOrDefault(record => string.Equals(record.Key, key, StringComparison.Ordinal));
if (foreign != null)
throw Overwrites(null, new[] { (foreign.Key, foreign.SchemaId, foreign.SchemaName) });
if (Entries.TryGetValue(key, out var held) && !Registry.Declared(held.StoredSchemaId) && !held.StoredSchemaReadWhole)
if (Entries.TryGetValue(key, out var held) && !HeldOwned(held, written))
throw Overwrites(null, new[] { (key, held.StoredSchemaId, held.Descriptor.SchemaName) });
}
private bool HeldOwned(CultStoredDocument held, CultDocumentDescriptor? written)
{
if (written == null)
return Registry.Declared(held.StoredSchemaId) || held.StoredSchemaReadWhole;
var published = held.StoredPublishedBy is { } loaded && string.Equals(loaded.Id, held.StoredSchemaId, StringComparison.Ordinal)
? loaded.Entry
: held.Descriptor.ToCatalogEntry();
return CultDocumentRegistry.WrittenOwns(written.ToCatalogEntry(), held.StoredSchemaId, new[] { published });
}
/// <summary>
/// Called when the cache removes keys it does not hold, outside a batch that commits through this store. Such a removal
/// removes nothing, but a store that holds a record at one of these keys under an id no registered type declares must
@ -4077,6 +4098,35 @@ namespace GameCult.Caching
? null
: (record.Key, record.SchemaId, CultDocumentRegistry.PublishedBy(record.SchemaId, catalog)?.SchemaName ?? string.Empty);
// The one per-key judge of a write over a durable catalog, for every store kind: the stored record at each key the write names
// is owned by the entry the write puts at that key (WrittenOwns), or, when the write only removes or checks the key, by the
// removal rule. heldAt gives the durable record at a key and writtenAt the catalog entry the write puts there (null for a
// removal or a check). Memoised per written schema id and stored id, so a large store pays one comparison per pair, and an
// answer for one written type is never reused for another.
internal static Func<string, (string Key, string SchemaId, string SchemaName)?> UndeclaredAt(
Func<string, CultPersistedRecord?> heldAt,
IReadOnlyCollection<CultSchemaCatalogEntry> catalog,
Func<string, CultSchemaCatalogEntry?> writtenAt,
Func<string, IReadOnlyCollection<CultSchemaCatalogEntry>, bool> removalOwns)
{
var answers = new Dictionary<(string? Written, string Stored), bool>();
return key =>
{
if (heldAt(key) is not { } held)
return null;
var written = writtenAt(key);
return Undeclared(held, catalog, (storedId, entries) =>
{
var asked = (written?.SchemaId, storedId);
if (!answers.TryGetValue(asked, out var answer))
answers[asked] = answer = written == null
? removalOwns(storedId, entries)
: CultDocumentRegistry.WrittenOwns(written, storedId, entries);
return answer;
});
};
}
// The apply half of rule 3, for every store kind, checked where the write lands under the store's lock. A key this write names
// whose durable record is under an id no registered type declares is refused, every such key at once, through Refuse, which
// reloads it as the store holds it: listed foreign when no type reads it, held read-only when one does.
@ -4484,7 +4534,7 @@ namespace GameCult.Caching
/// <param name="record">The record to store.</param>
/// <param name="catalogEntry">The schema catalog entry for the record's schema id.</param>
/// <param name="holdsIds">Whether the record's payload holds element ids, or null when the caller cannot tell.</param>
/// <param name="registry">The registry the payload was encoded with. Given, the write decides the header for copied records that registry can read, and its other types also claim stored records as a cache registering them would; it declares nothing, so it can only refuse more. <paramref name="catalogEntry"/> stays the only description of its own schema id and the registry's type for that id is never asked. Owned are a record whose id <paramref name="catalogEntry"/> declares (its own or one it lists compatible) and a record the entry reads whole when it is the picked claimant; refused are a record another registered type claims, one whose stored entry lists another registered type's id ahead of the entry's, and one that two registered types list or two versions of the schema could read. Omitted, it decides nothing for copied records and the one entry is the only claimant. It serves this call only: the store keeps no registry.</param>
/// <param name="registry">The registry the payload was encoded with. Given, the write decides the header for copied records that registry can read. It never decides ownership: the answer is the same lent or not. <paramref name="catalogEntry"/> alone owns the record at the key when it declares the stored id (its own or one it lists compatible), or when it is the type that reads the stored entry and reads it whole, so an older id of its own schema name it reads whole is its own and is restamped. Everything else is refused with the file left as it was: a record of another registered type included, whatever the registry holds, and a record it would read only by dropping persisted slots. Retyping a key is a removal and an add. It serves this call only: the store keeps no registry.</param>
/// <remarks>The call holds the store's gate while it runs. User code it runs must not enter a cache or another store; that throws <see cref="InvalidOperationException"/>.</remarks>
/// <exception cref="InvalidOperationException">The store is attached to a cache; write through the cache.</exception>
/// <exception cref="CultStoreUnreadableException">The file at the path is not a store this runtime can read.</exception>
@ -4507,7 +4557,7 @@ namespace GameCult.Caching
/// <param name="key">The record key the document is stored under.</param>
/// <param name="document">The document to store.</param>
/// <param name="storedAt">The stored-at stamp, or null for now.</param>
/// <param name="registry">The registry that describes and encodes the document, or null for <see cref="CultDocumentRegistry.Shared"/>. The document type's entry is taken from it and is the only description of that schema id. Its other types claim stored records as a cache registering them would, and it declares nothing, so it can only refuse more: the document's entry owns a record whose id it declares (its own or one it lists compatible), and a record it reads whole when it is the picked claimant; refused are a record another registered type claims, one whose stored entry lists another registered type's id ahead of the entry's, and one that two registered types list or two versions of the schema could read. A wider registry (Shared when none is passed) refuses at least as much. It serves this call only: the store keeps no registry.</param>
/// <param name="registry">The registry that describes and encodes the document, or null for <see cref="CultDocumentRegistry.Shared"/>. The document type's entry is taken from it, and the registry decides the header for copied records it can read. It never decides ownership, so the answer is the same for any registry that holds the type: the document's entry owns a record whose id it declares (its own or one it lists compatible), and a record of an older id of its own schema name that it reads whole; everything else is refused, another registered type's record included, and a record it would read only by dropping persisted slots. Retyping a key is a removal and an add. It serves this call only: the store keeps no registry.</param>
/// <remarks>The call holds the store's gate while it runs. User code it runs (property getters while the document is encoded, constructors while a copied record is read) must not enter a cache or another store; that throws <see cref="InvalidOperationException"/>, carried by the encoder's wrapper when the code ran inside it.</remarks>
/// <exception cref="InvalidOperationException">The store is attached to a cache; write through the cache.</exception>
/// <exception cref="CultStoreUnreadableException">The file at the path is not a store this runtime can read.</exception>
@ -4723,53 +4773,43 @@ namespace GameCult.Caching
using var fileLock = AcquireLock(wait);
if (fileLock == null)
return CultCommitOutcome.Contended;
RefuseUndeclaredOnDisk(ReadSnapshot(), keys, OwnsOnDisk());
RefuseUndeclaredOnDisk(ReadSnapshot(), keys, Array.Empty<CultStagedRecord>());
return CultCommitOutcome.Committed;
});
}
// A cache owns through its registry (CultDocumentRegistry.Owns). A cacheless CultMesh write owns what the catalog entries it
// writes declare: their own ids and the ids they list compatible, or an id whose publishing entry its claimant reads whole,
// the claimant being picked by the one precedence over the written entries and the lent registry's other types, and owning
// only when it is a written entry (variants:ruling:mesh-writes-bound-by-declaration). The keys judged are every key staged,
// written, removed or checked.
// Every key staged, written, removed or checked is judged, each by the one per-key judge (UndeclaredAt).
private void RefuseUndeclaredOnDisk(Durable? durable, IReadOnlyCollection<CultStagedRecord> upserts, IReadOnlyCollection<string> removals, IEnumerable<string> checks) =>
RefuseUndeclaredOnDisk(
durable,
_staged.Concat(upserts.Select(staged => staged.Record.Key)).Concat(removals).Concat(checks).Concat(StagedChecks),
OwnsOnDisk(upserts));
upserts);
// The owns predicate a write over the durable catalog asks. A cacheless writer declares what the entries it writes declare
// (variants:ruling:mesh-writes-bound-by-declaration) and reads a published entry with one lookup: the written entries shadow
// the lent registry's type for their ids, and the registry adds other claimants. The record is owned only when the claimant
// is a written entry, so the call is never more permissive than that declaration, and a registry can only refuse more.
private Func<string, IReadOnlyCollection<CultSchemaCatalogEntry>, bool> OwnsOnDisk(IReadOnlyCollection<CultStagedRecord>? upserts = null) =>
Cache != null || upserts == null
// A write's key is owned by the entry the write puts there, whoever the writer is (WrittenOwns). A key it only removes or
// checks is owned by the removal rule: a cache's registry (Owns), or, for a cacheless call that has no registry, some entry the
// call writes (a call that writes no entry owns nothing, variants:ruling:mesh-writes-bound-by-declaration).
private Func<string, IReadOnlyCollection<CultSchemaCatalogEntry>, bool> RemovalOwnsOnDisk(IReadOnlyCollection<CultStagedRecord> upserts) =>
Cache != null
? Registry.Owns
: (id, catalog) =>
{
var writes = upserts.Select(staged => staged.Entry).ToArray();
Func<string, bool> declared = candidate => writes.Any(entry =>
string.Equals(entry.SchemaId, candidate, StringComparison.Ordinal) ||
entry.CompatibleSchemaIds.Contains(candidate, StringComparer.Ordinal));
return CultDocumentRegistry.Owns(
id,
catalog,
declared,
new CatalogEntryLookup(writes, AttachedRegistry),
entry => entry,
claimant => writes.Any(entry => string.Equals(entry.SchemaId, claimant.SchemaId, StringComparison.Ordinal)));
};
: (id, catalog) => upserts.Any(staged => CultDocumentRegistry.WrittenOwns(staged.Entry, id, catalog));
// A refused key's durable version is what this store has now read of it.
private void RefuseUndeclaredOnDisk(Durable? durable, IEnumerable<string> written, Func<string, IReadOnlyCollection<CultSchemaCatalogEntry>, bool> owns)
private void RefuseUndeclaredOnDisk(Durable? durable, IEnumerable<string> written, IReadOnlyCollection<CultStagedRecord> upserts)
{
var undeclared = UndeclaredOnDisk(durable, owns);
if (undeclared.Count == 0)
return;
var keys = written.Distinct(StringComparer.Ordinal).ToArray();
var wanted = keys.ToHashSet(StringComparer.Ordinal);
var held = (durable?.Snapshot.Records ?? Array.Empty<CultPersistedRecord>())
.Where(record => wanted.Contains(record.Key))
.ToDictionary(record => record.Key, StringComparer.Ordinal);
var entries = upserts.ToDictionary(staged => staged.Record.Key, staged => staged.Entry, StringComparer.Ordinal);
var undeclaredAt = UndeclaredAt(
key => held.TryGetValue(key, out var record) ? record : null,
durable?.Snapshot.SchemaCatalog ?? Array.Empty<CultSchemaCatalogEntry>(),
key => entries.TryGetValue(key, out var entry) ? entry : null,
RemovalOwnsOnDisk(upserts));
RefuseUndeclaredOnDisk(
written,
key => undeclared.TryGetValue(key, out var carried) ? carried : null,
keys,
undeclaredAt,
key => Cache == null ? null : DurableAt(durable, key),
key => ForeignAt(durable, key),
key =>
@ -4781,23 +4821,6 @@ namespace GameCult.Caching
cacheless: Cache == null);
}
// Every durable record under an id the writer does not own, by key, named by the catalog entry that publishes its id. The
// answer is decided once per distinct schema id, so a large store pays one comparison per id, not per record.
private static Dictionary<string, (string Key, string SchemaId, string SchemaName)> UndeclaredOnDisk(Durable? durable, Func<string, IReadOnlyCollection<CultSchemaCatalogEntry>, bool> owns)
{
var undeclared = new Dictionary<string, (string Key, string SchemaId, string SchemaName)>(StringComparer.Ordinal);
var answers = new Dictionary<string, bool>(StringComparer.Ordinal);
bool OwnsOnce(string id, IReadOnlyCollection<CultSchemaCatalogEntry> catalog) =>
answers.TryGetValue(id, out var answer) ? answer : answers[id] = owns(id, catalog);
foreach (var record in durable?.Snapshot.Records ?? Array.Empty<CultPersistedRecord>())
{
if (Undeclared(record, durable!.Snapshot.SchemaCatalog, OwnsOnce) is { } carried)
undeclared[record.Key] = carried;
}
return undeclared;
}
// The durable record at a key as a load lists it when no registered type claims its id; null otherwise.
private CultForeignRecord? ForeignAt(Durable? durable, string key)
{

51
tests/GameCult.Caching.Tests/RestampOnWriteTests.cs
View file

@ -480,17 +480,18 @@ namespace GameCult.Caching.Tests
private static readonly CultDocumentRegistry NarrowShortRegistry = CultDocumentRegistry.ForTypes(new[] { typeof(NoteNarrow), typeof(NoteShort) });
// Which entry reads a persisted entry is one precedence: a cacheless writer lent a registry picks the claimant that registry
// picks, never one by schema name over its own entry. Here the entry is named and shaped like Narrow but lists Short's id.
// A write's ownership is the written entry's alone (variants:ruling:own-entry-reads-whole): the stored entry is named and shaped
// like Narrow but lists Short's id, and a registry that holds Short (which would read it by listing the id) changes nothing, for
// a cache and a cacheless writer alike.
[TestCase(false)]
[TestCase(true)]
public void ACachelessWriterPicksTheClaimantItsRegistryPicks(bool cacheless)
public void AWrittenEntryReadingTheRecordWholeOwnsItWhateverTheRegistryHolds(bool cacheless)
{
var path = SeedRenamed(listsNarrow: false, alsoLists: NarrowShortRegistry.GetRequired<NoteShort>().SchemaId, name: "tests.restamp_note");
var before = Fingerprint(path);
Assert.Throws<CultSchemaConflictException>(() => WriteNarrow(path, cacheless, NarrowShortRegistry));
Assert.That(Fingerprint(path), Is.EqualTo(before));
WriteNarrow(path, cacheless, NarrowShortRegistry);
Assert.That(Snapshot(path).Records.Single(record => record.Key == "note").SchemaId, Is.EqualTo(NarrowRegistry.GetRequired<NoteNarrow>().SchemaId));
}
[TestCase(false)]
@ -668,28 +669,6 @@ namespace GameCult.Caching.Tests
Assert.That(Fingerprint(path), Is.EqualTo(before));
}
// The shadow, where it is decided: a written entry answers for its schema id whatever the registry holds, and the registry
// still adds its other types.
[Test]
public void AWrittenEntryShadowsTheLentRegistrysTypeForItsId()
{
var written = NarrowRegistry.GetRequired<NoteNarrow>().ToCatalogEntry();
var lookup = new CatalogEntryLookup(new[] { written }, NarrowListerRegistry);
var lister = NarrowListerRegistry.GetRequired<NoteLister>();
Assert.That(lookup.Owner(written.SchemaId), Is.SameAs(written));
Assert.That(lookup.Named(written.SchemaName), Is.EqualTo(new[] { written }).Using<CultSchemaCatalogEntry>((x, y) => ReferenceEquals(x, y)));
Assert.That(lookup.Listing(written.SchemaId), Is.EqualTo(new[] { written }).Using<CultSchemaCatalogEntry>((x, y) => ReferenceEquals(x, y)));
Assert.That(lookup.Owner(lister.SchemaId)!.SchemaId, Is.EqualTo(lister.SchemaId));
Assert.That(lookup.Listing(ListerOldId).Select(entry => entry.SchemaId), Is.EqualTo(new[] { lister.SchemaId }));
Assert.That(lookup.Named(lister.SchemaName).Select(entry => entry.SchemaId), Is.EqualTo(new[] { lister.SchemaId }));
// A write of several entries shadows by the entry that has the id, not by all of them having it.
var other = CultDocumentRegistry.ForTypes(new[] { typeof(NoteOther) }).GetRequired<NoteOther>().ToCatalogEntry();
var several = new CatalogEntryLookup(new[] { other, written }, NarrowListerRegistry);
Assert.That(several.Named(written.SchemaName), Is.EqualTo(new[] { written }).Using<CultSchemaCatalogEntry>((x, y) => ReferenceEquals(x, y)));
}
// The registry's other types change nothing about a record the written type claims itself.
[Test]
public void AnotherRegisteredTypeDoesNotTakeAWritersOwnDriftRecord()
@ -701,16 +680,16 @@ namespace GameCult.Caching.Tests
Assert.That(Snapshot(path).Records.Single(record => record.Key == "note").SchemaId, Is.EqualTo(NarrowRegistry.GetRequired<NoteNarrow>().SchemaId));
}
// What the lent registry registers beyond the written type can only make the claim more ambiguous, never own more: a second
// version of the written schema's name refuses a record the write owned by name alone.
// What the lent registry registers beyond the written type changes nothing, in either direction: a second version of the
// written schema's name neither refuses nor widens a record the write owns by name alone.
[Test]
public void ASecondVersionOfTheWrittenNameInTheLentRegistryOnlyRefuses()
public void ASecondVersionOfTheWrittenNameInTheLentRegistryChangesNothing()
{
var path = SeedRenamed(listsNarrow: false, name: "tests.restamp_note");
var before = Fingerprint(path);
Assert.Throws<CultSchemaConflictException>(() => WriteNarrow(path, cacheless: true, CultDocumentRegistry.ForTypes(new[] { typeof(NoteNarrow), typeof(NoteGrown) })));
Assert.That(Fingerprint(path), Is.EqualTo(before));
WriteNarrow(path, cacheless: true, CultDocumentRegistry.ForTypes(new[] { typeof(NoteNarrow), typeof(NoteGrown) }));
Assert.That(Snapshot(path).Records.Single(record => record.Key == "note").SchemaId, Is.EqualTo(NarrowRegistry.GetRequired<NoteNarrow>().SchemaId));
}
// A record stored under an id its catalog entry only lists (no entry owns it) is published by that entry, and the type that
@ -765,8 +744,8 @@ namespace GameCult.Caching.Tests
var writes = new[] { narrow };
Func<string, bool> declares = id => id == narrow.SchemaId;
Assert.That(CultDocumentRegistry.Owns("tests.restamp.unpublished", writes, declares, new CatalogEntryLookup(writes, null), entry => entry, _ => true), Is.False);
Assert.That(CultDocumentRegistry.Owns(narrow.SchemaId, writes, declares, new CatalogEntryLookup(writes, null), entry => entry, _ => true), Is.True);
Assert.That(CultDocumentRegistry.Owns("tests.restamp.unpublished", writes, declares, new CatalogEntryLookup(writes), entry => entry), Is.False);
Assert.That(CultDocumentRegistry.Owns(narrow.SchemaId, writes, declares, new CatalogEntryLookup(writes), entry => entry), Is.True);
}
private static string VectorPath(string name)

2
unity/org.gamecult.cultlib/CHANGELOG.md
View file

@ -13,7 +13,7 @@ Releases as 3.0.0 (major).
- A store is written in the format its content needs, and CultLib 2.0 refuses the newer ones by name. A single-file store holding a document variant is written as `cultcache.store.v2`, one holding an element id as `cultcache.store.v3`; a directory store holding an element id is written as `cultcache.store.v5` (`v4` otherwise). A store is marked only when a record actually holds an id or a variant. Upgrade readers before writers: a 2.0 reader that opens such a store is refused, not misread.
- Writes are write sets. A flush or commit replaces the records it staged and copies every other record forward as stored, under the store's lock, instead of rewriting the whole store from the cache's memory. A record the cache does not know is kept, and an untouched record keeps a dropped member's slot through every write (never reuse a dropped slot; see the schema-compatibility contract).
- A write or removal at a key whose stored record the writer does not own is refused with `CultSchemaConflictException`, and the file is left as it was. A writer owns a record whose schema id it declares (owns or lists compatible) and a record its type reads whole under compatible drift: that record is writable and is restamped under the type's own id on write, as in 2.0.1. A record a type reads only by dropping persisted slots stays read-only until its id is declared compatible, which accepts the loss (this differs from 2.0.1, which shed the slots silently). A cache owns through its registry; a cacheless CultMesh write owns through the catalog entry it writes.
- A cacheless `WriteDocument`, `WriteRecord` or unattached-store `PushAll` owns a stored record in two ways only: the entries it writes declare the record's schema id (an entry's own id, or one it lists compatible), or the type picked to read the record is one of the written entries and reads it whole (no persisted slot unread). The type is picked as a cache registering the call's registry would pick it, except that a written entry is the only description of its schema id: `WriteDocument` takes its entry from the registry (`CultDocumentRegistry.Shared` when none is passed), `WriteRecord` writes the entry it is given, and a registry's own type for that id is never asked. A registry adds claimants and declares nothing, so it can only refuse more; without one the written entries are the only claimants. Refused, with the file left as it was: a record another registered type claims, under that type's current id or an old id it lists; a record whose stored entry lists another registered type's id ahead of the written one's, whatever name the stored entry carries; a record that two registered types list compatible, or that two registered versions of the written schema could read; a record the picked written entry would read only by dropping persisted slots; and the removal of any stored record by a call that writes no entry (a `Delete` then `PushAll`). As in 2.0.1, a record the written entry's own type reads whole under drift is owned and restamped. A registry-less `WriteDocument` reads `Shared` today, so any type loaded there can refuse the write; pass a registry to pin the answer.
- A write replaces the record at a key only when the type it writes there owns that record, decided per key and the same for a cache and for a cacheless `WriteDocument`, `WriteRecord` or unattached-store `PushAll`, in a single-file store and a directory store. The written type owns the stored record when its entry declares the stored id (its own id, or one it lists compatible), or when it is the type that reads the record's published entry and reads it whole (no persisted slot unread): an older id of its own schema name that it reads whole is its own and is restamped, as in 2.0.1. Anything else is refused with `CultSchemaConflictException` and the file left as it was: a record of another registered type, under its current id or an old id it lists, and a record the written type would read only by dropping persisted slots. No registry enters the decision, `CultDocumentRegistry.Shared` included, so a lent registry never changes the answer. This changes a cache: where 2.x let a cache holding two types replace one's record at a key with the other's, it now refuses; retype a key by removing it and adding the new record in separate commits. A removal or check goes to the registry's removal rule, and a cacheless call, which has no registry, removes a record only when an entry it writes owns it (a `Delete` then `PushAll` on an unattached store writes no entry and is refused).
- Registration refuses an owned object-list element type that has no `[CultElementId]` member (a public string with `[Key]`), naming the type and the list. Element ids let a write set merge a list by identity; a list of record references or plain values is a whole value and is not affected.
- The abstract and protected surface for store kinds outside this library changed. On `SingleFileBackingStore`, `SerializeSnapshot` and `DeserializeSnapshot` are replaced by the abstract `SerializeStore`, `SerializeRecord`, `SerializeCatalogEntry` and `ReadStore`, and the abstract `SerializePayload(object)` becomes `SerializePayload(object document, Type documentType)`: a store encodes a document as the type it is given. On `CacheBackingStore`, `Judging` is removed, `ToStoredDocument` takes the record's version (`record, version, catalog, deserialize, out report`), and these protected members are new for a store kind to implement or call: `HeaderFor`, `HeaderForWrite`, `Foreign`, `SetForeignRecords`, `Undeclared`, `RefuseUndeclared`, `RefuseUndeclaredOnDisk`, `RefuseRemovalOfUndeclared`, `Refuse`, `SameStoredIdentity`, `VersionOf`, `StoredAtOver`, `Overwrites`, `StagedChecks`, `StageChecks` and `JudgeChecks`. `MarkFlushSucceeded` is unchanged. `CacheBackingStore.ForeignRecords` is new and public: the records a store holds under a schema id its cache does not declare.
- `CultCommitRequest.Expected` is now `IReadOnlyList<(CultRecordKey Key, string? Version)>` (it was a key with a schema id and a stored-at), `CultCommitRequest.Checks` is new, and `ConditionsHold` takes `(IReadOnlyDictionary<string, string> durable, IEnumerable<CultStoredDocument> observed, IEnumerable<CultForeignRecord> foreign)` in place of `(IReadOnlyCollection<CultPersistedRecord>, IEnumerable<CultStoredDocument>)`. A store kind outside the library lists the record versions it holds and passes its foreign records.