CultCache: a record whose payload carries slots past its reader's, at any depth, is held read-only

The guard formatter counts each int-keyed object's slots into a thread-bound scope opened around the one payload decode;
ToStoredDocument turns the finding into CultStoredDocument.UnreadSlots (explained only for depth-0 slots the catalog comparison
reports ignored); HeldOwned refuses an unexplained mark whatever is declared; RefuseUndeclared throws the one refusal with
UnreadSlots; CultCache.IsWritable asks the same predicate. Vectors: extra-slot-unit-arm, extra-slot-store.cc.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
soul 2026-11-10 14:35:10 +00:00 • committed by Libby
commit c55d34fb32
14 changed files with 642 additions and 30 deletions

View file

@ -166,7 +166,7 @@ public static class CultDocumentMessagePackSerialization
private static (MessagePackSerializerOptions Options, IFormatterResolver[] Ahead) Compose(IFormatterResolver[] consumerResolvers)
{
// The guard goes first and wraps whatever formatter the rest of the composite would hand out, so a consumer
// The guard goes first and wraps whatever formatter the rest of the composite would hand out (a union, or a concrete int-keyed object for its slot count), so a consumer
// resolver still encodes an arm while the key check runs.
var ahead = consumerResolvers.Append(CultDocumentResolver.Instance).Append(CultMathResolver.Instance).Append(CultMathExprResolver.Instance).ToArray();
var inner = CompositeResolver.Create(ahead.Append(StandardResolver.Instance).ToArray());
@ -214,6 +214,19 @@ public static class CultDocumentMessagePackSerialization
?? throw new InvalidOperationException($"MessagePack returned null for Cult document type {type.FullName}.");
}
// The same decode, with the slots it read past noted (CultSlotScope). Opened around the one decode and after the depth check,
// so a payload past the nesting limit is refused, never marked.
internal static CultDecoded DeserializeMarked(Type type, byte[] payload, CultDocumentRegistry registry)
{
if (registry == null) throw new ArgumentNullException(nameof(registry));
registry.GetRequired(type);
CultPayloadDepth.Check(payload, type, PayloadDirection.Read);
using var scope = CultSlotScope.Open();
var document = MessagePackSerializer.Deserialize(type, payload, OptionsForDocument(type))
?? throw new InvalidOperationException($"MessagePack returned null for Cult document type {type.FullName}.");
return new CultDecoded(document, scope.Finding());
}
/// <summary>
/// Replaces slots of a serialized document payload (a MessagePack array indexed by slot) with already-encoded member
/// values, leaving every other slot byte for byte as it was. A slot past the end extends the array with nil.
@ -823,4 +836,7 @@ public class SingleFileMessagePackBackingStore : SingleFileBackingStore
{
return CultDocumentMessagePackSerialization.DeserializeUntyped(documentType, payload, Registry);
}
internal override CultDecoded DeserializeMarked(Type documentType, byte[] payload) =>
CultDocumentMessagePackSerialization.DeserializeMarked(documentType, payload, Registry);
}

View file

@ -0,0 +1,47 @@
using System;
using GameCult.Caching;
namespace GameCult.Caching.MessagePack;
/// <summary>
/// The marks one payload decode noted: the guard's slot formatter reports every object whose array held more slots than its type
/// declares, with the nesting depth it sat at, and DeserializeMarked reads the finding out. Thread-bound and open only around the one
/// decode, so a decode outside it (a CultMesh read, a codec) notes nothing.
/// </summary>
internal sealed class CultSlotScope : IDisposable
{
[ThreadStatic] private static CultSlotScope? current;
private readonly CultSlotScope? previous;
private string? typeName;
private int slotCount;
private int declaredCount;
private int occurrences;
private bool allAtDepthZero = true;
private CultSlotScope() => previous = current;
internal static CultSlotScope? Current => current;
internal static CultSlotScope Open() => current = new CultSlotScope();
// 0 for the document's own object; each object nested inside it adds one (a union adds none, its arm is the object).
internal int Depth { get; set; }
internal void Note(string name, int slots, int declared)
{
if (occurrences++ == 0)
{
typeName = name;
slotCount = slots;
declaredCount = declared;
}
if (Depth > 0) allAtDepthZero = false;
}
internal CultUnreadSlots? Finding() =>
occurrences == 0 ? null : new CultUnreadSlots(typeName!, slotCount, declaredCount, occurrences, allAtDepthZero, explained: false);
public void Dispose() => current = previous;
}

View file

@ -8,7 +8,7 @@ using MessagePack.Formatters;
namespace GameCult.Caching.MessagePack;
/// <summary>
/// The one place an unknown union arm or an unnamed subtype is decided. A union is an abstract class or interface carrying
/// The one sensor of an unknown union arm, an unnamed subtype and a slot past the reader's declared slots. A union is an abstract class or interface carrying
/// its own [Union] attributes (never inherited: a union whose arms are also a union has its own key set). MessagePack-CSharp
/// still encodes the [key, armSlots] shape; this wraps its formatter so that a key the union does not declare is refused on
/// read, and a value whose runtime type is not a declared arm is refused on write, where MessagePack alone yields null and nil.
@ -18,7 +18,58 @@ internal sealed class CultUnionGuard(IFormatterResolver inner) : IFormatterResol
public IMessagePackFormatter<T>? GetFormatter<T>()
{
var arms = typeof(T).GetCustomAttributes<UnionAttribute>(inherit: false).ToArray();
return arms.Length == 0 || inner.GetFormatter<T>() is not { } formatter ? null : new Formatter<T>(formatter, arms);
if (arms.Length == 0)
return DeclaredSlots(typeof(T)) is { } declared && inner.GetFormatter<T>() is { } objectFormatter
? new SlotFormatter<T>(objectFormatter, declared)
: null;
return inner.GetFormatter<T>() is not { } formatter ? null : new Formatter<T>(formatter, arms);
}
// The slots a concrete int-keyed [MessagePackObject] type declares (highest [Key] plus one), by MessagePack's member rule: public
// instance properties and fields, base keys included, [IgnoreMember] excluded. Null for any other type, which the guard leaves alone.
private static int? DeclaredSlots(Type type)
{
if (type.IsAbstract || type.IsInterface || type.IsGenericTypeDefinition) return null;
if (type.GetCustomAttribute<MessagePackObjectAttribute>(inherit: true) is not { KeyAsPropertyName: false }) return null;
var highest = -1;
const BindingFlags members = BindingFlags.Public | BindingFlags.Instance;
foreach (var member in type.GetFields(members).Cast<MemberInfo>().Concat(type.GetProperties(members)))
{
if (member.GetCustomAttribute<IgnoreMemberAttribute>(inherit: true) != null) continue;
if (member.GetCustomAttribute<KeyAttribute>(inherit: true)?.IntKey is { } key) highest = Math.Max(highest, key);
}
return highest + 1;
}
// Counts a concrete object's array header against its declared slots and notes the excess into the open CultSlotScope, then lets the
// inner formatter read it unchanged: no byte scan, no second decode. With no scope open it is a depth-free pass-through.
private sealed class SlotFormatter<T>(IMessagePackFormatter<T> inner, int declared) : IMessagePackFormatter<T>
{
public void Serialize(ref MessagePackWriter writer, T value, MessagePackSerializerOptions options) =>
inner.Serialize(ref writer, value, options);
public T Deserialize(ref MessagePackReader reader, MessagePackSerializerOptions options)
{
var scope = CultSlotScope.Current;
if (scope == null) return inner.Deserialize(ref reader, options);
var peek = reader.CreatePeekReader();
if (!peek.TryReadNil() && peek.NextMessagePackType == MessagePackType.Array)
{
var count = peek.ReadArrayHeader();
if (count > declared) scope.Note(typeof(T).Name, count, declared);
}
scope.Depth++;
try
{
return inner.Deserialize(ref reader, options);
}
finally
{
scope.Depth--;
}
}
}
private sealed class Formatter<T>(IMessagePackFormatter<T> inner, UnionAttribute[] arms) : IMessagePackFormatter<T>

View file

@ -636,7 +636,7 @@ public sealed class DirectoryMessagePackBackingStore : CacheBackingStore
record,
VersionOf(metadata.Payload),
catalog,
(type, payload) => CultDocumentMessagePackSerialization.DeserializeUntyped(type, payload, Registry),
(type, payload) => CultDocumentMessagePackSerialization.DeserializeMarked(type, payload, Registry),
out report);
// The manifest is this store's record of what is on disk: the stored id and the version a commit condition compares with.
stored.StoredSchemaId = metadata.SchemaId;

View file

@ -516,14 +516,22 @@ namespace GameCult.Caching
// resolved to Descriptor, until a write of this record puts it under Descriptor's id (a write of an owned record moves it).
public string StoredSchemaId { get; internal set; }
// True when the schema this record was loaded under left none of its persisted slots unread by Descriptor, so the type reads
// it whole and owns it (CultDocumentRegistry.Owns). Set by ToStoredDocument from the schema comparison's ignored slots,
// True when the schema this record was loaded under left none of its persisted slots unread by Descriptor and its payload
// carried none past Descriptor's, so the type reads it whole and owns it (CultDocumentRegistry.Owns). Set by ToStoredDocument from the schema comparison's ignored slots,
// before any variant override is shed. A record this cache constructed or wrote is stored under Descriptor's own id, which
// the registry declares, so only a loaded record under another id is ever asked. The staging half asks it;
// ConditionsHold never reads it. Only the store's loaded view carries it: the cache's resolved copies and flattens are never
// asked.
internal bool StoredSchemaReadWhole { get; set; }
/// <summary>
/// What the store's decode of this record's payload found past the slots this build declares, at any depth; null when none. Set by
/// ToStoredDocument on the store's loaded view only (the cache's resolved copy of a variant does not carry it; ask
/// <see cref="CultCache.IsWritable"/>). A mark that is not <see cref="CultUnreadSlots.Explained"/> keeps the record
/// unowned even under a declared id: it cannot be written or removed (HeldOwned).
/// </summary>
public CultUnreadSlots? UnreadSlots { get; internal set; }
// The catalog entry that publishes StoredSchemaId in the store this record was loaded from, with the id it was published for,
// so staging can ask WrittenOwns the question the store's judge asks, over the catalog the record came with. Null for a record
// this cache constructed or wrote, which is stored under Descriptor's own id and published by Descriptor's entry. A write that
@ -2656,6 +2664,17 @@ namespace GameCult.Caching
return _entries.TryGetValue(key.Value, out var stored) ? stored : null;
}
/// <summary>
/// Whether a write or removal of the key would be let through by every attached store: false when a store lists the key foreign,
/// or holds it unowned (undeclared drift that drops persisted slots, or a payload carrying slots past this build's, see
/// <see cref="CultStoredDocument.UnreadSlots"/>); true for a key no store holds. Live and advisory: the write still decides.
/// </summary>
public bool IsWritable(CultRecordKey key)
{
lock (Gate())
return _stores.All(entry => entry.Store.Refusal(key.Value, null) == null);
}
public object? Get(CultRecordKey key)
{
lock (Gate())
@ -3937,11 +3956,21 @@ namespace GameCult.Caching
string version,
IReadOnlyCollection<CultSchemaCatalogEntry> catalog,
Func<Type, byte[], object> deserializePayload,
out CultSchemaMigrationReport report) =>
ToStoredDocument(record, version, catalog, (type, payload) => new CultDecoded(deserializePayload(type, payload), null), out report);
// The same, for a decode that notes the extra slots it read past (DeserializeMarked): the mark is turned into
// CultStoredDocument.UnreadSlots here, the one place that knows both the payload's marks and the catalog comparison.
internal CultStoredDocument ToStoredDocument(
CultPersistedRecord record,
string version,
IReadOnlyCollection<CultSchemaCatalogEntry> catalog,
Func<Type, byte[], CultDecoded> deserializePayload,
out CultSchemaMigrationReport report)
{
var resolution = Registry.ResolvePersistedSchemaDetailed(record.SchemaId, catalog);
report = resolution.Report;
var readWhole = report.IgnoredExtraSlots.Length == 0;
CultUnreadSlots? unread = null;
if (record.Variant != null)
{
var descriptor = resolution.Descriptor;
@ -3955,13 +3984,22 @@ namespace GameCult.Caching
.ToArray();
if (shed.Length > 0)
{
// A slot the schema diff already reported is not reported twice.
// A slot the schema diff already reported is not reported twice. One it did not report is a mark nothing explains:
// the variant is read-only and is never rewritten.
var unreported = shed.Except(report.IgnoredExtraSlots).ToArray();
if (unreported.Length > 0)
unread = new CultUnreadSlots(
descriptor.DocumentType.Name,
unreported.Max() + 1,
descriptor.RichMembers.Select(member => member.Slot + 1).DefaultIfEmpty(0).Max(),
1,
allAtDepthZero: true,
explained: false);
report.IgnoredExtraSlots = report.IgnoredExtraSlots.Concat(unreported).OrderBy(slot => slot).ToArray();
report.Warnings = report.Warnings.Concat(unreported.Select(slot => new CultSchemaMigrationWarning
{
Code = "ignored_extra_slot",
Message = $"Variant {record.Key} overrides slot {slot}, which {descriptor.DocumentType.Name} does not have; the override is ignored and shed on flush."
Message = $"Variant {record.Key} overrides slot {slot}, which {descriptor.DocumentType.Name} does not have; the variant is read-only."
})).ToArray();
}
@ -3977,16 +4015,17 @@ namespace GameCult.Caching
null)
{
StoredSchemaId = record.SchemaId,
StoredSchemaReadWhole = readWhole,
StoredSchemaReadWhole = report.IgnoredExtraSlots.Length == 0,
UnreadSlots = unread,
StoredPublishedBy = PublishedByIn(record.SchemaId, catalog),
StoredVersion = version
};
}
object document;
CultDecoded decoded;
try
{
document = deserializePayload(resolution.Descriptor.DocumentType, record.Payload);
decoded = deserializePayload(resolution.Descriptor.DocumentType, record.Payload);
}
catch (Exception exception) when (exception is not OperationCanceledException)
{
@ -3995,14 +4034,17 @@ namespace GameCult.Caching
for (var cause = exception; cause != null; cause = cause.InnerException) causes.Add(cause.Message);
throw new InvalidDataException($"Record '{record.Key}' (schema '{record.SchemaId}') cannot be decoded: {string.Join(" <- ", causes.Distinct())}", exception);
}
// Explained only when every mark is at depth 0 and the schema comparison reported ignored slots under this id.
unread = decoded.UnreadSlots?.Explaining(decoded.UnreadSlots.AllAtDepthZero && report.IgnoredExtraSlots.Length > 0);
return new CultStoredDocument(
new CultRecordKey(record.Key),
record.StoredAt,
resolution.Descriptor,
document)
decoded.Document)
{
StoredSchemaId = record.SchemaId,
StoredSchemaReadWhole = readWhole,
StoredSchemaReadWhole = report.IgnoredExtraSlots.Length == 0 && unread == null,
UnreadSlots = unread,
StoredPublishedBy = PublishedByIn(record.SchemaId, catalog),
StoredVersion = version
};
@ -4041,22 +4083,37 @@ namespace GameCult.Caching
// catalog entry that published the held record's id when it was loaded, the question the store's judge asks of the file.
internal void RefuseUndeclared(string key, CultDocumentDescriptor? written)
{
var foreign = _foreignRecords.FirstOrDefault(record => string.Equals(record.Key, key, StringComparison.Ordinal));
if (foreign != null)
throw Overwrites(null, new[] { (foreign.Key, foreign.SchemaId, foreign.SchemaName) });
if (Entries.TryGetValue(key, out var held) && !HeldOwned(held, written))
throw Overwrites(null, new[] { (key, held.StoredSchemaId, held.Descriptor.SchemaName) });
if (Refusal(key, written) is { } refusal) throw refusal;
}
// The refusal of a write (of the written type) or a removal (written null) of the key, or null when this store's view lets it
// through. The one answer behind RefuseUndeclared and CultCache.IsWritable, so the query and the write cannot disagree.
internal CultSchemaConflictException? Refusal(string key, CultDocumentDescriptor? written)
{
var foreign = _foreignRecords.FirstOrDefault(record => string.Equals(record.Key, key, StringComparison.Ordinal));
if (foreign != null)
return Overwrites(null, new[] { (foreign.Key, foreign.SchemaId, foreign.SchemaName) });
if (!Entries.TryGetValue(key, out var held) || HeldOwned(held, written))
return null;
return held.UnreadSlots is { Explained: false } unread
? UnreadSlotsRefusal(held, key, unread)
: Overwrites(null, new[] { (key, held.StoredSchemaId, held.Descriptor.SchemaName) });
}
// The one ownership predicate of a held record. A mark nothing explains (anything but a depth-0 slot the catalog comparison
// reported ignored) keeps the record unowned whatever is declared: declaring the reader's own id consents to nothing.
private bool HeldOwned(CultStoredDocument held, CultDocumentDescriptor? written) =>
written == null
? Registry.Declared(held.StoredSchemaId) || held.StoredSchemaReadWhole
? !Unexplained(held) && (Registry.Declared(held.StoredSchemaId) || held.StoredSchemaReadWhole)
: UpsertOwned(held, written);
private static bool Unexplained(CultStoredDocument held) => held.UnreadSlots is { Explained: false };
// Whether a write of the written type may replace the record a cache holds: WrittenOwns over the entry that published the held
// record's id when it was loaded (a record this cache wrote is published by its own type). The one answer for a store-backed
// cache's staging and a store-less cache's.
internal static bool UpsertOwned(CultStoredDocument held, CultDocumentDescriptor written) =>
!Unexplained(held) &&
CultDocumentRegistry.WrittenOwns(
written.ToCatalogEntry(),
held.StoredSchemaId,
@ -4358,6 +4415,16 @@ namespace GameCult.Caching
refused[0].Key,
inner);
// The refusal of a write or removal of a record held read-only for the slots its payload carries past this build's. It names the
// type and the counts, never a slot value.
private CultSchemaConflictException UnreadSlotsRefusal(CultStoredDocument held, string key, CultUnreadSlots unread) => new(
$"{held.Descriptor.SchemaName} record {key} is read-only: {unread.TypeName} was read with {unread.SlotCount} slots where this build declares {unread.DeclaredCount} " +
$"({unread.Occurrences} object(s) in the record). Writing it would drop data a newer writer stored; read it with types that declare those slots, or write it from that build.",
held.StoredSchemaId,
new[] { held.Descriptor.SchemaName },
key,
unread);
// A flush or commit succeeded: nothing staged is left, writes or checks.
protected void MarkFlushSucceeded()
{
@ -4461,6 +4528,9 @@ namespace GameCult.Caching
protected abstract byte[] SerializePayload(object document, Type documentType);
protected abstract object DeserializePayload(Type documentType, byte[] payload);
// A decode that also notes the slots it read past. A store kind that does not override it never marks.
internal virtual CultDecoded DeserializeMarked(Type documentType, byte[] payload) => new(DeserializePayload(documentType, payload), null);
public override string ToString() => FileInfo.FullName;
public override void PullAll() => Held(PullAllCore);
@ -4495,7 +4565,7 @@ namespace GameCult.Caching
continue;
}
var stored = ToStoredDocument(record, version, snapshot.SchemaCatalog, DeserializePayload, out var report);
var stored = ToStoredDocument(record, version, snapshot.SchemaCatalog, DeserializeMarked, out var report);
reports.Add(report);
persisted[stored.Key.Value] = stored;
}
@ -4963,7 +5033,7 @@ namespace GameCult.Caching
var record = durable?.Snapshot.Records.FirstOrDefault(candidate => string.Equals(candidate.Key, key, StringComparison.Ordinal));
if (record == null || Foreign(record, durable!.Snapshot.SchemaCatalog, durable.Version(key)) != null)
return null;
return ToStoredDocument(record, durable.Version(key), durable.Snapshot.SchemaCatalog, DeserializePayload, out _);
return ToStoredDocument(record, durable.Version(key), durable.Snapshot.SchemaCatalog, DeserializeMarked, out _);
}
// The file now holds every record written, under the id and at the storedAt the write gave it, and the version of those bytes:

View file

@ -5,7 +5,8 @@ namespace GameCult.Caching
/// <summary>
/// A write the catalog cannot describe: two schemas that share a schema id and disagree on the schema name (both arrived with the
/// records, neither registered), a record whose schema id no chosen entry publishes, or a write that would replace or remove a
/// <see cref="CultForeignRecord"/>. Nothing is written, so the store stays as it was. It names the id, the schema names involved
/// <see cref="CultForeignRecord"/>, or a write or removal of a record held read-only because its payload carries slots past the
/// reader's declared slots (<see cref="UnreadSlots"/> is set). Nothing is written, so the store stays as it was. It names the id, the schema names involved
/// and a record key.
/// </summary>
public sealed class CultSchemaConflictException : InvalidOperationException
@ -27,8 +28,18 @@ namespace GameCult.Caching
RecordKey = recordKey;
}
/// <summary>The refusal of a record held read-only for the slots it carries past this build's: the same refusal, with the finding.</summary>
public CultSchemaConflictException(string message, string schemaId, string[] schemaNames, string recordKey, CultUnreadSlots unreadSlots)
: this(message, schemaId, schemaNames, recordKey)
{
UnreadSlots = unreadSlots;
}
public string SchemaId { get; }
public string[] SchemaNames { get; }
public string RecordKey { get; }
/// <summary>Set when the refusal is for unread slots; null for every other refusal.</summary>
public CultUnreadSlots? UnreadSlots { get; }
}
}

View file

@ -0,0 +1,53 @@
namespace GameCult.Caching
{
/// <summary>
/// What a store load found in a record's payload past the slots this build declares, at any depth: the first type read with extra
/// slots and how many slots it carried against how many it declares, and how many objects in the record did. It holds no slot
/// value, so it can be named in a refusal. A record carrying it is read-only (<see cref="CultCache.IsWritable"/>) unless every
/// mark is explained.
/// </summary>
public sealed class CultUnreadSlots
{
internal CultUnreadSlots(string typeName, int slotCount, int declaredCount, int occurrences, bool allAtDepthZero, bool explained)
{
TypeName = typeName;
SlotCount = slotCount;
DeclaredCount = declaredCount;
Occurrences = occurrences;
AllAtDepthZero = allAtDepthZero;
Explained = explained;
}
/// <summary>The first type read with extra slots.</summary>
public string TypeName { get; }
/// <summary>The slots that object carried.</summary>
public int SlotCount { get; }
/// <summary>The slots this build declares for that type (highest key plus one).</summary>
public int DeclaredCount { get; }
/// <summary>The objects in the record that carried extra slots.</summary>
public int Occurrences { get; }
/// <summary>
/// True only when every mark is at depth 0 and the schema comparison reported ignored slots under the id the record was stored
/// under: the loss a compatible-id declaration consents to (cultlib-gaps:ruling:slot-mark-consent). Any other mark leaves the
/// record read-only whatever is declared.
/// </summary>
public bool Explained { get; }
// Every mark the decode noted sat in the record's own object, the half of "explained" the payload decides.
internal bool AllAtDepthZero { get; }
internal CultUnreadSlots Explaining(bool explained) =>
new(TypeName, SlotCount, DeclaredCount, Occurrences, AllAtDepthZero, explained);
}
// A payload decoded by a store kind, with the extra slots the decode noted. A store kind that does not note them reports none.
internal readonly struct CultDecoded(object document, CultUnreadSlots? unreadSlots)
{
public object Document { get; } = document;
public CultUnreadSlots? UnreadSlots { get; } = unreadSlots;
}
}

View file

@ -0,0 +1,310 @@
#nullable enable
using System;
using System.Buffers;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using GameCult.Caching.MessagePack;
using MessagePack;
using NUnit.Framework;
using Surface = GameCult.Caching.Tests.UnionParityVectorTests.UnionSurface;
namespace GameCult.Caching.Tests
{
// cultlib-gaps:ruling:extra-trailing-slots and :slot-mark-consent. A record whose payload carries a slot past its reader's declared
// slots, at any depth, is held read-only: every staged write or removal of it is refused before anything is staged. A compatible-id
// declaration consents only to depth-0 slots the catalog comparison reports ignored under a listed compatible id.
public class SlotReadOnlyTests
{
private const string Extra = "extra-slot";
// Same members as UnionSurface under a new version, listing OldId compatible: the declaration that must not consent to a nested mark.
private const string OldId = "tests.slot_readonly.old";
[CultDocument("tests.union_surface", "tests.union_surface.v2", CompatibleSchemaIds = new[] { OldId })]
[MessagePackObject]
public sealed class SurfaceDeclaring
{
[Key(0)] [CultName] public string Id = string.Empty;
[Key(1)] public int Version;
[Key(2)] public UnionParityVectorTests.Node? Root;
[Key(3)] public List<UnionParityVectorTests.Outcome> Outcomes = new();
[Key(4)] public UnionParityVectorTests.Item? Part;
[Key(5)] public UnionParityVectorTests.CraftedItem? Crafted;
[Key(6)] public UnionParityVectorTests.Node? Missing;
}
[CultDocument("tests.slot_unit", "tests.slot_unit.v1")]
[MessagePackObject]
public sealed class UnitDoc
{
[Key(0)] [CultName] public string Name = string.Empty;
[Key(1)] public UnionParityVectorTests.UnitOutcome? Outcome;
}
// Slot 1 and 2 are retired gaps.
[CultDocument("tests.slot_gap", "tests.slot_gap.v1")]
[MessagePackObject]
public sealed class GapDoc
{
[Key(0)] [CultName] public string Name = string.Empty;
[Key(3)] public string Tail = string.Empty;
}
private static readonly CultDocumentRegistry SurfaceRegistry = CultDocumentRegistry.ForTypes(new[] { typeof(Surface) });
private static readonly CultDocumentRegistry DeclaringRegistry = CultDocumentRegistry.ForTypes(new[] { typeof(SurfaceDeclaring) });
private static readonly CultDocumentRegistry UnitRegistry = CultDocumentRegistry.ForTypes(new[] { typeof(UnitDoc) });
private static readonly CultDocumentRegistry GapRegistry = CultDocumentRegistry.ForTypes(new[] { typeof(GapDoc) });
private static readonly CultRecordKey Marked = new("marked");
private static readonly CultRecordKey Clean = new("clean");
private static readonly CultRecordKey Unheld = new("unheld");
private string _directory = string.Empty;
[SetUp]
public void CreateDirectory()
{
_directory = Path.Combine(Path.GetTempPath(), $"cultlib-slotro-{Guid.NewGuid():N}");
Directory.CreateDirectory(_directory);
}
[TearDown]
public void DeleteDirectory()
{
if (Directory.Exists(_directory)) Directory.Delete(_directory, recursive: true);
}
private static CultCache Open(string path, CultDocumentRegistry registry) =>
CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = registry });
private static byte[] Emit(WriterAction action)
{
var buffer = new ArrayBufferWriter<byte>();
var writer = new MessagePackWriter(buffer);
action(ref writer);
writer.Flush();
return buffer.WrittenSpan.ToArray();
}
private delegate void WriterAction(ref MessagePackWriter writer);
// A store of two records of the given type, the marked one carrying the payload and the clean one an ordinary value.
private string Seed(CultDocumentRegistry registry, Type type, Func<object> clean, byte[] markedPayload, string name = "slots.cc")
{
var path = Path.Combine(_directory, name);
using (var cache = Open(path, registry))
{
cache.UpsertAsync(type, clean(), Marked).GetAwaiter().GetResult();
cache.UpsertAsync(type, clean(), Clean).GetAwaiter().GetResult();
cache.FlushAsync().GetAwaiter().GetResult();
}
var snapshot = CultDocumentMessagePackSerialization.DeserializeSnapshot(File.ReadAllBytes(path));
snapshot.Records.Single(record => record.Key == Marked.Value).Payload = markedPayload;
File.WriteAllBytes(path, CultDocumentMessagePackSerialization.SerializeSnapshot(snapshot));
return path;
}
private string SeedSurface(string vector) =>
Seed(SurfaceRegistry, typeof(Surface), () => new Surface { Id = "x" }, UnionParityVectorTests.Committed(vector));
private static CultStoredDocument Stored(CultCache cache, CultRecordKey key) => cache.GetStored(key)!;
// ---- a mark at every depth, with the type and both counts ----
[TestCase("extra-slot-record.msgpack", "UnionSurface", 8, 7, 1)]
[TestCase("extra-slot-nested-record.msgpack", nameof(UnionParityVectorTests.Composition), 3, 2, 1)]
[TestCase("extra-slot-arm.msgpack", nameof(UnionParityVectorTests.Box), 3, 2, 1)]
[TestCase("extra-slot-list-element.msgpack", nameof(UnionParityVectorTests.Text), 3, 2, 1)]
[TestCase("depth-skipped-at-limit.msgpack", "UnionSurface", 8, 7, 1)]
public void AMarkAtAnyDepthHoldsTheRecordReadOnly(string vector, string typeName, int slots, int declared, int occurrences)
{
var path = SeedSurface(vector);
using var cache = Open(path, SurfaceRegistry);
var unread = Stored(cache, Marked).UnreadSlots;
Assert.That(unread, Is.Not.Null, vector);
Assert.That(unread!.TypeName, Is.EqualTo(typeName), vector);
Assert.That(unread.SlotCount, Is.EqualTo(slots), vector);
Assert.That(unread.DeclaredCount, Is.EqualTo(declared), vector);
Assert.That(unread.Occurrences, Is.EqualTo(occurrences), vector);
Assert.That(unread.Explained, Is.False, "the reader's own id is not a compatible-id declaration: nothing explains the mark");
Assert.That(cache.IsWritable(Marked), Is.False, vector);
Assert.That(cache.Get<Surface>(Marked), Is.Not.Null, "a read-only record is still served");
Assert.That(Stored(cache, Clean).UnreadSlots, Is.Null);
Assert.That(cache.IsWritable(Clean), Is.True);
Assert.That(cache.IsWritable(Unheld), Is.True, "a key no store holds is writable");
}
[Test]
public void AUnitArmWithAnExtraSlotHoldsTheRecordReadOnly()
{
var arm = UnionParityVectorTests.Committed("extra-slot-unit-arm.msgpack");
var payload = Emit((ref MessagePackWriter w) => { w.WriteArrayHeader(2); w.Write("u"); w.WriteRaw(arm); });
var path = Seed(UnitRegistry, typeof(UnitDoc), () => new UnitDoc { Name = "u" }, payload);
using var cache = Open(path, UnitRegistry);
var unread = Stored(cache, Marked).UnreadSlots;
Assert.That(unread, Is.Not.Null);
Assert.That(unread!.TypeName, Is.EqualTo(nameof(UnionParityVectorTests.Denied2)));
Assert.That(unread.SlotCount, Is.EqualTo(1));
Assert.That(unread.DeclaredCount, Is.EqualTo(0));
Assert.That(cache.IsWritable(Marked), Is.False);
Assert.That(cache.IsWritable(Clean), Is.True);
}
// ---- every staged write and removal of a marked key is refused, nothing else is ----
private static void AssertSlotRefusal(TestDelegate act, string what, string type, int slots, int declared)
{
var refusal = Assert.Throws<CultSchemaConflictException>(act, what)!;
Assert.That(refusal.UnreadSlots, Is.Not.Null, what);
Assert.That(refusal.RecordKey, Is.EqualTo("marked"), what);
Assert.That(refusal.Message, Does.Contain("is read-only: " + type + " was read with " + slots + " slots where this build declares " + declared)
.And.Contain("record marked"), what);
Assert.That(refusal.Message, Does.Not.Contain(Extra), $"{what}: the refusal never echoes a slot value");
}
[Test]
public void EveryWriteAndRemovalOfAMarkedRecordIsRefusedAndAnotherKeyStillWrites()
{
var path = SeedSurface("extra-slot-nested-record.msgpack");
var before = File.ReadAllBytes(path);
using (var cache = Open(path, SurfaceRegistry))
{
var composition = nameof(UnionParityVectorTests.Composition);
AssertSlotRefusal(() => cache.Commit(batch => batch.Upsert(new Surface { Id = "new" }, new CultRecordHandle<Surface>(Marked))), "upsert", composition, 3, 2);
AssertSlotRefusal(() => cache.Commit(batch => batch.UpsertVariant(Marked, Clean)), "variant", composition, 3, 2);
AssertSlotRefusal(() => cache.Commit(batch => batch.Remove(Marked)), "remove", composition, 3, 2);
AssertSlotRefusal(() => cache.Commit(batch => { batch.Upsert(new Surface { Id = "other" }, new CultRecordHandle<Surface>(Clean)); batch.Remove(Marked); }), "mixed batch", composition, 3, 2);
Assert.That(File.ReadAllBytes(path), Is.EqualTo(before), "refused writes leave the store as it was");
Assert.That(cache.Commit(batch => batch.Upsert(new Surface { Id = "changed" }, new CultRecordHandle<Surface>(Clean))), Is.True);
}
Assert.That(CultDocumentMessagePackSerialization.DeserializeSnapshot(File.ReadAllBytes(path)).Records.Count, Is.EqualTo(2));
}
// ---- consent ----
[Test]
public void ADeclaredIdDoesNotConsentToAnUnexplainedMark()
{
// The reader's own id is declared; the mark at depth 0 is in no compatible id's comparison: still read-only.
var path = SeedSurface("extra-slot-record.msgpack");
using var cache = Open(path, SurfaceRegistry);
AssertSlotRefusal(() => cache.Commit(batch => batch.Upsert(new Surface { Id = "new" }, new CultRecordHandle<Surface>(Marked))), "depth 0 under the reader's own id", "UnionSurface", 8, 7);
AssertSlotRefusal(() => cache.Commit(batch => batch.Remove(Marked)), "removal", "UnionSurface", 8, 7);
}
[Test]
public void AListedCompatibleIdDoesNotConsentToANestedMark()
{
// The record's stored id is one the reader lists compatible; the extra slot is nested, so the declaration does not reach it.
var path = SeedSurface("extra-slot-nested-record.msgpack");
Relabel(path, OldId);
using var cache = Open(path, DeclaringRegistry);
Assert.That(Stored(cache, Marked).StoredSchemaId, Is.EqualTo(OldId));
Assert.That(cache.IsWritable(Marked), Is.False);
AssertSlotRefusal(() => cache.Commit(batch => batch.Upsert(new SurfaceDeclaring { Id = "new" }, new CultRecordHandle<SurfaceDeclaring>(Marked))), "nested under a listed id", nameof(UnionParityVectorTests.Composition), 3, 2);
Assert.That(cache.IsWritable(Clean), Is.True, "the clean record under the listed id is consented to (declared)");
Assert.That(cache.Commit(batch => batch.Upsert(new SurfaceDeclaring { Id = "new" }, new CultRecordHandle<SurfaceDeclaring>(Clean))), Is.True);
}
private static void Relabel(string path, string id)
{
var snapshot = CultDocumentMessagePackSerialization.DeserializeSnapshot(File.ReadAllBytes(path));
snapshot.SchemaCatalog.Single().SchemaId = id;
foreach (var record in snapshot.Records) record.SchemaId = id;
File.WriteAllBytes(path, CultDocumentMessagePackSerialization.SerializeSnapshot(snapshot));
}
// ---- carried verbatim ----
[Test]
public void AMarkedRecordIsCarriedByteForByteWhenAnotherRecordIsWritten()
{
var path = Path.Combine(_directory, "extra-slot-store.cc");
File.WriteAllBytes(path, UnionParityVectorTests.Committed("extra-slot-store.cc"));
var before = CultDocumentMessagePackSerialization.DeserializeSnapshot(File.ReadAllBytes(path));
var original = before.Records.Single(record => record.Key == Marked.Value);
using (var cache = Open(path, SurfaceRegistry))
{
Assert.That(cache.IsWritable(Marked), Is.False);
Assert.That(cache.Commit(batch => batch.Upsert(new Surface { Id = "changed" }, new CultRecordHandle<Surface>(Clean))), Is.True);
}
var after = CultDocumentMessagePackSerialization.DeserializeSnapshot(File.ReadAllBytes(path));
var carried = after.Records.Single(record => record.Key == Marked.Value);
Assert.That(carried.Payload, Is.EqualTo(original.Payload));
Assert.That(carried.SchemaId, Is.EqualTo(original.SchemaId));
Assert.That(carried.StoredAt, Is.EqualTo(original.StoredAt));
Assert.That(after.SchemaCatalog.Select(CultDocumentMessagePackSerialization.SerializeCatalogEntry).Select(Convert.ToHexString),
Is.EqualTo(before.SchemaCatalog.Select(CultDocumentMessagePackSerialization.SerializeCatalogEntry).Select(Convert.ToHexString)));
Assert.That(after.Records.Single(record => record.Key == Clean.Value).Payload, Is.Not.EqualTo(before.Records.Single(record => record.Key == Clean.Value).Payload));
}
// ---- slots short of the reader's, and gaps below the highest key, do not mark ----
[Test]
public void MissingTrailingSlotsDefaultAndAGapSlotDoesNotMark()
{
var short1 = Emit((ref MessagePackWriter w) => { w.WriteArrayHeader(1); w.Write("short"); });
var path = Seed(GapRegistry, typeof(GapDoc), () => new GapDoc { Name = "g" }, short1);
using (var cache = Open(path, GapRegistry))
{
Assert.That(cache.Get<GapDoc>(Marked)!.Tail, Is.EqualTo(string.Empty), "a missing trailing slot keeps its default");
Assert.That(Stored(cache, Marked).UnreadSlots, Is.Null);
Assert.That(cache.IsWritable(Marked), Is.True);
Assert.That(cache.Commit(batch => batch.Upsert(new GapDoc { Name = "short", Tail = "t" }, new CultRecordHandle<GapDoc>(Marked))), Is.True);
}
var gapped = Emit((ref MessagePackWriter w) => { w.WriteArrayHeader(4); w.Write("gap"); w.WriteNil(); w.Write("in a retired slot"); w.Write("tail"); });
var gapPath = Seed(GapRegistry, typeof(GapDoc), () => new GapDoc { Name = "g" }, gapped, "gap.cc");
using (var cache = Open(gapPath, GapRegistry))
{
Assert.That(Stored(cache, Marked).UnreadSlots, Is.Null, "a non-nil value in a retired gap below the highest key does not mark");
Assert.That(cache.IsWritable(Marked), Is.True);
}
}
// ---- the catalog's consent still works, and is reported as explained ----
[CultDocument("tests.slot_wide", "tests.slot_wide.v1")]
[MessagePackObject]
public sealed class WideDoc
{
[Key(0)] [CultName] public string Name = string.Empty;
[Key(1)] public string Colour = string.Empty;
[Key(2)] public string Extra = string.Empty;
}
[CultDocument("tests.slot_wide", "tests.slot_wide.v2")]
[MessagePackObject]
public sealed class NarrowDoc
{
[Key(0)] [CultName] public string Name = string.Empty;
[Key(1)] public string Colour = string.Empty;
}
[Test]
public void ADepthZeroSlotTheCatalogReportsIsExplainedAndTheUndeclaredRecordStaysUnwritable([Values] bool directory)
{
var path = Path.Combine(_directory, directory ? "store" : "store.cc");
CacheBackingStore Store() => directory ? new DirectoryMessagePackBackingStore(path) : new SingleFileMessagePackBackingStore(path);
using (var writer = new CultCache(CultDocumentRegistry.ForTypes(new[] { typeof(WideDoc) })))
{
writer.AddBackingStore(Store());
Assert.That(writer.Commit(batch => batch.Upsert(new WideDoc { Name = "w", Extra = "kept?" }, new CultRecordHandle<WideDoc>(Marked))), Is.True);
}
using var reader = new CultCache(CultDocumentRegistry.ForTypes(new[] { typeof(NarrowDoc) }));
reader.AddBackingStore(Store());
var unread = Stored(reader, Marked).UnreadSlots;
Assert.That(unread, Is.Not.Null, "the store kind decodes through the marking path");
Assert.That(unread!.Explained, Is.True, "depth 0 and reported by the catalog comparison");
Assert.That(unread.TypeName, Is.EqualTo(nameof(NarrowDoc)));
Assert.That(unread.SlotCount, Is.EqualTo(3));
Assert.That(unread.DeclaredCount, Is.EqualTo(2));
Assert.That(reader.IsWritable(Marked), Is.False, "the wide id is undeclared: unowned, though the mark is explained");
Assert.Throws<CultSchemaConflictException>(() => reader.Commit(batch => batch.Remove(Marked)));
}
}
}

View file

@ -58,7 +58,7 @@ namespace GameCult.Caching.Tests
Assert.That(Convert.ToHexString(reference), Is.EqualTo(Convert.ToHexString(File.ReadAllBytes(path))), $"vector {name}");
}
private static byte[] Committed(string name)
internal static byte[] Committed(string name)
{
var path = VectorPath(name);
Assert.That(File.Exists(path), Is.True, $"vector {name} is committed");
@ -297,6 +297,9 @@ namespace GameCult.Caching.Tests
Surface(extra: new Extra(RootArm: true))),
new Case("extra-slot-list-element.msgpack", "read-only", "Node", 1, "list-element",
Surface(extra: new Extra(ListElement: true))),
// A bare UnitOutcome arm (the fixture's unit-arm union), not a UnionSurface payload: [2, [extra]] where Denied2 declares no slot.
new Case("extra-slot-unit-arm.msgpack", "read-only", "UnitOutcome", 2, "unit-arm",
Raw(ArmOf(2, ExtraSlot))),
};
private static byte[] OutcomesFile() => Raw((ref MessagePackWriter w) =>
@ -317,6 +320,7 @@ namespace GameCult.Caching.Tests
"Node" => typeof(Node).FullName!,
"Outcome" => typeof(Outcome).FullName!,
"CraftedItem" => typeof(CraftedItem).FullName!,
"UnitOutcome" => typeof(UnitOutcome).FullName!,
"UnionSurface" => typeof(UnionSurface).FullName!,
_ => throw new ArgumentOutOfRangeException(nameof(logical), logical, null)
};
@ -413,6 +417,34 @@ namespace GameCult.Caching.Tests
}
}
// The store the slot-readonly tests open and the other runtimes read: the extra-slot-nested-record payload as the record "marked"
// beside the ordinary value as "clean". The store's clock is pinned as union-surface.cc's.
[Test]
public void ExtraSlotStoreVectorHoldsAMarkedAndACleanRecord()
{
var work = Path.Combine(Path.GetTempPath(), $"cultlib-unionparity-{Guid.NewGuid():N}");
Directory.CreateDirectory(work);
try
{
var scratch = Path.Combine(work, "extra-slot-store.cc");
using (var cache = CultCacheMessagePack.Create(scratch, new CultCacheOpenOptions { Registry = Registry, StoreFlushOnDispose = true }))
{
cache.Clock = () => StoredAt;
cache.UpsertAsync(typeof(UnionSurface), Constructed(), new CultRecordKey("marked")).GetAwaiter().GetResult();
cache.UpsertAsync(typeof(UnionSurface), Constructed(), new CultRecordKey("clean")).GetAwaiter().GetResult();
cache.FlushAsync().GetAwaiter().GetResult();
}
var snapshot = CultDocumentMessagePackSerialization.DeserializeSnapshot(File.ReadAllBytes(scratch));
snapshot.Records.Single(record => record.Key == "marked").Payload = Committed("extra-slot-nested-record.msgpack");
Check("extra-slot-store.cc", CultDocumentMessagePackSerialization.SerializeSnapshot(snapshot));
}
finally
{
Directory.Delete(work, recursive: true);
}
}
// The value names every arm shape the vectors exist to hold other runtimes to.
[Test]
public void ConstructedValueExercisesEveryArmShape()
@ -475,8 +507,11 @@ namespace GameCult.Caching.Tests
else
{
Assert.That(outcome, Is.EqualTo("read-only"), file);
// The record is read whole; what makes it read-only is the slot rule of the slot-tolerance cut, not asserted here.
Assert.That(Describe(Decode(payload)), Is.EqualTo(golden), $"{file}: the extra slot is read past");
// The extra slot is read past; that the record then loads read-only is asserted in SlotReadOnlyTests, store by store.
if (union == "UnitOutcome")
Assert.That(MessagePackSerializer.Deserialize<UnitOutcome>(payload, UnitOptions), Is.TypeOf<Denied2>(), $"{file}: the extra slot is read past");
else
Assert.That(Describe(Decode(payload)), Is.EqualTo(golden), $"{file}: the extra slot is read past");
}
}
}

View file

@ -1039,7 +1039,7 @@ namespace GameCult.Caching.Tests
}
[Test]
public void AnOverrideOfASlotTheTypeNeverHadIsReportedOnEveryLoadAndStaysStoredUntilTheVariantIsWritten()
public void AnOverrideOfASlotTheTypeNeverHadIsReportedOnEveryLoadAndHoldsTheVariantReadOnly()
{
var path = WriteStore(PlainGear("hand-base"), OverridingRecord("hand-variant", "hand-base",
CultVariantOverride.Set(99, new byte[] { 0x01 }),
@ -1059,14 +1059,22 @@ namespace GameCult.Caching.Tests
{
Assert.That(reopened.BackingStores[0].LastSchemaMigrationReports.SelectMany(report => report.IgnoredExtraSlots), Is.EqualTo(new[] { 99 }));
Assert.That(reopened.Get<VariantGear>(new CultRecordKey("hand-variant"))!.Name, Is.EqualTo("hand variant"));
reopened.Commit(batch => batch.UpsertVariant(new CultRecordKey("hand-variant"), new CultRecordKey("hand-base"), new[]
var variant = new CultRecordKey("hand-variant");
Assert.That(reopened.IsWritable(variant), Is.False, "an override of a slot the type lacks, which the catalog did not report, is a mark nothing explains");
var overrides = new[]
{
reopened.Override<VariantGear>(nameof(VariantGear.Name), "hand variant"),
reopened.Override<VariantGear>(nameof(VariantGear.Code), "hand-code")
}));
};
var rewrite = Assert.Throws<CultSchemaConflictException>(() => reopened.Commit(batch => batch.UpsertVariant(variant, new CultRecordKey("hand-base"), overrides)))!;
Assert.That(rewrite.UnreadSlots, Is.Not.Null);
Assert.That(rewrite.UnreadSlots!.TypeName, Is.EqualTo(nameof(VariantGear)));
Assert.That(rewrite.UnreadSlots.SlotCount, Is.EqualTo(100));
Assert.Throws<CultSchemaConflictException>(() => reopened.Commit(batch => batch.Flatten(variant)));
Assert.Throws<CultSchemaConflictException>(() => reopened.Commit(batch => batch.Remove(variant)));
}
Assert.That(OverriddenSlotsOn(path, "hand-variant").OrderBy(slot => slot), Is.EqualTo(new[] { 0, 2 }), "a write of the variant stores only what it was given");
Assert.That(OverriddenSlotsOn(path, "hand-variant").OrderBy(slot => slot), Is.EqualTo(new[] { 0, 2, 99 }), "the refused writes left the variant as stored");
}
private static int[] OverriddenSlotsOn(string path, string key) =>

View file

@ -58,9 +58,19 @@ of it refused). A reader decodes the value of `union-surface.msgpack` from each;
| `extra-slot-nested-record.msgpack` | nested-record | Composition `comp` |
| `extra-slot-arm.msgpack` | arm | the Root Box arm |
| `extra-slot-list-element.msgpack` | list-element | Text `s1`, an element of Use.Slots |
| `extra-slot-unit-arm.msgpack` | unit-arm | not a UnionSurface: a bare UnitOutcome arm `[2, [extra]]` where Denied2 declares no slot (union `UnitOutcome`, key 2) |
The C# reference reads these today and its slot-tolerance cut is what makes them read-only; this cut asserts only
that the value is read.
`depth-skipped-at-limit.msgpack` (bounds file) is also `read-only`: its extra slot is a deeply nested value at depth 0.
Five rows of `outcomes.msgpack` are `read-only` (record, nested-record, arm, list-element, unit-arm).
`extra-slot-store.cc` is a single-file store holding two records of `tests.union_surface`: `marked` (the
`extra-slot-nested-record.msgpack` payload) and `clean` (the ordinary value); the store's clock is pinned to
2026-01-01T00:00:00Z as `union-surface.cc`'s.
Every runtime asserts the same for a `read-only` payload loaded as a store record: the value is read past the extra
slot, the record reports the unread slots (the first type read with them, the slots it carried and the slots the
reader declares) and is not writable, and a write or removal of it is refused before anything is staged. Opening
`extra-slot-store.cc` and writing `clean` leaves `marked` byte for byte, with its schema id and its catalog entry.
## Bounds (`bounds-outcomes.msgpack`)

Binary file not shown.

View file

@ -0,0 +1 @@
’‘ªextra-slot