CultLib/src/GameCult.Caching/Contracts/cultcache-persistence-format.md
Meta Vens 86dc2e33d0 Contract: the cache owns the schema-id stamp, records resolve by schema id, writes may not retype
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-09-30 11:04:16 +02:00

26 KiB

CultCache Persistence Format

This document is the canonical design receipt for the next CultCache storage format. GameCult.Caching in C# is the source of truth for this contract. Rust, TypeScript, and other siblings should follow once the shape stabilizes here.

The goal is to preserve CultCache's best property: application code should work with domain data, not with a little parade of storage metadata stapled onto every object until the object forgets what it was for.

Why This Exists

The current stack has two conflicting virtues:

  • domain ergonomics: users want plain typed objects
  • interop durability: caches in different languages need enough persisted information to read each other's data, compare schemas, and perform soft migration when shapes drift

The old C# shape leaned hard toward ergonomics by exposing DatabaseEntry.ID directly and letting backing stores carry a lot of implicit responsibility. The newer Rust draft leaned hard toward persistence machinery by wrapping payloads in a heavier envelope. Both capture part of the truth. Neither is the final machine.

The intended format keeps storage metadata out of domain objects while making the persisted store self-describing enough for cross-language readers and training-data archaeology.

Core Rule

Domain payloads contain domain truth only.

These are not domain fields and should not be exposed on app-facing types:

  • CultCache record key / GUID
  • persistence schema id
  • storage timestamp

Those belong to the CultCache persistence layer.

Transaction Visibility

A multi-record commit is an explicit batch of records that resolve to one home store. The store commits the batch as one durable step. Nothing in the batch is visible, to the committing flow or to observers, until the store has accepted it; a failed commit changes nothing on disk or in memory.

A batch may carry conditions: per-record (schemaId, storedAt) identity, or the whole store unchanged since it was last loaded. Conditions are evaluated under the store's exclusive lock against what is durably on disk. A failed condition is a lost race, not an error: the commit reports it and writes nothing.

Conditional commit is the only safe multi-process write. A plain flush and an unconditional commit are last-writer-wins and write the same bytes: a single file is replaced by the writer's whole view, a directory manifest takes the writer's staged keys over the current manifest. An unconditional commit also persists writes staged earlier and leaves the store clean. Processes sharing a store must all use conditional commit. The last writer wins only over a file it can read: a single-file flush or commit refuses a file whose header it cannot read (truncated, or a format it does not read) and leaves the file as it was.

A store that fails to hydrate on open (corrupt bytes, unresolvable schema) makes the open throw and is left byte-identical; it is never silently overwritten. Within one process the lock order is the cache's gate, then the store's file lock; a store's I/O blocks its cache's readers.

C# implements batch and conditional commit (CultCache.Commit, TryCommit, CultCacheBatch). Rust implements both. TypeScript and Python implement neither yet. Routing, batches and conditions are specified in cultcache-store-composition.md.

Canonical Store Shape

A persisted CultCache file should contain:

  1. store header
  2. embedded schema catalog
  3. record set

High-level shape:

{
  "formatVersion": "cultcache.store.v1",
  "catalog": {
    "catalogVersion": "cultcache.schema_catalog.v1",
    "schemas": []
  },
  "records": []
}

The on-disk encoding does not have to be JSON. MessagePack is the expected default. The shape above is explanatory.

Store Header

Required header fields:

  • formatVersion
  • catalog
  • records

Optional header fields:

  • createdAt
  • updatedAt
  • storeId
  • producer
  • notes

These are store-level bookkeeping fields, not per-record domain metadata.

Embedded Schema Catalog

The store must embed a catalog entry for every schema referenced by the records it contains.

A reader never recovers a schema from a record's payload: a record whose schema id the catalog does not publish (by id or as a compatible id) makes the store unreadable, whatever the payload's first field says. The catalog is the store's own description of itself.

A writer keeps the same invariant: every record it writes carries a schema id that some catalog entry it writes publishes, as that entry's id or as one of its compatible ids. A cache stamps a record with its registered schema's id, whatever id the record arrived under, when it loads the record and when it puts one. The cache is the only owner of that stamp: a store writer never changes the identity of a record it did not receive from its caller, so a record another writer changed since this cache read it is written back as that writer left it.

A cache resolves a record to a local type by the record's schema id: the registered type that owns the id, else one that lists it as compatible. The name the catalog entry carries is metadata. It names a local type only when no local type has the id, for a store written by a runtime whose schema ids this one cannot know (a Rust type's schema id is its entry type). A schema renamed under a stable id therefore opens in every runtime that holds the id.

A reader resolves a record's schema id to the entry that owns it, that entry's own schemaId, and only when no entry owns the id to an entry that lists it as a compatible id (the first such entry in catalog order).

A writer derives the catalog from the records it writes, and from nothing else. For each distinct schema id a record carries it chooses exactly one entry that publishes it: the entry that owns the id (a registered schema's descriptor, else an entry the record arrived with), else an entry that lists it as compatible (again registered first). When a registered descriptor and an arrived entry share an own id, the descriptor wins: a rename or a stale content hash takes the descriptor's name, hash, members and compatible ids as they are. Entries are written as chosen, deduplicated by own id, with no union of compatible ids and no dependence on the order records arrive in; an entry no record needs is not written. A write is refused, and the store left as it was, when two entries of one tier share an own id and disagree on the schema name, when records of different types share a schema id, when a write would change which type an existing record resolves to (the schema names of the records already in the store and of the records written differ), or when a record's id is published by no chosen entry. The retyping refusal has one exception, a rename: when the entries of both the existing records and the written ones own the id, the registered descriptor wins. (A C# cache stamps a record with its descriptor's own id, so it cannot write a record under an id its descriptor only lists, and the retyping refusal cannot arise there.) Each runtime refuses with one typed error naming the id, the schema names and a record key: CultSchemaConflictException (C#), SchemaConflictError (Rust, TypeScript, Python).

This is not decorative paperwork. It is what allows another CultCache implementation to inspect an old store, resolve the schema for each record, and decide whether it can:

  • read directly
  • soft-migrate with warnings
  • or refuse the data honestly

Schema Catalog Entry

Each schema descriptor should contain at least:

  • schemaId
  • schemaName
  • schemaVersion
  • contentHash
  • canonicalSchema

Recommended optional fields:

  • migrationFamily
  • supersedesSchemaIds
  • compatibleReaderFamilies
  • persistenceEncoding
  • notes

Example:

{
  "schemaId": "sha256:8b52c6...",
  "schemaName": "gamecult.player-data",
  "schemaVersion": "1.2.0",
  "contentHash": "sha256:52ff9c...",
  "migrationFamily": "player-data",
  "supersedesSchemaIds": ["sha256:0ce41a..."],
  "persistenceEncoding": "messagepack-array-v1",
  "canonicalSchema": {
    "kind": "messagepack-array-object",
    "fields": [
      { "slot": 0, "name": "Name", "type": "string", "required": true },
      { "slot": 1, "name": "Email", "type": "string", "required": true },
      { "slot": 2, "name": "Level", "type": "int32", "required": false, "default": 0 }
    ]
  }
}

schemaId

schemaId is the stable machine key. It should be derived from the canonical semantic persistence schema, not from:

  • repo path
  • C# type name
  • Rust type name
  • TypeScript interface name
  • generated file name

If two implementations describe the same persisted shape, they should derive the same schemaId regardless of origin language.

contentHash

contentHash is separate from schemaId.

  • schemaId answers: "is this the same semantic persistence shape?"
  • contentHash answers: "is this the exact same canonical schema document?"

This distinction matters because descriptions, examples, and other non-semantic surface details may change without changing the actual persisted shape.

Record Shape

Each persisted record should carry only the storage metadata needed to route and reconstruct the domain object:

  • key
  • schemaId
  • storedAt
  • payload

Example:

{
  "key": "e1439d7e-8da8-4b63-a0b1-238c815a8a17",
  "schemaId": "sha256:8b52c6...",
  "storedAt": "2026-05-08T12:00:00Z",
  "payload": "<messagepack bytes>"
}

The payload is the domain object only. It does not repeat the key, schema id, or stored timestamp.

Under cultcache.store.v1 and cultcache.store.v3 a record has exactly these four slots. A reader refuses any cultcache.store.* header it does not know and any record with more slots than its header allows, naming the version or the record's key and schema id. It never skips a slot it does not understand.

The headers are v1 (no variant, no element ids), v2 (holds a variant) and v3 (holds an element id, variants or not). A single-file store declares v3 when any record it holds carries an element id: a non-empty [CultElementId] value in a record's document, or in a variant's override values. A store that cannot read a variant's override values (no codec attached) counts the variant as holding one. A whole-store writer decides by what the store holds now, so the file drops back to v1 (or v2 with a variant) when the last id is taken away; a writer that touches only part of the file cannot see the rest, so a file already declared v3 stays v3 when it merges onto it. A directory store's manifest says cultcache.store.v5.directory-content-addressed-pages under the same rule, and v4 otherwise.

A single-file store is exactly one MessagePack array of three slots: the header, the schema catalog and the records. Every runtime refuses a file that is truncated, has bytes after the array (a second store appended included), is not an array, has fewer or more than three slots (a header alone included), whose first slot is not a header it reads, or whose catalog or records it cannot decode (a record with more slots than its header allows, a variant record it does not read). A runtime never skips a slot it does not understand. It refuses on open and on rewrite alike, because the rewrite asks the same full reader as the open. A zero-byte file and an empty array are an empty store. A flush or a commit replaces a file only when the runtime's own reader opens it, so one verdict covers open, flush and commit, and a file the runtime cannot read is left as it is. Each runtime refuses with one typed error: CultStoreUnreadableException (C#), StoreUnreadableError (Rust, TypeScript, Python), with the underlying cause attached. Rust, TypeScript and Python also open the legacy envelope array (an array of key/type/payload/storedAt maps) and replace it; C# has no legacy reader and refuses it. The shared byte vectors and each runtime's verdict are tests/vectors/document-variants-c2a/readability/manifest.txt.

Variant Records

A variant record is a record whose document is a base record's document with some members replaced. It stores the difference, never the resolved document. It carries a fifth slot:

[key, schemaId, storedAt, payload, [baseKey, overrides[]]]
  • payload is empty. An older reader that ignores the fifth slot reads an empty payload as a corrupt document and refuses, which is why the delta lives outside it.
  • schemaId is the type's schema id, the same as the base's. A variant has its base's concrete type, and lives in the same store (one home store per type).
  • An override is [op, path[], id, value]. op is 0 Set, 1 Insert or 2 Remove. A path step is [slot, elementId]: slot is the member's MessagePack [Key] and elementId is "" unless the step enters an object-list element. value is the member's own MessagePack encoding, inline. A top-level Set has one step and id ""; a reader that resolves only that shape refuses the others, naming the variant.
  • A store holding at least one variant declares cultcache.store.v2 (v3 when it holds an element id); a store holding none declares cultcache.store.v1 (or v3) and is byte-identical to what a runtime without variants writes. A v1 header over a variant record is refused.
  • A variant's resolved document is derived in memory when the cache admits the record or any record on its base chain, and is never written back. A plain write at a variant's key is refused; Flatten replaces a variant with a plain record holding its resolved document.
  • The C# runtime resolves variants. Every other runtime refuses a v2 store by name, and refuses a variant record inside a v3 store by its record key. CultMesh single-file reads, the directory store and CultNet refuse a variant record by name until each has its own cut.

Element Ids

Every object that sits in a list, array or dictionary value of a registered document type, union subtypes included, carries an id: one string member with an integer [Key] and [CultElementId]. The id identifies the element inside its list, so an override can name it (a path step is [slot, elementId]) and a base edit that reorders or inserts elements retargets nothing.

  • A random id is 12 lowercase hex characters. A derived id ([CultElementId(nameof(Source))], for content-addressed elements) is the invariant-culture text of its source member, which must be non-empty; it needs no other format. An id is unique within its list.
  • A write mints random ids for unset elements, including inside variant override values, and refuses a duplicate, a malformed random id or an empty derived source. A load refuses the same things, naming the record, the list and the id. A record written before ids existed loads, and its ids are minted from SHA-256 of the record key and the element's path, so every reader mints the same ids; they reach disk with the record's next write, or CultCache.MintElementIds().
  • A store that holds an id declares the v3 header (above), so a reader older than ids refuses it instead of skipping the id slot and rewriting the elements without ids. TypeScript, Python and Rust read v3 and keep a v3 header they found when they rewrite a store; they do not decide when a document carries ids.

Canonical Semantic Schema Hash

schemaId should be derived from a canonical semantic schema hash.

The canonicalization step must ignore noise that should not change storage identity, such as:

  • whitespace
  • key ordering
  • documentation prose
  • examples
  • source-generator quirks

It must preserve the parts that do define persistence compatibility:

  • field slot numbers
  • field types
  • required versus defaulted presence
  • nullability
  • collection/container semantics
  • enum/value constraints when those affect stored meaning
  • persistence encoding shape, such as MessagePack array layout

The output should be a stable string such as:

  • sha256:<hex>

This is the value persisted in each record's schemaId.

Soft Migration

Soft migration is a first-class requirement. The format must support a reader encountering a record whose embedded schema is not byte-identical to its local native schema while still consuming the record safely when the slot contract is compatible enough.

Reader flow:

  1. Read record header.
  2. Resolve schemaId against the embedded catalog.
  3. Resolve the local registered schema for the requested native type.
  4. Compare embedded versus local canonical persistence schemas.
  5. Choose one:
    • exact_match
    • soft_migrate
    • reject

Exact Match

Read directly when:

  • schema ids match exactly
  • or equivalent schema comparison says the persisted and local canonical shapes are identical

Soft Migration

Allowed when:

  • persisted schema has missing slots that the local schema can default
  • persisted schema has extra slots the local reader can safely ignore or preserve
  • slot meanings remain compatible

Soft migration should emit warnings, not silent shrugs.

Typical warnings:

  • missing slot defaulted
  • unknown extra slot ignored
  • nullable/default behavior widened
  • field preserved but not surfaced natively

Reject

Reject when:

  • slot meanings conflict
  • required local data cannot be reconstructed
  • stored type and local expected type are incompatible
  • persistence encoding families differ in a non-compatible way

Do not bluff here. A false clean read is worse than a refusal.

API Ergonomics

The public CultCache API should hide persistence metadata by default.

Desired feel:

  • callers work with PlayerData, not PlayerData plus storage sludge
  • cache methods own key generation or key routing
  • global and named-entry flows stay ergonomic
  • advanced callers can still inspect raw records and catalogs when needed

This implies a split between:

  • domain-facing API
  • raw persistence/interop API

Domain-Facing Surface

Domain callers should interact with:

  • typed adds/puts
  • typed gets
  • named lookups
  • index lookups
  • global entry helpers

They should not have to care about schemaId, storedAt, or raw payload bytes unless they are explicitly doing migration, tooling, or replication work.

Raw Persistence Surface

The lower-level API should still expose:

  • record inspection
  • schema catalog inspection
  • raw record import/export
  • migration diagnostics
  • exact payload preservation for bit-compatible neighbors

That is the cache's responsibility, not the domain model's.

Concurrent Store Policy

CultCache files are expected to become shared state, not private save blobs. Once multiple runtimes can read and write the same .cc file, the persistence contract must protect both durability and observation.

The store header should declare the write policy a runtime must obey before it mutates the file:

  • generation: monotonic store generation, incremented on each committed snapshot
  • contentHash: hash of the committed snapshot body
  • concurrency.policy: stable policy id, such as cultcache.concurrent.snapshot-lock.v1
  • concurrency.lockName: how to derive the sidecar lock path
  • concurrency.commit: commit primitive, such as atomic-replace

The header negotiates the protocol. It does not enforce the protocol by magic. Writers still need an external coordination primitive because replacing the main file also replaces whatever identity a platform may attach to that file handle.

The v1 concurrent single-file policy is:

  1. Readers may read lock-free when they can tolerate a stale-but-complete snapshot. Readers that need a stable read boundary take a shared sidecar lock.
  2. Writers take an exclusive sidecar lock derived from the .cc path.
  3. Writers re-read the current snapshot after taking the lock.
  4. Writers using conditional commit evaluate their conditions against the latest snapshot and write nothing if one fails. A plain flush and an unconditional commit compare nothing and are last-writer-wins over a file they can read; one whose header they cannot read is refused untouched.
  5. Writers write a temp file, flush it, atomically replace the .cc file, and release the lock.

This preserves the key invariant: every reader sees a complete snapshot. Only a conditional commit is protected against committing over unseen data.

Live Change Observation

A shared CultCache file is also a low-ceremony realtime database for local applications. The filesystem is the transport. CultCache is the typed database surface. The cache should therefore expose domain-level change streams, not force applications to subscribe to raw storage envelopes.

Recommended C# surface:

Observable<CultCacheChange<T>> Watch<T>();
Observable<T> WatchRecord<T>(CultRecordKey key);
Observable<T> WatchGlobal<T>();
Observable<T> WatchByName<T>(string name);

Change events should be emitted after a pulled or locally committed snapshot has been reconciled into the in-memory cache. Subscribers should see domain objects and cache-owned handles, not raw schemaId, storedAt, or MessagePack payload bytes unless they explicitly subscribe to the raw persistence lane.

The event model should distinguish at least:

  • added
  • updated
  • removed
  • schema_migrated
  • rejected

Local and remote-looking writes use the same reconciliation path. A write from another process is just a new committed generation discovered through file watching or polling, pulled into the cache, diffed against the current domain view, and published.

R3 Alignment

The C# reactive surface should align with R3, Cysharp/neuecc's modern Reactive Extensions implementation and successor line to UniRx. CultCache should not invent a bespoke observer API when the .NET ecosystem already has a well-understood shape for composable streams.

R3 should own the C# subscription ergonomics:

  • Observable<T> for public domain streams
  • Subject<T> or internal equivalents only at ownership boundaries
  • disposable subscriptions for lifecycle control
  • debounce/throttle operators for bursty filesystem notifications
  • scheduler hooks where Unity, desktop UI, or server callers need thread affinity

The invariant is simple: the persistence layer owns bytes and generations; the cache owns domain identity and diffing; R3 owns composition of observed changes.

CultNet Alignment

CultCache and CultNet should not grow rival schema religions.

The schema descriptor shape used by the CultCache embedded catalog should align with the CultNet schema catalog as closely as practical. In particular:

  • schemaId
  • schemaVersion
  • contentHash
  • canonical schema body

should mean the same thing across persistence and wire discovery.

Where CultNet needs additional wire-oriented metadata, extend around the shared core instead of redefining the core.

CultNet As A Sharding Layer

CultNet should be able to initialize directly as a CultCache distribution layer. The goal is a single distributed realtime database surface: clients work with typed CultCache documents, while CultNet decides which node owns the relevant shard and how committed changes reach subscribers.

This is closer to the application experience of RethinkDB or Firebase than a traditional request/response game protocol. CultNet should still keep its own shape: explicit schemas, MessagePack payloads, local-first operation where useful, and honest rejection when compatibility or authority is unclear.

The ownership split is:

  • CultCache owns document identity, schema compatibility, local indexing, persistence, and domain-level change diffing.
  • CultNet owns peer membership, shard routing, mutation transport, authorization, remote subscription fanout, and resynchronization after reconnect.
  • R3 owns C# stream composition for observed domain changes.

A distributed CultCache cluster should expose:

  • shard descriptors: which schema/key ranges or named partitions a node owns
  • shard epochs: monotonic authority versions for ownership changes
  • mutation messages: raw document put/delete records with source generation and shard epoch
  • snapshot messages: bounded catch-up from a known generation or empty local state
  • subscription messages: schema/key/index/global queries that stream matching domain changes
  • conflict reports: explicit outcomes when a mutation cannot be applied cleanly

Client-facing APIs should preserve the database illusion:

var db = await CultNetDatabase.ConnectAsync("localhost", 3075);

var player = await db.GetAsync<PlayerData>(playerKey);
await db.PutAsync(playerKey, player);

using var subscription = db
    .WatchByIndex<PlayerData>("Region", "eu-west")
    .Subscribe(change => Render(change.Document));

The illusion has limits and those limits should be visible in the contract. CultNet is not allowed to pretend that arbitrary split-brain writes merged cleanly. Each shard needs one current write authority, or a declared conflict strategy that can be explained without ritual smoke. If neither exists, the write is rejected and surfaced as a domain-visible failure.

The simplest coherent first implementation is primary-shard ownership:

  1. Each shard has one authoritative writer at a time.
  2. Followers subscribe to that shard's committed mutation stream.
  3. Clients may connect to any node; non-owner nodes forward writes to the shard owner or reject when forwarding is unavailable.
  4. Reconnect performs schema discovery, shard epoch comparison, then snapshot catch-up before live events resume.

More advanced replicated or offline-first conflict policies can come later, but they must be explicit shard policies rather than accidental behavior leaking out of transport retries.

Migration From Current C# Shape

The current C# DatabaseEntry base type exposes ID directly. That is useful today but not the desired long-term boundary.

Planned direction:

  1. define the v1 persistence format here
  2. implement store header, catalog, and record shape in C#
  3. move storage identity out of the domain-facing DatabaseEntry contract
  4. keep compatibility shims long enough to read older stores
  5. let Rust, TypeScript, and other siblings follow the stabilized C# contract

The transition should preserve the existing soft-migration advantage rather than trading it for fake purity.

Near-Term Rules

Until this format is implemented:

  • treat this document as the canonical target
  • do not add new storage metadata to domain payloads
  • prefer schema catalogs over repeated per-document metadata
  • preserve the distinction between semantic schema identity and exact schema document hash
  • keep cross-language interop as a first-class design constraint, not a later apology