26 KiB
CultCache Persistence Format
This document is the canonical design receipt for the next CultCache storage
format. GameCult.Caching in C# is the source of truth for this contract. Rust,
TypeScript, and other siblings should follow once the shape stabilizes here.
The goal is to preserve CultCache's best property: application code should work with domain data, not with a little parade of storage metadata stapled onto every object until the object forgets what it was for.
Why This Exists
The current stack has two conflicting virtues:
- domain ergonomics: users want plain typed objects
- interop durability: caches in different languages need enough persisted information to read each other's data, compare schemas, and perform soft migration when shapes drift
The old C# shape leaned hard toward ergonomics by exposing DatabaseEntry.ID
directly and letting backing stores carry a lot of implicit responsibility. The
newer Rust draft leaned hard toward persistence machinery by wrapping payloads
in a heavier envelope. Both capture part of the truth. Neither is the final
machine.
The intended format keeps storage metadata out of domain objects while making the persisted store self-describing enough for cross-language readers and training-data archaeology.
Core Rule
Domain payloads contain domain truth only.
These are not domain fields and should not be exposed on app-facing types:
- CultCache record key / GUID
- persistence schema id
- storage timestamp
Those belong to the CultCache persistence layer.
Transaction Visibility
A multi-record commit is an explicit batch of records that resolve to one home store. The store commits the batch as one durable step. Nothing in the batch is visible, to the committing flow or to observers, until the store has accepted it; a failed commit changes nothing on disk or in memory.
A batch may carry conditions: per-record (schemaId, storedAt) identity, or
the whole store unchanged since it was last loaded. Conditions are evaluated
under the store's exclusive lock against what is durably on disk. A failed
condition is a lost race, not an error: the commit reports it and writes
nothing.
Conditional commit is the only safe multi-process write. A plain flush and an unconditional commit are last-writer-wins and write the same bytes: a single file is replaced by the writer's whole view, a directory manifest takes the writer's staged keys over the current manifest. An unconditional commit also persists writes staged earlier and leaves the store clean. Processes sharing a store must all use conditional commit. The last writer wins only over a file it can read: a single-file flush or commit refuses a file whose header it cannot read (truncated, or a format it does not read) and leaves the file as it was.
A store that fails to hydrate on open (corrupt bytes, unresolvable schema) makes the open throw and is left byte-identical; it is never silently overwritten. Within one process the lock order is the cache's gate, then the store's file lock; a store's I/O blocks its cache's readers.
C# implements batch and conditional commit (CultCache.Commit, TryCommit,
CultCacheBatch). Rust implements both. TypeScript and Python implement
neither yet. Routing, batches and conditions are specified in
cultcache-store-composition.md.
Canonical Store Shape
A persisted CultCache file should contain:
- store header
- embedded schema catalog
- record set
High-level shape:
{
"formatVersion": "cultcache.store.v1",
"catalog": {
"catalogVersion": "cultcache.schema_catalog.v1",
"schemas": []
},
"records": []
}
The on-disk encoding does not have to be JSON. MessagePack is the expected default. The shape above is explanatory.
Store Header
Required header fields:
formatVersioncatalogrecords
Optional header fields:
createdAtupdatedAtstoreIdproducernotes
These are store-level bookkeeping fields, not per-record domain metadata.
Embedded Schema Catalog
The store must embed a catalog entry for every schema referenced by the records it contains.
A reader never recovers a schema from a record's payload: a record whose schema id the catalog does not publish (by id or as a compatible id) makes the store unreadable, whatever the payload's first field says. The catalog is the store's own description of itself.
A writer keeps the same invariant: every record it writes carries a schema id that some catalog entry it writes publishes, as that entry's id or as one of its compatible ids. A cache stamps a record with its registered schema's id, whatever id the record arrived under, when it loads the record and when it puts one. The cache is the only owner of that stamp: a store writer never changes the identity of a record it did not receive from its caller, so a record another writer changed since this cache read it is written back as that writer left it.
A cache resolves a record to a local type by the record's schema id: the registered type that owns the id, else one that lists it as compatible. The name the catalog entry carries is metadata. It names a local type only when no local type has the id, for a store written by a runtime whose schema ids this one cannot know (a Rust type's schema id is its entry type). A schema renamed under a stable id therefore opens in every runtime that holds the id.
A reader resolves a record's schema id to the entry that owns it, that entry's own schemaId, and
only when no entry owns the id to an entry that lists it as a compatible id (the first such entry
in catalog order).
A writer derives the catalog from the records it writes, and from nothing else. For each distinct
schema id a record carries it chooses exactly one entry that publishes it: the entry that owns the id
(a registered schema's descriptor, else an entry the record arrived with), else an entry that lists
it as compatible (again registered first). When a registered descriptor and an arrived entry share an
own id, the descriptor wins: a rename or a stale content hash takes the descriptor's name, hash,
members and compatible ids as they are. Entries are written as chosen, deduplicated by own id, with
no union of compatible ids and no dependence on the order records arrive in; an entry no record needs
is not written. A write is refused, and the store left as it was, when two entries of one tier share
an own id and disagree on the schema name, when records of different types share a schema id, when a
write would change which type an existing record resolves to (the schema names of the records already
in the store and of the records written differ), or when a record's id is published by no chosen entry.
The retyping refusal has one exception, a rename: when the entries of both the existing records and
the written ones own the id, the registered descriptor wins. (A C# cache stamps a record with its
descriptor's own id, so it cannot write a record under an id its descriptor only lists, and the
retyping refusal cannot arise there.) Each runtime refuses with one typed error naming the id, the
schema names and a record key: CultSchemaConflictException (C#), SchemaConflictError
(Rust, TypeScript, Python).
This is not decorative paperwork. It is what allows another CultCache implementation to inspect an old store, resolve the schema for each record, and decide whether it can:
- read directly
- soft-migrate with warnings
- or refuse the data honestly
Schema Catalog Entry
Each schema descriptor should contain at least:
schemaIdschemaNameschemaVersioncontentHashcanonicalSchema
Recommended optional fields:
migrationFamilysupersedesSchemaIdscompatibleReaderFamiliespersistenceEncodingnotes
Example:
{
"schemaId": "sha256:8b52c6...",
"schemaName": "gamecult.player-data",
"schemaVersion": "1.2.0",
"contentHash": "sha256:52ff9c...",
"migrationFamily": "player-data",
"supersedesSchemaIds": ["sha256:0ce41a..."],
"persistenceEncoding": "messagepack-array-v1",
"canonicalSchema": {
"kind": "messagepack-array-object",
"fields": [
{ "slot": 0, "name": "Name", "type": "string", "required": true },
{ "slot": 1, "name": "Email", "type": "string", "required": true },
{ "slot": 2, "name": "Level", "type": "int32", "required": false, "default": 0 }
]
}
}
schemaId
schemaId is the stable machine key. It should be derived from the canonical
semantic persistence schema, not from:
- repo path
- C# type name
- Rust type name
- TypeScript interface name
- generated file name
If two implementations describe the same persisted shape, they should derive
the same schemaId regardless of origin language.
contentHash
contentHash is separate from schemaId.
schemaIdanswers: "is this the same semantic persistence shape?"contentHashanswers: "is this the exact same canonical schema document?"
This distinction matters because descriptions, examples, and other non-semantic surface details may change without changing the actual persisted shape.
Record Shape
Each persisted record should carry only the storage metadata needed to route and reconstruct the domain object:
keyschemaIdstoredAtpayload
Example:
{
"key": "e1439d7e-8da8-4b63-a0b1-238c815a8a17",
"schemaId": "sha256:8b52c6...",
"storedAt": "2026-05-08T12:00:00Z",
"payload": "<messagepack bytes>"
}
The payload is the domain object only. It does not repeat the key, schema id, or stored timestamp.
Under cultcache.store.v1 and cultcache.store.v3 a record has exactly these four slots.
A reader refuses any cultcache.store.* header it does not know and any record with more
slots than its header allows, naming the version or the record's key and schema id. It
never skips a slot it does not understand.
The headers are v1 (no variant, no element ids), v2 (holds a variant) and v3 (holds
an element id, variants or not). A single-file store declares v3 when any record it
holds carries an element id: a non-empty [CultElementId] value in a record's document,
or in a variant's override values. A store that cannot read a variant's override values
(no codec attached) counts the variant as holding one. A whole-store writer decides by
what the store holds now, so the file drops back to v1 (or v2 with a variant) when
the last id is taken away; a writer that touches only part of the file cannot see the
rest, so a file already declared v3 stays v3 when it merges onto it. A directory
store's manifest says cultcache.store.v5.directory-content-addressed-pages under the
same rule, and v4 otherwise.
A single-file store is exactly one MessagePack array of three slots: the header, the schema
catalog and the records. Every runtime refuses a file that is truncated, has bytes after the
array (a second store appended included), is not an array, has fewer or more than three slots
(a header alone included), whose first slot is not a header it reads, or whose catalog or
records it cannot decode (a record with more slots than its header allows, a variant record
it does not read). A runtime never skips a slot it does not understand. It refuses on open
and on rewrite alike, because the rewrite asks the same full reader as the open. A zero-byte
file and an empty array are an empty store. A flush or a commit replaces a file only when the
runtime's own reader opens it, so one verdict covers open, flush and commit, and a file the
runtime cannot read is left as it is. Each runtime refuses with one typed error:
CultStoreUnreadableException (C#), StoreUnreadableError (Rust, TypeScript, Python), with
the underlying cause attached. Rust, TypeScript and Python also open the legacy envelope
array (an array of key/type/payload/storedAt maps) and replace it; C# has no legacy
reader and refuses it. The shared byte vectors and each runtime's verdict are
tests/vectors/document-variants-c2a/readability/manifest.txt.
Variant Records
A variant record is a record whose document is a base record's document with some members replaced. It stores the difference, never the resolved document. It carries a fifth slot:
[key, schemaId, storedAt, payload, [baseKey, overrides[]]]
payloadis empty. An older reader that ignores the fifth slot reads an empty payload as a corrupt document and refuses, which is why the delta lives outside it.schemaIdis the type's schema id, the same as the base's. A variant has its base's concrete type, and lives in the same store (one home store per type).- An override is
[op, path[], id, value].opis0Set,1Insert or2Remove. A path step is[slot, elementId]:slotis the member's MessagePack[Key]andelementIdis""unless the step enters an object-list element.valueis the member's own MessagePack encoding, inline. A top-level Set has one step andid""; a reader that resolves only that shape refuses the others, naming the variant. - A store holding at least one variant declares
cultcache.store.v2(v3when it holds an element id); a store holding none declarescultcache.store.v1(orv3) and is byte-identical to what a runtime without variants writes. Av1header over a variant record is refused. - A variant's resolved document is derived in memory when the cache admits the record
or any record on its base chain, and is never written back. A plain write at a
variant's key is refused;
Flattenreplaces a variant with a plain record holding its resolved document. - The C# runtime resolves variants. Every other runtime refuses a
v2store by name, and refuses a variant record inside av3store by its record key. CultMesh single-file reads, the directory store and CultNet refuse a variant record by name until each has its own cut.
Element Ids
Every object that sits in a list, array or dictionary value of a registered document
type, union subtypes included, carries an id: one string member with an integer [Key]
and [CultElementId]. The id identifies the element inside its list, so an override can
name it (a path step is [slot, elementId]) and a base edit that reorders or inserts
elements retargets nothing.
- A random id is 12 lowercase hex characters. A derived id
(
[CultElementId(nameof(Source))], for content-addressed elements) is the invariant-culture text of its source member, which must be non-empty; it needs no other format. An id is unique within its list. - A write mints random ids for unset elements, including inside variant override
values, and refuses a duplicate, a malformed random id or an empty derived source. A
load refuses the same things, naming the record, the list and the id. A record
written before ids existed loads, and its ids are minted from SHA-256 of the record
key and the element's path, so every reader mints the same ids; they reach disk with
the record's next write, or
CultCache.MintElementIds(). - A store that holds an id declares the
v3header (above), so a reader older than ids refuses it instead of skipping the id slot and rewriting the elements without ids. TypeScript, Python and Rust readv3and keep av3header they found when they rewrite a store; they do not decide when a document carries ids.
Canonical Semantic Schema Hash
schemaId should be derived from a canonical semantic schema hash.
The canonicalization step must ignore noise that should not change storage identity, such as:
- whitespace
- key ordering
- documentation prose
- examples
- source-generator quirks
It must preserve the parts that do define persistence compatibility:
- field slot numbers
- field types
- required versus defaulted presence
- nullability
- collection/container semantics
- enum/value constraints when those affect stored meaning
- persistence encoding shape, such as MessagePack array layout
The output should be a stable string such as:
sha256:<hex>
This is the value persisted in each record's schemaId.
Soft Migration
Soft migration is a first-class requirement. The format must support a reader encountering a record whose embedded schema is not byte-identical to its local native schema while still consuming the record safely when the slot contract is compatible enough.
Reader flow:
- Read record header.
- Resolve
schemaIdagainst the embedded catalog. - Resolve the local registered schema for the requested native type.
- Compare embedded versus local canonical persistence schemas.
- Choose one:
exact_matchsoft_migratereject
Exact Match
Read directly when:
- schema ids match exactly
- or equivalent schema comparison says the persisted and local canonical shapes are identical
Soft Migration
Allowed when:
- persisted schema has missing slots that the local schema can default
- persisted schema has extra slots the local reader can safely ignore or preserve
- slot meanings remain compatible
Soft migration should emit warnings, not silent shrugs.
Typical warnings:
- missing slot defaulted
- unknown extra slot ignored
- nullable/default behavior widened
- field preserved but not surfaced natively
Reject
Reject when:
- slot meanings conflict
- required local data cannot be reconstructed
- stored type and local expected type are incompatible
- persistence encoding families differ in a non-compatible way
Do not bluff here. A false clean read is worse than a refusal.
API Ergonomics
The public CultCache API should hide persistence metadata by default.
Desired feel:
- callers work with
PlayerData, notPlayerDataplus storage sludge - cache methods own key generation or key routing
- global and named-entry flows stay ergonomic
- advanced callers can still inspect raw records and catalogs when needed
This implies a split between:
- domain-facing API
- raw persistence/interop API
Domain-Facing Surface
Domain callers should interact with:
- typed adds/puts
- typed gets
- named lookups
- index lookups
- global entry helpers
They should not have to care about schemaId, storedAt, or raw payload bytes
unless they are explicitly doing migration, tooling, or replication work.
Raw Persistence Surface
The lower-level API should still expose:
- record inspection
- schema catalog inspection
- raw record import/export
- migration diagnostics
- exact payload preservation for bit-compatible neighbors
That is the cache's responsibility, not the domain model's.
Concurrent Store Policy
CultCache files are expected to become shared state, not private save blobs.
Once multiple runtimes can read and write the same .cc file, the persistence
contract must protect both durability and observation.
The store header should declare the write policy a runtime must obey before it mutates the file:
generation: monotonic store generation, incremented on each committed snapshotcontentHash: hash of the committed snapshot bodyconcurrency.policy: stable policy id, such ascultcache.concurrent.snapshot-lock.v1concurrency.lockName: how to derive the sidecar lock pathconcurrency.commit: commit primitive, such asatomic-replace
The header negotiates the protocol. It does not enforce the protocol by magic. Writers still need an external coordination primitive because replacing the main file also replaces whatever identity a platform may attach to that file handle.
The v1 concurrent single-file policy is:
- Readers may read lock-free when they can tolerate a stale-but-complete snapshot. Readers that need a stable read boundary take a shared sidecar lock.
- Writers take an exclusive sidecar lock derived from the
.ccpath. - Writers re-read the current snapshot after taking the lock.
- Writers using conditional commit evaluate their conditions against the latest snapshot and write nothing if one fails. A plain flush and an unconditional commit compare nothing and are last-writer-wins over a file they can read; one whose header they cannot read is refused untouched.
- Writers write a temp file, flush it, atomically replace the
.ccfile, and release the lock.
This preserves the key invariant: every reader sees a complete snapshot. Only a conditional commit is protected against committing over unseen data.
Live Change Observation
A shared CultCache file is also a low-ceremony realtime database for local applications. The filesystem is the transport. CultCache is the typed database surface. The cache should therefore expose domain-level change streams, not force applications to subscribe to raw storage envelopes.
Recommended C# surface:
Observable<CultCacheChange<T>> Watch<T>();
Observable<T> WatchRecord<T>(CultRecordKey key);
Observable<T> WatchGlobal<T>();
Observable<T> WatchByName<T>(string name);
Change events should be emitted after a pulled or locally committed snapshot has
been reconciled into the in-memory cache. Subscribers should see domain objects
and cache-owned handles, not raw schemaId, storedAt, or MessagePack payload
bytes unless they explicitly subscribe to the raw persistence lane.
The event model should distinguish at least:
addedupdatedremovedschema_migratedrejected
Local and remote-looking writes use the same reconciliation path. A write from another process is just a new committed generation discovered through file watching or polling, pulled into the cache, diffed against the current domain view, and published.
R3 Alignment
The C# reactive surface should align with R3, Cysharp/neuecc's modern Reactive Extensions implementation and successor line to UniRx. CultCache should not invent a bespoke observer API when the .NET ecosystem already has a well-understood shape for composable streams.
R3 should own the C# subscription ergonomics:
Observable<T>for public domain streamsSubject<T>or internal equivalents only at ownership boundaries- disposable subscriptions for lifecycle control
- debounce/throttle operators for bursty filesystem notifications
- scheduler hooks where Unity, desktop UI, or server callers need thread affinity
The invariant is simple: the persistence layer owns bytes and generations; the cache owns domain identity and diffing; R3 owns composition of observed changes.
CultNet Alignment
CultCache and CultNet should not grow rival schema religions.
The schema descriptor shape used by the CultCache embedded catalog should align with the CultNet schema catalog as closely as practical. In particular:
schemaIdschemaVersioncontentHash- canonical schema body
should mean the same thing across persistence and wire discovery.
Where CultNet needs additional wire-oriented metadata, extend around the shared core instead of redefining the core.
CultNet As A Sharding Layer
CultNet should be able to initialize directly as a CultCache distribution layer. The goal is a single distributed realtime database surface: clients work with typed CultCache documents, while CultNet decides which node owns the relevant shard and how committed changes reach subscribers.
This is closer to the application experience of RethinkDB or Firebase than a traditional request/response game protocol. CultNet should still keep its own shape: explicit schemas, MessagePack payloads, local-first operation where useful, and honest rejection when compatibility or authority is unclear.
The ownership split is:
- CultCache owns document identity, schema compatibility, local indexing, persistence, and domain-level change diffing.
- CultNet owns peer membership, shard routing, mutation transport, authorization, remote subscription fanout, and resynchronization after reconnect.
- R3 owns C# stream composition for observed domain changes.
A distributed CultCache cluster should expose:
- shard descriptors: which schema/key ranges or named partitions a node owns
- shard epochs: monotonic authority versions for ownership changes
- mutation messages: raw document put/delete records with source generation and shard epoch
- snapshot messages: bounded catch-up from a known generation or empty local state
- subscription messages: schema/key/index/global queries that stream matching domain changes
- conflict reports: explicit outcomes when a mutation cannot be applied cleanly
Client-facing APIs should preserve the database illusion:
var db = await CultNetDatabase.ConnectAsync("localhost", 3075);
var player = await db.GetAsync<PlayerData>(playerKey);
await db.PutAsync(playerKey, player);
using var subscription = db
.WatchByIndex<PlayerData>("Region", "eu-west")
.Subscribe(change => Render(change.Document));
The illusion has limits and those limits should be visible in the contract. CultNet is not allowed to pretend that arbitrary split-brain writes merged cleanly. Each shard needs one current write authority, or a declared conflict strategy that can be explained without ritual smoke. If neither exists, the write is rejected and surfaced as a domain-visible failure.
The simplest coherent first implementation is primary-shard ownership:
- Each shard has one authoritative writer at a time.
- Followers subscribe to that shard's committed mutation stream.
- Clients may connect to any node; non-owner nodes forward writes to the shard owner or reject when forwarding is unavailable.
- Reconnect performs schema discovery, shard epoch comparison, then snapshot catch-up before live events resume.
More advanced replicated or offline-first conflict policies can come later, but they must be explicit shard policies rather than accidental behavior leaking out of transport retries.
Migration From Current C# Shape
The current C# DatabaseEntry base type exposes ID directly. That is useful
today but not the desired long-term boundary.
Planned direction:
- define the v1 persistence format here
- implement store header, catalog, and record shape in C#
- move storage identity out of the domain-facing
DatabaseEntrycontract - keep compatibility shims long enough to read older stores
- let Rust, TypeScript, and other siblings follow the stabilized C# contract
The transition should preserve the existing soft-migration advantage rather than trading it for fake purity.
Near-Term Rules
Until this format is implemented:
- treat this document as the canonical target
- do not add new storage metadata to domain payloads
- prefer schema catalogs over repeated per-document metadata
- preserve the distinction between semantic schema identity and exact schema document hash
- keep cross-language interop as a first-class design constraint, not a later apology