CultLib/packages/cultcache-rs/README.md

CultCache Rust

cultcache-rs is the Rust port of the useful part of GameCult's CultCache: a polymorphism-aware in-memory cache that remembers through pluggable backing stores.

Consumer code should think in domain types:

let player = cache.get_required::<PlayerData>("player:meta")?;
cache.put("player:meta", &player)?;

The persistence layer should deal with envelopes, routing, MessagePack payload bytes, and schema identity. Application code should not paw at loose JSON files like a sad little bureaucrat with a clipboard.

What It Is

CultCache is a domain cache with persistence adapters.

• CultCache is the query and mutation surface.
• Domain structs implement DatabaseEntry.
• Entries are stored behind a type::key identity, so multiple entry types can share a logical key without colliding.
• Each payload is encoded directly from the known DatabaseEntry type into MessagePack bytes.
• Backing stores are adapters, not the public data model.
• Writes persist to the resolved backing store before the in-memory cache is updated.
• Type-specific backing stores beat generic backing stores.
• SingleFileMessagePackBackingStore is the first concrete store and guards file access with a sidecar lock file.

This is not an ORM, not a database, and not distributed consensus in a novelty hat. If multiple processes write the same backing file, use an external lock or a coordinator.

Current API

use cultcache_rs::{CultCache, DatabaseEntry, SingleFileMessagePackBackingStore};

#[derive(Clone, Debug, PartialEq, Eq, DatabaseEntry)]
#[cultcache(type = "settings")]
struct Settings {
    #[cultcache(key = 0)]
    theme: String,

    #[cultcache(key = 1, default)]
    retries: u32,
}

let mut cache = CultCache::new();
cache.register_entry_type::<Settings>()?;
cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new("cache.cc"));
cache.pull_all_backing_stores()?;

cache.put("app", &Settings {
    theme: "ash".to_string(),
    retries: 3,
})?;

let settings = cache.get_required::<Settings>("app")?;
# Ok::<(), anyhow::Error>(())

For a closed set of entries, generate a registry instead of repeating yourself:

use cultcache_rs::cultcache_registry;

cultcache_registry!(GameCultEntries {
    Settings,
    PlayerData,
});

let mut cache = CultCache::new();
cache.register_registry(GameCultEntries)?;

The important part is that callers retrieve domain values directly:

let settings: Settings = cache.get_required("app")?;
let all_settings: Vec<Settings> = cache.get_all()?;

The cache is the ergonomic surface. The backing store is the memory prosthetic.

How This Maps From Original CultCache

The original C# CultCache relies on runtime reflection:

• cacheable models inherit from DatabaseEntry
• the cache scans child classes at startup
• Get<T>(id) returns a T
• optional INamedEntry enables name lookup
• optional field/property indexes are registered by member name
• backing stores pull/push/delete DatabaseEntry values

Rust does not have C#-style assembly scanning or runtime subclass discovery. The closest honest Rust translation is:

• domain structs implement a marker trait, DatabaseEntry
• the derive macro generates a positional array formatter from explicit integer field slots
• the envelope carries the polymorphic type discriminator
• unknown persisted type ids fail closed instead of constructing arbitrary runtime types
• a generated registry should eventually replace repetitive manual registration

Manual register_entry_type::<T>() is still accepted, but entry identity no longer needs a hand-written impl: #[derive(DatabaseEntry)] declares it beside the domain type. cultcache_registry! provides the current generated resolver surface for a closed entry set.

Intended Rust Ergonomics

The target user-facing shape should be:

#[derive(Clone, DatabaseEntry)]
#[cultcache(type = "player")]
pub struct PlayerData {
    #[cultcache(key = 0)]
    pub id: String,

    #[cultcache(key = 1)]
    pub name: String,

    #[cultcache(key = 2)]
    pub faction: String,
}

let mut cache = CultCache::builder()
    .with_generated_entries(game_entries())
    .with_generic_store(SingleFileMessagePackBackingStore::new("cache.cc"))
    .build()?;

cache.pull_all_backing_stores()?;
let player = cache.get_required::<PlayerData>("player:ari")?;

The current cultcache_registry! macro is deliberately simple. The remaining ergonomic endpoint is automatic inventory or build-script generation so users do not have to maintain the registry list by hand.

The registry is the Rust equivalent of C# reflection and MessagePack resolver setup. It should know:

• entry type id
• generated MessagePack/serde formatter path
• Rust type name / schema name
• optional store domain route
• optional name key extractor
• optional secondary indexes
• optional global singleton marker

In other words: code generation should restore the original CultCache feeling without pretending Rust has runtime reflection hiding under the floorboards.

Current Surface

• CultCache::new
• register_entry_type::<T>
• add_generic_backing_store
• add_backing_store
• pull_all_backing_stores
• get::<T>
• get_required::<T>
• get_envelope::<T>
• get_required_envelope::<T>
• get_all::<T>
• get_all_with_keys::<T>
• soa::<T>
• put::<T>
• put_envelope::<T>
• update::<T>
• delete::<T>
• snapshot
• snapshot_soa
• load_soa
• SingleFileMessagePackBackingStore

Unions

A union that crosses runtimes is written [armKey, [armSlots...]], the shape the C# reference writes for [Union(key, type)]. Derive CultUnion on an enum: every variant carries #[cultcache(key = N)] and is a unit ([N, []]) or a newtype over a #[derive(CultSlots)] struct, which is a DatabaseEntry's slot array without the entry consts (a String or an Option is not an arm, and is a compile error). An arm key is the reference's Int32, negatives included, and is given once. A key the enum does not declare is a decode error naming the enum and the declared keys, and never quoting the payload. A null union is Option<Enum>. Plain serde enums are untouched and keep their bytes.

use cultcache_rs::{CultSlots, CultUnion};

#[derive(Clone, Debug, PartialEq, CultUnion)]
enum Node {
    #[cultcache(key = 0)]
    Box(BoxArm),
    #[cultcache(key = 1)]
    Text(TextArm),
    #[cultcache(key = 5)]
    Spacer, // a unit arm: [5, []]
}

#[derive(Clone, Debug, PartialEq, CultSlots)]
struct BoxArm {
    #[cultcache(key = 0)]
    id: String,
    #[cultcache(key = 1)]
    children: Vec<Node>, // recursive
}

#[derive(Clone, Debug, PartialEq, CultSlots)]
struct TextArm {
    #[cultcache(key = 0)]
    id: String,
    #[cultcache(key = 1)]
    content: String,
}

let tree = Node::Box(BoxArm { id: "root".into(), children: vec![Node::Spacer] });
let bytes = rmp_serde::to_vec(&tree)?;
assert_eq!(rmp_serde::from_slice::<Node>(&bytes)?, tree);
# Ok::<(), Box<dyn std::error::Error>>(())

Structure Of Arrays

The C# CultCache hot path exposes CPU-local SoA scans as:

Span<float> x = cache.Soa<PlayerTransform>()
    .Column<float>(nameof(PlayerTransform.PositionX))
    .Span;

Rust cannot discover those columns through runtime reflection, so each domain type declares its column extractors explicitly with SoaDocument or the cultcache_soa! helper:

cultcache_soa!(PlayerTransform {
    "position_x" => |row: &PlayerTransform| row.position_x,
    "health" => |row: &PlayerTransform| row.health,
});

let table = cache.soa::<PlayerTransform>()?;
let health = table.column::<i32>("health")?;

This is the same public shape as the C# rite: callers ask the cache for a typed table, then pull contiguous typed columns by name. The Rust version is explicit about the extractor table because the closed-world schema boundary is a virtue, not a missing reflection party trick.

Backing Store Routing

Generic store:

cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new("cache.cc"))?;

Type-specific store:

cache.add_backing_store(
    SingleFileMessagePackBackingStore::new("players.cc"),
    ["player"],
)?;

The contract, shared with every runtime, is one home store per document type (src/GameCult.Caching/Contracts/cultcache-store-composition.md in CultLib):

• a type-specific store owns its types; otherwise the generic store is the home
• a second generic store is an error, and so is a type claimed by a second store; both registrations return Err and attach nothing
• a write touches exactly one store; there are no mirrors
• a write whose type has no home store is an error

Persistence Semantics

put persists before mutating the in-memory cache. If persistence fails, the cache does not pretend the write succeeded.

The envelope is MessagePack, and the payload inside each envelope is also raw MessagePack bytes encoded from the registered concrete DatabaseEntry type. That avoids the old bootstrap path where payloads were normalized through serde_json::Value.

For bit-compatible neighbors, that also enables a real fast lane:

• get_envelope::<T>() exports the canonical persisted bytes for a typed entry
• put_envelope::<T>() ingests the same envelope into another cache instance without re-encoding the payload first

It still decodes once for validation and typed reads. We are not pretending one Vec<u8> became metaphysically zero-copy because we believed in it harder. But the stupid decode/re-encode loop is gone.

The DatabaseEntry derive does not trust Rust source field order as the durable schema. Every persisted member must declare a stable integer slot:

#[derive(Clone, DatabaseEntry)]
#[cultcache(type = "player")]
pub struct PlayerData {
    #[cultcache(key = 0)]
    pub id: String,

    #[cultcache(key = 1)]
    pub name: String,

    #[cultcache(key = 2, default)]
    pub level: u32,
}

The derive emits a tuple/array formatter. Gaps serialize as nil. Missing fields marked default use Default::default() when older payloads are read. Deleted field slots should stay reserved until an explicit store migration rewrites the data.

The single-file MessagePack store rewrites an atomic snapshot. That is a sane starting point for small typed state surfaces, settings, Epiphany agent memory, heartbeat state, and other compact control-plane data. Large corpora should use a sharded store or a real database instead of asking one file to become a warehouse and then acting wounded when physics invoices us.

The single-file store uses a sidecar lock file for shared reads and exclusive writes. That protects ordinary multi-process access to the same file. It is still not a multi-master replication protocol, and it is not a substitute for a coordinator when higher-level write ordering matters.

A write that fails part way, for example on a full disk, removes its staging file and leaves the store as it was. A read that fails tells you why through the error chain:

use cultcache_rs::{CacheBackingStore, CultCacheStoreUnreadable, CultCacheStoreUnreadableKind, SingleFileMessagePackBackingStore};

let store = SingleFileMessagePackBackingStore::new("cache.cc");
if let Err(error) = store.pull_all() {
    match error.downcast_ref::<CultCacheStoreUnreadable>().map(|unreadable| unreadable.kind) {
        // The bytes were read and are not a store: truncated, garbled, or invalid.
        Some(CultCacheStoreUnreadableKind::Undecodable) => {}
        // The header names a format this runtime does not read, such as a newer one.
        Some(CultCacheStoreUnreadableKind::UnsupportedFormat) => {}
        // The bytes could not be read at all: a std::io::Error is in the chain.
        _ => {}
    }
}

An I/O failure never carries CultCacheStoreUnreadable, and the chain under CultCacheStoreUnreadable never holds a std::io::Error.

Only a store file that does not exist (std::io::ErrorKind::NotFound) reads as an empty store. Any other failure to reach it, such as a symlink loop, a file where a parent directory should be, or a missing permission, is an I/O error. A store file that exists and holds zero bytes is Undecodable: no CultCache writer leaves one. A symbolic link whose target is gone is not a missing store: reading it is an I/O error, so a read-then-write (push, delete, the exchanges) fails before writing. Writes do not yet follow links: a write through a live link, and a push_all through a dangling one, replace the link with a regular file and leave its target alone. A refusal names no value read from the file; it names a record only by its key and schema id, and a header only when it has the shape cultcache.store.v<digits>.

A write that fails says what it left on disk:

use cultcache_rs::{CultCacheStoreWriteFailed, CultCacheStoreWriteFailedKind};

fn outcome(error: &anyhow::Error) {
    match error.downcast_ref::<CultCacheStoreWriteFailed>().map(|failed| failed.kind) {
        // The entries cannot be stored; retrying them fails the same way.
        Some(CultCacheStoreWriteFailedKind::Rejected) => {}
        // Staging, fsync or rename failed: the store file holds what it held before,
        // and a retry may succeed once the fault (a full disk, say) clears.
        Some(CultCacheStoreWriteFailedKind::NotReplaced) => {}
        // The rename landed and the directory sync failed: the new snapshot is in
        // place but may not survive a crash.
        Some(CultCacheStoreWriteFailedKind::ReplacedNotDurable) => {}
        _ => {}
    }
}

Near-Term Ergonomic Improvements

1. Derive macro

   • #[derive(DatabaseEntry)]
   • #[cultcache(type = "...")]
   • #[cultcache(key = N)] on every persisted field
   • tuple/array formatter generation landed
   • optional #[cultcache(name)], #[cultcache(index)], and #[cultcache(global)] are still future work

2. Generated registry

   • a CultCacheRegistry trait
   • cache.register_registry(GameCultEntries)
   • basic macro registry landed
   • automatic inventory/build-script generation and store routes are future work

3. Name and index lookups

   • cache.get_by_name::<T>("Potion")
   • cache.get_by_index::<T>("faction", "Lucent")
   • generated extractors instead of reflection strings

4. Schema/version metadata

   • schema version in the envelope or payload metadata
   • explicit migration hooks
   • refusal on unknown persisted entry types unless a migration/resolver is installed

5. Projection helpers

   • JSON projection for review and git diffs
   • vector stripping / large-field elision for agent memory surfaces

Why Not Skip Registration Immediately?

Rust can only deserialize polymorphic data into concrete types if something maps the persisted type discriminator to the Rust type. In C#, reflection and MessagePack resolvers can discover a closed DatabaseEntry inheritance tree at runtime and route it through known formatters. In Rust, that map has to come from somewhere:

• explicit registration
• a generated registry
• a macro inventory crate
• a hand-written resolver

Explicit registration is the simplest honest implementation. Generated registry is the ergonomic destination. Hand-written resolver tables are the punishment we give ourselves if we get lazy.

Security Model

CultCache should never parse arbitrary structured data into arbitrary runtime types. It only accepts persisted envelopes whose type discriminator resolves to a known DatabaseEntry type registered in the cache or generated registry.

That is the Rust version of the original trick:

• C# refuses to store objects outside the DatabaseEntry root type.
• Runtime-visible DatabaseEntry subclasses get dedicated formatters.
• Unknown or unregistered types are rejected.
• MessagePack payload bytes deserialize through known concrete formatters, not an open-world type loader or generic structured-value bridge.

The derive/registry work should preserve that shape. Its job is to remove manual registration ceremony, not to loosen the closed-world boundary.

Payload nesting is bounded at 256 MessagePack containers, checked before any recursion; past the bound the refusal is a typed PayloadTooDeep. Typed decode of stored bytes then grows its stack: the decoder checks the remaining stack at entry and serde_stacker checks it at every container, moving onto a fresh segment whenever less than 512 KiB remains, so a payload inside the bound decodes on a small thread instead of aborting the process. Envelope and store reads (single-file, redb, v1 snapshots) use the same grown decoder and keep accepting and refusing what they did, without the payload walk.

The guarantee holds for records the derives produce. MAX_RECORD_INLINE_BYTES (16 KiB) is the largest size_of a DatabaseEntry, CultSlots or CultUnion record may have, and the derives refuse a larger one at compile time: put its large data behind a Vec or a Box. A field of a type the derives did not produce that is larger than that, behind a Box or Vec, is outside the guarantee. The decoder's own frames have a fixed floor that no payload changes: at the bound a thread decodes a record of this size from about 56 KiB in a release build and 72 KiB in a debug build for a caller that only tests the result (is_ok), plus the caller's own frames. The figure depends on how the caller holds the result: one that moves the record out holds a second copy and needs more (about 104 KiB in debug). Stack growth was verified on x86_64-pc-windows-msvc by hand (decode_stack, revision 10a39dac, 13 tests, debug and release); a stacker bump or a change to the stack-size tests repeats it. The crate does not build for wasm32 (fs2), and no wasm behaviour is claimed.

Build dependencies

Stack growth adds these crates to every consumer's build, and a C toolchain (psm, stacker). On all targets: serde_stacker 0.1.14, stacker 0.1.25, psm 0.1.32, cc 1.8.0, find-msvc-tools 0.1.14, shlex 2.0.1 (MIT OR Apache-2.0), object 0.39.1 (Apache-2.0 OR MIT) and ar_archive_writer 0.5.3 (Apache-2.0 WITH LLVM-exception, build-only). On x86_64-pc-windows-msvc also libc 0.2.190 and windows-sys 0.61.2 (MIT OR Apache-2.0), beside the 0.59 the crate uses itself.

License

Private GameCult infrastructure for now. Public packaging can wait until the API grows enough teeth to deserve strangers.