CultLib/packages/cultcache-rs

CultCache Rust

cultcache-rs is the Rust port of the useful part of GameCult's CultCache: a polymorphism-aware in-memory cache that remembers through pluggable backing stores.

Consumer code should think in domain types:

let player = cache.get_required::<PlayerData>("player:meta")?;
cache.put("player:meta", &player)?;

The persistence layer should deal with envelopes, routing, MessagePack payload bytes, and schema identity. Application code should not paw at loose JSON files like a sad little bureaucrat with a clipboard.

What It Is

CultCache is a domain cache with persistence adapters.

• CultCache is the query and mutation surface.
• Domain structs implement DatabaseEntry.
• Entries are stored behind a type::key identity, so multiple entry types can share a logical key without colliding.
• Each payload is encoded directly from the known DatabaseEntry type into MessagePack bytes.
• Backing stores are adapters, not the public data model.
• Writes persist to the resolved backing store before the in-memory cache is updated.
• Type-specific backing stores beat generic backing stores.
• SingleFileMessagePackBackingStore is the first concrete store and guards file access with a sidecar lock file.

This is not an ORM, not a database, and not distributed consensus in a novelty hat. If multiple processes write the same backing file, use an external lock or a coordinator.

Current API

use cultcache_rs::{CultCache, DatabaseEntry, SingleFileMessagePackBackingStore};

#[derive(Clone, Debug, PartialEq, Eq, DatabaseEntry)]
#[cultcache(type = "settings")]
struct Settings {
    #[cultcache(key = 0)]
    theme: String,

    #[cultcache(key = 1, default)]
    retries: u32,
}

let mut cache = CultCache::new();
cache.register_entry_type::<Settings>()?;
cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new("cache.cc"));
cache.pull_all_backing_stores()?;

cache.put("app", &Settings {
    theme: "ash".to_string(),
    retries: 3,
})?;

let settings = cache.get_required::<Settings>("app")?;
# Ok::<(), anyhow::Error>(())

For a closed set of entries, generate a registry instead of repeating yourself:

use cultcache_rs::cultcache_registry;

cultcache_registry!(GameCultEntries {
    Settings,
    PlayerData,
});

let mut cache = CultCache::new();
cache.register_registry(GameCultEntries)?;

The important part is that callers retrieve domain values directly:

let settings: Settings = cache.get_required("app")?;
let all_settings: Vec<Settings> = cache.get_all()?;

The cache is the ergonomic surface. The backing store is the memory prosthetic.

How This Maps From Original CultCache

The original C# CultCache relies on runtime reflection:

• cacheable models inherit from DatabaseEntry
• the cache scans child classes at startup
• Get<T>(id) returns a T
• optional INamedEntry enables name lookup
• optional field/property indexes are registered by member name
• backing stores pull/push/delete DatabaseEntry values

Rust does not have C#-style assembly scanning or runtime subclass discovery. The closest honest Rust translation is:

• domain structs implement a marker trait, DatabaseEntry
• the derive macro generates a positional array formatter from explicit integer field slots
• the envelope carries the polymorphic type discriminator
• unknown persisted type ids fail closed instead of constructing arbitrary runtime types
• a generated registry should eventually replace repetitive manual registration

Manual register_entry_type::<T>() is still accepted, but entry identity no longer needs a hand-written impl: #[derive(DatabaseEntry)] declares it beside the domain type. cultcache_registry! provides the current generated resolver surface for a closed entry set.

Intended Rust Ergonomics

The target user-facing shape should be:

#[derive(Clone, DatabaseEntry)]
#[cultcache(type = "player")]
pub struct PlayerData {
    #[cultcache(key = 0)]
    pub id: String,

    #[cultcache(key = 1)]
    pub name: String,

    #[cultcache(key = 2)]
    pub faction: String,
}

let mut cache = CultCache::builder()
    .with_generated_entries(game_entries())
    .with_generic_store(SingleFileMessagePackBackingStore::new("cache.cc"))
    .build()?;

cache.pull_all_backing_stores()?;
let player = cache.get_required::<PlayerData>("player:ari")?;

The current cultcache_registry! macro is deliberately simple. The remaining ergonomic endpoint is automatic inventory or build-script generation so users do not have to maintain the registry list by hand.

The registry is the Rust equivalent of C# reflection and MessagePack resolver setup. It should know:

• entry type id
• generated MessagePack/serde formatter path
• Rust type name / schema name
• optional store domain route
• optional name key extractor
• optional secondary indexes
• optional global singleton marker

In other words: code generation should restore the original CultCache feeling without pretending Rust has runtime reflection hiding under the floorboards.

Current Surface

• CultCache::new
• register_entry_type::<T>
• add_generic_backing_store
• add_backing_store
• pull_all_backing_stores
• get::<T>
• get_required::<T>
• get_envelope::<T>
• get_required_envelope::<T>
• get_all::<T>
• get_all_with_keys::<T>
• soa::<T>
• put::<T>
• put_envelope::<T>
• update::<T>
• delete::<T>
• snapshot
• snapshot_soa
• load_soa
• SingleFileMessagePackBackingStore

Structure Of Arrays

The C# CultCache hot path exposes CPU-local SoA scans as:

Span<float> x = cache.Soa<PlayerTransform>()
    .Column<float>(nameof(PlayerTransform.PositionX))
    .Span;

Rust cannot discover those columns through runtime reflection, so each domain type declares its column extractors explicitly with SoaDocument or the cultcache_soa! helper:

cultcache_soa!(PlayerTransform {
    "position_x" => |row: &PlayerTransform| row.position_x,
    "health" => |row: &PlayerTransform| row.health,
});

let table = cache.soa::<PlayerTransform>()?;
let health = table.column::<i32>("health")?;

This is the same public shape as the C# rite: callers ask the cache for a typed table, then pull contiguous typed columns by name. The Rust version is explicit about the extractor table because the closed-world schema boundary is a virtue, not a missing reflection party trick.

Backing Store Routing

Generic store:

cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new("cache.cc"))?;

Type-specific store:

cache.add_backing_store(
    SingleFileMessagePackBackingStore::new("players.cc"),
    ["player"],
)?;

The contract, shared with every runtime, is one home store per document type (src/GameCult.Caching/Contracts/cultcache-store-composition.md in CultLib):

• a type-specific store owns its types; otherwise the generic store is the home
• a second generic store is an error, and so is a type claimed by a second store; both registrations return Err and attach nothing
• a write touches exactly one store; there are no mirrors
• a write whose type has no home store is an error

Persistence Semantics

put persists before mutating the in-memory cache. If persistence fails, the cache does not pretend the write succeeded.

The envelope is MessagePack, and the payload inside each envelope is also raw MessagePack bytes encoded from the registered concrete DatabaseEntry type. That avoids the old bootstrap path where payloads were normalized through serde_json::Value.

For bit-compatible neighbors, that also enables a real fast lane:

• get_envelope::<T>() exports the canonical persisted bytes for a typed entry
• put_envelope::<T>() ingests the same envelope into another cache instance without re-encoding the payload first

It still decodes once for validation and typed reads. We are not pretending one Vec<u8> became metaphysically zero-copy because we believed in it harder. But the stupid decode/re-encode loop is gone.

The DatabaseEntry derive does not trust Rust source field order as the durable schema. Every persisted member must declare a stable integer slot:

#[derive(Clone, DatabaseEntry)]
#[cultcache(type = "player")]
pub struct PlayerData {
    #[cultcache(key = 0)]
    pub id: String,

    #[cultcache(key = 1)]
    pub name: String,

    #[cultcache(key = 2, default)]
    pub level: u32,
}

The derive emits a tuple/array formatter. Gaps serialize as nil. Missing fields marked default use Default::default() when older payloads are read. Deleted field slots should stay reserved until an explicit store migration rewrites the data.

The single-file MessagePack store rewrites an atomic snapshot. That is a sane starting point for small typed state surfaces, settings, Epiphany agent memory, heartbeat state, and other compact control-plane data. Large corpora should use a sharded store or a real database instead of asking one file to become a warehouse and then acting wounded when physics invoices us.

The single-file store uses a sidecar lock file for shared reads and exclusive writes. That protects ordinary multi-process access to the same file. It is still not a multi-master replication protocol, and it is not a substitute for a coordinator when higher-level write ordering matters.

Near-Term Ergonomic Improvements

1. Derive macro

   • #[derive(DatabaseEntry)]
   • #[cultcache(type = "...")]
   • #[cultcache(key = N)] on every persisted field
   • tuple/array formatter generation landed
   • optional #[cultcache(name)], #[cultcache(index)], and #[cultcache(global)] are still future work

2. Generated registry

   • a CultCacheRegistry trait
   • cache.register_registry(GameCultEntries)
   • basic macro registry landed
   • automatic inventory/build-script generation and store routes are future work

3. Name and index lookups

   • cache.get_by_name::<T>("Potion")
   • cache.get_by_index::<T>("faction", "Lucent")
   • generated extractors instead of reflection strings

4. Schema/version metadata

   • schema version in the envelope or payload metadata
   • explicit migration hooks
   • refusal on unknown persisted entry types unless a migration/resolver is installed

5. Projection helpers

   • JSON projection for review and git diffs
   • vector stripping / large-field elision for agent memory surfaces

Why Not Skip Registration Immediately?

Rust can only deserialize polymorphic data into concrete types if something maps the persisted type discriminator to the Rust type. In C#, reflection and MessagePack resolvers can discover a closed DatabaseEntry inheritance tree at runtime and route it through known formatters. In Rust, that map has to come from somewhere:

• explicit registration
• a generated registry
• a macro inventory crate
• a hand-written resolver

Explicit registration is the simplest honest implementation. Generated registry is the ergonomic destination. Hand-written resolver tables are the punishment we give ourselves if we get lazy.

Security Model

CultCache should never parse arbitrary structured data into arbitrary runtime types. It only accepts persisted envelopes whose type discriminator resolves to a known DatabaseEntry type registered in the cache or generated registry.

That is the Rust version of the original trick:

• C# refuses to store objects outside the DatabaseEntry root type.
• Runtime-visible DatabaseEntry subclasses get dedicated formatters.
• Unknown or unregistered types are rejected.
• MessagePack payload bytes deserialize through known concrete formatters, not an open-world type loader or generic structured-value bridge.

The derive/registry work should preserve that shape. Its job is to remove manual registration ceremony, not to loosen the closed-world boundary.

License

Private GameCult infrastructure for now. Public packaging can wait until the API grows enough teeth to deserve strangers.