2026-05-30 00:39:39 +01:00
|
|
|
# CultCache Rust
|
|
|
|
|
|
|
|
|
|
`cultcache-rs` is the Rust port of the useful part of GameCult's CultCache:
|
|
|
|
|
a polymorphism-aware in-memory cache that remembers through pluggable backing
|
|
|
|
|
stores.
|
|
|
|
|
|
|
|
|
|
Consumer code should think in domain types:
|
|
|
|
|
|
|
|
|
|
```rust
|
|
|
|
|
let player = cache.get_required::<PlayerData>("player:meta")?;
|
|
|
|
|
cache.put("player:meta", &player)?;
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The persistence layer should deal with envelopes, routing, MessagePack payload
|
|
|
|
|
bytes, and schema identity. Application code should not paw at loose JSON files
|
|
|
|
|
like a sad little bureaucrat with a clipboard.
|
|
|
|
|
|
|
|
|
|
## What It Is
|
|
|
|
|
|
|
|
|
|
CultCache is a domain cache with persistence adapters.
|
|
|
|
|
|
|
|
|
|
- `CultCache` is the query and mutation surface.
|
|
|
|
|
- Domain structs implement `DatabaseEntry`.
|
|
|
|
|
- Entries are stored behind a `type::key` identity, so multiple entry types
|
|
|
|
|
can share a logical key without colliding.
|
|
|
|
|
- Each payload is encoded directly from the known `DatabaseEntry` type into
|
|
|
|
|
MessagePack bytes.
|
|
|
|
|
- Backing stores are adapters, not the public data model.
|
|
|
|
|
- Writes persist to the resolved backing store before the in-memory cache is
|
|
|
|
|
updated.
|
|
|
|
|
- Type-specific backing stores beat generic backing stores.
|
|
|
|
|
- `SingleFileMessagePackBackingStore` is the first concrete store and guards
|
|
|
|
|
file access with a sidecar lock file.
|
|
|
|
|
|
|
|
|
|
This is not an ORM, not a database, and not distributed consensus in a novelty
|
|
|
|
|
hat. If multiple processes write the same backing file, use an external lock or
|
|
|
|
|
a coordinator.
|
|
|
|
|
|
|
|
|
|
## Current API
|
|
|
|
|
|
|
|
|
|
```rust
|
|
|
|
|
use cultcache_rs::{CultCache, DatabaseEntry, SingleFileMessagePackBackingStore};
|
|
|
|
|
|
|
|
|
|
#[derive(Clone, Debug, PartialEq, Eq, DatabaseEntry)]
|
|
|
|
|
#[cultcache(type = "settings")]
|
|
|
|
|
struct Settings {
|
|
|
|
|
#[cultcache(key = 0)]
|
|
|
|
|
theme: String,
|
|
|
|
|
|
|
|
|
|
#[cultcache(key = 1, default)]
|
|
|
|
|
retries: u32,
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
let mut cache = CultCache::new();
|
|
|
|
|
cache.register_entry_type::<Settings>()?;
|
Return the Rust CultCache engine to CultLib
packages/cultcache-rs forked from GameCult/cultcache-rs at the 5c69bfc
monorepo import and the two lineages then diverged in both directions. The
standalone repo took 28 commits of engine work CultLib never saw: typed SoA
projection, atomic heterogeneous batch publication, single-entry and
conditional batch CAS, a transactional keyed redb backend with serialized
cross-process opens, lockless provider snapshot reads, RAII read-only shared
snapshots, snapshot-fenced replace/append, crash-atomic single-file snapshots,
compaction fences, portable and canonical byte fields, and unified retained
storage authority. That body is 3,714 lines with 48 tests against CultLib's
1,554 with 18.
CultLib is the canonical home because wire parity conformance lives here and
nothing outside it is tested for cross-runtime agreement. A runtime-specific
core-stack repo cannot hold the engine every other runtime must agree with.
The standalone body lands as the base rather than merging 28 commits backward.
It is already proven against a CultLib-derived cultnet-rs: Ghostlight vendors
exactly this pairing and compiles. Here, cultcache-rs passes 46 tests and the
cultnet-rs RUDP gate passes 20 with no source change.
Retained debt, not carried forward: a0b31b8 recovered schema-stamped records
missing catalog entries (SchemaStamped, infer_schema_name,
infer_schema_version_from_payload) and 743eaa5 hardened snapshot replacement
(SnapshotReplacementStage, CommittedSnapshotDurabilityUncertain, stale
candidate cleanup, with_exclusive_lock_using_unlock). Their tests were
white-box against internals this body does not have. The standalone
independently made single-file snapshots crash atomic and cleaned abandoned
staging files, so the durability guarantee is probably held under other names
-- probably is not proven. Re-establish both as black-box tests against the
new API before the standalone repo is retired.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 22:44:32 +02:00
|
|
|
cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new("cache.cc"));
|
2026-05-30 00:39:39 +01:00
|
|
|
cache.pull_all_backing_stores()?;
|
|
|
|
|
|
|
|
|
|
cache.put("app", &Settings {
|
|
|
|
|
theme: "ash".to_string(),
|
|
|
|
|
retries: 3,
|
|
|
|
|
})?;
|
|
|
|
|
|
|
|
|
|
let settings = cache.get_required::<Settings>("app")?;
|
|
|
|
|
# Ok::<(), anyhow::Error>(())
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
For a closed set of entries, generate a registry instead of repeating yourself:
|
|
|
|
|
|
|
|
|
|
```rust
|
|
|
|
|
use cultcache_rs::cultcache_registry;
|
|
|
|
|
|
|
|
|
|
cultcache_registry!(GameCultEntries {
|
|
|
|
|
Settings,
|
|
|
|
|
PlayerData,
|
|
|
|
|
});
|
|
|
|
|
|
|
|
|
|
let mut cache = CultCache::new();
|
|
|
|
|
cache.register_registry(GameCultEntries)?;
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The important part is that callers retrieve domain values directly:
|
|
|
|
|
|
|
|
|
|
```rust
|
|
|
|
|
let settings: Settings = cache.get_required("app")?;
|
|
|
|
|
let all_settings: Vec<Settings> = cache.get_all()?;
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The cache is the ergonomic surface. The backing store is the memory prosthetic.
|
|
|
|
|
|
|
|
|
|
## How This Maps From Original CultCache
|
|
|
|
|
|
|
|
|
|
The original C# CultCache relies on runtime reflection:
|
|
|
|
|
|
|
|
|
|
- cacheable models inherit from `DatabaseEntry`
|
|
|
|
|
- the cache scans child classes at startup
|
|
|
|
|
- `Get<T>(id)` returns a `T`
|
|
|
|
|
- optional `INamedEntry` enables name lookup
|
|
|
|
|
- optional field/property indexes are registered by member name
|
|
|
|
|
- backing stores pull/push/delete `DatabaseEntry` values
|
|
|
|
|
|
|
|
|
|
Rust does not have C#-style assembly scanning or runtime subclass discovery.
|
|
|
|
|
The closest honest Rust translation is:
|
|
|
|
|
|
|
|
|
|
- domain structs implement a marker trait, `DatabaseEntry`
|
|
|
|
|
- the derive macro generates a positional array formatter from explicit integer
|
|
|
|
|
field slots
|
|
|
|
|
- the envelope carries the polymorphic type discriminator
|
|
|
|
|
- unknown persisted type ids fail closed instead of constructing arbitrary
|
|
|
|
|
runtime types
|
|
|
|
|
- a generated registry should eventually replace repetitive manual
|
|
|
|
|
registration
|
|
|
|
|
|
|
|
|
|
Manual `register_entry_type::<T>()` is still accepted, but entry identity no
|
|
|
|
|
longer needs a hand-written impl: `#[derive(DatabaseEntry)]` declares it beside
|
|
|
|
|
the domain type. `cultcache_registry!` provides the current generated resolver
|
|
|
|
|
surface for a closed entry set.
|
|
|
|
|
|
|
|
|
|
## Intended Rust Ergonomics
|
|
|
|
|
|
|
|
|
|
The target user-facing shape should be:
|
|
|
|
|
|
|
|
|
|
```rust
|
|
|
|
|
#[derive(Clone, DatabaseEntry)]
|
|
|
|
|
#[cultcache(type = "player")]
|
|
|
|
|
pub struct PlayerData {
|
|
|
|
|
#[cultcache(key = 0)]
|
|
|
|
|
pub id: String,
|
|
|
|
|
|
|
|
|
|
#[cultcache(key = 1)]
|
|
|
|
|
pub name: String,
|
|
|
|
|
|
|
|
|
|
#[cultcache(key = 2)]
|
|
|
|
|
pub faction: String,
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
let mut cache = CultCache::builder()
|
|
|
|
|
.with_generated_entries(game_entries())
|
Return the Rust CultCache engine to CultLib
packages/cultcache-rs forked from GameCult/cultcache-rs at the 5c69bfc
monorepo import and the two lineages then diverged in both directions. The
standalone repo took 28 commits of engine work CultLib never saw: typed SoA
projection, atomic heterogeneous batch publication, single-entry and
conditional batch CAS, a transactional keyed redb backend with serialized
cross-process opens, lockless provider snapshot reads, RAII read-only shared
snapshots, snapshot-fenced replace/append, crash-atomic single-file snapshots,
compaction fences, portable and canonical byte fields, and unified retained
storage authority. That body is 3,714 lines with 48 tests against CultLib's
1,554 with 18.
CultLib is the canonical home because wire parity conformance lives here and
nothing outside it is tested for cross-runtime agreement. A runtime-specific
core-stack repo cannot hold the engine every other runtime must agree with.
The standalone body lands as the base rather than merging 28 commits backward.
It is already proven against a CultLib-derived cultnet-rs: Ghostlight vendors
exactly this pairing and compiles. Here, cultcache-rs passes 46 tests and the
cultnet-rs RUDP gate passes 20 with no source change.
Retained debt, not carried forward: a0b31b8 recovered schema-stamped records
missing catalog entries (SchemaStamped, infer_schema_name,
infer_schema_version_from_payload) and 743eaa5 hardened snapshot replacement
(SnapshotReplacementStage, CommittedSnapshotDurabilityUncertain, stale
candidate cleanup, with_exclusive_lock_using_unlock). Their tests were
white-box against internals this body does not have. The standalone
independently made single-file snapshots crash atomic and cleaned abandoned
staging files, so the durability guarantee is probably held under other names
-- probably is not proven. Re-establish both as black-box tests against the
new API before the standalone repo is retired.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 22:44:32 +02:00
|
|
|
.with_generic_store(SingleFileMessagePackBackingStore::new("cache.cc"))
|
2026-05-30 00:39:39 +01:00
|
|
|
.build()?;
|
|
|
|
|
|
|
|
|
|
cache.pull_all_backing_stores()?;
|
|
|
|
|
let player = cache.get_required::<PlayerData>("player:ari")?;
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The current `cultcache_registry!` macro is deliberately simple. The remaining
|
|
|
|
|
ergonomic endpoint is automatic inventory or build-script generation so users
|
|
|
|
|
do not have to maintain the registry list by hand.
|
|
|
|
|
|
|
|
|
|
The registry is the Rust equivalent of C# reflection and MessagePack resolver
|
|
|
|
|
setup. It should know:
|
|
|
|
|
|
|
|
|
|
- entry type id
|
|
|
|
|
- generated MessagePack/serde formatter path
|
|
|
|
|
- Rust type name / schema name
|
|
|
|
|
- optional store domain route
|
|
|
|
|
- optional name key extractor
|
|
|
|
|
- optional secondary indexes
|
|
|
|
|
- optional global singleton marker
|
|
|
|
|
|
|
|
|
|
In other words: code generation should restore the original CultCache feeling
|
|
|
|
|
without pretending Rust has runtime reflection hiding under the floorboards.
|
|
|
|
|
|
|
|
|
|
## Current Surface
|
|
|
|
|
|
|
|
|
|
- `CultCache::new`
|
|
|
|
|
- `register_entry_type::<T>`
|
|
|
|
|
- `add_generic_backing_store`
|
|
|
|
|
- `add_backing_store`
|
|
|
|
|
- `pull_all_backing_stores`
|
|
|
|
|
- `get::<T>`
|
|
|
|
|
- `get_required::<T>`
|
|
|
|
|
- `get_envelope::<T>`
|
|
|
|
|
- `get_required_envelope::<T>`
|
|
|
|
|
- `get_all::<T>`
|
Return the Rust CultCache engine to CultLib
packages/cultcache-rs forked from GameCult/cultcache-rs at the 5c69bfc
monorepo import and the two lineages then diverged in both directions. The
standalone repo took 28 commits of engine work CultLib never saw: typed SoA
projection, atomic heterogeneous batch publication, single-entry and
conditional batch CAS, a transactional keyed redb backend with serialized
cross-process opens, lockless provider snapshot reads, RAII read-only shared
snapshots, snapshot-fenced replace/append, crash-atomic single-file snapshots,
compaction fences, portable and canonical byte fields, and unified retained
storage authority. That body is 3,714 lines with 48 tests against CultLib's
1,554 with 18.
CultLib is the canonical home because wire parity conformance lives here and
nothing outside it is tested for cross-runtime agreement. A runtime-specific
core-stack repo cannot hold the engine every other runtime must agree with.
The standalone body lands as the base rather than merging 28 commits backward.
It is already proven against a CultLib-derived cultnet-rs: Ghostlight vendors
exactly this pairing and compiles. Here, cultcache-rs passes 46 tests and the
cultnet-rs RUDP gate passes 20 with no source change.
Retained debt, not carried forward: a0b31b8 recovered schema-stamped records
missing catalog entries (SchemaStamped, infer_schema_name,
infer_schema_version_from_payload) and 743eaa5 hardened snapshot replacement
(SnapshotReplacementStage, CommittedSnapshotDurabilityUncertain, stale
candidate cleanup, with_exclusive_lock_using_unlock). Their tests were
white-box against internals this body does not have. The standalone
independently made single-file snapshots crash atomic and cleaned abandoned
staging files, so the durability guarantee is probably held under other names
-- probably is not proven. Re-establish both as black-box tests against the
new API before the standalone repo is retired.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 22:44:32 +02:00
|
|
|
- `get_all_with_keys::<T>`
|
|
|
|
|
- `soa::<T>`
|
2026-05-30 00:39:39 +01:00
|
|
|
- `put::<T>`
|
|
|
|
|
- `put_envelope::<T>`
|
|
|
|
|
- `update::<T>`
|
|
|
|
|
- `delete::<T>`
|
|
|
|
|
- `snapshot`
|
Return the Rust CultCache engine to CultLib
packages/cultcache-rs forked from GameCult/cultcache-rs at the 5c69bfc
monorepo import and the two lineages then diverged in both directions. The
standalone repo took 28 commits of engine work CultLib never saw: typed SoA
projection, atomic heterogeneous batch publication, single-entry and
conditional batch CAS, a transactional keyed redb backend with serialized
cross-process opens, lockless provider snapshot reads, RAII read-only shared
snapshots, snapshot-fenced replace/append, crash-atomic single-file snapshots,
compaction fences, portable and canonical byte fields, and unified retained
storage authority. That body is 3,714 lines with 48 tests against CultLib's
1,554 with 18.
CultLib is the canonical home because wire parity conformance lives here and
nothing outside it is tested for cross-runtime agreement. A runtime-specific
core-stack repo cannot hold the engine every other runtime must agree with.
The standalone body lands as the base rather than merging 28 commits backward.
It is already proven against a CultLib-derived cultnet-rs: Ghostlight vendors
exactly this pairing and compiles. Here, cultcache-rs passes 46 tests and the
cultnet-rs RUDP gate passes 20 with no source change.
Retained debt, not carried forward: a0b31b8 recovered schema-stamped records
missing catalog entries (SchemaStamped, infer_schema_name,
infer_schema_version_from_payload) and 743eaa5 hardened snapshot replacement
(SnapshotReplacementStage, CommittedSnapshotDurabilityUncertain, stale
candidate cleanup, with_exclusive_lock_using_unlock). Their tests were
white-box against internals this body does not have. The standalone
independently made single-file snapshots crash atomic and cleaned abandoned
staging files, so the durability guarantee is probably held under other names
-- probably is not proven. Re-establish both as black-box tests against the
new API before the standalone repo is retired.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 22:44:32 +02:00
|
|
|
- `snapshot_soa`
|
|
|
|
|
- `load_soa`
|
2026-05-30 00:39:39 +01:00
|
|
|
- `SingleFileMessagePackBackingStore`
|
|
|
|
|
|
Return the Rust CultCache engine to CultLib
packages/cultcache-rs forked from GameCult/cultcache-rs at the 5c69bfc
monorepo import and the two lineages then diverged in both directions. The
standalone repo took 28 commits of engine work CultLib never saw: typed SoA
projection, atomic heterogeneous batch publication, single-entry and
conditional batch CAS, a transactional keyed redb backend with serialized
cross-process opens, lockless provider snapshot reads, RAII read-only shared
snapshots, snapshot-fenced replace/append, crash-atomic single-file snapshots,
compaction fences, portable and canonical byte fields, and unified retained
storage authority. That body is 3,714 lines with 48 tests against CultLib's
1,554 with 18.
CultLib is the canonical home because wire parity conformance lives here and
nothing outside it is tested for cross-runtime agreement. A runtime-specific
core-stack repo cannot hold the engine every other runtime must agree with.
The standalone body lands as the base rather than merging 28 commits backward.
It is already proven against a CultLib-derived cultnet-rs: Ghostlight vendors
exactly this pairing and compiles. Here, cultcache-rs passes 46 tests and the
cultnet-rs RUDP gate passes 20 with no source change.
Retained debt, not carried forward: a0b31b8 recovered schema-stamped records
missing catalog entries (SchemaStamped, infer_schema_name,
infer_schema_version_from_payload) and 743eaa5 hardened snapshot replacement
(SnapshotReplacementStage, CommittedSnapshotDurabilityUncertain, stale
candidate cleanup, with_exclusive_lock_using_unlock). Their tests were
white-box against internals this body does not have. The standalone
independently made single-file snapshots crash atomic and cleaned abandoned
staging files, so the durability guarantee is probably held under other names
-- probably is not proven. Re-establish both as black-box tests against the
new API before the standalone repo is retired.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 22:44:32 +02:00
|
|
|
## Structure Of Arrays
|
|
|
|
|
|
|
|
|
|
The C# CultCache hot path exposes CPU-local SoA scans as:
|
|
|
|
|
|
|
|
|
|
```csharp
|
|
|
|
|
Span<float> x = cache.Soa<PlayerTransform>()
|
|
|
|
|
.Column<float>(nameof(PlayerTransform.PositionX))
|
|
|
|
|
.Span;
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Rust cannot discover those columns through runtime reflection, so each domain
|
|
|
|
|
type declares its column extractors explicitly with `SoaDocument` or the
|
|
|
|
|
`cultcache_soa!` helper:
|
|
|
|
|
|
|
|
|
|
```rust
|
|
|
|
|
cultcache_soa!(PlayerTransform {
|
|
|
|
|
"position_x" => |row: &PlayerTransform| row.position_x,
|
|
|
|
|
"health" => |row: &PlayerTransform| row.health,
|
|
|
|
|
});
|
|
|
|
|
|
|
|
|
|
let table = cache.soa::<PlayerTransform>()?;
|
|
|
|
|
let health = table.column::<i32>("health")?;
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
This is the same public shape as the C# rite: callers ask the cache for a typed
|
|
|
|
|
table, then pull contiguous typed columns by name. The Rust version is explicit
|
|
|
|
|
about the extractor table because the closed-world schema boundary is a virtue,
|
|
|
|
|
not a missing reflection party trick.
|
|
|
|
|
|
2026-05-30 00:39:39 +01:00
|
|
|
## Backing Store Routing
|
|
|
|
|
|
|
|
|
|
Generic store:
|
|
|
|
|
|
|
|
|
|
```rust
|
Give every cultcache-rs entry type one home store
Cut 5 (Rust). The four mirror loops in put, put_envelope, put_raw_envelope and
delete are deleted, and so is put_prepared_batch's route.len() != 1 branch.
resolve_route_index returns the single home store (the store that claims the
type, else the generic store) and replaces resolve_route_indices.
add_backing_store and add_generic_backing_store return Result and refuse a
second generic store or a type already claimed by another store, attaching
nothing when they refuse. Callers take the Result: the crate's tests, the
cultcache_interop example, cultnet-rs's tests and interop peer example, and
CultMeshNode::create_node in cultmesh-rs.
Tests: second_generic_store_is_rejected, type_claimed_twice_is_rejected;
type_specific_store_routes_before_generic_store stays, now attaching the routed
store before the generic one. No bytes change.
The store-composition contract's runtime table now records Rust, TypeScript
and Python as conforming to one home store per type, and the README routing
section drops its non-conformance note.
Version 0.1.0 -> 0.2.0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-14 00:38:41 +02:00
|
|
|
cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new("cache.cc"))?;
|
2026-05-30 00:39:39 +01:00
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Type-specific store:
|
|
|
|
|
|
|
|
|
|
```rust
|
|
|
|
|
cache.add_backing_store(
|
Return the Rust CultCache engine to CultLib
packages/cultcache-rs forked from GameCult/cultcache-rs at the 5c69bfc
monorepo import and the two lineages then diverged in both directions. The
standalone repo took 28 commits of engine work CultLib never saw: typed SoA
projection, atomic heterogeneous batch publication, single-entry and
conditional batch CAS, a transactional keyed redb backend with serialized
cross-process opens, lockless provider snapshot reads, RAII read-only shared
snapshots, snapshot-fenced replace/append, crash-atomic single-file snapshots,
compaction fences, portable and canonical byte fields, and unified retained
storage authority. That body is 3,714 lines with 48 tests against CultLib's
1,554 with 18.
CultLib is the canonical home because wire parity conformance lives here and
nothing outside it is tested for cross-runtime agreement. A runtime-specific
core-stack repo cannot hold the engine every other runtime must agree with.
The standalone body lands as the base rather than merging 28 commits backward.
It is already proven against a CultLib-derived cultnet-rs: Ghostlight vendors
exactly this pairing and compiles. Here, cultcache-rs passes 46 tests and the
cultnet-rs RUDP gate passes 20 with no source change.
Retained debt, not carried forward: a0b31b8 recovered schema-stamped records
missing catalog entries (SchemaStamped, infer_schema_name,
infer_schema_version_from_payload) and 743eaa5 hardened snapshot replacement
(SnapshotReplacementStage, CommittedSnapshotDurabilityUncertain, stale
candidate cleanup, with_exclusive_lock_using_unlock). Their tests were
white-box against internals this body does not have. The standalone
independently made single-file snapshots crash atomic and cleaned abandoned
staging files, so the durability guarantee is probably held under other names
-- probably is not proven. Re-establish both as black-box tests against the
new API before the standalone repo is retired.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 22:44:32 +02:00
|
|
|
SingleFileMessagePackBackingStore::new("players.cc"),
|
2026-05-30 00:39:39 +01:00
|
|
|
["player"],
|
Give every cultcache-rs entry type one home store
Cut 5 (Rust). The four mirror loops in put, put_envelope, put_raw_envelope and
delete are deleted, and so is put_prepared_batch's route.len() != 1 branch.
resolve_route_index returns the single home store (the store that claims the
type, else the generic store) and replaces resolve_route_indices.
add_backing_store and add_generic_backing_store return Result and refuse a
second generic store or a type already claimed by another store, attaching
nothing when they refuse. Callers take the Result: the crate's tests, the
cultcache_interop example, cultnet-rs's tests and interop peer example, and
CultMeshNode::create_node in cultmesh-rs.
Tests: second_generic_store_is_rejected, type_claimed_twice_is_rejected;
type_specific_store_routes_before_generic_store stays, now attaching the routed
store before the generic one. No bytes change.
The store-composition contract's runtime table now records Rust, TypeScript
and Python as conforming to one home store per type, and the README routing
section drops its non-conformance note.
Version 0.1.0 -> 0.2.0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-14 00:38:41 +02:00
|
|
|
)?;
|
2026-05-30 00:39:39 +01:00
|
|
|
```
|
|
|
|
|
|
2026-09-13 16:01:32 +02:00
|
|
|
The contract, shared with every runtime, is one home store per document type
|
|
|
|
|
(`src/GameCult.Caching/Contracts/cultcache-store-composition.md` in CultLib):
|
2026-05-30 00:39:39 +01:00
|
|
|
|
2026-09-13 16:01:32 +02:00
|
|
|
- a type-specific store owns its types; otherwise the generic store is the home
|
Give every cultcache-rs entry type one home store
Cut 5 (Rust). The four mirror loops in put, put_envelope, put_raw_envelope and
delete are deleted, and so is put_prepared_batch's route.len() != 1 branch.
resolve_route_index returns the single home store (the store that claims the
type, else the generic store) and replaces resolve_route_indices.
add_backing_store and add_generic_backing_store return Result and refuse a
second generic store or a type already claimed by another store, attaching
nothing when they refuse. Callers take the Result: the crate's tests, the
cultcache_interop example, cultnet-rs's tests and interop peer example, and
CultMeshNode::create_node in cultmesh-rs.
Tests: second_generic_store_is_rejected, type_claimed_twice_is_rejected;
type_specific_store_routes_before_generic_store stays, now attaching the routed
store before the generic one. No bytes change.
The store-composition contract's runtime table now records Rust, TypeScript
and Python as conforming to one home store per type, and the README routing
section drops its non-conformance note.
Version 0.1.0 -> 0.2.0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-14 00:38:41 +02:00
|
|
|
- a second generic store is an error, and so is a type claimed by a second
|
|
|
|
|
store; both registrations return `Err` and attach nothing
|
2026-09-13 16:01:32 +02:00
|
|
|
- a write touches exactly one store; there are no mirrors
|
Give every cultcache-rs entry type one home store
Cut 5 (Rust). The four mirror loops in put, put_envelope, put_raw_envelope and
delete are deleted, and so is put_prepared_batch's route.len() != 1 branch.
resolve_route_index returns the single home store (the store that claims the
type, else the generic store) and replaces resolve_route_indices.
add_backing_store and add_generic_backing_store return Result and refuse a
second generic store or a type already claimed by another store, attaching
nothing when they refuse. Callers take the Result: the crate's tests, the
cultcache_interop example, cultnet-rs's tests and interop peer example, and
CultMeshNode::create_node in cultmesh-rs.
Tests: second_generic_store_is_rejected, type_claimed_twice_is_rejected;
type_specific_store_routes_before_generic_store stays, now attaching the routed
store before the generic one. No bytes change.
The store-composition contract's runtime table now records Rust, TypeScript
and Python as conforming to one home store per type, and the README routing
section drops its non-conformance note.
Version 0.1.0 -> 0.2.0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-14 00:38:41 +02:00
|
|
|
- a write whose type has no home store is an error
|
2026-05-30 00:39:39 +01:00
|
|
|
|
|
|
|
|
## Persistence Semantics
|
|
|
|
|
|
|
|
|
|
`put` persists before mutating the in-memory cache. If persistence fails, the
|
|
|
|
|
cache does not pretend the write succeeded.
|
|
|
|
|
|
|
|
|
|
The envelope is MessagePack, and the payload inside each envelope is also raw
|
|
|
|
|
MessagePack bytes encoded from the registered concrete `DatabaseEntry` type.
|
|
|
|
|
That avoids the old bootstrap path where payloads were normalized through
|
|
|
|
|
`serde_json::Value`.
|
|
|
|
|
|
|
|
|
|
For bit-compatible neighbors, that also enables a real fast lane:
|
|
|
|
|
|
|
|
|
|
- `get_envelope::<T>()` exports the canonical persisted bytes for a typed entry
|
|
|
|
|
- `put_envelope::<T>()` ingests the same envelope into another cache instance
|
|
|
|
|
without re-encoding the payload first
|
|
|
|
|
|
|
|
|
|
It still decodes once for validation and typed reads. We are not pretending one
|
|
|
|
|
`Vec<u8>` became metaphysically zero-copy because we believed in it harder. But
|
|
|
|
|
the stupid decode/re-encode loop is gone.
|
|
|
|
|
|
|
|
|
|
The `DatabaseEntry` derive does not trust Rust source field order as the durable
|
|
|
|
|
schema. Every persisted member must declare a stable integer slot:
|
|
|
|
|
|
|
|
|
|
```rust
|
|
|
|
|
#[derive(Clone, DatabaseEntry)]
|
|
|
|
|
#[cultcache(type = "player")]
|
|
|
|
|
pub struct PlayerData {
|
|
|
|
|
#[cultcache(key = 0)]
|
|
|
|
|
pub id: String,
|
|
|
|
|
|
|
|
|
|
#[cultcache(key = 1)]
|
|
|
|
|
pub name: String,
|
|
|
|
|
|
|
|
|
|
#[cultcache(key = 2, default)]
|
|
|
|
|
pub level: u32,
|
|
|
|
|
}
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The derive emits a tuple/array formatter. Gaps serialize as nil. Missing fields
|
|
|
|
|
marked `default` use `Default::default()` when older payloads are read. Deleted
|
|
|
|
|
field slots should stay reserved until an explicit store migration rewrites the
|
|
|
|
|
data.
|
|
|
|
|
|
Return the Rust CultCache engine to CultLib
packages/cultcache-rs forked from GameCult/cultcache-rs at the 5c69bfc
monorepo import and the two lineages then diverged in both directions. The
standalone repo took 28 commits of engine work CultLib never saw: typed SoA
projection, atomic heterogeneous batch publication, single-entry and
conditional batch CAS, a transactional keyed redb backend with serialized
cross-process opens, lockless provider snapshot reads, RAII read-only shared
snapshots, snapshot-fenced replace/append, crash-atomic single-file snapshots,
compaction fences, portable and canonical byte fields, and unified retained
storage authority. That body is 3,714 lines with 48 tests against CultLib's
1,554 with 18.
CultLib is the canonical home because wire parity conformance lives here and
nothing outside it is tested for cross-runtime agreement. A runtime-specific
core-stack repo cannot hold the engine every other runtime must agree with.
The standalone body lands as the base rather than merging 28 commits backward.
It is already proven against a CultLib-derived cultnet-rs: Ghostlight vendors
exactly this pairing and compiles. Here, cultcache-rs passes 46 tests and the
cultnet-rs RUDP gate passes 20 with no source change.
Retained debt, not carried forward: a0b31b8 recovered schema-stamped records
missing catalog entries (SchemaStamped, infer_schema_name,
infer_schema_version_from_payload) and 743eaa5 hardened snapshot replacement
(SnapshotReplacementStage, CommittedSnapshotDurabilityUncertain, stale
candidate cleanup, with_exclusive_lock_using_unlock). Their tests were
white-box against internals this body does not have. The standalone
independently made single-file snapshots crash atomic and cleaned abandoned
staging files, so the durability guarantee is probably held under other names
-- probably is not proven. Re-establish both as black-box tests against the
new API before the standalone repo is retired.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-03 22:44:32 +02:00
|
|
|
The single-file MessagePack store rewrites an atomic snapshot. That is a sane
|
|
|
|
|
starting point for small typed state surfaces, settings, Epiphany agent memory,
|
2026-05-30 00:39:39 +01:00
|
|
|
heartbeat state, and other compact control-plane data. Large corpora should use
|
|
|
|
|
a sharded store or a real database instead of asking one file to become a
|
|
|
|
|
warehouse and then acting wounded when physics invoices us.
|
|
|
|
|
|
|
|
|
|
The single-file store uses a sidecar lock file for shared reads and exclusive
|
|
|
|
|
writes. That protects ordinary multi-process access to the same file. It is
|
|
|
|
|
still not a multi-master replication protocol, and it is not a substitute for a
|
|
|
|
|
coordinator when higher-level write ordering matters.
|
|
|
|
|
|
|
|
|
|
## Near-Term Ergonomic Improvements
|
|
|
|
|
|
|
|
|
|
1. **Derive macro**
|
|
|
|
|
- `#[derive(DatabaseEntry)]`
|
|
|
|
|
- `#[cultcache(type = "...")]`
|
|
|
|
|
- `#[cultcache(key = N)]` on every persisted field
|
|
|
|
|
- tuple/array formatter generation landed
|
|
|
|
|
- optional `#[cultcache(name)]`, `#[cultcache(index)]`, and
|
|
|
|
|
`#[cultcache(global)]` are still future work
|
|
|
|
|
|
|
|
|
|
2. **Generated registry**
|
|
|
|
|
- a `CultCacheRegistry` trait
|
|
|
|
|
- `cache.register_registry(GameCultEntries)`
|
|
|
|
|
- basic macro registry landed
|
|
|
|
|
- automatic inventory/build-script generation and store routes are future work
|
|
|
|
|
|
|
|
|
|
3. **Name and index lookups**
|
|
|
|
|
- `cache.get_by_name::<T>("Potion")`
|
|
|
|
|
- `cache.get_by_index::<T>("faction", "Lucent")`
|
|
|
|
|
- generated extractors instead of reflection strings
|
|
|
|
|
|
|
|
|
|
4. **Schema/version metadata**
|
|
|
|
|
- schema version in the envelope or payload metadata
|
|
|
|
|
- explicit migration hooks
|
|
|
|
|
- refusal on unknown persisted entry types unless a migration/resolver is
|
|
|
|
|
installed
|
|
|
|
|
|
|
|
|
|
5. **Projection helpers**
|
|
|
|
|
- JSON projection for review and git diffs
|
|
|
|
|
- vector stripping / large-field elision for agent memory surfaces
|
|
|
|
|
|
|
|
|
|
## Why Not Skip Registration Immediately?
|
|
|
|
|
|
|
|
|
|
Rust can only deserialize polymorphic data into concrete types if something
|
|
|
|
|
maps the persisted `type` discriminator to the Rust type. In C#, reflection and
|
|
|
|
|
MessagePack resolvers can discover a closed `DatabaseEntry` inheritance tree at
|
|
|
|
|
runtime and route it through known formatters. In Rust, that map has to come from
|
|
|
|
|
somewhere:
|
|
|
|
|
|
|
|
|
|
- explicit registration
|
|
|
|
|
- a generated registry
|
|
|
|
|
- a macro inventory crate
|
|
|
|
|
- a hand-written resolver
|
|
|
|
|
|
|
|
|
|
Explicit registration is the simplest honest implementation. Generated registry
|
|
|
|
|
is the ergonomic destination. Hand-written resolver tables are the punishment
|
|
|
|
|
we give ourselves if we get lazy.
|
|
|
|
|
|
|
|
|
|
## Security Model
|
|
|
|
|
|
|
|
|
|
CultCache should never parse arbitrary structured data into arbitrary runtime
|
|
|
|
|
types. It only accepts persisted envelopes whose `type` discriminator resolves
|
|
|
|
|
to a known `DatabaseEntry` type registered in the cache or generated registry.
|
|
|
|
|
|
|
|
|
|
That is the Rust version of the original trick:
|
|
|
|
|
|
|
|
|
|
- C# refuses to store objects outside the `DatabaseEntry` root type.
|
|
|
|
|
- Runtime-visible `DatabaseEntry` subclasses get dedicated formatters.
|
|
|
|
|
- Unknown or unregistered types are rejected.
|
|
|
|
|
- MessagePack payload bytes deserialize through known concrete formatters, not
|
|
|
|
|
an open-world type loader or generic structured-value bridge.
|
|
|
|
|
|
|
|
|
|
The derive/registry work should preserve that shape. Its job is to remove
|
|
|
|
|
manual registration ceremony, not to loosen the closed-world boundary.
|
|
|
|
|
|
|
|
|
|
## License
|
|
|
|
|
|
|
|
|
|
Private GameCult infrastructure for now. Public packaging can wait until the
|
|
|
|
|
API grows enough teeth to deserve strangers.
|