Return the Rust CultCache engine to CultLib

packages/cultcache-rs forked from GameCult/cultcache-rs at the 5c69bfc
monorepo import and the two lineages then diverged in both directions. The
standalone repo took 28 commits of engine work CultLib never saw: typed SoA
projection, atomic heterogeneous batch publication, single-entry and
conditional batch CAS, a transactional keyed redb backend with serialized
cross-process opens, lockless provider snapshot reads, RAII read-only shared
snapshots, snapshot-fenced replace/append, crash-atomic single-file snapshots,
compaction fences, portable and canonical byte fields, and unified retained
storage authority. That body is 3,714 lines with 48 tests against CultLib's
1,554 with 18.

CultLib is the canonical home because wire parity conformance lives here and
nothing outside it is tested for cross-runtime agreement. A runtime-specific
core-stack repo cannot hold the engine every other runtime must agree with.

The standalone body lands as the base rather than merging 28 commits backward.
It is already proven against a CultLib-derived cultnet-rs: Ghostlight vendors
exactly this pairing and compiles. Here, cultcache-rs passes 46 tests and the
cultnet-rs RUDP gate passes 20 with no source change.

Retained debt, not carried forward: a0b31b8 recovered schema-stamped records
missing catalog entries (SchemaStamped, infer_schema_name,
infer_schema_version_from_payload) and 743eaa5 hardened snapshot replacement
(SnapshotReplacementStage, CommittedSnapshotDurabilityUncertain, stale
candidate cleanup, with_exclusive_lock_using_unlock). Their tests were
white-box against internals this body does not have. The standalone
independently made single-file snapshots crash atomic and cleaned abandoned
staging files, so the durability guarantee is probably held under other names
-- probably is not proven. Re-establish both as black-box tests against the
new API before the standalone repo is retired.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

packages/cultcache-rs/Cargo.toml

@@ -17,14 +17,15 @@ chrono = { version = "0.4", features = ["serde"] }
 cultcache-rs-derive = { path = "cultcache-rs-derive" }
 fs2 = "0.4"
 rmp-serde = "1.3"
+redb = "4.1.0"
 serde = { version = "1", features = ["derive"] }
 serde_bytes = "0.11"
 serde_json = "1"
 uuid = { version = "1", features = ["v4"] }
 
-[target.'cfg(windows)'.dependencies]
-windows-sys = { version = "0.59", features = ["Win32_Storage_FileSystem"] }
-
 [dev-dependencies]
 pretty_assertions = "1.4"
 tempfile = "3"
+
+[target.'cfg(windows)'.dependencies]
+windows-sys = { version = "0.59", features = ["Win32_Foundation", "Win32_Storage_FileSystem"] }

packages/cultcache-rs/README.md

@@ -53,7 +53,7 @@ struct Settings {
 
 let mut cache = CultCache::new();
 cache.register_entry_type::<Settings>()?;
-cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new("cache.msgpack"));
+cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new("cache.cc"));
 cache.pull_all_backing_stores()?;
 
 cache.put("app", &Settings {
@@ -136,7 +136,7 @@ pub struct PlayerData {
 
 let mut cache = CultCache::builder()
     .with_generated_entries(game_entries())
-    .with_generic_store(SingleFileMessagePackBackingStore::new("cache.msgpack"))
+    .with_generic_store(SingleFileMessagePackBackingStore::new("cache.cc"))
     .build()?;
 
 cache.pull_all_backing_stores()?;
@@ -173,26 +173,59 @@ without pretending Rust has runtime reflection hiding under the floorboards.
 - `get_envelope::<T>`
 - `get_required_envelope::<T>`
 - `get_all::<T>`
+- `get_all_with_keys::<T>`
+- `soa::<T>`
 - `put::<T>`
 - `put_envelope::<T>`
 - `update::<T>`
 - `delete::<T>`
 - `snapshot`
+- `snapshot_soa`
+- `load_soa`
 - `SingleFileMessagePackBackingStore`
 
+## Structure Of Arrays
+
+The C# CultCache hot path exposes CPU-local SoA scans as:
+
+```csharp
+Span<float> x = cache.Soa<PlayerTransform>()
+    .Column<float>(nameof(PlayerTransform.PositionX))
+    .Span;
+```
+
+Rust cannot discover those columns through runtime reflection, so each domain
+type declares its column extractors explicitly with `SoaDocument` or the
+`cultcache_soa!` helper:
+
+```rust
+cultcache_soa!(PlayerTransform {
+    "position_x" => |row: &PlayerTransform| row.position_x,
+    "health" => |row: &PlayerTransform| row.health,
+});
+
+let table = cache.soa::<PlayerTransform>()?;
+let health = table.column::<i32>("health")?;
+```
+
+This is the same public shape as the C# rite: callers ask the cache for a typed
+table, then pull contiguous typed columns by name. The Rust version is explicit
+about the extractor table because the closed-world schema boundary is a virtue,
+not a missing reflection party trick.
+
 ## Backing Store Routing
 
 Generic store:
 
 ```rust
-cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new("cache.msgpack"));
+cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new("cache.cc"));
 ```
 
 Type-specific store:
 
 ```rust
 cache.add_backing_store(
-    SingleFileMessagePackBackingStore::new("players.msgpack"),
+    SingleFileMessagePackBackingStore::new("players.cc"),
     ["player"],
 );
 ```
@@ -250,38 +283,8 @@ marked `default` use `Default::default()` when older payloads are read. Deleted
 field slots should stay reserved until an explicit store migration rewrites the
 data.
 
-The single-file MessagePack store rewrites one complete snapshot. Under the
-exclusive sidecar lock it removes abandoned candidates without following them,
-creates a unique same-directory candidate with create-new semantics, writes and
-flushes it, and then replaces the committed pathname without deleting that
-pathname first. A process interruption before replacement leaves the previous
-snapshot authoritative. Once replacement succeeds, readers observe the complete
-new snapshot rather than a partial file.
-
-Unix requests a parent-directory flush after replacement. Windows uses
-`MoveFileExW` with replace-existing and write-through flags. If the final Unix
-directory flush fails, the method returns a downcastable
-`CommittedSnapshotDurabilityUncertain`: the complete new snapshot may already be
-visible, so the caller must fail-stop or reopen and reconcile before issuing
-higher-level work again. Treating that error as "not committed" and blindly
-retrying can duplicate commands or receipts.
-
-Abandoned `.<uuid>.tmp` candidates are never read as committed state and are
-removed by the next exclusive writer. Cleanup recognizes only the exact
-canonical UUID form emitted for that target, including targets whose native
-filename is not UTF-8; similar `.tmp` names remain untouched. Candidate creation
-uses create-new semantics, so an unexpected file or symlink collision fails
-instead of being opened or truncated. The canonical candidate filename namespace
-is reserved to the store.
-
-These guarantees assume a local filesystem that honors the platform rename and
-flush primitives, and writers that honor the sidecar lock. Network filesystems,
-faulty storage hardware, and unrelated writers may provide weaker behavior.
-Fault-injection tests prove the process-visible old-or-new outcomes around the
-replacement boundary; they do not simulate sudden power loss or prove that a
-device honored a successful flush.
-
-This is a sane starting point for small typed state surfaces, settings, Epiphany agent memory,
+The single-file MessagePack store rewrites an atomic snapshot. That is a sane
+starting point for small typed state surfaces, settings, Epiphany agent memory,
 heartbeat state, and other compact control-plane data. Large corpora should use
 a sharded store or a real database instead of asking one file to become a
 warehouse and then acting wounded when physics invoices us.
@@ -290,9 +293,6 @@ The single-file store uses a sidecar lock file for shared reads and exclusive
 writes. That protects ordinary multi-process access to the same file. It is
 still not a multi-master replication protocol, and it is not a substitute for a
 coordinator when higher-level write ordering matters.
-An explicit unlock error cannot replace the write outcome: dropping the lock-file
-handle is the final OS lock-release path, so a committed-durability-uncertain
-result remains inspectable rather than being masked by cleanup noise.
 
 ## Near-Term Ergonomic Improvements
 

packages/cultcache-rs/cultcache-rs-derive/src/lib.rs

@@ -75,6 +75,7 @@ pub fn derive_database_entry(input: TokenStream) -> TokenStream {
         };
         let mut key: Option<usize> = None;
         let mut default = false;
+        let mut bytes = false;
         for attribute in field
             .attrs
             .iter()
@@ -88,6 +89,9 @@ pub fn derive_database_entry(input: TokenStream) -> TokenStream {
                 } else if meta.path.is_ident("default") {
                     default = true;
                     Ok(())
+                } else if meta.path.is_ident("bytes") {
+                    bytes = true;
+                    Ok(())
                 } else {
                     Err(meta.error("unsupported field cultcache attribute"))
                 }
@@ -117,6 +121,7 @@ pub fn derive_database_entry(input: TokenStream) -> TokenStream {
             ident: field_ident,
             ty: field.ty,
             default,
+            bytes,
         });
     }
     slots.sort_by_key(|slot| slot.key);
@@ -125,8 +130,14 @@ pub fn derive_database_entry(input: TokenStream) -> TokenStream {
     let serialize_elements = (0..=max_slot).map(|slot| {
         if let Some(field) = slots.iter().find(|field| field.key == slot) {
             let field_ident = &field.ident;
-            quote! {
-                tuple.serialize_element(&self.#field_ident)?;
+            if field.bytes {
+                quote! {
+                    tuple.serialize_element(::serde_bytes::Bytes::new(&self.#field_ident))?;
+                }
+            } else {
+                quote! {
+                    tuple.serialize_element(&self.#field_ident)?;
+                }
             }
         } else {
             quote! {
@@ -138,8 +149,14 @@ pub fn derive_database_entry(input: TokenStream) -> TokenStream {
     let option_declarations = slots.iter().map(|field| {
         let variable = format_ident!("slot_{}", field.key);
         let ty = &field.ty;
-        quote! {
-            let mut #variable: ::std::option::Option<#ty> = ::std::option::Option::None;
+        if field.bytes {
+            quote! {
+                let mut #variable: ::std::option::Option<::serde_bytes::ByteBuf> = ::std::option::Option::None;
+            }
+        } else {
+            quote! {
+                let mut #variable: ::std::option::Option<#ty> = ::std::option::Option::None;
+            }
         }
     });
 
@@ -147,9 +164,17 @@ pub fn derive_database_entry(input: TokenStream) -> TokenStream {
         if let Some(field) = slots.iter().find(|field| field.key == slot) {
             let variable = format_ident!("slot_{}", field.key);
             let ty = &field.ty;
-            quote! {
-                #slot => {
-                    #variable = seq.next_element::<#ty>()?;
+            if field.bytes {
+                quote! {
+                    #slot => {
+                        #variable = seq.next_element::<::serde_bytes::ByteBuf>()?;
+                    }
+                }
+            } else {
+                quote! {
+                    #slot => {
+                        #variable = seq.next_element::<#ty>()?;
+                    }
                 }
             }
         } else {
@@ -164,7 +189,16 @@ pub fn derive_database_entry(input: TokenStream) -> TokenStream {
     let build_fields = slots.iter().map(|field| {
         let field_ident = &field.ident;
         let variable = format_ident!("slot_{}", field.key);
-        if field.default {
+        if field.bytes && field.default {
+            quote! {
+                #field_ident: #variable.unwrap_or_default().into_vec()
+            }
+        } else if field.bytes {
+            let field_name = field_ident.to_string();
+            quote! {
+                #field_ident: #variable.ok_or_else(|| ::serde::de::Error::missing_field(#field_name))?.into_vec()
+            }
+        } else if field.default {
             quote! {
                 #field_ident: #variable.unwrap_or_default()
             }
@@ -240,4 +274,5 @@ struct CultCacheSlot {
     ident: syn::Ident,
     ty: Type,
     default: bool,
+    bytes: bool,
 }