A cache whose load was refused refuses every write and removal; depth pinned at load; map names the live owner

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

docs/cultcache-union-parity-map.md

@@ -259,7 +259,7 @@ landed in `ead5260c`, Python in `5618d8dd`, Rust in `c028c511`. TypeScript is no
 | Runtime | Owner of the decision (constant and walker) | Doors routed through it |
 |---|---|---|
 | C# | `CultPayloadDepth.Check` (`src/GameCult.Caching.MessagePack/CultPayloadDepth.cs`), limit `CultMessagePackSecurity.MaximumPayloadDepth`. MessagePack's own graph counter is set to the same value as a backstop only. | Write: `CultDocumentMessagePackSerialization.SerializeUntyped`. Read: `DeserializeUntyped`. The cache store (`SerializePayload`/`DeserializePayload`), the directory store, `CultMesh` and single-file documents all call those two; C# has no separate raw-envelope or prepared-batch door. |
-| Rust | `cultcache_rs::check_payload_depth` and `MAX_PAYLOAD_DEPTH` (`packages/cultcache-rs/src/lib.rs`). | Read: `decode_payload` (get, list, `validate_envelope`/`put_envelope`, `finish_prepared_entry`). Write: `put`/`prepare_entry`/`prepare_entry_named` reach it through `finish_prepared_entry`, which decodes what it just encoded. Raw envelope: `put_raw_envelope`, the door CultMesh network puts use (`cultmesh-rs` `lib.rs`, two call sites). Prepared batch: `put_prepared_batch` via `require_payload_depth`. cultnet-rs: every free-form or skippable wire payload decodes through `cultcache_rs::decode_payload` (message decode in `contracts.rs`, `provider_session`, `media_stream_wire`, `replication`, `service_identity`). |
+| Rust | `check_record_payload` (private; depth, then no bytes after the first value) over `first_value_end`, and `MAX_PAYLOAD_DEPTH`; `cultcache_rs::check_payload_depth` is the depth-only probe for outside callers (`packages/cultcache-rs/src/lib.rs`). | Read: `decode_payload` (get, list, `validate_envelope`/`put_envelope`, `finish_prepared_entry`). Write: `put`/`prepare_entry`/`prepare_entry_named` reach it through `finish_prepared_entry`, which decodes what it just encoded. Raw envelope: `put_raw_envelope`, the door CultMesh network puts use (`cultmesh-rs` `lib.rs`, two call sites). Prepared batch: `put_prepared_batch` via `require_record_payload`. Load: `pull_all_backing_stores` and `load_soa` run `require_record_payload` on every record, so a store holding a payload past the bound (or with bytes after it) is refused at load, and the cache then refuses every write and removal (`CultCacheLoadRefused`) until a load succeeds. cultnet-rs: every free-form or skippable wire payload decodes through `cultcache_rs::decode_payload` (message decode in `contracts.rs`, `provider_session`, `media_stream_wire`, `replication`, `service_identity`). |
 | Python | `_refuse_past_depth_limit` and `MAX_PAYLOAD_DEPTH` (`packages/cultcache-py/src/cultcache_py/documents.py`); a msgpack `StackError` on read maps to the same error. | `DocumentDefinition.encode_payload` (write) and `decode_payload` (read). `CultCache.put`, `put_envelope`, `put_envelopes` and load-time hydration go through them; there is no raw-envelope door. cultmesh-py round trips call the same pair. |
 
 - **Doors deliberately not routed.** cultnet-rs `idunn_lifecycle_brake`, `provider_health_contracts` and

packages/cultcache-rs/CHANGELOG.md

@@ -8,6 +8,7 @@ only in git history.
 
 ### Breaking
 
+- A cache whose last `pull_all_backing_stores` was refused (a tail, a payload past the nesting bound, a duplicate unique-index value, a retype conflict, a record from the wrong store) refuses every `put`, `put_envelope`, `put_raw_envelope`, `put_prepared_batch`, `update` and `delete` with `CultCacheLoadRefused` until a pull succeeds. Before, it stayed writable over an empty image, and the next batch write replaced the store with only the new records.
 - `decode_payload`, `put_raw_envelope` and prepared batches refuse a payload with bytes after its first MessagePack value (`payload has {n} bytes after its first MessagePack value`), by count; the tail is never walked or quoted. They used to read the first value and keep or drop the tail, so a record with a stray tail was served and rewritten without it. A store holding such a record (single-file, redb, `load_soa`) is refused at load, naming the type, the key and the count, and stays unreadable until the record is repaired; the same check now runs on every record at load, as the C# and Python runtimes do at open. The nesting bound is part of that check, so a stored payload nested past it is refused at load too, where it used to load and refuse on read. Behaviour change in a public API: the next release is a minor bump (0.x breaking).
 - A single-file store written as `cultcache.store.v3` is read and written: a store the C# runtime marked v3 (it holds a record with an element id: an owned list element with a `[CultElementId]` member) keeps its header here, and is carried through every rewrite with its ids; this runtime never drops the mark. A runtime from before this change refuses a v3 store by name. Rolling back to an older release after a store has been written as v3 is one-way: there is no carry-back, so restore from a copy taken before upgrading.
 - A write over a file the reader refuses (a zero-byte file, a truncated store, a stamp this runtime does not read) refuses with `CultCacheStoreUnreadable` and leaves the file as it is. A whole-store write used to replace such a file. This reverses 0.3.1, where `push_all` replaced a file with no readable stamp (healing it); the stamp rule now has one owner, the reader every write asks under the lock, and it accepts v1 and v3 (0.3.1 refused v3).

packages/cultcache-rs/src/lib.rs

@@ -2772,8 +2772,30 @@ pub struct CultCache {
     /// Records a store holds that no registered type claims: (store index, the envelope as the store read it). Derived from the last
     /// pull and replaced with it; written nowhere else. It decides only refusals at its keys.
     foreign: Vec<(usize, CultCacheEnvelope)>,
+    /// Why the last pull of the backing stores was refused; `None` once a pull succeeds. While set, every write and removal is refused.
+    load_refused: Option<String>,
 }
 
+/// A write or removal refused because the cache's last load of its backing stores was refused. The cache holds no
+/// image of those stores, so any write would persist an image that lacks the records it never read. Repair the store
+/// named by `reason` (which names a type, a key and a clause, never a payload) and load again.
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct CultCacheLoadRefused {
+    pub reason: String,
+}
+
+impl std::fmt::Display for CultCacheLoadRefused {
+    fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(
+            formatter,
+            "CultCache refuses writes and removals: its load of the backing stores failed ({}); repair the store and load again",
+            self.reason
+        )
+    }
+}
+
+impl std::error::Error for CultCacheLoadRefused {}
+
 /// A record a store holds that no registered type claims: carried, never served.
 #[derive(Clone, Debug, PartialEq, Eq)]
 pub struct ForeignRecord {
@@ -2835,6 +2857,7 @@ impl CultCache {
             projectors: BTreeMap::new(),
             stores: Vec::new(),
             foreign: Vec::new(),
+            load_refused: None,
         }
     }
 
@@ -2971,6 +2994,32 @@ impl CultCache {
     /// Reads every store into a staged image and replaces the cache's view
     /// only when all of them loaded, so a refused load admits nothing.
     pub fn pull_all_backing_stores(&mut self) -> Result<()> {
+        match self.stage_pull() {
+            Ok((loaded, foreign, holders)) => {
+                self.entries = loaded;
+                self.foreign = foreign;
+                self.index_holders = holders;
+                self.load_refused = None;
+                Ok(())
+            }
+            Err(error) => {
+                self.load_refused = Some(format!("{error:#}"));
+                Err(error)
+            }
+        }
+    }
+
+    /// The one gate of every write and removal: a cache whose load was refused holds no image of its stores.
+    fn require_loaded(&self) -> Result<()> {
+        match &self.load_refused {
+            Some(reason) => Err(CultCacheLoadRefused { reason: reason.clone() }.into()),
+            None => Ok(()),
+        }
+    }
+
+    /// The image every store holds, judged whole, installed nowhere.
+    #[allow(clippy::type_complexity)]
+    fn stage_pull(&self) -> Result<(BTreeMap<(String, String), CultCacheEnvelope>, Vec<(usize, CultCacheEnvelope)>, IndexHolders)> {
         let mut loaded = BTreeMap::new();
         let mut foreign = Vec::new();
         let known_types: BTreeSet<String> = self.definitions.keys().cloned().collect();
@@ -3010,10 +3059,7 @@ impl CultCache {
         }
         self.refuse_second_globals(&loaded)?;
         let holders = self.judge_image(&loaded)?;
-        self.entries = loaded;
-        self.foreign = foreign;
-        self.index_holders = holders;
-        Ok(())
+        Ok((loaded, foreign, holders))
     }
 
     /// The records the last pull found in a store that no registered type claims, sorted by key (UTF-16 order). The cache does not
@@ -3185,6 +3231,7 @@ impl CultCache {
     }
 
     pub fn put<T: DatabaseEntry>(&mut self, key: impl Into<String>, value: &T) -> Result<T> {
+        self.require_loaded()?;
         let (entry, parsed) = self.prepare_entry(key, value)?;
         self.refuse_second_global(&self.entries, &entry.r#type, &entry.key)?;
         self.refuse_retype(&self.entries, &self.foreign, &entry)?;
@@ -3261,6 +3308,7 @@ impl CultCache {
     }
 
     pub fn put_prepared_batch(&mut self, staged: Vec<CultCacheEnvelope>) -> Result<()> {
+        self.require_loaded()?;
         if staged.is_empty() {
             return Err(anyhow!("CultCache batch must contain at least one entry"));
         }
@@ -3327,6 +3375,7 @@ impl CultCache {
     }
 
     pub fn put_envelope<T: DatabaseEntry>(&mut self, entry: CultCacheEnvelope) -> Result<T> {
+        self.require_loaded()?;
         let parsed = self.validate_envelope::<T>(&entry)?;
         self.refuse_claimed_id(&entry)?;
         self.refuse_second_global(&self.entries, &entry.r#type, &entry.key)?;
@@ -3346,6 +3395,7 @@ impl CultCache {
     /// admission; CultCache still enforces registered type, identity, timestamp,
     /// the payload nesting bound ([`MAX_PAYLOAD_DEPTH`]), routing, and persistence.
     pub fn put_raw_envelope(&mut self, entry: CultCacheEnvelope) -> Result<()> {
+        self.require_loaded()?;
         if !self.definitions.contains_key(&entry.r#type) {
             return Err(anyhow!(
                 "No entry type registered for CultCache envelope type {:?}",
@@ -3419,6 +3469,7 @@ impl CultCache {
     }
 
     pub fn delete<T: DatabaseEntry>(&mut self, key: &str) -> Result<bool> {
+        self.require_loaded()?;
         self.require_entry_type::<T>()?;
         let id = entry_id_parts(T::TYPE, key);
         let Some(entry) = self.entries.get(&id).cloned() else {

packages/cultcache-rs/tests/union_parity.rs

@@ -6,6 +6,7 @@ use std::path::PathBuf;
 use cultcache_rs::CultCache;
 use cultcache_rs::CacheBackingStore;
 use cultcache_rs::CultCacheEnvelope;
+use cultcache_rs::CultCacheLoadRefused;
 use cultcache_rs::CultCacheSoaTable;
 use cultcache_rs::MAX_PAYLOAD_DEPTH;
 use cultcache_rs::PayloadTooDeep;
@@ -830,6 +831,81 @@ fn depth_beats_a_tail() {
     assert_eq!(error.downcast_ref::<PayloadTooDeep>(), Some(&TOO_DEEP), "{error:#}");
 }
 
+// Depth is refused where the payload is loaded: a store holding one nested past the bound is not read.
+#[test]
+fn depth_is_refused_at_load() {
+    let mut deep = vec![0x91u8; MAX_PAYLOAD_DEPTH + 1];
+    deep.push(0xc0);
+    let temp = tempfile::tempdir().unwrap();
+    let path = temp.path().join("deep.cc");
+    SingleFileMessagePackBackingStore::new(&path).push(&envelope("k", &deep)).unwrap();
+    let mut cache = CultCache::new();
+    cache.register_entry_type::<UnionSurface>().unwrap();
+    cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&path)).unwrap();
+    let error = cache.pull_all_backing_stores().unwrap_err();
+    assert_eq!(error.downcast_ref::<PayloadTooDeep>(), Some(&TOO_DEEP), "{error:#}");
+    assert!(cache.snapshot().is_empty());
+}
+
+// A cache whose load was refused refuses every write and removal and leaves the store as it was: a
+// later write would otherwise persist the empty image over the records it never read.
+#[test]
+fn a_refused_load_refuses_every_write() {
+    let tail = {
+        let mut bytes = vector("union-surface.msgpack");
+        bytes.push(0xc0);
+        bytes
+    };
+    let good = envelope("good", &vector("union-surface.msgpack"));
+    let bad = envelope("bad", &tail);
+    for redb in [false, true] {
+        let temp = tempfile::tempdir().unwrap();
+        let path = temp.path().join(if redb { "seed.redb" } else { "seed.cc" });
+        let mut cache = CultCache::new();
+        cache.register_entry_type::<UnionSurface>().unwrap();
+        if redb {
+            let mut store = RedbMessagePackBackingStore::new(&path).unwrap();
+            store.push(&good).unwrap();
+            store.push(&bad).unwrap();
+            drop(store);
+            cache.add_generic_backing_store(RedbMessagePackBackingStore::new(&path).unwrap()).unwrap();
+        } else {
+            let mut store = SingleFileMessagePackBackingStore::new(&path);
+            store.push(&good).unwrap();
+            store.push(&bad).unwrap();
+            cache.add_generic_backing_store(SingleFileMessagePackBackingStore::new(&path)).unwrap();
+        }
+        let load = cache.pull_all_backing_stores().unwrap_err();
+        assert!(format!("{load:#}").contains("bytes after"), "{load:#}");
+        let before = std::fs::read(&path).unwrap();
+        let value = constructed();
+        let staged = cache.prepare_entry("new", &value).unwrap().0;
+        let results: Vec<(&str, anyhow::Error)> = vec![
+            ("put", cache.put("new", &value).unwrap_err()),
+            ("put_prepared_batch", cache.put_prepared_batch(vec![staged.clone()]).unwrap_err()),
+            ("put_envelope", cache.put_envelope::<UnionSurface>(staged.clone()).unwrap_err()),
+            ("put_raw_envelope", cache.put_raw_envelope(staged.clone()).unwrap_err()),
+            ("update", cache.update::<UnionSurface, _>("good", |current| current.unwrap_or_else(constructed)).unwrap_err()),
+            ("delete", cache.delete::<UnionSurface>("good").unwrap_err()),
+            ("delete a key the image lacks", cache.delete::<UnionSurface>("absent").unwrap_err()),
+        ];
+        for (door, error) in results {
+            let refusal = error.downcast_ref::<CultCacheLoadRefused>().unwrap_or_else(|| panic!("{door} redb={redb}: {error:#}"));
+            assert!(refusal.to_string().contains("bytes after"), "{door}: {refusal}");
+        }
+        assert_eq!(std::fs::read(&path).unwrap(), before, "redb={redb}: the store changed");
+        assert!(cache.snapshot().is_empty());
+        // Repairing the store and loading again lifts it.
+        if redb {
+            RedbMessagePackBackingStore::new(&path).unwrap().delete(&bad).unwrap();
+        } else {
+            SingleFileMessagePackBackingStore::new(&path).delete(&bad).unwrap();
+        }
+        cache.pull_all_backing_stores().unwrap();
+        cache.put("new", &value).unwrap();
+    }
+}
+
 // A payload that ends early is not bytes after the value: the decoder refuses it, not the check.
 #[test]
 fn a_truncated_payload_is_the_decoders_not_bytes_after() {