packages/cultcache-py/CHANGELOG.md

@@ -16,7 +16,7 @@
 
 ### Breaking
 
-- `push_all(envelopes, soft=False)` on the single-file and JSONL stores and on the `BackingStore` protocol takes the soft flag, as TypeScript, Rust and C# do: a soft push writes only where nothing is at the path; over a store the reader reads, empty included, it returns without writing, and over a file the reader cannot read it refuses (`StoreUnreadableError` for the single-file store). The `BackingStore` protocol's default `push_all` cannot tell an absent store from an empty one (a pull returns a list), so a soft push through it always skips a store whose pull returns; a store kind that can tell nothing-at-the-path overrides `push_all`. A store kind that overrides `push_all(envelopes)` without the keyword fails when a caller passes `soft`.
+- `push_all(envelopes, soft=False)` on the single-file and JSONL stores and on the `BackingStore` protocol takes the soft flag, as TypeScript, Rust and C# do: a soft push writes only where nothing is at the path; over a store the reader reads, empty included, it returns without writing, and over a file the reader cannot read it refuses (`StoreUnreadableError` for the single-file store). The `BackingStore` protocol has `pull_or_absent()`: the envelopes the reader reads, `None` when nothing is at the path, a raise when the path cannot be read; its default is `pull_all()`, which cannot say absent, so a store that inherits it is always read. The default `push_all` decides soft by it (writes where it says absent, skips over a store it read, refuses where it raises), and the single-file and JSONL stores expose their pull through it; an inheriting store that can tell nothing-at-the-path overrides `pull_or_absent`, not `push_all`. Before, a soft push through the default always skipped. A store kind that overrides `push_all(envelopes)` without the keyword fails when a caller passes `soft`.
 - A write at a store path that is a link, live or dangling, is refused with the new `StoreLinkedError` (an `OSError`, exported from `cultcache_py`; `.path` names the path) before the lock and the replace, for the single-file store and for `JsonLinesBackingStore` (`push`, `push_all`, `delete`), and nothing is written. Before, a write at a live link replaced the link with a regular file, and at a dangling link both stores raised `FileNotFoundError`. A caller catching `FileNotFoundError` there now catches `StoreLinkedError` (`OSError`); configure the link's target as the store path. On Windows a junction or any reparse point counts as a link. Reads through a live link work.
 - A record whose payload carries a slot past its reader's declared slots, at any depth (the document, a nested record, a list element, a union arm, a unit arm), is read, held read-only, and every write or removal of its key (`put`, `put_global`, `update`, `put_envelope`, `put_envelopes`, `delete`, `delete_global`) is refused with `SchemaConflictError` before any store is touched (`cultlib-gaps:ruling:extra-trailing-slots`). Reading dropped those slots silently and the next write shed them. `SchemaConflictError.unread_slots` is an `UnreadSlots(type_name, slot_count, declared_count, occurrences, depth0_only)` and the message names the type and both counts, never a slot value. A depth-0 mark is consented, and the record writable, only when the record is stored under an id the document lists in `compatible_schema_ids` and that id's catalog entry has slots past the document's, covering the mark's slot count (`cultlib-gaps:ruling:slot-mark-consent`); a nested mark, or a mark under the document's own id, keeps the record read-only whatever is declared. Missing trailing slots keep their declared defaults. An untouched marked record keeps its stored bytes through every other write. Added `CultCache.is_writable(document, key)` (advisory; the write still decides: it also answers False for a foreign-carried key, for a key held as a type the document does not own, and for a global document at a non-global key; `delete` asks the same judges, so removing a key held as another type is refused with `SchemaConflictError` unless the document owns that record's schema id (its own or one it lists compatible), in which case the removal removes that record, its file row and its lookups, as a write of the key would replace it; it returned silently and removed nothing), `DocumentDefinition.decode_payload_marked`, and `UnreadSlots`. A document with a custom `decode` (no slot codec) never marks. A store holding a variant record is still unreadable here. Implies a minor bump while the version is 0.x (0.3.0 to 0.4.0): code that wrote back a record from a newer writer now gets `SchemaConflictError`.
 - A payload is one MessagePack value: `DocumentDefinition.decode_payload` refuses bytes after it, by count, with `ValueError` `<type> payload has N bytes after its first MessagePack value (refused on read).` (never the bytes, never the depth refusal; a payload that ends early stays the decoder's), as the C# reference does (`tests/vectors/union-parity/bytes-after-outcomes.msgpack`). It raised `msgpack.ExtraData` (a `ValueError`, so only a caller naming `msgpack.ExtraData` is affected); this is a minor-version bump while pre-1.0. The refused set is unchanged. Cache loads, `get` and every definition built on the MessagePack decoder go through it; a user `payload_decoder` that raises `ExtraData` is refused the same way. Depth is judged first, so a value nested past 256 with a tail is the depth refusal, as in C#.

packages/cultcache-py/src/cultcache_py/backing_store.py

@@ -67,12 +67,19 @@ class BackingStore(Protocol):
     def delete(self, type: str, key: str) -> None:
         ...
 
+    def pull_or_absent(self) -> list[CultCacheEnvelope] | None:
+        """The envelopes the reader reads (empty included), None when nothing is at the path; raises when the path cannot be read.
+
+        The default is pull_all, which cannot say absent: a store that inherits it is always a store the reader read. A store kind
+        that can tell nothing-at-the-path overrides this, and the default push_all then writes where nothing is on a soft push."""
+        return self.pull_all()
+
     def push_all(self, envelopes: list[CultCacheEnvelope], soft: bool = False) -> None:
-        # A pull that returns is a store the reader read, empty included: the protocol cannot say "absent", so a soft push
-        # through this default always skips. A store that can tell nothing-at-the-path overrides push_all.
-        pulled = self.pull_all()
-        if soft:
+        # Soft: write only where pull_or_absent says nothing is; skip over a store it read, empty included; its raise refuses.
+        pulled = self.pull_or_absent()
+        if soft and pulled is not None:
             return
+        pulled = pulled or []
         existing = {(envelope.type, envelope.key): envelope for envelope in pulled}
         for envelope in envelopes:
             existing[(envelope.type, envelope.key)] = envelope

packages/cultcache-py/src/cultcache_py/stores.py

@@ -78,6 +78,9 @@ class JsonLinesBackingStore:
     def pull_all(self) -> list[CultCacheEnvelope]:
         return self._pull() or []
 
+    def pull_or_absent(self) -> list[CultCacheEnvelope] | None:
+        return self._pull()
+
     def _pull(self) -> list[CultCacheEnvelope] | None:
         """The envelopes the reader reads, or None when nothing is at the path."""
         with self._lock:
@@ -152,6 +155,9 @@ class SingleFileMessagePackBackingStore:
     def pull_all(self) -> list[CultCacheEnvelope]:
         return self._pull() or []
 
+    def pull_or_absent(self) -> list[CultCacheEnvelope] | None:
+        return self._pull()
+
     def _pull(self) -> list[CultCacheEnvelope] | None:
         """The envelopes the reader reads, or None when nothing is at the path."""
         with self._lock:

packages/cultcache-py/tests/test_soft_push_all.py

@@ -69,8 +69,33 @@ class SoftPushAllTests(unittest.TestCase):
             JsonLinesBackingStore(self.path).push_all([envelope("a")], soft=True)
         self.assertEqual(junk, self.path.read_bytes())
 
-    def test_the_protocol_default_push_all_skips_what_its_pull_read_empty_included(self) -> None:
+    def test_the_protocol_default_push_all_decides_soft_by_what_pull_or_absent_says(self) -> None:
         class Memory(BackingStore):
+            def __init__(self) -> None:
+                self.items: list[CultCacheEnvelope] | None = None  # None: nothing at the path
+
+            def pull_all(self) -> list[CultCacheEnvelope]:
+                return list(self.items or [])
+
+            def pull_or_absent(self) -> list[CultCacheEnvelope] | None:
+                return None if self.items is None else list(self.items)
+
+            def _replace_all(self, envelopes: list[CultCacheEnvelope]) -> None:
+                self.items = list(envelopes)
+
+        store = Memory()
+        store.push_all([envelope("a")], soft=True)  # nothing at the path: writes
+        self.assertEqual(["a"], [item.key for item in store.items or []])
+        store.push_all([envelope("b")], soft=True)  # a store with records: skips
+        self.assertEqual(["a"], [item.key for item in store.items or []])
+        store.items = []
+        store.push_all([envelope("b")], soft=True)  # a store the reader read, empty: skips
+        self.assertEqual([], store.items)
+        store.push_all([envelope("b")])  # plain: lands
+        self.assertEqual(["b"], [item.key for item in store.items])
+
+    def test_a_store_that_only_pulls_all_is_read_so_a_soft_push_skips_it(self) -> None:
+        class PullOnly(BackingStore):
             def __init__(self) -> None:
                 self.items: list[CultCacheEnvelope] = []
 
@@ -80,14 +105,18 @@ class SoftPushAllTests(unittest.TestCase):
             def _replace_all(self, envelopes: list[CultCacheEnvelope]) -> None:
                 self.items = list(envelopes)
 
-        store = Memory()
-        store.push_all([envelope("a")], soft=True)  # the pull read an empty store: skips
+        store = PullOnly()
+        store.push_all([envelope("a")], soft=True)
         self.assertEqual([], store.items)
-        store.push_all([envelope("a")])  # plain: lands
-        store.push_all([envelope("b")], soft=True)  # a store with records: skips
-        self.assertEqual(["a"], [item.key for item in store.items])
-        store.push_all([envelope("b")])
-        self.assertEqual(["a", "b"], sorted(item.key for item in store.items))
+
+    def test_the_single_file_and_jsonl_stores_say_absent_and_read_through_pull_or_absent(self) -> None:
+        for make in (SingleFileMessagePackBackingStore, JsonLinesBackingStore):
+            with self.subTest(make.__name__):
+                path = self.dir / make.__name__
+                store = make(path)
+                self.assertIsNone(store.pull_or_absent())
+                store.push_all([envelope("a")])
+                self.assertEqual(["a"], [item.key for item in store.pull_or_absent() or []])
 
     def test_the_protocol_default_push_all_refuses_where_its_pull_refuses(self) -> None:
         class Unreadable(BackingStore):
@@ -96,6 +125,9 @@ class SoftPushAllTests(unittest.TestCase):
             def pull_all(self) -> list[CultCacheEnvelope]:
                 raise StoreUnreadableError(Path("x"), ValueError("junk"))
 
+            def pull_or_absent(self) -> list[CultCacheEnvelope] | None:
+                raise StoreUnreadableError(Path("x"), ValueError("junk"))
+
             def _replace_all(self, envelopes: list[CultCacheEnvelope]) -> None:
                 Unreadable.replaced = True