Depth wins over a tail; one owner reads a payload tree, the witness bundle helper uses it

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

packages/cultcache-py/CHANGELOG.md

@@ -10,7 +10,7 @@
 ### Breaking
 
 - A record whose payload carries a slot past its reader's declared slots, at any depth (the document, a nested record, a list element, a union arm, a unit arm), is read, held read-only, and every write or removal of its key (`put`, `put_global`, `update`, `put_envelope`, `put_envelopes`, `delete`, `delete_global`) is refused with `SchemaConflictError` before any store is touched (`cultlib-gaps:ruling:extra-trailing-slots`). Reading dropped those slots silently and the next write shed them. `SchemaConflictError.unread_slots` is an `UnreadSlots(type_name, slot_count, declared_count, occurrences, depth0_only)` and the message names the type and both counts, never a slot value. A depth-0 mark is consented, and the record writable, only when the record is stored under an id the document lists in `compatible_schema_ids` and that id's catalog entry has slots past the document's, covering the mark's slot count (`cultlib-gaps:ruling:slot-mark-consent`); a nested mark, or a mark under the document's own id, keeps the record read-only whatever is declared. Missing trailing slots keep their declared defaults. An untouched marked record keeps its stored bytes through every other write. Added `CultCache.is_writable(document, key)` (advisory; the write still decides: it also answers False for a foreign-carried key, for a key held as a type the document does not own, and for a global document at a non-global key; `delete` asks the same judges, so removing a key held as another type is refused with `SchemaConflictError` unless the document owns that record's schema id (its own or one it lists compatible), in which case the removal removes that record, its file row and its lookups, as a write of the key would replace it; it returned silently and removed nothing), `DocumentDefinition.decode_payload_marked`, and `UnreadSlots`. A document with a custom `decode` (no slot codec) never marks. A store holding a variant record is still unreadable here. Implies a minor bump while the version is 0.x (0.3.0 to 0.4.0): code that wrote back a record from a newer writer now gets `SchemaConflictError`.
-- A payload is one MessagePack value: `DocumentDefinition.decode_payload` refuses bytes after it, by count, with `ValueError` `<type> payload has N bytes after its first MessagePack value (refused on read).` (never the bytes, never the depth refusal; a payload that ends early stays the decoder's), as the C# reference does (`tests/vectors/union-parity/bytes-after-outcomes.msgpack`). It raised `msgpack.ExtraData` (a `ValueError`, so only a caller naming `msgpack.ExtraData` is affected); this is a minor-version bump while pre-1.0. The refused set is unchanged. Cache loads, `get` and every definition built on the MessagePack decoder go through it; a user `payload_decoder` that raises `ExtraData` is refused the same way.
+- A payload is one MessagePack value: `DocumentDefinition.decode_payload` refuses bytes after it, by count, with `ValueError` `<type> payload has N bytes after its first MessagePack value (refused on read).` (never the bytes, never the depth refusal; a payload that ends early stays the decoder's), as the C# reference does (`tests/vectors/union-parity/bytes-after-outcomes.msgpack`). It raised `msgpack.ExtraData` (a `ValueError`, so only a caller naming `msgpack.ExtraData` is affected); this is a minor-version bump while pre-1.0. The refused set is unchanged. Cache loads, `get` and every definition built on the MessagePack decoder go through it; a user `payload_decoder` that raises `ExtraData` is refused the same way. Depth is judged first, so a value nested past 256 with a tail is the depth refusal, as in C#. `read_payload_tree(type_name, payload, decoder=None)` is the exported owner of this read for payloads read outside a definition.
 - A cache holds one record per key whatever its type (`variants:ruling:cache-refuses-retype`). A `put`, `put_envelope`, `put_envelopes` or `put_global` of a second type under a key the cache holds, or under a key a store carries foreign, is refused with `SchemaConflictError` naming the key, even with no store, and changes nothing; retype a key by deleting it, then putting the new type. A write may replace another registered type's record only when the written type owns its schema id (declares it or lists it compatible) and both types share one home store. A load that reads one key from two stores, whatever the types, is refused with `CultCacheError` naming the key and both types, and admits nothing. A store that holds a record no registered type claims now loads and is carried: the cache does not serve it, the store keeps its bytes through every write, `delete` or `put` at its key is refused with `SchemaConflictError`, and registering a type that declares its schema id claims it at the next load. It raised `CultCacheError` "Unknown persisted document type". Two global types share the key `__global__`, so a second global type in one cache is now refused, and a store that already holds two global types at `__global__` (a JSONL store written by caches that never pulled could produce one) no longer loads: the pull raises `CultCacheError` and admits nothing. C# refuses a second persisted global on pull the same way (`cultcache-store-composition.md`, "Globals never invented"); repair such a file by deleting one of the two records with the runtime that wrote it.
 - Added `foreign_records()` and the frozen `ForeignRecord(key, schema_id, schema_name, stored_at)`, sorted by key in UTF-16 code-unit order. `stores.retyped` and `stores.written_owns` lose their underscore: the cache judge and the single-file store's commit guard ask one predicate.
 

packages/cultcache-py/src/cultcache_py/__init__.py

@@ -16,6 +16,7 @@ from .documents import (
     define_document_type,
     define_record_codec,
     define_union,
+    read_payload_tree,
 )
 from .leaves import SlotCodec
 from .stores import JsonLinesBackingStore, SchemaConflictError, SingleFileMessagePackBackingStore, StoreUnreadableError
@@ -40,6 +41,7 @@ __all__ = [
     "StoreUnreadableError",
     "UnionArm",
     "UnreadSlots",
+    "read_payload_tree",
     "leaves",
     "database_entry_field",
     "define_database_entry_type",

packages/cultcache-py/src/cultcache_py/documents.py

@@ -158,6 +158,26 @@ def _messagepack_payload_decoder(payload: bytes) -> Any:
     return msgpack.unpackb(payload, raw=False)
 
 
+def read_payload_tree(type_name: str, payload: bytes, decoder: Callable[[bytes], Any] | None = None) -> Any:
+    """The one read of a payload's tree: exactly one MessagePack value, within the nesting limit. Depth is judged first,
+    on the value read so far, so a tail never pre-empts it (C# throws at container 257 before it reaches the tail); then a
+    tail is refused by count and never echoed. Raised outside the handler so no `__context__` keeps the tail."""
+    after = -1
+    unpacked: Any = None
+    try:
+        tree = (decoder or _messagepack_payload_decoder)(payload)
+    except _STACK_FAULTS:
+        raise CultCachePayloadDepthError(type_name, "read") from None
+    except msgpack.ExtraData as error:
+        after = len(error.extra)
+        unpacked = error.unpacked
+    if after >= 0:
+        _refuse_past_depth_limit(unpacked, type_name, "read")
+        raise ValueError(f"{type_name} payload has {after} bytes after its first MessagePack value (refused on read).")
+    _refuse_past_depth_limit(tree, type_name, "read")
+    return tree
+
+
 @dataclass(frozen=True)
 class DatabaseEntryField:
     name: str
@@ -234,18 +254,7 @@ class DocumentDefinition(Generic[T]):
     def decode_payload_marked(self, payload: bytes) -> tuple[T, UnreadSlots | None]:
         """The value and what the decode found past the declared slots, at any depth. The payload depth check runs first: a
         payload past the limit is refused, never marked."""
-        after = -1
-        try:
-            tree = self.payload_decoder(payload)
-        except _STACK_FAULTS:
-            raise CultCachePayloadDepthError(self.type, "read") from None
-        except msgpack.ExtraData as error:
-            after = len(error.extra)
-        if after >= 0:
-            # A payload is one MessagePack value: the tail is refused by count, never echoed. Raised outside the handler so
-            # no __context__ keeps error.extra (`from None` only hides the context from the traceback).
-            raise ValueError(f"{self.type} payload has {after} bytes after its first MessagePack value (refused on read).")
-        _refuse_past_depth_limit(tree, self.type, "read")
+        tree = read_payload_tree(self.type, payload, self.payload_decoder)
         outer, opened = _current_scope(), _SlotScope()
         _scope.open = opened
         try:

packages/cultnet-py/CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ## Unreleased (breaking; version 0.x, so the next minor)
 
+- `decode_witness_artifact_bundle_payload` (and `CultNetWitnessArtifactBundle.from_payload`) read the payload through `cultcache_py.read_payload_tree`: bytes after its first MessagePack value raise `ValueError` `cultnet.witness_artifact_bundle payload has N bytes after its first MessagePack value (refused on read).` instead of `msgpack.ExtraData`, and a payload nested past 256 raises `CultCachePayloadDepthError`. Needs the `cultcache-py` that exports `read_payload_tree`.
 - `apply_raw_document_record` and `apply_raw_snapshot` raise `SchemaConflictError` when a remote record is of another type than the one the cache holds at its key (`cultcache-py` holds one record per key whatever its type); the cache is unchanged, and the caller decides whether to drop the record or the peer. `cultmesh-py`'s server answers such a put with `message_invalid`; its client's `read_next_change` lets the error out.
 - `CultNetDatabaseChange` carries `commit_id` and `commit_count` (`commitId`/`commitCount`; None outside a multi-record commit, as C# writes nil): `from_wire` refuses an empty or non-string id and a count that is not an integer of at least 2, `to_wire` writes them when set, and the published `database_change_raw` schema lists both. Python applies no batch. The `simulation_fact` document's `shard`, `frame`, `subject` and `claim_kind` aliases are `roles` (many facts share each); the catalog bytes do not move.
 - Decodes the write receipt frames of `contracts/cultnet/interop/write-receipt.md`: `parse_message` validates `cultnet.write_admitted.v0` and `cultnet.error.v1` against their closed schemas (unknown keys, empty or non-string ids, an unknown operation or code are refused; `error.v1` requires `messageId`, string or null; `limit_reached` is `error.v1` only). Added `CULTNET_ERROR_V1_CODES`, `CULTNET_WRITE_RECEIPT_MESSAGE_VERSIONS`, `write_content_hash` and `hello_advertises_write_receipts`; `hello` takes `supported_message_versions`. `tests/test_write_receipt.py` replays the vectors' `contentHash`, `wire` and `schemaInvalid` sections. Not implemented here: the client write tracker (`scenarios`) and the server rules (`dedupe`, `blankFields`); this package has no receipt server and no write client.

packages/cultnet-py/src/cultnet_py/messages.py

@@ -5,6 +5,7 @@ from dataclasses import dataclass
 from typing import Any
 
 import msgpack
+from cultcache_py import read_payload_tree
 
 from .unicode_messageid_tables import DEFAULT_IGNORABLE, LETTER_OR_NUMBER
 
@@ -544,7 +545,7 @@ def encode_witness_artifact_bundle_payload(bundle: dict[str, Any] | CultNetWitne
 
 
 def decode_witness_artifact_bundle_payload(payload: bytes) -> dict[str, Any]:
-    slots = msgpack.unpackb(payload, raw=False)
+    slots = read_payload_tree("cultnet.witness_artifact_bundle", payload)
     if not isinstance(slots, list) or len(slots) < 8:
         raise ValueError("CultNet witness artifact bundle payload must be an 8-slot MessagePack array")
     return {