cultcache-py: convert keeps the source store's header; README and CHANGELOG state the format limits exactly; stale references fixed
All checks were successful
CultMesh portability and performance / ubuntu-latest reactive semantics and leases (pull_request) Successful in 24m50s

Soul pass on #179: a v3 source converted to v1 (now carried, pinned by the v3-base vector and
mutation-checked); README states the one-digit JSON exception once and the object-slot byte
difference from C#; CHANGELOG records the entry-type codec inheritance, the doubled walk and the
accepted JSON depth refusal; the store-reader docstring and test name no longer overstate
'codec alone'; stale payload_decoder/_refuse_past_depth_limit references fixed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Libby 2026-11-17 02:21:05 +00:00
commit b36cf5ed91
8 changed files with 42 additions and 24 deletions

View file

@ -260,7 +260,7 @@ landed in `ead5260c`, Python in `5618d8dd`, Rust in `c028c511`. TypeScript is no
|---|---|---|
| C# | `CultPayloadDepth.Check` (`src/GameCult.Caching.MessagePack/CultPayloadDepth.cs`), limit `CultMessagePackSecurity.MaximumPayloadDepth`. MessagePack's own graph counter is set to the same value as a backstop only. | Write: `CultDocumentMessagePackSerialization.SerializeUntyped`. Read: `DeserializeUntyped`. The cache store (`SerializePayload`/`DeserializePayload`), the directory store, `CultMesh` and single-file documents all call those two; C# has no separate raw-envelope or prepared-batch door. |
| Rust | `check_record_payload` (private; depth, then no bytes after the first value) over `first_value_end`, and `MAX_PAYLOAD_DEPTH`; `cultcache_rs::check_payload_depth` is the depth-only probe for outside callers (`packages/cultcache-rs/src/lib.rs`). | Read: `decode_payload` (get, list, `validate_envelope`/`put_envelope`, `finish_prepared_entry`). Write: `put`/`prepare_entry`/`prepare_entry_named` reach it through `finish_prepared_entry`, which decodes what it just encoded. Raw envelope: `put_raw_envelope`, the door CultMesh network puts use (`cultmesh-rs` `lib.rs`, two call sites). Prepared batch: `put_prepared_batch` via `require_record_payload`. Load: `pull_all_backing_stores` and `load_soa` run `require_record_payload` on every record, so a store holding a payload past the bound (or with bytes after it) is refused at load, and the cache then refuses every write and removal (`CultCacheLoadRefused`) until a load succeeds. cultnet-rs: every free-form or skippable wire payload decodes through `cultcache_rs::decode_payload` (message decode in `contracts.rs`, `provider_session`, `media_stream_wire`, `replication`, `service_identity`). |
| Python | `_refuse_past_depth_limit` and `MAX_PAYLOAD_DEPTH` (`packages/cultcache-py/src/cultcache_py/documents.py`); a msgpack `StackError` on read maps to the same error. | `DocumentDefinition.encode_payload` (write) and `decode_payload` (read). `CultCache.put`, `put_envelope`, `put_envelopes` and load-time hydration go through them; there is no raw-envelope door. cultmesh-py round trips call the same pair. |
| Python | `_containers` (the one depth walk, through `_check_payload_tree`) and `MAX_PAYLOAD_DEPTH` (`packages/cultcache-py/src/cultcache_py/documents.py`); a msgpack `StackError` on read maps to the same error. | `DocumentDefinition.encode_payload` (write) and `decode_payload` (read). `CultCache.put`, `put_envelope`, `put_envelopes` and load-time hydration go through them; there is no raw-envelope door. cultmesh-py round trips call the same pair. |
- **Doors deliberately not routed.** cultnet-rs `idunn_lifecycle_brake`, `provider_health_contracts` and
`runtime_authority_contracts` decode records whose positional array length is checked exactly before decoding. A

View file

@ -14,7 +14,7 @@ raw = msgpack.unpackb(open(path, "rb").read(), raw=False)
print("store header", raw[0], "catalog entry", raw[1][0][:5] if raw[1] else None)
print("record", raw[2][0][:3], "payload", raw[2][0][3][:40])
# MessagePack-default reader, today's _messagepack pair standing in for the proposed default
new = cc.define_document_type("brokkr.sync.var.v0", payload_encoder=lambda t: msgpack.packb(t, use_bin_type=True), payload_decoder=lambda b: msgpack.unpackb(b, raw=False))
new = cc.define_document_type("brokkr.sync.var.v0") # historical: the probe stood a msgpack pair in for the proposed default, now the default
try: open_cache(new); print("new reader: ACCEPTED")
except Exception as e: print("new reader refused:", type(e).__module__, type(e).__name__, str(e)[:80])
# opt-in JSON pair reads it

View file

@ -3,7 +3,7 @@ import sys, os
sys.path.insert(0, r"F:\Projects\CultLib\packages\cultcache-py\src"); sys.path.insert(0, os.path.dirname(__file__))
import proto_leaves as L
from cultcache_py import CultCache, SingleFileMessagePackBackingStore
from cultcache_py.documents import define_document_type
from cultcache_py.documents import PayloadCodec, define_document_type
from cultcache_py.backing_store import CultCacheSchemaCatalogMember as M
NS = "GameCult.Caching.Tests.PyLeavesProbe"
@ -25,7 +25,7 @@ def payload(doc_id):
def write(path, members, tree_for):
doc = define_document_type(
"cultcache.py-leaves-probe", encode=lambda v: v, decode=lambda v: v,
payload_encoder=L.write, payload_decoder=lambda b: b, name=lambda v: v[0],
payload_codec=PayloadCodec("leaves-probe", L.write, lambda b: b), name=lambda v: v[0],
schema_name="cultcache.py-leaves-probe", schema_version="cultcache.py_leaves_probe.v1",
members=[M(slot=s, member_name=n, type_name=t, is_name=(s == 0)) for s, n, t in members])
if os.path.exists(path): os.remove(path)

View file

@ -4,13 +4,13 @@
### Changed
- Breaking: `define_document_type` writes and reads MessagePack payloads by default (`cultlib-gaps:ruling:py-msgpack-default`); it wrote JSON until 0.3. A definition's `payload_codec` (`PayloadCodec`, with `MESSAGEPACK_PAYLOAD_CODEC` and `JSON_PAYLOAD_CODEC` exported) owns its payload format, replacing the `payload_encoder` and `payload_decoder` parameters and fields, so no definition carries a second author of the format. A store written by 0.3 or earlier holds JSON: define its types with `payload_codec=JSON_PAYLOAD_CODEC` to read them, or write a new MessagePack store with `convert(source, destination, definitions)`, which never writes its source. The MessagePack codec takes the keys, integers, timestamps and nesting the C# reference's object reader takes and refuses the rest on read and on write (`CultCachePayloadFormatError`, `PayloadCodecFault`; see the README, Payload format); the single-file store's reader takes the same map keys. A 0.3 JSON payload that is an object or list is refused by count as a payload with bytes after its first MessagePack value, a `ValueError`. This is the same minor bump as the rest of this section: 0.4.0 is not yet tagged.
- Breaking: `define_document_type` writes and reads MessagePack payloads by default (`cultlib-gaps:ruling:py-msgpack-default`); it wrote JSON until 0.3. A definition's `payload_codec` (`PayloadCodec`, with `MESSAGEPACK_PAYLOAD_CODEC` and `JSON_PAYLOAD_CODEC` exported) owns its payload format, replacing the `payload_encoder` and `payload_decoder` parameters and fields, so no definition carries a second author of the format. A store written by 0.3 or earlier holds JSON: define its types with `payload_codec=JSON_PAYLOAD_CODEC` to read them, or write a new MessagePack store with `convert(source, destination, definitions)`, which never writes its source. The MessagePack codec takes the keys, integers, timestamps and nesting the C# reference's object reader takes and refuses the rest on read and on write (`CultCachePayloadFormatError`, `PayloadCodecFault`; see the README, Payload format); the single-file store's reader takes the same map keys. A 0.3 JSON payload that is an object or list is refused by count as a payload with bytes after its first MessagePack value, a `ValueError`. `define_database_entry_type` inherits the checked codec too: main wrote `{b'k': 1}` in an untyped slot, which is now refused on write, and a bin map key in an entry payload is refused on read where main read it, so a main-era store with a bin-keyed entry fails to load. The checked codec walks each tree twice (the depth walk, then the carried-value check), which roughly doubles entry and document codec cost; the double walk is left in place. The JSON codec's `depth_faults=(RecursionError,)` gives a JSON payload nested past the interpreter's stack the typed `CultCachePayloadDepthError`, which resolves the deferred `cut-union-bounds-py.s2.depth-typed-refusal-lost` for the JSON codec only. This is the same minor bump as the rest of this section: 0.4.0 is not yet tagged.
- `parse_stored_at` (was `_parse_stored_at`) is public: cultmesh-py judges a peer's `storedAt` by it. A peer's `document_put_raw` whose `storedAt` is missing, empty or not a stamp is refused `message_invalid` (field `document`) and nothing is minted or stored (variants:ruling:raw-put-without-stamp-refused). A subscribed server's stampless put now raises `ValueError` from `CultMeshClient.read_next_change` instead of being stored under a minted stamp.
### Added
- `CultCache.update` and `update_global` return `UpdateResult(outcome, document)`; `UpdateOutcome` is `UPDATED`, `DECLINED` or `LOST_RACE`. The single-file store gains `read(key)` (the record and its content version, the SHA-256 C# computes) and `commit_if(envelope, expected)`, the version is the SHA-256 of the record re-encoded as `[key, schemaId, storedAt, payload]` with minimal MessagePack encodings, which is the SHA-256 of the record's stored bytes exactly when the writer encoded it minimally (every writer in this library does; a record another writer encoded with a wider-than-needed length or integer has the version of its minimal form, not of its bytes). Both looked up by the cache and not `BackingStore` protocol members (variants:cut_spec:cut-update-with-retry-py).
- `read_payload_tree(type_name, payload, decoder=None)`: the exported owner of the one-value, depth-first payload read that `decode_payload` uses, for payloads read outside a definition (`cultnet-py`'s witness bundle decode needs it, so `cultnet-py` now requires `cultcache-py>=0.4.0`). The manifest version is 0.4.0, the minor these Unreleased breaking changes imply, set in this change so the workspace install resolves the new bound; the `## Unreleased` heading becomes `## [0.4.0]` when it is tagged.
- `read_payload_tree(type_name, payload, decoder=None, depth_faults=(msgpack.exceptions.StackError,))`: the exported owner of the one-value, depth-first payload read that `decode_payload` uses, for payloads read outside a definition (`cultnet-py`'s witness bundle decode needs it, so `cultnet-py` now requires `cultcache-py>=0.4.0`). The manifest version is 0.4.0, the minor these Unreleased breaking changes imply, set in this change so the workspace install resolves the new bound; the `## Unreleased` heading becomes `## [0.4.0]` when it is tagged.
### Changed
@ -29,7 +29,7 @@
- `push_all(envelopes, soft=False)` on the single-file and JSONL stores and on the `BackingStore` protocol takes the soft flag, as TypeScript, Rust and C# do: a soft push writes only where nothing is at the path; over a store the reader reads, empty included, it returns without writing, and over a file the reader cannot read it refuses (`StoreUnreadableError` for the single-file store). The `BackingStore` protocol has `pull_or_absent()`: the envelopes the reader reads, `None` when nothing is at the path, a raise when the path cannot be read; its default is `pull_all()`, which cannot say absent, so a store that inherits it is always read. The default `push_all` decides soft by it (writes where it says absent, skips over a store it read, refuses where it raises), and the single-file and JSONL stores expose their pull through it; an inheriting store that can tell nothing-at-the-path overrides `pull_or_absent`, not `push_all`. Before, a soft push through the default always skipped. A store kind that overrides `push_all(envelopes)` without the keyword fails when a caller passes `soft`.
- A write at a store path that is a link, live or dangling, is refused with the new `StoreLinkedError` (an `OSError`, exported from `cultcache_py`; `.path` names the path) before the lock and the replace, for the single-file store and for `JsonLinesBackingStore` (`push`, `push_all`, `delete`), and nothing is written. Before, a write at a live link replaced the link with a regular file, and at a dangling link both stores raised `FileNotFoundError`. A caller catching `FileNotFoundError` there now catches `StoreLinkedError` (`OSError`); configure the link's target as the store path. On Windows a junction or any reparse point counts as a link. Reads through a live link work.
- A record whose payload carries a slot past its reader's declared slots, at any depth (the document, a nested record, a list element, a union arm, a unit arm), is read, held read-only, and every write or removal of its key (`put`, `put_global`, `update`, `put_envelope`, `put_envelopes`, `delete`, `delete_global`) is refused with `SchemaConflictError` before any store is touched (`cultlib-gaps:ruling:extra-trailing-slots`). Reading dropped those slots silently and the next write shed them. `SchemaConflictError.unread_slots` is an `UnreadSlots(type_name, slot_count, declared_count, occurrences, depth0_only)` and the message names the type and both counts, never a slot value. A depth-0 mark is consented, and the record writable, only when the record is stored under an id the document lists in `compatible_schema_ids` and that id's catalog entry has slots past the document's, covering the mark's slot count (`cultlib-gaps:ruling:slot-mark-consent`); a nested mark, or a mark under the document's own id, keeps the record read-only whatever is declared. Missing trailing slots keep their declared defaults. An untouched marked record keeps its stored bytes through every other write. Added `CultCache.is_writable(document, key)` (advisory; the write still decides: it also answers False for a foreign-carried key, for a key held as a type the document does not own, and for a global document at a non-global key; `delete` asks the same judges, so removing a key held as another type is refused with `SchemaConflictError` unless the document owns that record's schema id (its own or one it lists compatible), in which case the removal removes that record, its file row and its lookups, as a write of the key would replace it; it returned silently and removed nothing), `DocumentDefinition.decode_payload_marked`, and `UnreadSlots`. A document with a custom `decode` (no slot codec) never marks. A store holding a variant record is still unreadable here. Implies a minor bump while the version is 0.x (0.3.0 to 0.4.0): code that wrote back a record from a newer writer now gets `SchemaConflictError`.
- A payload is one MessagePack value: `DocumentDefinition.decode_payload` refuses bytes after it, by count, with `ValueError` `<type> payload has N bytes after its first MessagePack value (refused on read).` (never the bytes, never the depth refusal; a payload that ends early stays the decoder's), as the C# reference does (`tests/vectors/union-parity/bytes-after-outcomes.msgpack`). It raised `msgpack.ExtraData` (a `ValueError`, so only a caller naming `msgpack.ExtraData` is affected); this is a minor-version bump while pre-1.0. The refused set is unchanged. Cache loads, `get` and every definition built on the MessagePack decoder go through it; a user `payload_decoder` that raises `ExtraData` is refused the same way. Depth is judged first, so a value nested past 256 with a tail is the depth refusal, as in C#.
- A payload is one MessagePack value: `DocumentDefinition.decode_payload` refuses bytes after it, by count, with `ValueError` `<type> payload has N bytes after its first MessagePack value (refused on read).` (never the bytes, never the depth refusal; a payload that ends early stays the decoder's), as the C# reference does (`tests/vectors/union-parity/bytes-after-outcomes.msgpack`). It raised `msgpack.ExtraData` (a `ValueError`, so only a caller naming `msgpack.ExtraData` is affected); this is a minor-version bump while pre-1.0. The refused set is unchanged. Cache loads, `get` and every definition built on a codec go through it; a codec whose `decode` raises `ExtraData` is refused the same way. Depth is judged first, so a value nested past 256 with a tail is the depth refusal, as in C#.
- A cache holds one record per key whatever its type (`variants:ruling:cache-refuses-retype`). A `put`, `put_envelope`, `put_envelopes` or `put_global` of a second type under a key the cache holds, or under a key a store carries foreign, is refused with `SchemaConflictError` naming the key, even with no store, and changes nothing; retype a key by deleting it, then putting the new type. A write may replace another registered type's record only when the written type owns its schema id (declares it or lists it compatible) and both types share one home store. A load that reads one key from two stores, whatever the types, is refused with `CultCacheError` naming the key and both types, and admits nothing. A store that holds a record no registered type claims now loads and is carried: the cache does not serve it, the store keeps its bytes through every write, `delete` or `put` at its key is refused with `SchemaConflictError`, and registering a type that declares its schema id claims it at the next load. It raised `CultCacheError` "Unknown persisted document type". Two global types share the key `__global__`, so a second global type in one cache is now refused, and a store that already holds two global types at `__global__` (a JSONL store written by caches that never pulled could produce one) no longer loads: the pull raises `CultCacheError` and admits nothing. C# refuses a second persisted global on pull the same way (`cultcache-store-composition.md`, "Globals never invented"); repair such a file by deleting one of the two records with the runtime that wrote it.
- Added `foreign_records()` and the frozen `ForeignRecord(key, schema_id, schema_name, stored_at)`, sorted by key in UTF-16 code-unit order. `stores.retyped` and `stores.written_owns` lose their underscore: the cache judge and the single-file store's commit guard ask one predicate.

View file

@ -154,11 +154,13 @@ A store whose elements hold an id is marked `cultcache.store.v3` by the C# runti
A definition's `payload_codec` owns its payload bytes. The default,
`MESSAGEPACK_PAYLOAD_CODEC`, writes MessagePack; a store holds no format marker,
so the registered definition decides how every record is read, and a record in
another format is refused, never guessed at: a JSON object or list read as MessagePack
is refused by count (`<type> payload has N bytes after its first MessagePack value`,
a `ValueError`), and a payload the codec cannot read or carry is a
`CultCachePayloadFormatError` or `CultCachePayloadDepthError`.
so the registered definition decides how every record is read. A record in another
format is refused when it is not one value of the definition's codec: a JSON object,
list, string, `true` or `null` read as MessagePack is refused by count (`<type>
payload has N bytes after its first MessagePack value`, a `ValueError`), and a payload
the codec cannot read or carry is a `CultCachePayloadFormatError` or
`CultCachePayloadDepthError`. The one exception is a JSON document that is a single
digit (`0` to `9`), below.
- Map keys are `str`, integer, float, `bool` or `msgpack.Timestamp`, as the C#
reference's object reader takes them; `None`, `bytes`, list and map keys are
@ -186,8 +188,9 @@ a `ValueError`), and a payload the codec cannot read or carry is a
leaves no destination; a killed run leaves the source as it was and a partial
destination, so delete the destination and run again.
- A JSON payload that is a single digit (`0` to `9`) is also a valid one-byte
MessagePack integer and reads as 48 to 57 under the default. Documents that
are objects or lists never meet this.
MessagePack integer and reads as 48 to 57 under the default, with no refusal. Only a
document whose whole value is such a digit meets this; objects, lists and
longer numbers never do.
- A `PayloadCodec` you build owns the nesting depth of nodes the library's walk
cannot see (anything that is not a mapping, sequence or set), and may raise
`PayloadCodecFault(code, kind)` for a key or integer its format cannot carry.
@ -290,6 +293,10 @@ carries the value.
## Wire Parity
Bytes match the C# reference for typed slots. An untyped (`object`) slot differs in
what it writes: C# writes a boxed integer 1 as `d2 00000001`, this runtime writes the
fixint `01`; each runtime reads the other's bytes.
See [docs/python-runtime-parity.md](../../docs/python-runtime-parity.md) for
the evidence map across the three Python packages.

View file

@ -1063,7 +1063,7 @@ def convert(
) -> int:
"""Write the records of `source` to a new store of the same kind at `destination`, the records of `definitions`
rewritten from `source_codec` to each definition's own codec, and return the count rewritten. Records of other types
keep their bytes. Stop every writer of the source before calling. `destination` must not exist: the call creates it
keep their bytes. Stop every writer of the source before calling. The destination keeps the header the source was read under (a `cultcache.store.v3` source stays v3). `destination` must not exist: the call creates it
exclusively, and it is complete only once the call returns. A refusal (a record, a failed write) or an interrupt
after the create removes it and leaves the source as it was. A killed run leaves the source intact and a partial
destination: delete the destination and run again. A file another process creates at the destination between the
@ -1114,7 +1114,7 @@ def convert(
if failure is not None:
raise failure
converted.append(envelope)
target.write(type(source)._file_bytes(converted))
target.write(source._file_bytes(converted))
target.flush()
os.fsync(target.fileno())
except BaseException:

View file

@ -237,10 +237,9 @@ class SingleFileMessagePackBackingStore:
with store_write_lock(self.path, self.lock_timeout):
yield
@staticmethod
def _file_bytes(envelopes: list[CultCacheEnvelope]) -> bytes:
"""The bytes of a new store holding exactly `envelopes`, as a write would leave it."""
return msgpack.packb(_encode_snapshot(envelopes, [], STORE_FORMAT_VERSION)[0], use_bin_type=True) # type: ignore[no-any-return]
def _file_bytes(self, envelopes: list[CultCacheEnvelope]) -> bytes:
"""The bytes of a new store holding exactly `envelopes`, under the header this store read (as a write keeps it)."""
return msgpack.packb(_encode_snapshot(envelopes, [], self._format)[0], use_bin_type=True) # type: ignore[no-any-return]
def _replace_all(self, supplied: list[CultCacheEnvelope], arrived: list[CultCacheEnvelope]) -> dict[str, str]:
snapshot, stamps = _encode_snapshot(supplied, arrived, self._format)
@ -678,9 +677,9 @@ def _decode_snapshot(decoded: Any) -> tuple[str, list[CultCacheEnvelope]] | None
def _decode_legacy_envelopes(decoded: Any) -> list[CultCacheEnvelope]:
"""A legacy envelope holds its payload as an inline MessagePack subtree or as bytes. Extracting the subtree's bytes
decides no format: the inline value is MessagePack by construction, and the registered codec still reads it or
refuses it."""
"""A legacy envelope holds its payload as an inline MessagePack subtree or as bytes. The store's reader judged the
subtree's map keys when it unpacked the whole file (`_unpack_tree`, whatever the record's type or codec), so a nil or
bin key anywhere in the file refuses the store; the registered codec then reads the repacked bytes or refuses them."""
if not isinstance(decoded, list):
raise ValueError("CultCache MessagePack store is not a recognized snapshot")
envelopes: list[CultCacheEnvelope] = []

View file

@ -625,7 +625,7 @@ class PayloadCodecTests(unittest.TestCase):
refused.pull_all_backing_stores()
self.assertEqual(caught.exception.code, "not_one_value")
def test_a_legacy_inline_payload_key_is_judged_by_the_codec_alone(self) -> None:
def test_a_legacy_inline_payload_key_is_judged_by_the_store_reader_for_every_record(self) -> None:
def store_of(payload: object, name: str) -> SingleFileMessagePackBackingStore:
path = self.directory / f"inline-{name}.cc"
path.write_bytes(
@ -685,6 +685,18 @@ class ConvertTests(unittest.TestCase):
cache.pull_all_backing_stores()
return cache.get_all(document)
def test_convert_keeps_the_header_the_source_was_read_under(self) -> None:
vector = Path(__file__).resolve().parents[3] / "tests" / "vectors" / "document-variants-c2a" / "v3-base.msgpack"
source_path = self.directory / "v3.cc"
source_path.write_bytes(vector.read_bytes())
destination = self.directory / "out.cc"
self.assertEqual(convert(SingleFileMessagePackBackingStore(source_path), destination, []), 0)
header = msgpack.unpackb(destination.read_bytes(), raw=False)[0]
self.assertEqual(header, "cultcache.store.v3")
again = SingleFileMessagePackBackingStore(destination)
self.assertEqual(sorted(e.key for e in again.pull_all()), ["alpha", "beta"])
self.assertEqual(again._format, "cultcache.store.v3")
def test_convert_writes_a_new_store_of_the_same_kind(self) -> None:
records = {"k1": b"5", "k2": b"53", "k3": b'{"n": 1}'}
definition = define_document_type("tests.mig", encode=lambda value: value)