cultcache-py: define_document_type defaults to MessagePack payloads; a definition's PayloadCodec owns its format; convert writes a new store (py-msgpack-default)
Some checks failed
CultMesh portability and performance / ubuntu-latest reactive semantics and leases (pull_request) Has been cancelled

Ports hands/py-msgpack-default (20 commits) onto main as one squash. Main's one-value
payload read (bytes after the first value refused by count, depth first, plain
ValueError), update-with-retry, stamp hardening and store lock/commit path are kept;
the codec replaces payload_encoder/payload_decoder. JSON_PAYLOAD_CODEC reads 0.3
stores; convert(source, destination, definitions) rewrites them to a new store.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Libby 2026-11-17 02:06:35 +00:00
commit 2267c81866
12 changed files with 1577 additions and 87 deletions

View file

@ -4,6 +4,7 @@
### Changed
- Breaking: `define_document_type` writes and reads MessagePack payloads by default (`cultlib-gaps:ruling:py-msgpack-default`); it wrote JSON until 0.3. A definition's `payload_codec` (`PayloadCodec`, with `MESSAGEPACK_PAYLOAD_CODEC` and `JSON_PAYLOAD_CODEC` exported) owns its payload format, replacing the `payload_encoder` and `payload_decoder` parameters and fields, so no definition carries a second author of the format. A store written by 0.3 or earlier holds JSON: define its types with `payload_codec=JSON_PAYLOAD_CODEC` to read them, or write a new MessagePack store with `convert(source, destination, definitions)`, which never writes its source. The MessagePack codec takes the keys, integers, timestamps and nesting the C# reference's object reader takes and refuses the rest on read and on write (`CultCachePayloadFormatError`, `PayloadCodecFault`; see the README, Payload format); the single-file store's reader takes the same map keys. A 0.3 JSON payload that is an object or list is refused by count as a payload with bytes after its first MessagePack value, a `ValueError`. This is the same minor bump as the rest of this section: 0.4.0 is not yet tagged.
- `parse_stored_at` (was `_parse_stored_at`) is public: cultmesh-py judges a peer's `storedAt` by it. A peer's `document_put_raw` whose `storedAt` is missing, empty or not a stamp is refused `message_invalid` (field `document`) and nothing is minted or stored (variants:ruling:raw-put-without-stamp-refused). A subscribed server's stampless put now raises `ValueError` from `CultMeshClient.read_next_change` instead of being stored under a minted stamp.
### Added

View file

@ -31,7 +31,7 @@ discovery, sessions, local server) build on it, mirroring `cultcache-rs` /
second store, or a second generic store, raises `CultCacheError`; a write
touches exactly one store and nothing is mirrored; once any store is
attached, a write whose type has no home raises
- the JSONL store uses only the Python standard library for bootstrap consumers
- `define_document_type(...)` payloads are MessagePack by default; `JSON_PAYLOAD_CODEC` reads stores written by 0.3 and earlier
- `define_database_entry_type(...)` emits Rust/C#-style slot-indexed
MessagePack array payloads for cross-runtime `DatabaseEntry` contracts
@ -131,7 +131,7 @@ reclassified; a unique key declared a role answers no lookups and stops refusing
Backing stores persist envelopes, not caller domain objects. The cache decodes payloads through registered document definitions and rejects unknown type discriminators. This keeps Python's dynamic runtime from turning persistence into an open polymorphic sewer with a cheerful docstring.
`JsonLinesBackingStore` is the dependency-free control-plane store. It rewrites an atomic JSONL snapshot and base64-encodes payload bytes. It is designed for compact state spines, settings, ledgers, and bootstrap surfaces.
`JsonLinesBackingStore` is the control-plane store. It rewrites an atomic JSONL snapshot and base64-encodes the payload bytes its definition's codec wrote. It is designed for compact state spines, settings, ledgers, and bootstrap surfaces.
`SingleFileMessagePackBackingStore` follows the shared CultCache v1 store shape
used by the TypeScript, Rust, and C# runtimes:
@ -150,10 +150,52 @@ Large corpora should use a sharded store or a real database. A single snapshot f
A store whose elements hold an id is marked `cultcache.store.v3` by the C# runtime. This runtime reads and writes a v3 single-file store, keeps the header it read and carries the ids through a rewrite; it never drops the mark, so it has no carry-back. A cultcache-py from before element ids refuses a v3 store by name, so rolling back to it after a store has been written as v3 is one-way: restore from a copy taken before upgrading, and upgrade every reader before any writer. (The C# line is different: CultLib 1.x does not refuse; see the GameCult.Caching README.)
## Payload format
A definition's `payload_codec` owns its payload bytes. The default,
`MESSAGEPACK_PAYLOAD_CODEC`, writes MessagePack; a store holds no format marker,
so the registered definition decides how every record is read, and a record in
another format is refused, never guessed at: a JSON object or list read as MessagePack
is refused by count (`<type> payload has N bytes after its first MessagePack value`,
a `ValueError`), and a payload the codec cannot read or carry is a
`CultCachePayloadFormatError` or `CultCachePayloadDepthError`.
- Map keys are `str`, integer, float, `bool` or `msgpack.Timestamp`, as the C#
reference's object reader takes them; `None`, `bytes`, list and map keys are
refused, on read and on write. Two keys Python holds as one are refused on read
as `duplicate_map_key`, which includes `1` beside `1.0` or `True`: the reference
keeps those apart and a Python `dict` cannot. The same keys are refused on
write as `duplicate_map_key`, which includes a `Float32` key beside an equal
int, bool or float key, and two NaN keys, which are one key: the codec never
writes bytes its own reader refuses. One pair of keys is the exception to the reference, see Wire Parity.
- Arrays and maps nest at most 256 deep (`MAX_PAYLOAD_DEPTH`), on read and on
write, in the codec and in the definition alike.
- Integers beyond 64 bits (outside `-(2**63)..2**64-1`) are refused on write as
`integer_out_of_range`; MessagePack has no wider integer.
- Timestamps lie within years 1 to 9999 (`.NET` `DateTime`), and no extension type
but the timestamp exists: a `msgpack.ExtType` value or a timestamp outside that
range is refused on read and on write as `value_kind` or `timestamp_out_of_range`.
- A store written by cultcache-py 0.3 or earlier holds JSON payloads. Define each
type with `payload_codec=JSON_PAYLOAD_CODEC` to read it, or write it as a new store
with `convert(source, destination, definitions)`. Stop every writer of the source
first. `destination` must not exist; every record of the listed definitions is
rewritten under the definitions' own codec, records of other types keep their
bytes, and the new file is fsynced before the call returns. Open it with the
default definitions, then swap it in yourself. The source is never written. A
refused record raises `CultCacheError` naming the type, record key and code, and
leaves no destination; a killed run leaves the source as it was and a partial
destination, so delete the destination and run again.
- A JSON payload that is a single digit (`0` to `9`) is also a valid one-byte
MessagePack integer and reads as 48 to 57 under the default. Documents that
are objects or lists never meet this.
- A `PayloadCodec` you build owns the nesting depth of nodes the library's walk
cannot see (anything that is not a mapping, sequence or set), and may raise
`PayloadCodecFault(code, kind)` for a key or integer its format cannot carry.
## DatabaseEntry Slot Contract
For cross-runtime cache entries, use `define_database_entry_type(...)` instead
of the generic JSON formatter:
For cross-runtime cache entries, use `define_database_entry_type(...)`, which
fixes the MessagePack slot-array payload:
```python
from dataclasses import dataclass
@ -258,6 +300,12 @@ python -m cultcache_py.interop write --file cache.cc --runtime-id python
python -m cultcache_py.interop read --file cache.cc
```
One deliberate exception to leaf acceptance following the reference: Python reads `float32` and `float64` map keys as
one float, so a map holding a `float32` NaN key beside a `float64` NaN key is refused on read and on write as
`duplicate_map_key`. C# reads that map as two keys, from the bytes `82 ca ffc00000 01 cb fff8000000000000 02` (fixint
values) and from the bytes it writes itself, `82 ca ffc00000 d2 00000001 cb fff8000000000000 d2 00000002` (`float.NaN`
and `double.NaN` keys with values 1 and 2, written as int32); every pair of NaN keys of one width is refused by both runtimes.
`packages/cultcache-ts/test/cult-cache.test.ts` includes Python in the shared
CultCache v1 parity matrix with TypeScript, Rust, and C#. `cultcache_py` ships
a `py.typed` marker so downstream type checkers can inspect the package surface.

View file

@ -1,13 +1,18 @@
from .backing_store import BackingStore, CultCacheEnvelope
from .backing_store import CultCacheSchemaCatalogEntry, CultCacheSchemaCatalogMember
from . import leaves
from .cache import CultCache, CultCacheBuilder, ForeignRecord, UpdateOutcome, UpdateResult
from .cache import CultCache, CultCacheBuilder, ForeignRecord, UpdateOutcome, UpdateResult, convert
from .documents import (
MAX_PAYLOAD_DEPTH,
JSON_PAYLOAD_CODEC,
MAX_SLOT_KEY,
MESSAGEPACK_PAYLOAD_CODEC,
CultCachePayloadDepthError,
CultCachePayloadFormatError,
DatabaseEntryField,
DocumentDefinition,
PayloadCodec,
PayloadCodecFault,
UnionArm,
UnreadSlots,
database_entry_field,
@ -26,15 +31,20 @@ __all__ = [
"CultCache",
"CultCacheBuilder",
"CultCachePayloadDepthError",
"CultCachePayloadFormatError",
"CultCacheEnvelope",
"CultCacheSchemaCatalogEntry",
"CultCacheSchemaCatalogMember",
"DatabaseEntryField",
"DocumentDefinition",
"ForeignRecord",
"JSON_PAYLOAD_CODEC",
"JsonLinesBackingStore",
"MAX_PAYLOAD_DEPTH",
"MAX_SLOT_KEY",
"MESSAGEPACK_PAYLOAD_CODEC",
"PayloadCodec",
"PayloadCodecFault",
"SchemaConflictError",
"SingleFileMessagePackBackingStore",
"SlotCodec",
@ -52,4 +62,5 @@ __all__ = [
"define_document_type",
"define_record_codec",
"define_union",
"convert",
]

View file

@ -3,14 +3,24 @@
import enum
import functools
import math
import os
import threading
from decimal import Decimal
from dataclasses import dataclass, field, replace
from pathlib import Path
from typing import Any, Callable, Generic, Iterable, TypeVar
from .backing_store import BackingStore, CultCacheEnvelope
from .documents import DocumentDefinition, UnreadSlots, extract_value
from .stores import SchemaConflictError, parse_stored_at, retyped, stored_at_over, written_owns
from .documents import JSON_PAYLOAD_CODEC, DocumentDefinition, PayloadCodec, UnreadSlots, extract_value
from .stores import (
JsonLinesBackingStore,
SchemaConflictError,
SingleFileMessagePackBackingStore,
parse_stored_at,
retyped,
stored_at_over,
written_owns,
)
T = TypeVar("T")
F = TypeVar("F", bound=Callable[..., Any])
@ -1042,3 +1052,74 @@ class CultCache:
if document is not None:
return document
return self._state.documents_by_schema_name.get(envelope.type)
def convert(
source: SingleFileMessagePackBackingStore | JsonLinesBackingStore,
destination: str | os.PathLike[str],
definitions: Iterable[DocumentDefinition[Any]],
*,
source_codec: PayloadCodec = JSON_PAYLOAD_CODEC,
) -> int:
"""Write the records of `source` to a new store of the same kind at `destination`, the records of `definitions`
rewritten from `source_codec` to each definition's own codec, and return the count rewritten. Records of other types
keep their bytes. Stop every writer of the source before calling. `destination` must not exist: the call creates it
exclusively, and it is complete only once the call returns. A refusal (a record, a failed write) or an interrupt
after the create removes it and leaves the source as it was. A killed run leaves the source intact and a partial
destination: delete the destination and run again. A file another process creates at the destination between the
existence check and the create is refused and left alone. The source is only read, never written; swap the new
store in yourself once it opens. A count of 0 still writes the destination.
"""
if not isinstance(source, (SingleFileMessagePackBackingStore, JsonLinesBackingStore)):
raise TypeError(f"convert takes a library file store, not {type(source).__name__}")
if not source.path.exists():
raise CultCacheError("convert: argument source does not exist (source_missing); there is nothing to convert.")
by_type: dict[str, DocumentDefinition[Any]] = {}
for definition in definitions:
by_type[definition.type] = definition
by_type.setdefault(definition.catalog_entry().schema_name, definition)
destination = Path(destination)
destination.parent.mkdir(parents=True, exist_ok=True)
exists = CultCacheError(
"convert: argument destination already exists (destination_exists); convert writes only a new store, "
"and the source is unchanged."
)
if destination.exists():
raise exists
target = None
failure: CultCacheError | None = None
try:
try:
target = open(destination, "xb")
except FileExistsError:
pass
else:
with target:
converted: list[CultCacheEnvelope] = []
count = 0
for envelope in source.pull_all():
definition = by_type.get(envelope.type)
if definition is not None:
try:
value = replace(definition, payload_codec=source_codec).decode_payload(envelope.payload)
envelope = replace(envelope, payload=definition.encode_payload(value))
except Exception as refusal:
code = getattr(refusal, "code", type(refusal).__name__)
failure = CultCacheError(
f"Cannot convert {envelope.type} record {envelope.key}: {code}; "
"no store was written and the source is unchanged."
)
else:
count += 1
if failure is not None:
raise failure
converted.append(envelope)
target.write(type(source)._file_bytes(converted))
target.flush()
os.fsync(target.fileno())
except BaseException:
destination.unlink(missing_ok=True)
raise
if target is None:
raise exists
return count

View file

@ -1,6 +1,7 @@
from __future__ import annotations
import json
import struct
import threading
from dataclasses import MISSING, dataclass, field
from collections.abc import Mapping, Sequence, Set
@ -8,7 +9,6 @@ from itertools import chain
from typing import Any, Callable, Generic, Iterable, TypeVar
import msgpack
from msgpack.exceptions import StackError as _MsgpackStackError
from .backing_store import CultCacheSchemaCatalogEntry, CultCacheSchemaCatalogMember
from .leaves import Float32, SlotCodec
@ -28,9 +28,6 @@ or skipped, the outermost one is depth 1; depth 256 is accepted and 257 refused,
MAX_SLOT_KEY = 65535
"""The largest slot key a record may define: the largest MessagePack array16 length."""
_STACK_FAULTS: tuple[type[BaseException], ...] = (_MsgpackStackError,)
@dataclass(frozen=True)
class UnreadSlots:
"""What a payload decode found past the slots this runtime declares, at any depth: the first type read with extra slots, how
@ -89,22 +86,85 @@ class CultCachePayloadDepthError(ValueError):
)
def _refuse_past_depth_limit(tree: Any, type_name: str, direction: str) -> None:
"""The one nesting count. Walks a payload tree iteratively (every mapping, sequence and set that is not text or
bytes is a container), refusing the first container past `MAX_PAYLOAD_DEPTH`."""
stack = [(tree, 1)] if _is_container(tree) else []
while stack:
node, depth = stack.pop()
if depth > MAX_PAYLOAD_DEPTH:
raise CultCachePayloadDepthError(type_name, direction)
children = chain(node.keys(), node.values()) if isinstance(node, Mapping) else node
stack.extend((child, depth + 1) for child in children if _is_container(child))
class PayloadCodecFault(ValueError):
"""A codec raises this for a payload its format cannot carry. `code` names the rule (the library's are
`map_key_kind`, `duplicate_map_key`, `integer_out_of_range`, `depth_limit`, `value_kind` and
`timestamp_out_of_range`) and `kind` the kind of node at fault (a MessagePack
kind such as `nil` or `bin`); neither is ever a payload value. A consumer-built codec may raise it."""
def __init__(self, code: str, kind: str = "") -> None:
self.code = code
self.kind = kind
super().__init__(code)
class CultCachePayloadFormatError(ValueError):
"""A payload is not one value of its definition's codec (read), or holds a key or integer the codec cannot carry.
Names the type, the codec, the direction and a `code`; `detail` is the fault's class or the kind of the key or
extension value. Never a payload byte or value."""
def __init__(self, type_name: str, codec: str, direction: str, code: str, detail: str) -> None:
self.type_name = type_name
self.codec = codec
self.direction = direction
self.code = code
if code == "not_one_value":
message = f"{type_name} payload is not one {codec} value ({detail}); refused on read."
elif code == "map_key_kind":
message = (
f"{type_name} payload has a map key of kind {detail}; "
f"{codec} payloads take str, int, float, bool or timestamp keys (refused on {direction})."
)
elif code == "duplicate_map_key":
message = f"{type_name} payload has two map keys Python holds as one; refused on {direction}."
elif code == "integer_out_of_range":
message = f"{type_name} payload has an integer beyond 64 bits; refused on write."
elif code == "value_kind":
message = (
f"{type_name} payload holds a value of kind {detail}; "
f"{codec} payloads carry no extension type but the timestamp (refused on {direction})."
)
elif code == "timestamp_out_of_range":
message = f"{type_name} payload holds a timestamp outside years 1 to 9999; refused on {direction}."
else:
message = f"{type_name} payload is refused by the {codec} codec ({code}); refused on {direction}."
super().__init__(message)
def _is_container(node: Any) -> bool:
return isinstance(node, _CONTAINERS) and not isinstance(node, _SCALAR_SEQUENCES)
def _children(node: Any) -> Iterable[Any]:
return chain(node.keys(), node.values()) if isinstance(node, Mapping) else node
def _containers(tree: Any) -> Iterable[Any]:
"""The one depth walk over a payload tree, for every codec and every caller. Iterative, so a cycle ends at the limit:
yields every container (a mapping, sequence or set that is not text or bytes; a mapping's keys and values are both
children), the outermost at depth 1, and raises `PayloadCodecFault('depth_limit')` at the first one past
`MAX_PAYLOAD_DEPTH`."""
stack = [(tree, 1)] if _is_container(tree) else []
while stack:
node, depth = stack.pop()
if depth > MAX_PAYLOAD_DEPTH:
raise PayloadCodecFault("depth_limit")
yield node
stack.extend((child, depth + 1) for child in _children(node) if _is_container(child))
def _check_payload_tree(tree: Any, type_name: str, direction: str) -> None:
"""The definition's translation of the depth walk into its typed refusal."""
try:
for _ in _containers(tree):
pass
except PayloadCodecFault:
refusal = CultCachePayloadDepthError(type_name, direction)
else:
return
raise refusal
def _default_encode(value: Any) -> Any:
if hasattr(value, "model_dump"):
return value.model_dump()
@ -119,14 +179,6 @@ def _default_decode(value: Any) -> Any:
return value
def _json_payload_encoder(value: Any) -> bytes:
return json.dumps(value, ensure_ascii=False, sort_keys=True).encode("utf-8")
def _json_payload_decoder(payload: bytes) -> Any:
return json.loads(payload.decode("utf-8"))
def _pack_tree(tree: Any) -> bytes:
"""The bytes of the tree the leaves produced. `packb` writes everything but a `Float32`, which it refuses with a
TypeError; only then does an iterative walk pack each array and map header and each value, a `Float32` through a
@ -154,27 +206,147 @@ def _pack_tree(tree: Any) -> bytes:
return b"".join(out)
@dataclass(frozen=True)
class PayloadCodec:
"""A codec owns one payload format: how a tree becomes bytes and back, which of its decode faults mean the payload
nests past the interpreter's own stack, and which trees it can carry: it raises `PayloadCodecFault` for a key, an
integer or a nesting its format cannot hold, and nothing it writes is refused by its own `decode`. A consumer-built codec owns
the depth of nodes the depth walk cannot see (anything that is not a Mapping, Sequence or Set)."""
name: str
encode: Callable[[Any], bytes]
decode: Callable[[bytes], Any]
depth_faults: tuple[type[BaseException], ...] = ()
_INTEGER_RANGE = (-(2**63), 2**64 - 1)
_TIMESTAMP_SECONDS = (-62135596800, 253402300799)
_NAN_KEY = object()
_KEY_TYPES = (str, int, float, Float32, msgpack.Timestamp)
def _node_kind(node: Any) -> str:
"""The MessagePack kind of a node, for a refusal: a kind, never a value."""
if node is None:
return "nil"
if isinstance(node, (bytes, bytearray, memoryview)):
return "bin"
if isinstance(node, Mapping):
return "map"
if isinstance(node, msgpack.ExtType):
return "ext"
if isinstance(node, _CONTAINERS):
return "array"
return "unsupported"
def _key_image(key: Any) -> Any:
"""A map key as the reader holds it, and the one key identity of read and write: a `Float32` is written as float32
and read back as that float, and every NaN is one key (the reference refuses two of them)."""
if isinstance(key, Float32):
key = struct.unpack(">f", struct.pack(">f", key.value))[0]
return _NAN_KEY if isinstance(key, float) and key != key else key
def _checked_map(pairs: list[tuple[Any, Any]]) -> dict[Any, Any]:
"""The read hook of every MessagePack map: the keys the C# reference's object reader accepts (str, int, float,
bool, timestamp) and no duplicate."""
for key, _ in pairs:
if not isinstance(key, _KEY_TYPES):
raise PayloadCodecFault("map_key_kind", _node_kind(key))
if len(dict.fromkeys(_key_image(key) for key, _ in pairs)) != len(pairs):
raise PayloadCodecFault("duplicate_map_key")
return dict(pairs)
def _check_scalar(node: Any) -> None:
if isinstance(node, int) and not _INTEGER_RANGE[0] <= node <= _INTEGER_RANGE[1]:
raise PayloadCodecFault("integer_out_of_range")
if isinstance(node, msgpack.Timestamp) and not _TIMESTAMP_SECONDS[0] <= node.seconds <= _TIMESTAMP_SECONDS[1]:
raise PayloadCodecFault("timestamp_out_of_range")
def _check_carried(tree: Any) -> None:
"""The one check of what a MessagePack payload carries, on read and on write: the keys, integers, timestamps and
nesting the C# reference's object reader takes, no extension type but the timestamp, and no two keys that decode
as one (a `Float32` beside an equal int, bool or float, or two NaNs), so nothing this codec writes is refused by
its own decode."""
_check_scalar(tree)
for node in _containers(tree):
if isinstance(node, msgpack.ExtType):
raise PayloadCodecFault("value_kind", "ext")
for child in _children(node):
_check_scalar(child)
if isinstance(node, Mapping):
for key in node:
if not isinstance(key, _KEY_TYPES):
raise PayloadCodecFault("map_key_kind", _node_kind(key))
if len(dict.fromkeys(_key_image(key) for key in node)) != len(node):
raise PayloadCodecFault("duplicate_map_key")
def _unpack_tree(payload: bytes) -> Any:
return msgpack.unpackb(payload, raw=False, strict_map_key=False, object_pairs_hook=_checked_map)
def _unpack_checked_tree(payload: bytes) -> Any:
tree = _unpack_tree(payload)
_check_carried(tree)
return tree
def _pack_checked_tree(tree: Any) -> bytes:
_check_carried(tree)
return _pack_tree(tree)
def _dump_json(tree: Any) -> bytes:
return json.dumps(tree, ensure_ascii=False, sort_keys=True).encode("utf-8")
def _load_json(payload: bytes) -> Any:
return json.loads(payload.decode("utf-8"))
MESSAGEPACK_PAYLOAD_CODEC = PayloadCodec(
"MessagePack", _pack_checked_tree, _unpack_checked_tree, depth_faults=(msgpack.exceptions.StackError,)
)
"""The default payload format. It carries what the C# reference's object reader accepts: str, int, float, bool and
timestamp keys, no duplicate key (two NaN keys are one), integers within -(2**63)..2**64-1, timestamps within years 1 to
9999, no extension type but the timestamp, and nesting no deeper than `MAX_PAYLOAD_DEPTH`. One check refuses the rest on
read and on write, so it never writes a record it cannot read."""
JSON_PAYLOAD_CODEC = PayloadCodec("JSON", _dump_json, _load_json, depth_faults=(RecursionError,))
"""The 0.3 default, kept for stores written before MessagePack: it writes exactly the 0.3 bytes. Never a default."""
def _messagepack_payload_decoder(payload: bytes) -> Any:
return msgpack.unpackb(payload, raw=False)
def read_payload_tree(type_name: str, payload: bytes, decoder: Callable[[bytes], Any] | None = None) -> Any:
def read_payload_tree(
type_name: str,
payload: bytes,
decoder: Callable[[bytes], Any] | None = None,
depth_faults: tuple[type[BaseException], ...] = (msgpack.exceptions.StackError,),
) -> Any:
"""The one read of a payload's tree: exactly one MessagePack value, within the nesting limit. Depth is judged first,
on the value read so far, so a tail never pre-empts it (C# throws at container 257 before it reaches the tail); then a
tail is refused by count and never echoed. Raised outside the handler so no `__context__` keeps the tail."""
tail is refused by count and never echoed. Raised outside the handler so no `__context__` keeps the tail. A
definition passes its codec's `decode` and `depth_faults`; with none this reads plain MessagePack."""
after = -1
unpacked: Any = None
try:
tree = (decoder or _messagepack_payload_decoder)(payload)
except _STACK_FAULTS:
except depth_faults:
raise CultCachePayloadDepthError(type_name, "read") from None
except msgpack.ExtraData as error:
after = len(error.extra)
unpacked = error.unpacked
if after >= 0:
_refuse_past_depth_limit(unpacked, type_name, "read")
_check_payload_tree(unpacked, type_name, "read")
raise ValueError(f"{type_name} payload has {after} bytes after its first MessagePack value (refused on read).")
_refuse_past_depth_limit(tree, type_name, "read")
_check_payload_tree(tree, type_name, "read")
return tree
@ -212,6 +384,12 @@ class UnionArm:
value: Any = None
def _fault_refusal(type_name: str, codec: PayloadCodec, direction: str, fault: PayloadCodecFault) -> CultCachePayloadFormatError | CultCachePayloadDepthError:
if fault.code == "depth_limit":
return CultCachePayloadDepthError(type_name, direction)
return CultCachePayloadFormatError(type_name, codec.name, direction, fault.code, fault.kind)
@dataclass(frozen=True)
class DocumentDefinition(Generic[T]):
type: str
@ -226,8 +404,7 @@ class DocumentDefinition(Generic[T]):
roles: dict[str, str | Extractor[T]] = field(default_factory=dict)
"""Selection aliases (C# `[CultRole]`): many records may share a value. Never indexed for lookup, never refuse; the
member's catalog `index_alias` is the same either way."""
payload_encoder: Callable[[Any], bytes] = _json_payload_encoder
payload_decoder: Callable[[bytes], Any] = _json_payload_decoder
payload_codec: PayloadCodec = MESSAGEPACK_PAYLOAD_CODEC
schema_id: str | None = None
schema_name: str | None = None
schema_version: str | None = None
@ -245,8 +422,13 @@ class DocumentDefinition(Generic[T]):
if self.validate:
self.validate(value)
tree = self.encode(value)
_refuse_past_depth_limit(tree, self.type, "write")
return self.payload_encoder(tree)
_check_payload_tree(tree, self.type, "write")
codec = self.payload_codec
try:
return codec.encode(tree)
except PayloadCodecFault as fault:
refusal = _fault_refusal(self.type, codec, "write", fault)
raise refusal
def decode_payload(self, payload: bytes) -> T:
return self.decode_payload_marked(payload)[0]
@ -254,7 +436,23 @@ class DocumentDefinition(Generic[T]):
def decode_payload_marked(self, payload: bytes) -> tuple[T, UnreadSlots | None]:
"""The value and what the decode found past the declared slots, at any depth. The payload depth check runs first: a
payload past the limit is refused, never marked."""
tree = read_payload_tree(self.type, payload, self.payload_decoder)
codec = self.payload_codec
def read(raw: bytes) -> Any:
try:
return codec.decode(raw)
except (msgpack.ExtraData, *codec.depth_faults):
raise
except PayloadCodecFault as fault:
refusal = _fault_refusal(self.type, codec, "read", fault)
except ValueError as fault:
refusal = CultCachePayloadFormatError(self.type, codec.name, "read", "not_one_value", type(fault).__name__)
raise refusal
tree = read_payload_tree(self.type, payload, read, codec.depth_faults)
return self._decode_tree(tree)
def _decode_tree(self, tree: Any) -> tuple[T, UnreadSlots | None]:
outer, opened = _current_scope(), _SlotScope()
_scope.open = opened
try:
@ -325,8 +523,7 @@ def define_document_type(
name: str | Extractor[T] | None = None,
indexes: dict[str, str | Extractor[T]] | None = None,
roles: dict[str, str | Extractor[T]] | None = None,
payload_encoder: Callable[[Any], bytes] = _json_payload_encoder,
payload_decoder: Callable[[bytes], Any] = _json_payload_decoder,
payload_codec: PayloadCodec | None = None,
schema_id: str | None = None,
schema_name: str | None = None,
schema_version: str | None = None,
@ -335,6 +532,10 @@ def define_document_type(
compatible_schema_ids: Iterable[str] | None = None,
members: Iterable[CultCacheSchemaCatalogMember] | Callable[[], tuple[CultCacheSchemaCatalogMember, ...]] | None = None,
) -> DocumentDefinition[T]:
"""Define a document type. Its payload is the MessagePack of what `encode` returns; pass
`payload_codec=JSON_PAYLOAD_CODEC` to read a store written by cultcache-py 0.3 or earlier, and `convert` to write
it as a new MessagePack store."""
codec = {} if payload_codec is None else {"payload_codec": payload_codec}
return DocumentDefinition(
type=type,
encode=encode,
@ -344,8 +545,7 @@ def define_document_type(
name=name,
indexes=indexes or {},
roles=roles or {},
payload_encoder=payload_encoder,
payload_decoder=payload_decoder,
**codec,
schema_id=schema_id,
schema_name=schema_name,
schema_version=schema_version,
@ -410,8 +610,6 @@ def define_database_entry_type(
name=name,
indexes=indexes,
roles=roles,
payload_encoder=_pack_tree,
payload_decoder=_messagepack_payload_decoder,
schema_id=schema_id,
schema_name=schema_name,
schema_version=schema_version,

View file

@ -17,7 +17,7 @@ from typing import Any, Callable, Iterator, TypeVar
import msgpack
from .documents import UnreadSlots
from .documents import UnreadSlots, _unpack_tree
from .backing_store import (
CultCacheEnvelope,
CultCacheSchemaCatalogEntry,
@ -144,8 +144,8 @@ class JsonLinesBackingStore:
change(existing)
self._replace_all(list(existing.values()))
def _replace_all(self, envelopes: list[CultCacheEnvelope]) -> None:
self.path.parent.mkdir(parents=True, exist_ok=True)
@staticmethod
def _file_bytes(envelopes: list[CultCacheEnvelope]) -> bytes:
lines = []
for envelope in sorted(envelopes, key=lambda item: (item.type, item.key)):
lines.append(json.dumps({
@ -154,7 +154,11 @@ class JsonLinesBackingStore:
"payload": base64.b64encode(envelope.payload).decode("ascii"),
"stored_at": envelope.stored_at,
}, ensure_ascii=False, sort_keys=True))
commit_store_file(self.path, ("\n".join(lines) + ("\n" if lines else "")).encode("utf-8"))
return ("\n".join(lines) + ("\n" if lines else "")).encode("utf-8")
def _replace_all(self, envelopes: list[CultCacheEnvelope]) -> None:
self.path.parent.mkdir(parents=True, exist_ok=True)
commit_store_file(self.path, self._file_bytes(envelopes))
class SingleFileMessagePackBackingStore:
@ -181,7 +185,7 @@ class SingleFileMessagePackBackingStore:
if data is None:
return None
try:
decoded = msgpack.unpackb(data, raw=False)
decoded = _unpack_tree(data)
snapshot = _decode_snapshot(decoded)
if snapshot is not None:
self._format, envelopes = snapshot
@ -233,6 +237,11 @@ class SingleFileMessagePackBackingStore:
with store_write_lock(self.path, self.lock_timeout):
yield
@staticmethod
def _file_bytes(envelopes: list[CultCacheEnvelope]) -> bytes:
"""The bytes of a new store holding exactly `envelopes`, as a write would leave it."""
return msgpack.packb(_encode_snapshot(envelopes, [], STORE_FORMAT_VERSION)[0], use_bin_type=True) # type: ignore[no-any-return]
def _replace_all(self, supplied: list[CultCacheEnvelope], arrived: list[CultCacheEnvelope]) -> dict[str, str]:
snapshot, stamps = _encode_snapshot(supplied, arrived, self._format)
commit_store_file(self.path, msgpack.packb(snapshot, use_bin_type=True))
@ -669,6 +678,9 @@ def _decode_snapshot(decoded: Any) -> tuple[str, list[CultCacheEnvelope]] | None
def _decode_legacy_envelopes(decoded: Any) -> list[CultCacheEnvelope]:
"""A legacy envelope holds its payload as an inline MessagePack subtree or as bytes. Extracting the subtree's bytes
decides no format: the inline value is MessagePack by construction, and the registered codec still reads it or
refuses it."""
if not isinstance(decoded, list):
raise ValueError("CultCache MessagePack store is not a recognized snapshot")
envelopes: list[CultCacheEnvelope] = []

View file

@ -1,9 +1,12 @@
from __future__ import annotations
import base64
import json
import os
import tempfile
import threading
import unittest
import msgpack
from dataclasses import asdict, dataclass, replace
from pathlib import Path
from cultcache_py.backing_store import CultCacheEnvelope
@ -120,11 +123,6 @@ class CultCacheTests(unittest.TestCase):
self.assertIsNone(cache.get_key_by_index(document, "category", "Rare"))
def test_database_entry_formatter_uses_slot_indexed_messagepack_array(self) -> None:
try:
import msgpack # type: ignore
except ModuleNotFoundError:
self.skipTest("msgpack optional dependency is not installed")
@dataclass
class Settings:
theme: str
@ -159,11 +157,6 @@ class CultCacheTests(unittest.TestCase):
self.assertEqual([member.slot for member in first.members], [0, 2])
def test_messagepack_store_writes_v1_snapshot(self) -> None:
try:
import msgpack # type: ignore
except ModuleNotFoundError:
self.skipTest("msgpack optional dependency is not installed")
with tempfile.TemporaryDirectory() as tmp:
document = define_database_entry_type(
"settings",
@ -1570,6 +1563,39 @@ class CultCacheTests(unittest.TestCase):
SingleFileMessagePackBackingStore(path).pull_all()
self.assertNotIsInstance(caught.exception, FileNotFoundError)
def test_json_lines_pull_all_reads_hand_written_lines_and_names_a_malformed_one(self) -> None:
def line(**fields: object) -> str:
return json.dumps(fields, ensure_ascii=False)
first = base64.b64encode(b"one").decode("ascii")
second = base64.b64encode(b"two").decode("ascii")
with tempfile.TemporaryDirectory() as tmp:
path = Path(tmp) / "hand.jsonl"
lines = [
line(key="é-ключ", type="tests.t", payload=first, storedAt="2026-01-01T00:00:00Z", schemaId="schema.one"),
"",
line(key="k2", type="tests.t", payload=second, stored_at="2026-02-02T00:00:00Z", schema_id="schema.two"),
]
path.write_bytes(("\n".join(lines) + "\n").encode("utf-8"))
pulled = JsonLinesBackingStore(path).pull_all()
self.assertEqual(
[(e.key, e.type, e.payload, e.stored_at, e.schema_id) for e in pulled],
[
("é-ключ", "tests.t", b"one", "2026-01-01T00:00:00Z", "schema.one"),
("k2", "tests.t", b"two", "2026-02-02T00:00:00Z", "schema.two"),
],
)
written = Path(tmp) / "written.jsonl"
JsonLinesBackingStore(written).push(CultCacheEnvelope.create(key="k", type="tests.t", payload=b"x"))
raw = written.read_bytes()
self.assertTrue(raw.endswith(b"\n"))
self.assertNotIn(b"\r", raw)
broken = Path(tmp) / "broken.jsonl"
broken.write_bytes(("\n".join([lines[0], "", line(key="k3", type="tests.t")]) + "\n").encode("utf-8"))
with self.assertRaises(ValueError) as caught:
JsonLinesBackingStore(broken).pull_all()
self.assertEqual(str(caught.exception), f"Malformed CultCache JSONL envelope at {broken}:3: missing 'payload'")
# The element-id marker: tests/vectors/document-variants-c2a/v3-base.msgpack is v1-base with its header replaced.
_C2A_VECTORS = Path(__file__).resolve().parents[3] / "tests" / "vectors" / "document-variants-c2a"

File diff suppressed because it is too large Load diff

View file

@ -2,6 +2,7 @@
import collections
import dataclasses
import json
import shutil
import tempfile
import traceback
@ -15,8 +16,12 @@ from cultcache_py import (
MAX_PAYLOAD_DEPTH,
MAX_SLOT_KEY,
CultCache,
JSON_PAYLOAD_CODEC,
MESSAGEPACK_PAYLOAD_CODEC,
PayloadCodecFault,
CultCachePayloadDepthError,
DatabaseEntryField,
PayloadCodec,
SlotCodec,
SingleFileMessagePackBackingStore,
UnionArm,
@ -595,17 +600,18 @@ class BoundsVectorTests(unittest.TestCase):
with self.assertRaises(CultCachePayloadDepthError) as caught:
tree.encode_payload(_nest(5000))
self.assertIn("refused on write", str(caught.exception))
document = define_document_type("tests.json_tree")
with self.assertRaises(CultCachePayloadDepthError) as read:
document.decode_payload(b"[" * 300 + b"]" * 300)
self.assertIn("refused on read", str(read.exception))
for codec, deep in ((None, b"\x91" * 300 + b"\xc0"), (JSON_PAYLOAD_CODEC, b"[" * 300 + b"]" * 300)):
document = define_document_type("tests.json_tree") if codec is None else define_document_type("tests.json_tree", payload_codec=codec)
with self.assertRaises(CultCachePayloadDepthError) as read:
document.decode_payload(deep)
self.assertIn("refused on read", str(read.exception))
def test_a_codec_that_exhausts_the_interpreter_stack_keeps_its_own_recursion_error_on_each_side(self) -> None:
def boom(value: object) -> object:
return boom(value)
document = define_document_type("tests.boom", encode=boom, decode=boom)
for run in (lambda: document.encode_payload("x"), lambda: document.decode_payload(b'"x"')):
for run in (lambda: document.encode_payload("x"), lambda: document.decode_payload(b"\xa1x")):
try:
run()
except RecursionError as caught:
@ -620,11 +626,11 @@ class BoundsVectorTests(unittest.TestCase):
"tests.union_surface payload nests 257 containers deep; the limit is 256 (refused on read).",
)
def test_a_json_payload_past_the_interpreters_own_stack_keeps_the_decoders_recursion_error(self) -> None:
document = define_document_type("tests.json_tree")
with self.assertRaises(RecursionError) as caught:
def test_json_text_past_the_interpreters_own_stack_is_the_depth_refusal(self) -> None:
document = define_document_type("tests.json_tree", payload_codec=JSON_PAYLOAD_CODEC)
with self.assertRaises(CultCachePayloadDepthError) as caught:
document.decode_payload(b"[" * 100000)
self.assertNotIsInstance(caught.exception, CultCachePayloadDepthError)
self.assertIn("refused on read", str(caught.exception))
def test_a_refusal_reports_the_depth_where_the_walk_stopped_whatever_the_payloads_real_depth(self) -> None:
document = define_document_type("tests.json_tree", encode=lambda value: value)
@ -633,7 +639,7 @@ class BoundsVectorTests(unittest.TestCase):
document.encode_payload(_nest(depth))
self.assertEqual((write.exception.depth, write.exception.limit), (257, 256), depth)
with self.assertRaises(CultCachePayloadDepthError) as read:
document.decode_payload(b"[" * depth + b"]" * depth)
document.decode_payload(b"\x91" * depth + b"\xc0")
self.assertEqual((read.exception.depth, read.exception.limit), (257, 256), depth)
def test_the_validator_sees_the_value_on_write_and_the_decoded_value_on_read(self) -> None:
@ -650,6 +656,9 @@ class BoundsVectorTests(unittest.TestCase):
document.encode_payload(loop)
_STUB_CODEC = PayloadCodec("test", lambda tree: b"ok", lambda payload: payload)
def _nest(depth: int, mixed: bool = False) -> object:
value: object = "leaf"
for level in range(depth):
@ -738,18 +747,24 @@ class PayloadDepthRuleTests(unittest.TestCase):
def test_the_limit_is_256_and_depth_256_is_accepted_on_read_and_write_while_257_is_refused(self) -> None:
self.assertEqual(MAX_PAYLOAD_DEPTH, 256)
document = define_document_type("tests.json_tree", encode=lambda value: value)
for mixed in (False, True):
at_limit = _nest(256, mixed)
self.assertEqual(document.decode_payload(document.encode_payload(at_limit)), at_limit)
past = _nest(257, mixed)
with self.assertRaises(CultCachePayloadDepthError):
document.encode_payload(past)
with self.assertRaises(CultCachePayloadDepthError):
document.decode_payload(document.payload_encoder(past))
for codec in (None, JSON_PAYLOAD_CODEC):
document = define_document_type("tests.json_tree", encode=lambda value: value, **({} if codec is None else {"payload_codec": codec}))
for mixed in (False, True):
at_limit = _nest(256, mixed)
self.assertEqual(document.decode_payload(document.encode_payload(at_limit)), at_limit)
past = _nest(257, mixed)
with self.assertRaises(CultCachePayloadDepthError):
document.encode_payload(past)
raw = json.dumps(past).encode() if codec is JSON_PAYLOAD_CODEC else msgpack.packb(past, use_bin_type=True)
with self.assertRaises(CultCachePayloadDepthError):
document.decode_payload(raw)
if codec is None:
with self.assertRaises(PayloadCodecFault) as refused:
MESSAGEPACK_PAYLOAD_CODEC.encode(past)
self.assertEqual(refused.exception.code, "depth_limit")
def test_a_tuple_and_a_container_dict_key_count_as_containers(self) -> None:
document = define_document_type("tests.tuple_tree", encode=lambda value: value, payload_encoder=lambda tree: b"ok")
document = define_document_type("tests.tuple_tree", encode=lambda value: value, payload_codec=_STUB_CODEC)
self.assertEqual(document.encode_payload(_as_tuple(_nest(256))), b"ok")
with self.assertRaises(CultCachePayloadDepthError):
document.encode_payload(_as_tuple(_nest(257)))
@ -765,19 +780,21 @@ class PayloadDepthRuleTests(unittest.TestCase):
"userlist": lambda inner: collections.UserList([inner]),
}
for name, wrap in wrappers.items():
document = define_document_type(f"tests.{name}", encode=lambda value: value, payload_encoder=lambda tree: b"ok")
document = define_document_type(f"tests.{name}", encode=lambda value: value, payload_codec=_STUB_CODEC)
at_limit: object = "leaf"
for _ in range(256):
at_limit = wrap(at_limit)
self.assertEqual(document.encode_payload(at_limit), b"ok", name)
with self.assertRaises(CultCachePayloadDepthError, msg=name):
document.encode_payload(wrap(at_limit))
decoding = define_document_type(f"tests.{name}.read", payload_decoder=lambda payload, w=wrap: w(at_limit))
decoding = define_document_type(
f"tests.{name}.read", payload_codec=PayloadCodec("test", lambda tree: b"ok", lambda payload, w=wrap: w(at_limit))
)
with self.assertRaises(CultCachePayloadDepthError, msg=name):
decoding.decode_payload(b"x")
def test_text_and_bytes_are_leaves_however_long(self) -> None:
document = define_document_type("tests.leaves", encode=lambda value: value, payload_encoder=lambda tree: b"ok")
document = define_document_type("tests.leaves", encode=lambda value: value, payload_codec=_STUB_CODEC)
self.assertEqual(document.encode_payload([["x" * 100000, b"y" * 100000, bytearray(5), memoryview(b"z")]]), b"ok")
def test_a_scalar_payload_has_no_depth(self) -> None:

View file

@ -963,8 +963,6 @@ def binding(
document_type,
encode=encode_slots,
decode=decode_slots,
payload_encoder=lambda value: msgpack.packb(value, use_bin_type=True),
payload_decoder=lambda payload: msgpack.unpackb(payload, raw=False),
schema_id=schema_id,
schema_name=document_type,
schema_version=schema_version,

View file

@ -4,7 +4,7 @@ from dataclasses import dataclass
from typing import Any, Iterable
from cultcache_py import CultCache, CultCacheEnvelope
from cultcache_py.documents import DocumentDefinition
from cultcache_py.documents import MESSAGEPACK_PAYLOAD_CODEC, DocumentDefinition
from cultcache_py.stores import parse_stored_at
from .shard_log import CultNetShardLogResponse
@ -190,9 +190,12 @@ def resolve_document_for_raw_record(
def _infer_schema_version_from_payload(payload: bytes) -> str | None:
"""Routing only, as the C# `CultNetDocumentRegistry.TryResolveDescriptorByPayloadSchema` does: reads a schema stamp
off a raw record whose schema id is unknown, to pick a definition. It decides no format; the definition it picks
has a codec that decodes the payload or refuses it. The codec's reader reads the stamp, so the router decides no key."""
import msgpack # type: ignore
try:
decoded = msgpack.unpackb(payload, raw=False)
decoded = MESSAGEPACK_PAYLOAD_CODEC.decode(payload)
except msgpack.ExtraData as extra:
# The hint is read from the first value; the tail is not this function's to judge. The routed definition refuses
# it by count when the record is applied, as it does for a record routed by a registered schema id. Treating a

View file

@ -25,6 +25,7 @@ from cultnet_py.compare_csharp import (
)
from cultnet_py.interop_peer import (
append_shard_log_put,
binding,
build_state,
handle_server_message,
raw_snapshot_response,
@ -1976,8 +1977,7 @@ class CultNetTests(unittest.TestCase):
from cultcache_py import define_document_type
from cultnet_py import resolve_document_for_raw_record
document = define_document_type("tests.tail_note", schema_id="tests.tail_note", schema_version="tests.tail_note.v1",
payload_encoder=msgpack.packb, payload_decoder=lambda payload: msgpack.unpackb(payload, raw=False))
document = define_document_type("tests.tail_note", schema_id="tests.tail_note", schema_version="tests.tail_note.v1")
known = document.catalog_entry().schema_id
head = msgpack.packb(["tests.tail_note.v1", "x"], use_bin_type=True)
@ -2119,6 +2119,42 @@ class CultNetTests(unittest.TestCase):
self.assertEqual([change.change_kind for change in typed_applied_log], ["updated", "removed"])
self.assertIsNone(cache.get(document, "item:1"))
def test_the_stamp_router_judges_no_map_key_and_the_codec_alone_does(self) -> None:
import msgpack # type: ignore
from cultnet_py import resolve_document_for_raw_record
document = define_database_entry_type(
"replica.stamped",
[("schema_version", 0), ("value", 1)],
schema_id="replica.stamped.current",
schema_name="replica.stamped",
schema_version="replica.stamped.v1",
)
by_schema_id = {"replica.stamped.current": document, "replica.stamped": document}
def routed(tree: object) -> object:
record = {"payload": msgpack.packb(tree, use_bin_type=True)}
return resolve_document_for_raw_record(by_schema_id, "sha256:unknown", record)
for name, tree in (
("array", ["replica.stamped.v1", {1: "a"}]),
("map", {"schemaVersion": "replica.stamped.v1", "v": {1: "a"}}),
("nested", ["replica.stamped.v1", {"a": {2: [{3: 4}]}}]),
("snake", {"schema_version": "replica.stamped.v1", "v": {1: "a"}}),
):
with self.subTest(form=name):
self.assertIs(routed(tree), document)
for name, tree in (
("array", ["replica.stamped.v1", {None: "a"}]),
("map", {"schemaVersion": "replica.stamped.v1", "v": {None: "a"}}),
("not-a-string", {"schemaVersion": 5, "v": {1: "a"}}),
("unknown-name", ["replica.other.v1", {1: "a"}]),
):
with self.subTest(form=name):
with self.assertRaises(KeyError) as caught:
routed(tree)
self.assertEqual(caught.exception.args, ("sha256:unknown",))
def test_cultnet_replication_helpers_recover_schema_stamped_raw_records(self) -> None:
document = define_database_entry_type(
"replica.runtime-policy",
@ -2678,6 +2714,12 @@ class CultNetTests(unittest.TestCase):
self.assertEqual(response["shardLogSequence"], 1)
self.assertEqual([record["recordKey"] for record in response["documents"]], ["note:logged"])
def test_python_interop_peer_binding_writes_plain_messagepack_slots(self) -> None:
# The bytes the binding wrote before it stopped passing its own codec lambdas: packb of the slot array.
note = binding("tests.note", "tests.note.v1", "tests.note.v1", "{}", lambda value: ["a", 1], lambda slots: slots)
self.assertEqual(note.document.encode_payload({}), bytes.fromhex("92a16101"))
self.assertEqual(note.document.decode_payload(bytes.fromhex("92a16101")), ["a", 1])
def test_python_interop_peer_refuses_v1_selection_messages(self) -> None:
# R-2, owed by Cut 1 (docs/cultnet-selection-cut.md, section 9): a v0-only peer refuses any
# non-v0 message loudly - cultnet.error.v0 { code: "unsupported_schema_version" } - rather than