cultcache-py: define_document_type defaults to MessagePack payloads; a definition's PayloadCodec owns its format; convert writes a new store (py-msgpack-default)
Some checks failed
CultMesh portability and performance / ubuntu-latest reactive semantics and leases (pull_request) Has been cancelled
Some checks failed
CultMesh portability and performance / ubuntu-latest reactive semantics and leases (pull_request) Has been cancelled
Ports hands/py-msgpack-default (20 commits) onto main as one squash. Main's one-value payload read (bytes after the first value refused by count, depth first, plain ValueError), update-with-retry, stamp hardening and store lock/commit path are kept; the codec replaces payload_encoder/payload_decoder. JSON_PAYLOAD_CODEC reads 0.3 stores; convert(source, destination, definitions) rewrites them to a new store. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
75188fdefa
commit
2267c81866
12 changed files with 1577 additions and 87 deletions
|
|
@ -4,6 +4,7 @@
|
|||
|
||||
### Changed
|
||||
|
||||
- Breaking: `define_document_type` writes and reads MessagePack payloads by default (`cultlib-gaps:ruling:py-msgpack-default`); it wrote JSON until 0.3. A definition's `payload_codec` (`PayloadCodec`, with `MESSAGEPACK_PAYLOAD_CODEC` and `JSON_PAYLOAD_CODEC` exported) owns its payload format, replacing the `payload_encoder` and `payload_decoder` parameters and fields, so no definition carries a second author of the format. A store written by 0.3 or earlier holds JSON: define its types with `payload_codec=JSON_PAYLOAD_CODEC` to read them, or write a new MessagePack store with `convert(source, destination, definitions)`, which never writes its source. The MessagePack codec takes the keys, integers, timestamps and nesting the C# reference's object reader takes and refuses the rest on read and on write (`CultCachePayloadFormatError`, `PayloadCodecFault`; see the README, Payload format); the single-file store's reader takes the same map keys. A 0.3 JSON payload that is an object or list is refused by count as a payload with bytes after its first MessagePack value, a `ValueError`. This is the same minor bump as the rest of this section: 0.4.0 is not yet tagged.
|
||||
- `parse_stored_at` (was `_parse_stored_at`) is public: cultmesh-py judges a peer's `storedAt` by it. A peer's `document_put_raw` whose `storedAt` is missing, empty or not a stamp is refused `message_invalid` (field `document`) and nothing is minted or stored (variants:ruling:raw-put-without-stamp-refused). A subscribed server's stampless put now raises `ValueError` from `CultMeshClient.read_next_change` instead of being stored under a minted stamp.
|
||||
|
||||
### Added
|
||||
|
|
|
|||
|
|
@ -31,7 +31,7 @@ discovery, sessions, local server) build on it, mirroring `cultcache-rs` /
|
|||
second store, or a second generic store, raises `CultCacheError`; a write
|
||||
touches exactly one store and nothing is mirrored; once any store is
|
||||
attached, a write whose type has no home raises
|
||||
- the JSONL store uses only the Python standard library for bootstrap consumers
|
||||
- `define_document_type(...)` payloads are MessagePack by default; `JSON_PAYLOAD_CODEC` reads stores written by 0.3 and earlier
|
||||
- `define_database_entry_type(...)` emits Rust/C#-style slot-indexed
|
||||
MessagePack array payloads for cross-runtime `DatabaseEntry` contracts
|
||||
|
||||
|
|
@ -131,7 +131,7 @@ reclassified; a unique key declared a role answers no lookups and stops refusing
|
|||
|
||||
Backing stores persist envelopes, not caller domain objects. The cache decodes payloads through registered document definitions and rejects unknown type discriminators. This keeps Python's dynamic runtime from turning persistence into an open polymorphic sewer with a cheerful docstring.
|
||||
|
||||
`JsonLinesBackingStore` is the dependency-free control-plane store. It rewrites an atomic JSONL snapshot and base64-encodes payload bytes. It is designed for compact state spines, settings, ledgers, and bootstrap surfaces.
|
||||
`JsonLinesBackingStore` is the control-plane store. It rewrites an atomic JSONL snapshot and base64-encodes the payload bytes its definition's codec wrote. It is designed for compact state spines, settings, ledgers, and bootstrap surfaces.
|
||||
|
||||
`SingleFileMessagePackBackingStore` follows the shared CultCache v1 store shape
|
||||
used by the TypeScript, Rust, and C# runtimes:
|
||||
|
|
@ -150,10 +150,52 @@ Large corpora should use a sharded store or a real database. A single snapshot f
|
|||
|
||||
A store whose elements hold an id is marked `cultcache.store.v3` by the C# runtime. This runtime reads and writes a v3 single-file store, keeps the header it read and carries the ids through a rewrite; it never drops the mark, so it has no carry-back. A cultcache-py from before element ids refuses a v3 store by name, so rolling back to it after a store has been written as v3 is one-way: restore from a copy taken before upgrading, and upgrade every reader before any writer. (The C# line is different: CultLib 1.x does not refuse; see the GameCult.Caching README.)
|
||||
|
||||
## Payload format
|
||||
|
||||
A definition's `payload_codec` owns its payload bytes. The default,
|
||||
`MESSAGEPACK_PAYLOAD_CODEC`, writes MessagePack; a store holds no format marker,
|
||||
so the registered definition decides how every record is read, and a record in
|
||||
another format is refused, never guessed at: a JSON object or list read as MessagePack
|
||||
is refused by count (`<type> payload has N bytes after its first MessagePack value`,
|
||||
a `ValueError`), and a payload the codec cannot read or carry is a
|
||||
`CultCachePayloadFormatError` or `CultCachePayloadDepthError`.
|
||||
|
||||
- Map keys are `str`, integer, float, `bool` or `msgpack.Timestamp`, as the C#
|
||||
reference's object reader takes them; `None`, `bytes`, list and map keys are
|
||||
refused, on read and on write. Two keys Python holds as one are refused on read
|
||||
as `duplicate_map_key`, which includes `1` beside `1.0` or `True`: the reference
|
||||
keeps those apart and a Python `dict` cannot. The same keys are refused on
|
||||
write as `duplicate_map_key`, which includes a `Float32` key beside an equal
|
||||
int, bool or float key, and two NaN keys, which are one key: the codec never
|
||||
writes bytes its own reader refuses. One pair of keys is the exception to the reference, see Wire Parity.
|
||||
- Arrays and maps nest at most 256 deep (`MAX_PAYLOAD_DEPTH`), on read and on
|
||||
write, in the codec and in the definition alike.
|
||||
- Integers beyond 64 bits (outside `-(2**63)..2**64-1`) are refused on write as
|
||||
`integer_out_of_range`; MessagePack has no wider integer.
|
||||
- Timestamps lie within years 1 to 9999 (`.NET` `DateTime`), and no extension type
|
||||
but the timestamp exists: a `msgpack.ExtType` value or a timestamp outside that
|
||||
range is refused on read and on write as `value_kind` or `timestamp_out_of_range`.
|
||||
- A store written by cultcache-py 0.3 or earlier holds JSON payloads. Define each
|
||||
type with `payload_codec=JSON_PAYLOAD_CODEC` to read it, or write it as a new store
|
||||
with `convert(source, destination, definitions)`. Stop every writer of the source
|
||||
first. `destination` must not exist; every record of the listed definitions is
|
||||
rewritten under the definitions' own codec, records of other types keep their
|
||||
bytes, and the new file is fsynced before the call returns. Open it with the
|
||||
default definitions, then swap it in yourself. The source is never written. A
|
||||
refused record raises `CultCacheError` naming the type, record key and code, and
|
||||
leaves no destination; a killed run leaves the source as it was and a partial
|
||||
destination, so delete the destination and run again.
|
||||
- A JSON payload that is a single digit (`0` to `9`) is also a valid one-byte
|
||||
MessagePack integer and reads as 48 to 57 under the default. Documents that
|
||||
are objects or lists never meet this.
|
||||
- A `PayloadCodec` you build owns the nesting depth of nodes the library's walk
|
||||
cannot see (anything that is not a mapping, sequence or set), and may raise
|
||||
`PayloadCodecFault(code, kind)` for a key or integer its format cannot carry.
|
||||
|
||||
## DatabaseEntry Slot Contract
|
||||
|
||||
For cross-runtime cache entries, use `define_database_entry_type(...)` instead
|
||||
of the generic JSON formatter:
|
||||
For cross-runtime cache entries, use `define_database_entry_type(...)`, which
|
||||
fixes the MessagePack slot-array payload:
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass
|
||||
|
|
@ -258,6 +300,12 @@ python -m cultcache_py.interop write --file cache.cc --runtime-id python
|
|||
python -m cultcache_py.interop read --file cache.cc
|
||||
```
|
||||
|
||||
One deliberate exception to leaf acceptance following the reference: Python reads `float32` and `float64` map keys as
|
||||
one float, so a map holding a `float32` NaN key beside a `float64` NaN key is refused on read and on write as
|
||||
`duplicate_map_key`. C# reads that map as two keys, from the bytes `82 ca ffc00000 01 cb fff8000000000000 02` (fixint
|
||||
values) and from the bytes it writes itself, `82 ca ffc00000 d2 00000001 cb fff8000000000000 d2 00000002` (`float.NaN`
|
||||
and `double.NaN` keys with values 1 and 2, written as int32); every pair of NaN keys of one width is refused by both runtimes.
|
||||
|
||||
`packages/cultcache-ts/test/cult-cache.test.ts` includes Python in the shared
|
||||
CultCache v1 parity matrix with TypeScript, Rust, and C#. `cultcache_py` ships
|
||||
a `py.typed` marker so downstream type checkers can inspect the package surface.
|
||||
|
|
|
|||
|
|
@ -1,13 +1,18 @@
|
|||
from .backing_store import BackingStore, CultCacheEnvelope
|
||||
from .backing_store import CultCacheSchemaCatalogEntry, CultCacheSchemaCatalogMember
|
||||
from . import leaves
|
||||
from .cache import CultCache, CultCacheBuilder, ForeignRecord, UpdateOutcome, UpdateResult
|
||||
from .cache import CultCache, CultCacheBuilder, ForeignRecord, UpdateOutcome, UpdateResult, convert
|
||||
from .documents import (
|
||||
MAX_PAYLOAD_DEPTH,
|
||||
JSON_PAYLOAD_CODEC,
|
||||
MAX_SLOT_KEY,
|
||||
MESSAGEPACK_PAYLOAD_CODEC,
|
||||
CultCachePayloadDepthError,
|
||||
CultCachePayloadFormatError,
|
||||
DatabaseEntryField,
|
||||
DocumentDefinition,
|
||||
PayloadCodec,
|
||||
PayloadCodecFault,
|
||||
UnionArm,
|
||||
UnreadSlots,
|
||||
database_entry_field,
|
||||
|
|
@ -26,15 +31,20 @@ __all__ = [
|
|||
"CultCache",
|
||||
"CultCacheBuilder",
|
||||
"CultCachePayloadDepthError",
|
||||
"CultCachePayloadFormatError",
|
||||
"CultCacheEnvelope",
|
||||
"CultCacheSchemaCatalogEntry",
|
||||
"CultCacheSchemaCatalogMember",
|
||||
"DatabaseEntryField",
|
||||
"DocumentDefinition",
|
||||
"ForeignRecord",
|
||||
"JSON_PAYLOAD_CODEC",
|
||||
"JsonLinesBackingStore",
|
||||
"MAX_PAYLOAD_DEPTH",
|
||||
"MAX_SLOT_KEY",
|
||||
"MESSAGEPACK_PAYLOAD_CODEC",
|
||||
"PayloadCodec",
|
||||
"PayloadCodecFault",
|
||||
"SchemaConflictError",
|
||||
"SingleFileMessagePackBackingStore",
|
||||
"SlotCodec",
|
||||
|
|
@ -52,4 +62,5 @@ __all__ = [
|
|||
"define_document_type",
|
||||
"define_record_codec",
|
||||
"define_union",
|
||||
"convert",
|
||||
]
|
||||
|
|
|
|||
|
|
@ -3,14 +3,24 @@
|
|||
import enum
|
||||
import functools
|
||||
import math
|
||||
import os
|
||||
import threading
|
||||
from decimal import Decimal
|
||||
from dataclasses import dataclass, field, replace
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable, Generic, Iterable, TypeVar
|
||||
|
||||
from .backing_store import BackingStore, CultCacheEnvelope
|
||||
from .documents import DocumentDefinition, UnreadSlots, extract_value
|
||||
from .stores import SchemaConflictError, parse_stored_at, retyped, stored_at_over, written_owns
|
||||
from .documents import JSON_PAYLOAD_CODEC, DocumentDefinition, PayloadCodec, UnreadSlots, extract_value
|
||||
from .stores import (
|
||||
JsonLinesBackingStore,
|
||||
SchemaConflictError,
|
||||
SingleFileMessagePackBackingStore,
|
||||
parse_stored_at,
|
||||
retyped,
|
||||
stored_at_over,
|
||||
written_owns,
|
||||
)
|
||||
|
||||
T = TypeVar("T")
|
||||
F = TypeVar("F", bound=Callable[..., Any])
|
||||
|
|
@ -1042,3 +1052,74 @@ class CultCache:
|
|||
if document is not None:
|
||||
return document
|
||||
return self._state.documents_by_schema_name.get(envelope.type)
|
||||
|
||||
|
||||
def convert(
|
||||
source: SingleFileMessagePackBackingStore | JsonLinesBackingStore,
|
||||
destination: str | os.PathLike[str],
|
||||
definitions: Iterable[DocumentDefinition[Any]],
|
||||
*,
|
||||
source_codec: PayloadCodec = JSON_PAYLOAD_CODEC,
|
||||
) -> int:
|
||||
"""Write the records of `source` to a new store of the same kind at `destination`, the records of `definitions`
|
||||
rewritten from `source_codec` to each definition's own codec, and return the count rewritten. Records of other types
|
||||
keep their bytes. Stop every writer of the source before calling. `destination` must not exist: the call creates it
|
||||
exclusively, and it is complete only once the call returns. A refusal (a record, a failed write) or an interrupt
|
||||
after the create removes it and leaves the source as it was. A killed run leaves the source intact and a partial
|
||||
destination: delete the destination and run again. A file another process creates at the destination between the
|
||||
existence check and the create is refused and left alone. The source is only read, never written; swap the new
|
||||
store in yourself once it opens. A count of 0 still writes the destination.
|
||||
"""
|
||||
if not isinstance(source, (SingleFileMessagePackBackingStore, JsonLinesBackingStore)):
|
||||
raise TypeError(f"convert takes a library file store, not {type(source).__name__}")
|
||||
if not source.path.exists():
|
||||
raise CultCacheError("convert: argument source does not exist (source_missing); there is nothing to convert.")
|
||||
by_type: dict[str, DocumentDefinition[Any]] = {}
|
||||
for definition in definitions:
|
||||
by_type[definition.type] = definition
|
||||
by_type.setdefault(definition.catalog_entry().schema_name, definition)
|
||||
destination = Path(destination)
|
||||
destination.parent.mkdir(parents=True, exist_ok=True)
|
||||
exists = CultCacheError(
|
||||
"convert: argument destination already exists (destination_exists); convert writes only a new store, "
|
||||
"and the source is unchanged."
|
||||
)
|
||||
if destination.exists():
|
||||
raise exists
|
||||
target = None
|
||||
failure: CultCacheError | None = None
|
||||
try:
|
||||
try:
|
||||
target = open(destination, "xb")
|
||||
except FileExistsError:
|
||||
pass
|
||||
else:
|
||||
with target:
|
||||
converted: list[CultCacheEnvelope] = []
|
||||
count = 0
|
||||
for envelope in source.pull_all():
|
||||
definition = by_type.get(envelope.type)
|
||||
if definition is not None:
|
||||
try:
|
||||
value = replace(definition, payload_codec=source_codec).decode_payload(envelope.payload)
|
||||
envelope = replace(envelope, payload=definition.encode_payload(value))
|
||||
except Exception as refusal:
|
||||
code = getattr(refusal, "code", type(refusal).__name__)
|
||||
failure = CultCacheError(
|
||||
f"Cannot convert {envelope.type} record {envelope.key}: {code}; "
|
||||
"no store was written and the source is unchanged."
|
||||
)
|
||||
else:
|
||||
count += 1
|
||||
if failure is not None:
|
||||
raise failure
|
||||
converted.append(envelope)
|
||||
target.write(type(source)._file_bytes(converted))
|
||||
target.flush()
|
||||
os.fsync(target.fileno())
|
||||
except BaseException:
|
||||
destination.unlink(missing_ok=True)
|
||||
raise
|
||||
if target is None:
|
||||
raise exists
|
||||
return count
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import struct
|
||||
import threading
|
||||
from dataclasses import MISSING, dataclass, field
|
||||
from collections.abc import Mapping, Sequence, Set
|
||||
|
|
@ -8,7 +9,6 @@ from itertools import chain
|
|||
from typing import Any, Callable, Generic, Iterable, TypeVar
|
||||
|
||||
import msgpack
|
||||
from msgpack.exceptions import StackError as _MsgpackStackError
|
||||
|
||||
from .backing_store import CultCacheSchemaCatalogEntry, CultCacheSchemaCatalogMember
|
||||
from .leaves import Float32, SlotCodec
|
||||
|
|
@ -28,9 +28,6 @@ or skipped, the outermost one is depth 1; depth 256 is accepted and 257 refused,
|
|||
MAX_SLOT_KEY = 65535
|
||||
"""The largest slot key a record may define: the largest MessagePack array16 length."""
|
||||
|
||||
_STACK_FAULTS: tuple[type[BaseException], ...] = (_MsgpackStackError,)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class UnreadSlots:
|
||||
"""What a payload decode found past the slots this runtime declares, at any depth: the first type read with extra slots, how
|
||||
|
|
@ -89,22 +86,85 @@ class CultCachePayloadDepthError(ValueError):
|
|||
)
|
||||
|
||||
|
||||
def _refuse_past_depth_limit(tree: Any, type_name: str, direction: str) -> None:
|
||||
"""The one nesting count. Walks a payload tree iteratively (every mapping, sequence and set that is not text or
|
||||
bytes is a container), refusing the first container past `MAX_PAYLOAD_DEPTH`."""
|
||||
stack = [(tree, 1)] if _is_container(tree) else []
|
||||
while stack:
|
||||
node, depth = stack.pop()
|
||||
if depth > MAX_PAYLOAD_DEPTH:
|
||||
raise CultCachePayloadDepthError(type_name, direction)
|
||||
children = chain(node.keys(), node.values()) if isinstance(node, Mapping) else node
|
||||
stack.extend((child, depth + 1) for child in children if _is_container(child))
|
||||
class PayloadCodecFault(ValueError):
|
||||
"""A codec raises this for a payload its format cannot carry. `code` names the rule (the library's are
|
||||
`map_key_kind`, `duplicate_map_key`, `integer_out_of_range`, `depth_limit`, `value_kind` and
|
||||
`timestamp_out_of_range`) and `kind` the kind of node at fault (a MessagePack
|
||||
kind such as `nil` or `bin`); neither is ever a payload value. A consumer-built codec may raise it."""
|
||||
|
||||
def __init__(self, code: str, kind: str = "") -> None:
|
||||
self.code = code
|
||||
self.kind = kind
|
||||
super().__init__(code)
|
||||
|
||||
|
||||
class CultCachePayloadFormatError(ValueError):
|
||||
"""A payload is not one value of its definition's codec (read), or holds a key or integer the codec cannot carry.
|
||||
Names the type, the codec, the direction and a `code`; `detail` is the fault's class or the kind of the key or
|
||||
extension value. Never a payload byte or value."""
|
||||
|
||||
def __init__(self, type_name: str, codec: str, direction: str, code: str, detail: str) -> None:
|
||||
self.type_name = type_name
|
||||
self.codec = codec
|
||||
self.direction = direction
|
||||
self.code = code
|
||||
if code == "not_one_value":
|
||||
message = f"{type_name} payload is not one {codec} value ({detail}); refused on read."
|
||||
elif code == "map_key_kind":
|
||||
message = (
|
||||
f"{type_name} payload has a map key of kind {detail}; "
|
||||
f"{codec} payloads take str, int, float, bool or timestamp keys (refused on {direction})."
|
||||
)
|
||||
elif code == "duplicate_map_key":
|
||||
message = f"{type_name} payload has two map keys Python holds as one; refused on {direction}."
|
||||
elif code == "integer_out_of_range":
|
||||
message = f"{type_name} payload has an integer beyond 64 bits; refused on write."
|
||||
elif code == "value_kind":
|
||||
message = (
|
||||
f"{type_name} payload holds a value of kind {detail}; "
|
||||
f"{codec} payloads carry no extension type but the timestamp (refused on {direction})."
|
||||
)
|
||||
elif code == "timestamp_out_of_range":
|
||||
message = f"{type_name} payload holds a timestamp outside years 1 to 9999; refused on {direction}."
|
||||
else:
|
||||
message = f"{type_name} payload is refused by the {codec} codec ({code}); refused on {direction}."
|
||||
super().__init__(message)
|
||||
|
||||
|
||||
def _is_container(node: Any) -> bool:
|
||||
return isinstance(node, _CONTAINERS) and not isinstance(node, _SCALAR_SEQUENCES)
|
||||
|
||||
|
||||
def _children(node: Any) -> Iterable[Any]:
|
||||
return chain(node.keys(), node.values()) if isinstance(node, Mapping) else node
|
||||
|
||||
|
||||
def _containers(tree: Any) -> Iterable[Any]:
|
||||
"""The one depth walk over a payload tree, for every codec and every caller. Iterative, so a cycle ends at the limit:
|
||||
yields every container (a mapping, sequence or set that is not text or bytes; a mapping's keys and values are both
|
||||
children), the outermost at depth 1, and raises `PayloadCodecFault('depth_limit')` at the first one past
|
||||
`MAX_PAYLOAD_DEPTH`."""
|
||||
stack = [(tree, 1)] if _is_container(tree) else []
|
||||
while stack:
|
||||
node, depth = stack.pop()
|
||||
if depth > MAX_PAYLOAD_DEPTH:
|
||||
raise PayloadCodecFault("depth_limit")
|
||||
yield node
|
||||
stack.extend((child, depth + 1) for child in _children(node) if _is_container(child))
|
||||
|
||||
|
||||
def _check_payload_tree(tree: Any, type_name: str, direction: str) -> None:
|
||||
"""The definition's translation of the depth walk into its typed refusal."""
|
||||
try:
|
||||
for _ in _containers(tree):
|
||||
pass
|
||||
except PayloadCodecFault:
|
||||
refusal = CultCachePayloadDepthError(type_name, direction)
|
||||
else:
|
||||
return
|
||||
raise refusal
|
||||
|
||||
|
||||
def _default_encode(value: Any) -> Any:
|
||||
if hasattr(value, "model_dump"):
|
||||
return value.model_dump()
|
||||
|
|
@ -119,14 +179,6 @@ def _default_decode(value: Any) -> Any:
|
|||
return value
|
||||
|
||||
|
||||
def _json_payload_encoder(value: Any) -> bytes:
|
||||
return json.dumps(value, ensure_ascii=False, sort_keys=True).encode("utf-8")
|
||||
|
||||
|
||||
def _json_payload_decoder(payload: bytes) -> Any:
|
||||
return json.loads(payload.decode("utf-8"))
|
||||
|
||||
|
||||
def _pack_tree(tree: Any) -> bytes:
|
||||
"""The bytes of the tree the leaves produced. `packb` writes everything but a `Float32`, which it refuses with a
|
||||
TypeError; only then does an iterative walk pack each array and map header and each value, a `Float32` through a
|
||||
|
|
@ -154,27 +206,147 @@ def _pack_tree(tree: Any) -> bytes:
|
|||
return b"".join(out)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PayloadCodec:
|
||||
"""A codec owns one payload format: how a tree becomes bytes and back, which of its decode faults mean the payload
|
||||
nests past the interpreter's own stack, and which trees it can carry: it raises `PayloadCodecFault` for a key, an
|
||||
integer or a nesting its format cannot hold, and nothing it writes is refused by its own `decode`. A consumer-built codec owns
|
||||
the depth of nodes the depth walk cannot see (anything that is not a Mapping, Sequence or Set)."""
|
||||
|
||||
name: str
|
||||
encode: Callable[[Any], bytes]
|
||||
decode: Callable[[bytes], Any]
|
||||
depth_faults: tuple[type[BaseException], ...] = ()
|
||||
|
||||
|
||||
_INTEGER_RANGE = (-(2**63), 2**64 - 1)
|
||||
_TIMESTAMP_SECONDS = (-62135596800, 253402300799)
|
||||
_NAN_KEY = object()
|
||||
_KEY_TYPES = (str, int, float, Float32, msgpack.Timestamp)
|
||||
|
||||
|
||||
def _node_kind(node: Any) -> str:
|
||||
"""The MessagePack kind of a node, for a refusal: a kind, never a value."""
|
||||
if node is None:
|
||||
return "nil"
|
||||
if isinstance(node, (bytes, bytearray, memoryview)):
|
||||
return "bin"
|
||||
if isinstance(node, Mapping):
|
||||
return "map"
|
||||
if isinstance(node, msgpack.ExtType):
|
||||
return "ext"
|
||||
if isinstance(node, _CONTAINERS):
|
||||
return "array"
|
||||
return "unsupported"
|
||||
|
||||
|
||||
def _key_image(key: Any) -> Any:
|
||||
"""A map key as the reader holds it, and the one key identity of read and write: a `Float32` is written as float32
|
||||
and read back as that float, and every NaN is one key (the reference refuses two of them)."""
|
||||
if isinstance(key, Float32):
|
||||
key = struct.unpack(">f", struct.pack(">f", key.value))[0]
|
||||
return _NAN_KEY if isinstance(key, float) and key != key else key
|
||||
|
||||
|
||||
def _checked_map(pairs: list[tuple[Any, Any]]) -> dict[Any, Any]:
|
||||
"""The read hook of every MessagePack map: the keys the C# reference's object reader accepts (str, int, float,
|
||||
bool, timestamp) and no duplicate."""
|
||||
for key, _ in pairs:
|
||||
if not isinstance(key, _KEY_TYPES):
|
||||
raise PayloadCodecFault("map_key_kind", _node_kind(key))
|
||||
if len(dict.fromkeys(_key_image(key) for key, _ in pairs)) != len(pairs):
|
||||
raise PayloadCodecFault("duplicate_map_key")
|
||||
return dict(pairs)
|
||||
|
||||
|
||||
def _check_scalar(node: Any) -> None:
|
||||
if isinstance(node, int) and not _INTEGER_RANGE[0] <= node <= _INTEGER_RANGE[1]:
|
||||
raise PayloadCodecFault("integer_out_of_range")
|
||||
if isinstance(node, msgpack.Timestamp) and not _TIMESTAMP_SECONDS[0] <= node.seconds <= _TIMESTAMP_SECONDS[1]:
|
||||
raise PayloadCodecFault("timestamp_out_of_range")
|
||||
|
||||
|
||||
def _check_carried(tree: Any) -> None:
|
||||
"""The one check of what a MessagePack payload carries, on read and on write: the keys, integers, timestamps and
|
||||
nesting the C# reference's object reader takes, no extension type but the timestamp, and no two keys that decode
|
||||
as one (a `Float32` beside an equal int, bool or float, or two NaNs), so nothing this codec writes is refused by
|
||||
its own decode."""
|
||||
_check_scalar(tree)
|
||||
for node in _containers(tree):
|
||||
if isinstance(node, msgpack.ExtType):
|
||||
raise PayloadCodecFault("value_kind", "ext")
|
||||
for child in _children(node):
|
||||
_check_scalar(child)
|
||||
if isinstance(node, Mapping):
|
||||
for key in node:
|
||||
if not isinstance(key, _KEY_TYPES):
|
||||
raise PayloadCodecFault("map_key_kind", _node_kind(key))
|
||||
if len(dict.fromkeys(_key_image(key) for key in node)) != len(node):
|
||||
raise PayloadCodecFault("duplicate_map_key")
|
||||
|
||||
|
||||
def _unpack_tree(payload: bytes) -> Any:
|
||||
return msgpack.unpackb(payload, raw=False, strict_map_key=False, object_pairs_hook=_checked_map)
|
||||
|
||||
|
||||
def _unpack_checked_tree(payload: bytes) -> Any:
|
||||
tree = _unpack_tree(payload)
|
||||
_check_carried(tree)
|
||||
return tree
|
||||
|
||||
|
||||
def _pack_checked_tree(tree: Any) -> bytes:
|
||||
_check_carried(tree)
|
||||
return _pack_tree(tree)
|
||||
|
||||
|
||||
def _dump_json(tree: Any) -> bytes:
|
||||
return json.dumps(tree, ensure_ascii=False, sort_keys=True).encode("utf-8")
|
||||
|
||||
|
||||
def _load_json(payload: bytes) -> Any:
|
||||
return json.loads(payload.decode("utf-8"))
|
||||
|
||||
|
||||
MESSAGEPACK_PAYLOAD_CODEC = PayloadCodec(
|
||||
"MessagePack", _pack_checked_tree, _unpack_checked_tree, depth_faults=(msgpack.exceptions.StackError,)
|
||||
)
|
||||
"""The default payload format. It carries what the C# reference's object reader accepts: str, int, float, bool and
|
||||
timestamp keys, no duplicate key (two NaN keys are one), integers within -(2**63)..2**64-1, timestamps within years 1 to
|
||||
9999, no extension type but the timestamp, and nesting no deeper than `MAX_PAYLOAD_DEPTH`. One check refuses the rest on
|
||||
read and on write, so it never writes a record it cannot read."""
|
||||
|
||||
JSON_PAYLOAD_CODEC = PayloadCodec("JSON", _dump_json, _load_json, depth_faults=(RecursionError,))
|
||||
"""The 0.3 default, kept for stores written before MessagePack: it writes exactly the 0.3 bytes. Never a default."""
|
||||
|
||||
|
||||
def _messagepack_payload_decoder(payload: bytes) -> Any:
|
||||
return msgpack.unpackb(payload, raw=False)
|
||||
|
||||
|
||||
def read_payload_tree(type_name: str, payload: bytes, decoder: Callable[[bytes], Any] | None = None) -> Any:
|
||||
def read_payload_tree(
|
||||
type_name: str,
|
||||
payload: bytes,
|
||||
decoder: Callable[[bytes], Any] | None = None,
|
||||
depth_faults: tuple[type[BaseException], ...] = (msgpack.exceptions.StackError,),
|
||||
) -> Any:
|
||||
"""The one read of a payload's tree: exactly one MessagePack value, within the nesting limit. Depth is judged first,
|
||||
on the value read so far, so a tail never pre-empts it (C# throws at container 257 before it reaches the tail); then a
|
||||
tail is refused by count and never echoed. Raised outside the handler so no `__context__` keeps the tail."""
|
||||
tail is refused by count and never echoed. Raised outside the handler so no `__context__` keeps the tail. A
|
||||
definition passes its codec's `decode` and `depth_faults`; with none this reads plain MessagePack."""
|
||||
after = -1
|
||||
unpacked: Any = None
|
||||
try:
|
||||
tree = (decoder or _messagepack_payload_decoder)(payload)
|
||||
except _STACK_FAULTS:
|
||||
except depth_faults:
|
||||
raise CultCachePayloadDepthError(type_name, "read") from None
|
||||
except msgpack.ExtraData as error:
|
||||
after = len(error.extra)
|
||||
unpacked = error.unpacked
|
||||
if after >= 0:
|
||||
_refuse_past_depth_limit(unpacked, type_name, "read")
|
||||
_check_payload_tree(unpacked, type_name, "read")
|
||||
raise ValueError(f"{type_name} payload has {after} bytes after its first MessagePack value (refused on read).")
|
||||
_refuse_past_depth_limit(tree, type_name, "read")
|
||||
_check_payload_tree(tree, type_name, "read")
|
||||
return tree
|
||||
|
||||
|
||||
|
|
@ -212,6 +384,12 @@ class UnionArm:
|
|||
value: Any = None
|
||||
|
||||
|
||||
def _fault_refusal(type_name: str, codec: PayloadCodec, direction: str, fault: PayloadCodecFault) -> CultCachePayloadFormatError | CultCachePayloadDepthError:
|
||||
if fault.code == "depth_limit":
|
||||
return CultCachePayloadDepthError(type_name, direction)
|
||||
return CultCachePayloadFormatError(type_name, codec.name, direction, fault.code, fault.kind)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DocumentDefinition(Generic[T]):
|
||||
type: str
|
||||
|
|
@ -226,8 +404,7 @@ class DocumentDefinition(Generic[T]):
|
|||
roles: dict[str, str | Extractor[T]] = field(default_factory=dict)
|
||||
"""Selection aliases (C# `[CultRole]`): many records may share a value. Never indexed for lookup, never refuse; the
|
||||
member's catalog `index_alias` is the same either way."""
|
||||
payload_encoder: Callable[[Any], bytes] = _json_payload_encoder
|
||||
payload_decoder: Callable[[bytes], Any] = _json_payload_decoder
|
||||
payload_codec: PayloadCodec = MESSAGEPACK_PAYLOAD_CODEC
|
||||
schema_id: str | None = None
|
||||
schema_name: str | None = None
|
||||
schema_version: str | None = None
|
||||
|
|
@ -245,8 +422,13 @@ class DocumentDefinition(Generic[T]):
|
|||
if self.validate:
|
||||
self.validate(value)
|
||||
tree = self.encode(value)
|
||||
_refuse_past_depth_limit(tree, self.type, "write")
|
||||
return self.payload_encoder(tree)
|
||||
_check_payload_tree(tree, self.type, "write")
|
||||
codec = self.payload_codec
|
||||
try:
|
||||
return codec.encode(tree)
|
||||
except PayloadCodecFault as fault:
|
||||
refusal = _fault_refusal(self.type, codec, "write", fault)
|
||||
raise refusal
|
||||
|
||||
def decode_payload(self, payload: bytes) -> T:
|
||||
return self.decode_payload_marked(payload)[0]
|
||||
|
|
@ -254,7 +436,23 @@ class DocumentDefinition(Generic[T]):
|
|||
def decode_payload_marked(self, payload: bytes) -> tuple[T, UnreadSlots | None]:
|
||||
"""The value and what the decode found past the declared slots, at any depth. The payload depth check runs first: a
|
||||
payload past the limit is refused, never marked."""
|
||||
tree = read_payload_tree(self.type, payload, self.payload_decoder)
|
||||
codec = self.payload_codec
|
||||
|
||||
def read(raw: bytes) -> Any:
|
||||
try:
|
||||
return codec.decode(raw)
|
||||
except (msgpack.ExtraData, *codec.depth_faults):
|
||||
raise
|
||||
except PayloadCodecFault as fault:
|
||||
refusal = _fault_refusal(self.type, codec, "read", fault)
|
||||
except ValueError as fault:
|
||||
refusal = CultCachePayloadFormatError(self.type, codec.name, "read", "not_one_value", type(fault).__name__)
|
||||
raise refusal
|
||||
|
||||
tree = read_payload_tree(self.type, payload, read, codec.depth_faults)
|
||||
return self._decode_tree(tree)
|
||||
|
||||
def _decode_tree(self, tree: Any) -> tuple[T, UnreadSlots | None]:
|
||||
outer, opened = _current_scope(), _SlotScope()
|
||||
_scope.open = opened
|
||||
try:
|
||||
|
|
@ -325,8 +523,7 @@ def define_document_type(
|
|||
name: str | Extractor[T] | None = None,
|
||||
indexes: dict[str, str | Extractor[T]] | None = None,
|
||||
roles: dict[str, str | Extractor[T]] | None = None,
|
||||
payload_encoder: Callable[[Any], bytes] = _json_payload_encoder,
|
||||
payload_decoder: Callable[[bytes], Any] = _json_payload_decoder,
|
||||
payload_codec: PayloadCodec | None = None,
|
||||
schema_id: str | None = None,
|
||||
schema_name: str | None = None,
|
||||
schema_version: str | None = None,
|
||||
|
|
@ -335,6 +532,10 @@ def define_document_type(
|
|||
compatible_schema_ids: Iterable[str] | None = None,
|
||||
members: Iterable[CultCacheSchemaCatalogMember] | Callable[[], tuple[CultCacheSchemaCatalogMember, ...]] | None = None,
|
||||
) -> DocumentDefinition[T]:
|
||||
"""Define a document type. Its payload is the MessagePack of what `encode` returns; pass
|
||||
`payload_codec=JSON_PAYLOAD_CODEC` to read a store written by cultcache-py 0.3 or earlier, and `convert` to write
|
||||
it as a new MessagePack store."""
|
||||
codec = {} if payload_codec is None else {"payload_codec": payload_codec}
|
||||
return DocumentDefinition(
|
||||
type=type,
|
||||
encode=encode,
|
||||
|
|
@ -344,8 +545,7 @@ def define_document_type(
|
|||
name=name,
|
||||
indexes=indexes or {},
|
||||
roles=roles or {},
|
||||
payload_encoder=payload_encoder,
|
||||
payload_decoder=payload_decoder,
|
||||
**codec,
|
||||
schema_id=schema_id,
|
||||
schema_name=schema_name,
|
||||
schema_version=schema_version,
|
||||
|
|
@ -410,8 +610,6 @@ def define_database_entry_type(
|
|||
name=name,
|
||||
indexes=indexes,
|
||||
roles=roles,
|
||||
payload_encoder=_pack_tree,
|
||||
payload_decoder=_messagepack_payload_decoder,
|
||||
schema_id=schema_id,
|
||||
schema_name=schema_name,
|
||||
schema_version=schema_version,
|
||||
|
|
|
|||
|
|
@ -17,7 +17,7 @@ from typing import Any, Callable, Iterator, TypeVar
|
|||
|
||||
import msgpack
|
||||
|
||||
from .documents import UnreadSlots
|
||||
from .documents import UnreadSlots, _unpack_tree
|
||||
from .backing_store import (
|
||||
CultCacheEnvelope,
|
||||
CultCacheSchemaCatalogEntry,
|
||||
|
|
@ -144,8 +144,8 @@ class JsonLinesBackingStore:
|
|||
change(existing)
|
||||
self._replace_all(list(existing.values()))
|
||||
|
||||
def _replace_all(self, envelopes: list[CultCacheEnvelope]) -> None:
|
||||
self.path.parent.mkdir(parents=True, exist_ok=True)
|
||||
@staticmethod
|
||||
def _file_bytes(envelopes: list[CultCacheEnvelope]) -> bytes:
|
||||
lines = []
|
||||
for envelope in sorted(envelopes, key=lambda item: (item.type, item.key)):
|
||||
lines.append(json.dumps({
|
||||
|
|
@ -154,7 +154,11 @@ class JsonLinesBackingStore:
|
|||
"payload": base64.b64encode(envelope.payload).decode("ascii"),
|
||||
"stored_at": envelope.stored_at,
|
||||
}, ensure_ascii=False, sort_keys=True))
|
||||
commit_store_file(self.path, ("\n".join(lines) + ("\n" if lines else "")).encode("utf-8"))
|
||||
return ("\n".join(lines) + ("\n" if lines else "")).encode("utf-8")
|
||||
|
||||
def _replace_all(self, envelopes: list[CultCacheEnvelope]) -> None:
|
||||
self.path.parent.mkdir(parents=True, exist_ok=True)
|
||||
commit_store_file(self.path, self._file_bytes(envelopes))
|
||||
|
||||
|
||||
class SingleFileMessagePackBackingStore:
|
||||
|
|
@ -181,7 +185,7 @@ class SingleFileMessagePackBackingStore:
|
|||
if data is None:
|
||||
return None
|
||||
try:
|
||||
decoded = msgpack.unpackb(data, raw=False)
|
||||
decoded = _unpack_tree(data)
|
||||
snapshot = _decode_snapshot(decoded)
|
||||
if snapshot is not None:
|
||||
self._format, envelopes = snapshot
|
||||
|
|
@ -233,6 +237,11 @@ class SingleFileMessagePackBackingStore:
|
|||
with store_write_lock(self.path, self.lock_timeout):
|
||||
yield
|
||||
|
||||
@staticmethod
|
||||
def _file_bytes(envelopes: list[CultCacheEnvelope]) -> bytes:
|
||||
"""The bytes of a new store holding exactly `envelopes`, as a write would leave it."""
|
||||
return msgpack.packb(_encode_snapshot(envelopes, [], STORE_FORMAT_VERSION)[0], use_bin_type=True) # type: ignore[no-any-return]
|
||||
|
||||
def _replace_all(self, supplied: list[CultCacheEnvelope], arrived: list[CultCacheEnvelope]) -> dict[str, str]:
|
||||
snapshot, stamps = _encode_snapshot(supplied, arrived, self._format)
|
||||
commit_store_file(self.path, msgpack.packb(snapshot, use_bin_type=True))
|
||||
|
|
@ -669,6 +678,9 @@ def _decode_snapshot(decoded: Any) -> tuple[str, list[CultCacheEnvelope]] | None
|
|||
|
||||
|
||||
def _decode_legacy_envelopes(decoded: Any) -> list[CultCacheEnvelope]:
|
||||
"""A legacy envelope holds its payload as an inline MessagePack subtree or as bytes. Extracting the subtree's bytes
|
||||
decides no format: the inline value is MessagePack by construction, and the registered codec still reads it or
|
||||
refuses it."""
|
||||
if not isinstance(decoded, list):
|
||||
raise ValueError("CultCache MessagePack store is not a recognized snapshot")
|
||||
envelopes: list[CultCacheEnvelope] = []
|
||||
|
|
|
|||
|
|
@ -1,9 +1,12 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import tempfile
|
||||
import threading
|
||||
import unittest
|
||||
import msgpack
|
||||
from dataclasses import asdict, dataclass, replace
|
||||
from pathlib import Path
|
||||
from cultcache_py.backing_store import CultCacheEnvelope
|
||||
|
|
@ -120,11 +123,6 @@ class CultCacheTests(unittest.TestCase):
|
|||
self.assertIsNone(cache.get_key_by_index(document, "category", "Rare"))
|
||||
|
||||
def test_database_entry_formatter_uses_slot_indexed_messagepack_array(self) -> None:
|
||||
try:
|
||||
import msgpack # type: ignore
|
||||
except ModuleNotFoundError:
|
||||
self.skipTest("msgpack optional dependency is not installed")
|
||||
|
||||
@dataclass
|
||||
class Settings:
|
||||
theme: str
|
||||
|
|
@ -159,11 +157,6 @@ class CultCacheTests(unittest.TestCase):
|
|||
self.assertEqual([member.slot for member in first.members], [0, 2])
|
||||
|
||||
def test_messagepack_store_writes_v1_snapshot(self) -> None:
|
||||
try:
|
||||
import msgpack # type: ignore
|
||||
except ModuleNotFoundError:
|
||||
self.skipTest("msgpack optional dependency is not installed")
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
document = define_database_entry_type(
|
||||
"settings",
|
||||
|
|
@ -1570,6 +1563,39 @@ class CultCacheTests(unittest.TestCase):
|
|||
SingleFileMessagePackBackingStore(path).pull_all()
|
||||
self.assertNotIsInstance(caught.exception, FileNotFoundError)
|
||||
|
||||
def test_json_lines_pull_all_reads_hand_written_lines_and_names_a_malformed_one(self) -> None:
|
||||
def line(**fields: object) -> str:
|
||||
return json.dumps(fields, ensure_ascii=False)
|
||||
|
||||
first = base64.b64encode(b"one").decode("ascii")
|
||||
second = base64.b64encode(b"two").decode("ascii")
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
path = Path(tmp) / "hand.jsonl"
|
||||
lines = [
|
||||
line(key="é-ключ", type="tests.t", payload=first, storedAt="2026-01-01T00:00:00Z", schemaId="schema.one"),
|
||||
"",
|
||||
line(key="k2", type="tests.t", payload=second, stored_at="2026-02-02T00:00:00Z", schema_id="schema.two"),
|
||||
]
|
||||
path.write_bytes(("\n".join(lines) + "\n").encode("utf-8"))
|
||||
pulled = JsonLinesBackingStore(path).pull_all()
|
||||
self.assertEqual(
|
||||
[(e.key, e.type, e.payload, e.stored_at, e.schema_id) for e in pulled],
|
||||
[
|
||||
("é-ключ", "tests.t", b"one", "2026-01-01T00:00:00Z", "schema.one"),
|
||||
("k2", "tests.t", b"two", "2026-02-02T00:00:00Z", "schema.two"),
|
||||
],
|
||||
)
|
||||
written = Path(tmp) / "written.jsonl"
|
||||
JsonLinesBackingStore(written).push(CultCacheEnvelope.create(key="k", type="tests.t", payload=b"x"))
|
||||
raw = written.read_bytes()
|
||||
self.assertTrue(raw.endswith(b"\n"))
|
||||
self.assertNotIn(b"\r", raw)
|
||||
broken = Path(tmp) / "broken.jsonl"
|
||||
broken.write_bytes(("\n".join([lines[0], "", line(key="k3", type="tests.t")]) + "\n").encode("utf-8"))
|
||||
with self.assertRaises(ValueError) as caught:
|
||||
JsonLinesBackingStore(broken).pull_all()
|
||||
self.assertEqual(str(caught.exception), f"Malformed CultCache JSONL envelope at {broken}:3: missing 'payload'")
|
||||
|
||||
|
||||
# The element-id marker: tests/vectors/document-variants-c2a/v3-base.msgpack is v1-base with its header replaced.
|
||||
_C2A_VECTORS = Path(__file__).resolve().parents[3] / "tests" / "vectors" / "document-variants-c2a"
|
||||
|
|
|
|||
1053
packages/cultcache-py/tests/test_payload_codec.py
Normal file
1053
packages/cultcache-py/tests/test_payload_codec.py
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -2,6 +2,7 @@
|
|||
|
||||
import collections
|
||||
import dataclasses
|
||||
import json
|
||||
import shutil
|
||||
import tempfile
|
||||
import traceback
|
||||
|
|
@ -15,8 +16,12 @@ from cultcache_py import (
|
|||
MAX_PAYLOAD_DEPTH,
|
||||
MAX_SLOT_KEY,
|
||||
CultCache,
|
||||
JSON_PAYLOAD_CODEC,
|
||||
MESSAGEPACK_PAYLOAD_CODEC,
|
||||
PayloadCodecFault,
|
||||
CultCachePayloadDepthError,
|
||||
DatabaseEntryField,
|
||||
PayloadCodec,
|
||||
SlotCodec,
|
||||
SingleFileMessagePackBackingStore,
|
||||
UnionArm,
|
||||
|
|
@ -595,17 +600,18 @@ class BoundsVectorTests(unittest.TestCase):
|
|||
with self.assertRaises(CultCachePayloadDepthError) as caught:
|
||||
tree.encode_payload(_nest(5000))
|
||||
self.assertIn("refused on write", str(caught.exception))
|
||||
document = define_document_type("tests.json_tree")
|
||||
with self.assertRaises(CultCachePayloadDepthError) as read:
|
||||
document.decode_payload(b"[" * 300 + b"]" * 300)
|
||||
self.assertIn("refused on read", str(read.exception))
|
||||
for codec, deep in ((None, b"\x91" * 300 + b"\xc0"), (JSON_PAYLOAD_CODEC, b"[" * 300 + b"]" * 300)):
|
||||
document = define_document_type("tests.json_tree") if codec is None else define_document_type("tests.json_tree", payload_codec=codec)
|
||||
with self.assertRaises(CultCachePayloadDepthError) as read:
|
||||
document.decode_payload(deep)
|
||||
self.assertIn("refused on read", str(read.exception))
|
||||
|
||||
def test_a_codec_that_exhausts_the_interpreter_stack_keeps_its_own_recursion_error_on_each_side(self) -> None:
|
||||
def boom(value: object) -> object:
|
||||
return boom(value)
|
||||
|
||||
document = define_document_type("tests.boom", encode=boom, decode=boom)
|
||||
for run in (lambda: document.encode_payload("x"), lambda: document.decode_payload(b'"x"')):
|
||||
for run in (lambda: document.encode_payload("x"), lambda: document.decode_payload(b"\xa1x")):
|
||||
try:
|
||||
run()
|
||||
except RecursionError as caught:
|
||||
|
|
@ -620,11 +626,11 @@ class BoundsVectorTests(unittest.TestCase):
|
|||
"tests.union_surface payload nests 257 containers deep; the limit is 256 (refused on read).",
|
||||
)
|
||||
|
||||
def test_a_json_payload_past_the_interpreters_own_stack_keeps_the_decoders_recursion_error(self) -> None:
|
||||
document = define_document_type("tests.json_tree")
|
||||
with self.assertRaises(RecursionError) as caught:
|
||||
def test_json_text_past_the_interpreters_own_stack_is_the_depth_refusal(self) -> None:
|
||||
document = define_document_type("tests.json_tree", payload_codec=JSON_PAYLOAD_CODEC)
|
||||
with self.assertRaises(CultCachePayloadDepthError) as caught:
|
||||
document.decode_payload(b"[" * 100000)
|
||||
self.assertNotIsInstance(caught.exception, CultCachePayloadDepthError)
|
||||
self.assertIn("refused on read", str(caught.exception))
|
||||
|
||||
def test_a_refusal_reports_the_depth_where_the_walk_stopped_whatever_the_payloads_real_depth(self) -> None:
|
||||
document = define_document_type("tests.json_tree", encode=lambda value: value)
|
||||
|
|
@ -633,7 +639,7 @@ class BoundsVectorTests(unittest.TestCase):
|
|||
document.encode_payload(_nest(depth))
|
||||
self.assertEqual((write.exception.depth, write.exception.limit), (257, 256), depth)
|
||||
with self.assertRaises(CultCachePayloadDepthError) as read:
|
||||
document.decode_payload(b"[" * depth + b"]" * depth)
|
||||
document.decode_payload(b"\x91" * depth + b"\xc0")
|
||||
self.assertEqual((read.exception.depth, read.exception.limit), (257, 256), depth)
|
||||
|
||||
def test_the_validator_sees_the_value_on_write_and_the_decoded_value_on_read(self) -> None:
|
||||
|
|
@ -650,6 +656,9 @@ class BoundsVectorTests(unittest.TestCase):
|
|||
document.encode_payload(loop)
|
||||
|
||||
|
||||
_STUB_CODEC = PayloadCodec("test", lambda tree: b"ok", lambda payload: payload)
|
||||
|
||||
|
||||
def _nest(depth: int, mixed: bool = False) -> object:
|
||||
value: object = "leaf"
|
||||
for level in range(depth):
|
||||
|
|
@ -738,18 +747,24 @@ class PayloadDepthRuleTests(unittest.TestCase):
|
|||
|
||||
def test_the_limit_is_256_and_depth_256_is_accepted_on_read_and_write_while_257_is_refused(self) -> None:
|
||||
self.assertEqual(MAX_PAYLOAD_DEPTH, 256)
|
||||
document = define_document_type("tests.json_tree", encode=lambda value: value)
|
||||
for mixed in (False, True):
|
||||
at_limit = _nest(256, mixed)
|
||||
self.assertEqual(document.decode_payload(document.encode_payload(at_limit)), at_limit)
|
||||
past = _nest(257, mixed)
|
||||
with self.assertRaises(CultCachePayloadDepthError):
|
||||
document.encode_payload(past)
|
||||
with self.assertRaises(CultCachePayloadDepthError):
|
||||
document.decode_payload(document.payload_encoder(past))
|
||||
for codec in (None, JSON_PAYLOAD_CODEC):
|
||||
document = define_document_type("tests.json_tree", encode=lambda value: value, **({} if codec is None else {"payload_codec": codec}))
|
||||
for mixed in (False, True):
|
||||
at_limit = _nest(256, mixed)
|
||||
self.assertEqual(document.decode_payload(document.encode_payload(at_limit)), at_limit)
|
||||
past = _nest(257, mixed)
|
||||
with self.assertRaises(CultCachePayloadDepthError):
|
||||
document.encode_payload(past)
|
||||
raw = json.dumps(past).encode() if codec is JSON_PAYLOAD_CODEC else msgpack.packb(past, use_bin_type=True)
|
||||
with self.assertRaises(CultCachePayloadDepthError):
|
||||
document.decode_payload(raw)
|
||||
if codec is None:
|
||||
with self.assertRaises(PayloadCodecFault) as refused:
|
||||
MESSAGEPACK_PAYLOAD_CODEC.encode(past)
|
||||
self.assertEqual(refused.exception.code, "depth_limit")
|
||||
|
||||
def test_a_tuple_and_a_container_dict_key_count_as_containers(self) -> None:
|
||||
document = define_document_type("tests.tuple_tree", encode=lambda value: value, payload_encoder=lambda tree: b"ok")
|
||||
document = define_document_type("tests.tuple_tree", encode=lambda value: value, payload_codec=_STUB_CODEC)
|
||||
self.assertEqual(document.encode_payload(_as_tuple(_nest(256))), b"ok")
|
||||
with self.assertRaises(CultCachePayloadDepthError):
|
||||
document.encode_payload(_as_tuple(_nest(257)))
|
||||
|
|
@ -765,19 +780,21 @@ class PayloadDepthRuleTests(unittest.TestCase):
|
|||
"userlist": lambda inner: collections.UserList([inner]),
|
||||
}
|
||||
for name, wrap in wrappers.items():
|
||||
document = define_document_type(f"tests.{name}", encode=lambda value: value, payload_encoder=lambda tree: b"ok")
|
||||
document = define_document_type(f"tests.{name}", encode=lambda value: value, payload_codec=_STUB_CODEC)
|
||||
at_limit: object = "leaf"
|
||||
for _ in range(256):
|
||||
at_limit = wrap(at_limit)
|
||||
self.assertEqual(document.encode_payload(at_limit), b"ok", name)
|
||||
with self.assertRaises(CultCachePayloadDepthError, msg=name):
|
||||
document.encode_payload(wrap(at_limit))
|
||||
decoding = define_document_type(f"tests.{name}.read", payload_decoder=lambda payload, w=wrap: w(at_limit))
|
||||
decoding = define_document_type(
|
||||
f"tests.{name}.read", payload_codec=PayloadCodec("test", lambda tree: b"ok", lambda payload, w=wrap: w(at_limit))
|
||||
)
|
||||
with self.assertRaises(CultCachePayloadDepthError, msg=name):
|
||||
decoding.decode_payload(b"x")
|
||||
|
||||
def test_text_and_bytes_are_leaves_however_long(self) -> None:
|
||||
document = define_document_type("tests.leaves", encode=lambda value: value, payload_encoder=lambda tree: b"ok")
|
||||
document = define_document_type("tests.leaves", encode=lambda value: value, payload_codec=_STUB_CODEC)
|
||||
self.assertEqual(document.encode_payload([["x" * 100000, b"y" * 100000, bytearray(5), memoryview(b"z")]]), b"ok")
|
||||
|
||||
def test_a_scalar_payload_has_no_depth(self) -> None:
|
||||
|
|
|
|||
|
|
@ -963,8 +963,6 @@ def binding(
|
|||
document_type,
|
||||
encode=encode_slots,
|
||||
decode=decode_slots,
|
||||
payload_encoder=lambda value: msgpack.packb(value, use_bin_type=True),
|
||||
payload_decoder=lambda payload: msgpack.unpackb(payload, raw=False),
|
||||
schema_id=schema_id,
|
||||
schema_name=document_type,
|
||||
schema_version=schema_version,
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ from dataclasses import dataclass
|
|||
from typing import Any, Iterable
|
||||
|
||||
from cultcache_py import CultCache, CultCacheEnvelope
|
||||
from cultcache_py.documents import DocumentDefinition
|
||||
from cultcache_py.documents import MESSAGEPACK_PAYLOAD_CODEC, DocumentDefinition
|
||||
from cultcache_py.stores import parse_stored_at
|
||||
|
||||
from .shard_log import CultNetShardLogResponse
|
||||
|
|
@ -190,9 +190,12 @@ def resolve_document_for_raw_record(
|
|||
|
||||
|
||||
def _infer_schema_version_from_payload(payload: bytes) -> str | None:
|
||||
"""Routing only, as the C# `CultNetDocumentRegistry.TryResolveDescriptorByPayloadSchema` does: reads a schema stamp
|
||||
off a raw record whose schema id is unknown, to pick a definition. It decides no format; the definition it picks
|
||||
has a codec that decodes the payload or refuses it. The codec's reader reads the stamp, so the router decides no key."""
|
||||
import msgpack # type: ignore
|
||||
try:
|
||||
decoded = msgpack.unpackb(payload, raw=False)
|
||||
decoded = MESSAGEPACK_PAYLOAD_CODEC.decode(payload)
|
||||
except msgpack.ExtraData as extra:
|
||||
# The hint is read from the first value; the tail is not this function's to judge. The routed definition refuses
|
||||
# it by count when the record is applied, as it does for a record routed by a registered schema id. Treating a
|
||||
|
|
|
|||
|
|
@ -25,6 +25,7 @@ from cultnet_py.compare_csharp import (
|
|||
)
|
||||
from cultnet_py.interop_peer import (
|
||||
append_shard_log_put,
|
||||
binding,
|
||||
build_state,
|
||||
handle_server_message,
|
||||
raw_snapshot_response,
|
||||
|
|
@ -1976,8 +1977,7 @@ class CultNetTests(unittest.TestCase):
|
|||
from cultcache_py import define_document_type
|
||||
from cultnet_py import resolve_document_for_raw_record
|
||||
|
||||
document = define_document_type("tests.tail_note", schema_id="tests.tail_note", schema_version="tests.tail_note.v1",
|
||||
payload_encoder=msgpack.packb, payload_decoder=lambda payload: msgpack.unpackb(payload, raw=False))
|
||||
document = define_document_type("tests.tail_note", schema_id="tests.tail_note", schema_version="tests.tail_note.v1")
|
||||
known = document.catalog_entry().schema_id
|
||||
head = msgpack.packb(["tests.tail_note.v1", "x"], use_bin_type=True)
|
||||
|
||||
|
|
@ -2119,6 +2119,42 @@ class CultNetTests(unittest.TestCase):
|
|||
self.assertEqual([change.change_kind for change in typed_applied_log], ["updated", "removed"])
|
||||
self.assertIsNone(cache.get(document, "item:1"))
|
||||
|
||||
def test_the_stamp_router_judges_no_map_key_and_the_codec_alone_does(self) -> None:
|
||||
import msgpack # type: ignore
|
||||
from cultnet_py import resolve_document_for_raw_record
|
||||
|
||||
document = define_database_entry_type(
|
||||
"replica.stamped",
|
||||
[("schema_version", 0), ("value", 1)],
|
||||
schema_id="replica.stamped.current",
|
||||
schema_name="replica.stamped",
|
||||
schema_version="replica.stamped.v1",
|
||||
)
|
||||
by_schema_id = {"replica.stamped.current": document, "replica.stamped": document}
|
||||
|
||||
def routed(tree: object) -> object:
|
||||
record = {"payload": msgpack.packb(tree, use_bin_type=True)}
|
||||
return resolve_document_for_raw_record(by_schema_id, "sha256:unknown", record)
|
||||
|
||||
for name, tree in (
|
||||
("array", ["replica.stamped.v1", {1: "a"}]),
|
||||
("map", {"schemaVersion": "replica.stamped.v1", "v": {1: "a"}}),
|
||||
("nested", ["replica.stamped.v1", {"a": {2: [{3: 4}]}}]),
|
||||
("snake", {"schema_version": "replica.stamped.v1", "v": {1: "a"}}),
|
||||
):
|
||||
with self.subTest(form=name):
|
||||
self.assertIs(routed(tree), document)
|
||||
for name, tree in (
|
||||
("array", ["replica.stamped.v1", {None: "a"}]),
|
||||
("map", {"schemaVersion": "replica.stamped.v1", "v": {None: "a"}}),
|
||||
("not-a-string", {"schemaVersion": 5, "v": {1: "a"}}),
|
||||
("unknown-name", ["replica.other.v1", {1: "a"}]),
|
||||
):
|
||||
with self.subTest(form=name):
|
||||
with self.assertRaises(KeyError) as caught:
|
||||
routed(tree)
|
||||
self.assertEqual(caught.exception.args, ("sha256:unknown",))
|
||||
|
||||
def test_cultnet_replication_helpers_recover_schema_stamped_raw_records(self) -> None:
|
||||
document = define_database_entry_type(
|
||||
"replica.runtime-policy",
|
||||
|
|
@ -2678,6 +2714,12 @@ class CultNetTests(unittest.TestCase):
|
|||
self.assertEqual(response["shardLogSequence"], 1)
|
||||
self.assertEqual([record["recordKey"] for record in response["documents"]], ["note:logged"])
|
||||
|
||||
def test_python_interop_peer_binding_writes_plain_messagepack_slots(self) -> None:
|
||||
# The bytes the binding wrote before it stopped passing its own codec lambdas: packb of the slot array.
|
||||
note = binding("tests.note", "tests.note.v1", "tests.note.v1", "{}", lambda value: ["a", 1], lambda slots: slots)
|
||||
self.assertEqual(note.document.encode_payload({}), bytes.fromhex("92a16101"))
|
||||
self.assertEqual(note.document.decode_payload(bytes.fromhex("92a16101")), ["a", 1])
|
||||
|
||||
def test_python_interop_peer_refuses_v1_selection_messages(self) -> None:
|
||||
# R-2, owed by Cut 1 (docs/cultnet-selection-cut.md, section 9): a v0-only peer refuses any
|
||||
# non-v0 message loudly - cultnet.error.v0 { code: "unsupported_schema_version" } - rather than
|
||||
|
|
|
|||
Loading…
Reference in a new issue