cultcache-py: indexes are unique at commit and load, roles name selection aliases; simulation_fact aliases become roles

One judge (_refuse_index_sharing) behind every write, load and registration; canonical index text as C# IndexValue;
the index handoff in _remove_lookups and str() rendering are deleted.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Libby 2026-11-11 05:18:28 +00:00
commit 7c7ab30f4d
9 changed files with 606 additions and 20 deletions

View file

@ -167,8 +167,8 @@ fn first_call() -> anyhow::Result<()> {
## Python first call
A document is a definition with an encoder and decoder; `name=` and `indexes=` register the name lookup
and indexes. The source is
A document is a definition with an encoder and decoder; `name=` registers the name lookup, `indexes=` unique lookups and
`roles=` many-row selection aliases. The source is
[`packages/cultcache-py/tests/test_first_call.py`](../../packages/cultcache-py/tests/test_first_call.py),
run by `python -m unittest tests.test_first_call` in `packages/cultcache-py` with `src` on `PYTHONPATH`
and `msgpack` installed (the store needs it). A second test fails if the block below stops matching the
@ -194,7 +194,6 @@ item_doc = define_document_type(
encode=lambda item: asdict(item),
decode=lambda raw: Item(**raw),
name="name",
indexes={"category": "category"},
)

View file

@ -9,6 +9,8 @@
### Breaking
- `indexes` are unique, as C# `[CultIndex]`: a write, load or `register_index` that leaves two records of a type holding one non-blank value under one alias is refused (`CultCacheError`) naming both keys and the alias, and writes nothing. Index text is the C# canonical text (`True`/`False`, integers as digits, floats as the exact expansion, blank values never indexed), so an empty or whitespace string is no longer a value. A new `roles` map names the many-row selection aliases (never indexed for lookup, same catalog `index_alias`). Classify every `indexes` key per the README before updating. The verdict is the contract, the text is informational.
- A single-file store written as `cultcache.store.v3` is read and written: a record that holds an element id (an owned list element with an `[CultElementId]` member, carried through every write) marks the store v3, and a v3 store is carried through a rewrite with its ids. A runtime from before this change refuses a v3 store by name.
- A write over a file the reader refuses (`StoreUnreadableError`, from `push`, `push_all` and `delete`) refuses and leaves the file as it is. A flush used to replace such a file. A write set cannot merge with a disk it cannot read, so the file is repaired or removed by its owner before the store writes again.
- Added `StoreUnreadableError` (a `ValueError`) for a store file this runtime cannot read, and `SchemaConflictError` (a `ValueError`) for a

View file

@ -73,7 +73,7 @@ store.
- `register_document_type(...)`
- `register_registry(...)`
- `register_name_lookup(...)`
- `register_index(...)`
- `register_index(...)` (a unique index)
- `add_backing_store(...)`
- `add_generic_store(...)`
- `pull_all_backing_stores()`
@ -95,6 +95,33 @@ store.
- `delete_global(...)`
- `snapshot()`
## Indexes And Roles
`indexes=` are unique lookups (variants:ruling:unique-indexes; C# `[CultIndex]`): within one document type and one alias
no two records hold one non-blank value, judged on the result of every write (a batch is judged whole, so a swap inside
one `put_envelopes` lands) and of every load and `register_index`. A refusal is a `CultCacheError`,
`Records <k1> and <k2> (<schema name>) both hold unique index '<alias>'; an index value names one record.`, keys in
ordinal order, never the value; it writes nothing, and a refused load or registration installs nothing. The contract is
the verdict; the text is informational. Blank values (absent, empty, or only C# `White_Space` characters) are never
indexed and never conflict. Two types may share an alias and a value. Values compare as the C# canonical text: strings as
is (never trimmed), `True`/`False`, integers as digits, a non-integral float as its exact decimal expansion, NaN and
infinities no value.
`roles=` name selection aliases (C# `[CultRole]`): many records may share a value, nothing is indexed or refused,
`get_by_index` on a role answers `None`, and the member's catalog `index_alias` is the same either way, so catalog bytes
do not move. An alias is declared in one of the two. `CultMeshDatabase.watch_by_index` reads `indexes` only.
Migrating an existing `indexes` key, per key, before bumping:
1. Nothing looks records up by the key (`get_by_index`, `get_key_by_index`, `node.get_by_index`, `watch_by_index`): move it to `roles`.
2. Something looks it up: it stays unique only when two live records can never hold one non-blank value (the record key
is derived from that value alone, or it is an outside identity: email, sku, external id). Enums, statuses, kinds,
categories, parent or foreign ids, shared hashes and timestamps are roles; select them with `get_all` plus a filter.
3. Open each real `.cc` once with the new definitions. A refusal names both keys and the alias and writes nothing.
Misclassifying: a role declared unique makes a store that holds a shared value refuse to open until the key is
reclassified; a unique key declared a role answers no lookups and stops refusing duplicates.
## Persistence Model
Backing stores persist envelopes, not caller domain objects. The cache decodes payloads through registered document definitions and rejects unknown type discriminators. This keeps Python's dynamic runtime from turning persistence into an open polymorphic sewer with a cheerful docstring.

View file

@ -1,9 +1,11 @@
from __future__ import annotations
import functools
import math
import threading
from decimal import Decimal
from dataclasses import dataclass, field, replace
from typing import Any, Callable, Generic, TypeVar
from typing import Any, Callable, Generic, Iterable, TypeVar
from .backing_store import BackingStore, CultCacheEnvelope
from .documents import DocumentDefinition, extract_value
@ -52,6 +54,47 @@ class CultCacheError(RuntimeError):
pass
# Blank is no value: empty or only C# White_Space characters (the 25 chars string.IsNullOrWhiteSpace is true for).
# An explicit set, not str.isspace(): Python also counts U+001C-001F, which C# does not, and omits nothing C# counts.
_BLANK_INDEX_CHARS = frozenset(
chr(code)
for code in (
*range(0x09, 0x0E),
0x20,
0x85,
0xA0,
0x1680,
*range(0x2000, 0x200B),
0x2028,
0x2029,
0x202F,
0x205F,
0x3000,
)
)
def _index_text(value: Any) -> str | None:
"""The canonical text of an index value, as C# IndexValue and RenderCanonicalNumber, or None when it names no
value: a string is as is (never trimmed), a boolean is `True`/`False`, an integer is its digits, a non-integral
finite float is its exact decimal expansion, NaN and infinities and blank strings are no value."""
if value is None:
return None
if isinstance(value, bool): # before int: bool is an int subclass
return "True" if value else "False"
if isinstance(value, int):
return str(value)
if isinstance(value, float):
if not math.isfinite(value):
return None
if value.is_integer():
return str(int(value)) # -0.0 is 0
text = format(Decimal(value), "f") # Decimal(float) is exact
return text.rstrip("0").rstrip(".") if "." in text else text
text = value if isinstance(value, str) else str(value)
return None if all(char in _BLANK_INDEX_CHARS for char in text) else text
@dataclass
class _State:
documents: dict[str, DocumentDefinition[Any]] = field(default_factory=dict)
@ -438,6 +481,12 @@ class CultCache:
(envelope, value, self._lookup_keys_for(value, name_extractor, index_extractors))
for envelope, value in records
]
self._refuse_index_sharing(
type,
{envelope.key: keys for envelope, _, keys in admitted},
self._state.indexes,
[envelope.key for envelope, _ in records],
)
store = self._store_for_type(type)
legacy_key = self._state.legacy_global_keys.get(type)
if store is not None:
@ -552,9 +601,9 @@ class CultCache:
name = None if name_extractor is None else extract_value(value, name_extractor)
indexes: dict[str, str] = {}
for index, extractor in index_extractors.items():
index_value = extract_value(value, extractor)
index_value = _index_text(extract_value(value, extractor))
if index_value is not None:
indexes[index] = str(index_value)
indexes[index] = index_value
return _LookupKeys(None if name is None else str(name), indexes)
def _derive_lookups(
@ -571,17 +620,48 @@ class CultCache:
name_extractor = name_extractors.get(type)
type_index_extractors = index_extractors.get(type, {})
for key, value in values.items():
keys = self._lookup_keys_for(value, name_extractor, type_index_extractors)
lookup_keys.setdefault(type, {})[key] = keys
lookup_keys.setdefault(type, {})[key] = self._lookup_keys_for(value, name_extractor, type_index_extractors)
# Judged whole over every alias: a held value can have been mutated in place since it was admitted.
self._refuse_index_sharing(type, lookup_keys.get(type, {}), {}, ())
for key, keys in lookup_keys.get(type, {}).items():
if keys.name is not None:
names[(type, keys.name)] = key
for index, index_value in keys.indexes.items():
indexes[(type, index, index_value)] = key
return names, indexes, lookup_keys
def _refuse_index_sharing(
self,
type: str,
landing: dict[str, _LookupKeys],
held: dict[tuple[str, str, str], str],
replaced: Iterable[str],
) -> None:
"""The one index rule (variants:ruling:unique-indexes): within one type and one unique alias, no two records
hold one non-blank value, judged on the result. The holders of a value are the landing keys projecting it plus
the held key unless this admission replaces it; two distinct holders refuse, naming the two smallest keys
(ordinal) and the alias, never the value. Aliases are judged in ordinal order, values in ordinal order."""
released = set(replaced)
holders: dict[tuple[str, str], set[str]] = {}
for key, keys in landing.items():
for index, index_value in keys.indexes.items():
holders.setdefault((index, index_value), set()).add(key)
for (index, index_value), keys in sorted(holders.items()):
held_key = held.get((type, index, index_value))
if held_key is not None and held_key not in released:
keys = keys | {held_key}
if len(keys) > 1:
first, second = sorted(keys)[:2]
document = self._state.documents.get(type)
schema_name = (document.schema_name if document is not None else None) or type
raise CultCacheError(
f"Records {first} and {second} ({schema_name}) both hold unique index '{index}'; "
"an index value names one record."
)
def _remove_lookups(self, type: str, key: str) -> None:
"""Removes the lookups the held record produced at admission, handing a name or index value to
another held record that produced it too. Reads stored lookup keys only; runs no extractor."""
"""Removes the lookups the held record produced at admission, handing a name to another held
record that produced it too (an index value has one holder). Reads stored lookup keys only; runs no extractor."""
held = self._state.lookup_keys.get(type, {})
keys = held.get(key)
if keys is None:
@ -596,9 +676,6 @@ class CultCache:
if self._state.indexes.get(lookup_key) != key:
continue
del self._state.indexes[lookup_key]
for candidate_key, candidate in held.items():
if candidate_key != key and candidate.indexes.get(index) == index_value:
self._state.indexes[lookup_key] = candidate_key
def _assert_registered(self, document: DocumentDefinition[Any]) -> None:
if self._state.documents.get(document.type) is not document:

View file

@ -157,6 +157,11 @@ class DocumentDefinition(Generic[T]):
global_document: bool = False
name: str | Extractor[T] | None = None
indexes: dict[str, str | Extractor[T]] = field(default_factory=dict)
"""Unique lookups (C# `[CultIndex]`): no two records of the type hold one non-blank value, judged at every commit and
load. `get_by_index` answers the one holder."""
roles: dict[str, str | Extractor[T]] = field(default_factory=dict)
"""Selection aliases (C# `[CultRole]`): many records may share a value. Never indexed for lookup, never refuse; the
member's catalog `index_alias` is the same either way."""
payload_encoder: Callable[[Any], bytes] = _json_payload_encoder
payload_decoder: Callable[[bytes], Any] = _json_payload_decoder
schema_id: str | None = None
@ -232,6 +237,14 @@ class DocumentDefinition(Generic[T]):
return catalog_entry
def _refuse_shared_aliases(type: str, indexes: dict[str, Any] | None, roles: dict[str, Any] | None) -> None:
for alias in roles or ():
if alias in (indexes or {}):
raise ValueError(
f"Document type {type} declares '{alias}' as both an index and a role; an index already names its role."
)
def define_document_type(
type: str,
*,
@ -241,6 +254,7 @@ def define_document_type(
global_document: bool = False,
name: str | Extractor[T] | None = None,
indexes: dict[str, str | Extractor[T]] | None = None,
roles: dict[str, str | Extractor[T]] | None = None,
payload_encoder: Callable[[Any], bytes] = _json_payload_encoder,
payload_decoder: Callable[[bytes], Any] = _json_payload_decoder,
schema_id: str | None = None,
@ -251,6 +265,7 @@ def define_document_type(
compatible_schema_ids: Iterable[str] | None = None,
members: Iterable[CultCacheSchemaCatalogMember] | Callable[[], tuple[CultCacheSchemaCatalogMember, ...]] | None = None,
) -> DocumentDefinition[T]:
_refuse_shared_aliases(type, indexes, roles)
return DocumentDefinition(
type=type,
encode=encode,
@ -259,6 +274,7 @@ def define_document_type(
global_document=global_document,
name=name,
indexes=indexes or {},
roles=roles or {},
payload_encoder=payload_encoder,
payload_decoder=payload_decoder,
schema_id=schema_id,
@ -280,6 +296,7 @@ def define_database_entry_type(
global_document: bool = False,
name: str | Extractor[T] | None = None,
indexes: dict[str, str | Extractor[T]] | None = None,
roles: dict[str, str | Extractor[T]] | None = None,
schema_id: str | None = None,
schema_name: str | None = None,
schema_version: str | None = None,
@ -296,7 +313,7 @@ def define_database_entry_type(
normalized = _normalize_fields(type, fields)
member_name_lookup = name if isinstance(name, str) else None
index_lookup = indexes or {}
index_lookup = {**(roles or {}), **(indexes or {})}
def members() -> tuple[CultCacheSchemaCatalogMember, ...]:
return tuple(
@ -323,6 +340,7 @@ def define_database_entry_type(
global_document=global_document,
name=name,
indexes=indexes,
roles=roles,
payload_encoder=_pack_tree,
payload_decoder=_messagepack_payload_decoder,
schema_id=schema_id,

View file

@ -103,20 +103,20 @@ class CultCacheTests(unittest.TestCase):
self.assertEqual(cache.get_key_by_name(document, "Elixir"), "item:potion")
self.assertEqual(cache.get_by_index(document, "category", "Rare").name, "Elixir")
def test_incremental_lookup_delete_restores_duplicate_owner(self) -> None:
def test_incremental_lookup_delete_restores_duplicate_name_owner(self) -> None:
document = item_doc()
cache = CultCache()
cache.register_document_type(document)
cache.put(document, "item:first", Item(name="Potion", category="Consumable", value=50))
cache.put(document, "item:second", Item(name="Potion", category="Consumable", value=60))
cache.put(document, "item:second", Item(name="Potion", category="Rare", value=60))
self.assertEqual(cache.get_key_by_name(document, "Potion"), "item:second")
self.assertEqual(cache.get_key_by_index(document, "category", "Consumable"), "item:second")
cache.delete(document, "item:second")
self.assertEqual(cache.get_key_by_name(document, "Potion"), "item:first")
self.assertEqual(cache.get_key_by_index(document, "category", "Consumable"), "item:first")
self.assertIsNone(cache.get_key_by_index(document, "category", "Rare"))
def test_database_entry_formatter_uses_slot_indexed_messagepack_array(self) -> None:
try:

View file

@ -24,7 +24,6 @@ item_doc = define_document_type(
encode=lambda item: asdict(item),
decode=lambda raw: Item(**raw),
name="name",
indexes={"category": "category"},
)

View file

@ -0,0 +1,464 @@
from __future__ import annotations
import itertools
import shutil
import tempfile
import unittest
from dataclasses import asdict, dataclass
from decimal import Decimal
from pathlib import Path
import msgpack
from cultcache_py import (
CultCache,
CultCacheEnvelope,
SingleFileMessagePackBackingStore,
define_database_entry_type,
define_document_type,
leaves as t,
database_entry_field,
)
from cultcache_py.cache import CultCacheError, _index_text
# variants:ruling:unique-indexes. `indexes` are unique within one type, `roles` are selection aliases. The vectors in
# tests/vectors/unique-index are written by the C# reference; the rest of the behaviour is pinned here.
VECTORS = Path(__file__).resolve().parents[3] / "tests" / "vectors" / "unique-index"
@dataclass
class Item:
sku: object = None
serial: object = None
group: object = None
def refusal(key_a: str, key_b: str, schema_name: str, alias: str) -> str:
return f"Records {key_a} and {key_b} ({schema_name}) both hold unique index '{alias}'; an index value names one record."
def item_doc(type: str = "item", *, indexes=None, roles=None):
return define_document_type(
type,
encode=asdict,
decode=lambda raw: Item(**raw),
indexes={"sku": "sku", "serial": "serial"} if indexes is None else indexes,
roles={"group": "group"} if roles is None else roles,
)
def envelope(document, key: str, item: Item) -> CultCacheEnvelope:
return CultCacheEnvelope.create(key=key, type=document.type, payload=document.encode_payload(item))
def cache_of(*documents, store=None) -> CultCache:
builder = CultCache.builder()
for document in documents:
builder.register_document_type(document)
if store is not None:
builder.add_generic_store(store)
return builder.build()
class RecordingStore:
"""A store that counts what reaches it; the judge must refuse before any store call."""
def __init__(self) -> None:
self.pushed: list[str] = []
def pull_all(self):
return []
def push(self, envelope) -> None:
self.pushed.append(envelope.key)
def push_all(self, envelopes) -> None:
self.pushed.extend(envelope.key for envelope in envelopes)
def delete(self, type, key) -> None:
pass
class IndexTextTests(unittest.TestCase):
def test_canonical_text(self) -> None:
cases = [
("S-1", "S-1"),
(" a ", " a "),
("", ""),
("\x1f", "\x1f"),
("\x1c", "\x1c"),
("\x1d", "\x1d"),
("\x1e", "\x1e"),
("᠎", "᠎"),
("​", "​"),
(True, "True"),
(False, "False"),
(7, "7"),
(-7, "-7"),
(9223372036854775807, "9223372036854775807"),
(18446744073709551615, "18446744073709551615"),
(0.0, "0"),
(-0.0, "0"),
(2.0, "2"),
(-3.0, "-3"),
(1e21, "1000000000000000000000"),
(0.1, "0.1000000000000000055511151231257827021181583404541015625"),
(-0.5, "-0.5"),
(2.5e-07, "0.00000024999999999999998868702795647156467140348468092270195484161376953125"),
]
for value, text in cases:
with self.subTest(value=value):
self.assertEqual(_index_text(value), text)
self.assertIsInstance(_index_text(value), str)
def test_no_value(self) -> None:
blank = ["", " ", "\t", "\n", "\x0b", "\x0c", "\r", "\x85", "\xa0", " ", "
", "
", " ",
" ", " ", "  \x85 ", *map(chr, range(0x2000, 0x200B))]
for value in [None, float("nan"), float("inf"), float("-inf"), *blank]:
with self.subTest(value=repr(value)):
self.assertIsNone(_index_text(value))
def test_blank_set_is_exactly_the_csharp_set(self) -> None:
csharp = {*range(0x09, 0x0E), 0x20, 0x85, 0xA0, 0x1680, *range(0x2000, 0x200B), 0x2028, 0x2029, 0x202F, 0x205F, 0x3000}
self.assertEqual(len(csharp), 25)
got = {code for code in range(0x110000) if 0xD800 > code or code > 0xDFFF if _index_text(chr(code)) is None}
self.assertEqual(got, csharp)
# Python's isspace is the wrong test: it also counts U+001C-001F.
self.assertTrue("\x1f".isspace())
self.assertEqual(_index_text("\x1f"), "\x1f")
def test_floats_render_exactly_not_through_str(self) -> None:
for value in (0.1, 1e-7, 123456789.123456789, 5e-324, 1.7976931348623157e308):
with self.subTest(value=value):
text = _index_text(value)
self.assertNotIn("e", text.lower())
self.assertEqual(Decimal(text), Decimal(value))
self.assertFalse(text.endswith("0") and "." in text)
def test_bool_is_not_an_integer(self) -> None:
self.assertEqual(_index_text(True), "True")
self.assertNotEqual(_index_text(True), "1")
class UniqueIndexTests(unittest.TestCase):
def test_a_write_of_a_held_value_at_another_key_is_refused_by_every_write_path(self) -> None:
for alias in ("sku", "serial"): # two unique indexes: each is judged
other = "serial" if alias == "sku" else "sku"
paths = {
"put": lambda cache, d, k, i: cache.put(d, k, i),
"put_envelope": lambda cache, d, k, i: cache.put_envelope(d, envelope(d, k, i)),
"put_envelopes": lambda cache, d, k, i: cache.put_envelopes(d, [envelope(d, k, i)]),
"update": lambda cache, d, k, i: cache.update(d, k, lambda _held: i),
}
for name, write in paths.items():
with self.subTest(alias=alias, path=name):
document = item_doc()
store = RecordingStore()
cache = cache_of(document, store=store)
cache.put(document, "item:b", Item(sku="CANARY-b", serial="canary-b"))
cache.put(document, "item:a", Item(sku="CANARY-a", serial="canary-a"))
before = cache.snapshot()
pushed = list(store.pushed)
taken = "CANARY-b" if alias == "sku" else "canary-b"
landing = Item(**{alias: taken, other: "fresh"})
with self.assertRaises(CultCacheError) as caught:
write(cache, document, "item:a", landing)
self.assertEqual(str(caught.exception), refusal("item:a", "item:b", "item", alias))
self.assertNotIn("anary", str(caught.exception))
self.assertEqual(cache.snapshot(), before)
self.assertEqual(store.pushed, pushed)
self.assertEqual(cache.get_key_by_index(document, alias, taken), "item:b")
self.assertIsNone(cache.get_key_by_index(document, other, "fresh"))
def test_a_record_rewriting_its_own_key_keeps_its_value(self) -> None:
document = item_doc()
cache = cache_of(document)
cache.put(document, "item:a", Item(sku="S", serial="1", group="g"))
cache.put(document, "item:a", Item(sku="S", serial="2", group="h"))
self.assertEqual(cache.get_key_by_index(document, "sku", "S"), "item:a")
self.assertIsNone(cache.get_key_by_index(document, "serial", "1"))
self.assertEqual(cache.get_key_by_index(document, "serial", "2"), "item:a")
def test_the_two_smallest_keys_are_named_in_ordinal_order(self) -> None:
document = item_doc()
cache = cache_of(document)
cache.put(document, "item:m", Item(sku="S"))
with self.assertRaises(CultCacheError) as caught:
cache.put_envelopes(document, [envelope(document, "item:z", Item(sku="S")), envelope(document, "item:A", Item(sku="S"))])
self.assertEqual(str(caught.exception), refusal("item:A", "item:m", "item", "sku"))
def test_put_envelopes_holding_one_value_twice_is_refused_whole(self) -> None:
document = item_doc()
store = RecordingStore()
cache = cache_of(document, store=store)
with self.assertRaises(CultCacheError) as caught:
cache.put_envelopes(
document,
[
envelope(document, "item:b", Item(sku="S")),
envelope(document, "item:c", Item(sku="T")),
envelope(document, "item:a", Item(sku="S")),
],
)
self.assertEqual(str(caught.exception), refusal("item:a", "item:b", "item", "sku"))
self.assertEqual(cache.snapshot(), {})
self.assertEqual(store.pushed, [])
self.assertIsNone(cache.get_key_by_index(document, "sku", "T"))
def test_verdict_does_not_depend_on_batch_order(self) -> None:
document = item_doc()
rows = [("item:a", Item(sku="S", serial="1")), ("item:b", Item(sku="T", serial="1")), ("item:c", Item(sku="U"))]
for order in itertools.permutations(rows):
with self.subTest(order=[key for key, _ in order]):
cache = cache_of(document)
with self.assertRaises(CultCacheError) as caught:
cache.put_envelopes(document, [envelope(document, key, item) for key, item in order])
self.assertEqual(str(caught.exception), refusal("item:a", "item:b", "item", "serial"))
self.assertEqual(cache.snapshot(), {})
def test_put_envelopes_swapping_two_records_values_lands(self) -> None:
document = item_doc()
cache = cache_of(document)
cache.put(document, "item:a", Item(sku="S", serial="1"))
cache.put(document, "item:b", Item(sku="T", serial="2"))
cache.put_envelopes(
document,
[envelope(document, "item:a", Item(sku="T", serial="2")), envelope(document, "item:b", Item(sku="S", serial="1"))],
)
self.assertEqual(cache.get_key_by_index(document, "sku", "S"), "item:b")
self.assertEqual(cache.get_key_by_index(document, "sku", "T"), "item:a")
self.assertEqual(cache.get_key_by_index(document, "serial", "1"), "item:b")
self.assertEqual(cache.get_key_by_index(document, "serial", "2"), "item:a")
def test_a_batch_may_take_a_value_the_batch_releases(self) -> None:
document = item_doc()
cache = cache_of(document)
cache.put(document, "item:a", Item(sku="S"))
cache.put_envelopes(document, [envelope(document, "item:a", Item(sku="T")), envelope(document, "item:b", Item(sku="S"))])
self.assertEqual(cache.get_key_by_index(document, "sku", "S"), "item:b")
self.assertEqual(cache.get_key_by_index(document, "sku", "T"), "item:a")
def test_delete_frees_the_value(self) -> None:
document = item_doc()
cache = cache_of(document)
cache.put(document, "item:a", Item(sku="S"))
cache.delete(document, "item:a")
self.assertIsNone(cache.get_key_by_index(document, "sku", "S"))
cache.put(document, "item:b", Item(sku="S"))
self.assertEqual(cache.get_key_by_index(document, "sku", "S"), "item:b")
def test_blank_values_never_conflict(self) -> None:
document = item_doc()
cache = cache_of(document)
blanks = ["", " ", "\x85", " ", None]
for number, blank in enumerate(blanks):
cache.put(document, f"item:blank{number}", Item(sku=blank, serial=blank))
cache.put_envelopes(document, [envelope(document, "item:more", Item(sku=" ")), envelope(document, "item:more2", Item(sku=""))])
for blank in ("", " ", "\x85"):
self.assertIsNone(cache.get_key_by_index(document, "sku", blank))
# U+001F and U+FEFF are values.
cache.put(document, "item:us", Item(sku="\x1f"))
cache.put(document, "item:bom", Item(sku=""))
self.assertEqual(cache.get_key_by_index(document, "sku", "\x1f"), "item:us")
self.assertEqual(cache.get_key_by_index(document, "sku", ""), "item:bom")
with self.assertRaises(CultCacheError):
cache.put(document, "item:us2", Item(sku="\x1f"))
def test_values_compare_as_the_canonical_text(self) -> None:
document = item_doc()
cache = cache_of(document)
cache.put(document, "item:a", Item(sku=1))
for same in (1, 1.0, "1"):
with self.subTest(same=same), self.assertRaises(CultCacheError):
cache.put(document, "item:b", Item(sku=same))
cache.put(document, "item:true", Item(sku=True))
self.assertEqual(cache.get_key_by_index(document, "sku", "True"), "item:true")
self.assertEqual(cache.get_key_by_index(document, "sku", "1"), "item:a")
cache.put(document, "item:f", Item(sku=0.1))
self.assertEqual(cache.get_key_by_index(document, "sku", "0.1000000000000000055511151231257827021181583404541015625"), "item:f")
def test_two_types_may_share_an_alias_and_a_value(self) -> None:
one, two = item_doc("one"), item_doc("two")
cache = cache_of(one, two)
cache.put(one, "k:1", Item(sku="S", serial="1"))
cache.put(two, "k:2", Item(sku="S", serial="1"))
self.assertEqual(cache.get_key_by_index(one, "sku", "S"), "k:1")
self.assertEqual(cache.get_key_by_index(two, "sku", "S"), "k:2")
cache.delete(one, "k:1")
self.assertEqual(cache.get_key_by_index(two, "sku", "S"), "k:2")
def test_roles_may_share_values_and_are_not_looked_up(self) -> None:
document = item_doc()
cache = cache_of(document)
cache.put(document, "item:a", Item(sku="1", group="g"))
cache.put(document, "item:b", Item(sku="2", group="g"))
self.assertIsNone(cache.get_key_by_index(document, "group", "g"))
self.assertIsNone(cache.get_by_index(document, "group", "g"))
def test_an_alias_in_both_indexes_and_roles_is_refused(self) -> None:
with self.assertRaisesRegex(ValueError, "both an index and a role"):
define_document_type("bad", indexes={"a": "a"}, roles={"a": "a"})
with self.assertRaisesRegex(ValueError, "both an index and a role"):
define_database_entry_type("bad", [("a", 0)], indexes={"a": "a"}, roles={"a": "a"})
define_document_type("fine", indexes={"a": "a"}, roles={"b": "a"})
def test_register_index_over_held_duplicates_installs_nothing(self) -> None:
document = item_doc(indexes={}, roles={})
cache = cache_of(document)
cache.put(document, "item:b", Item(sku="S", serial="1"))
cache.put(document, "item:a", Item(sku="S", serial="2"))
with self.assertRaises(CultCacheError) as caught:
cache.register_index(document, "sku", "sku")
self.assertEqual(str(caught.exception), refusal("item:a", "item:b", "item", "sku"))
self.assertIsNone(cache.get_key_by_index(document, "sku", "S"))
cache.register_index(document, "serial", "serial") # an installed alias is not poisoned by the refused one
self.assertEqual(cache.get_key_by_index(document, "serial", "2"), "item:a")
# Nothing was installed, so a write sharing the refused alias's value is still admitted.
cache.put(document, "item:c", Item(sku="S", serial="3"))
def test_registration_names_an_installed_alias_a_caller_mutated_into_a_collision(self) -> None:
document = item_doc(indexes={"sku": "sku"}, roles={})
cache = cache_of(document)
cache.put(document, "item:x", Item(sku="1", serial="s1"))
cache.put(document, "item:y", Item(sku="2", serial="s2"))
cache.get_required(document, "item:y").sku = "1" # the cache hands out its held value
with self.assertRaises(CultCacheError) as caught:
cache.register_index(document, "serial", "serial")
self.assertEqual(str(caught.exception), refusal("item:x", "item:y", "item", "sku"))
def test_derive_names_the_ordinal_first_alias_then_value(self) -> None:
document = item_doc(indexes={}, roles={})
cache = cache_of(document)
cache.put(document, "item:1", Item(sku="a", serial="p", group="x"))
cache.put(document, "item:2", Item(sku="a", serial="p", group="x"))
cache.put(document, "item:3", Item(sku="b", serial="q", group="y"))
cache.put(document, "item:4", Item(sku="b", serial="q", group="y"))
with self.assertRaises(CultCacheError) as caught:
cache.register_index(document, "serial", "serial")
self.assertEqual(str(caught.exception), refusal("item:1", "item:2", "item", "serial"))
def test_a_document_in_two_unique_indexes_loads_and_registers_through_every_path(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
path = Path(tmp) / "items.cc"
document = item_doc()
cache = cache_of(document, store=SingleFileMessagePackBackingStore(path))
cache.put(document, "item:a", Item(sku="S", serial="1"))
cache.put(document, "item:b", Item(sku="T", serial="2"))
# a store written by a looser definition, then opened by this one
loose = item_doc(indexes={}, roles={})
writer = cache_of(loose, store=SingleFileMessagePackBackingStore(path))
writer.pull_all_backing_stores()
writer.put(loose, "item:c", Item(sku="U", serial="2"))
before = path.read_bytes()
reader = cache_of(document, store=SingleFileMessagePackBackingStore(path))
with self.assertRaises(CultCacheError) as caught:
reader.pull_all_backing_stores()
self.assertEqual(str(caught.exception), refusal("item:b", "item:c", "item", "serial"))
self.assertEqual(path.read_bytes(), before)
self.assertEqual(reader.snapshot(), {})
def test_a_refused_load_keeps_the_held_state(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
path = Path(tmp) / "items.cc"
loose = item_doc(indexes={}, roles={})
writer = cache_of(loose, store=SingleFileMessagePackBackingStore(path))
writer.put(loose, "item:a", Item(sku="S"))
writer.put(loose, "item:b", Item(sku="S"))
document = item_doc()
cache = cache_of(document, store=SingleFileMessagePackBackingStore(path))
with self.assertRaises(CultCacheError):
cache.pull_all_backing_stores()
self.assertIsNone(cache.get_key_by_index(document, "sku", "S"))
self.assertEqual(cache.get_all(document), [])
class SimulationFactTests(unittest.TestCase):
def test_simulation_fact_catalog_entry_is_unchanged_by_indexes_to_roles(self) -> None:
import re
from cultnet_py.cultmesh_contracts import SIMULATION_FACT_DOCUMENT_TYPE, simulation_fact_document as document
aliases = {"shard": "shard_id", "frame": "frame", "subject": "subject_id", "claim_kind": "claim_kind"}
self.assertEqual(document.roles, aliases)
self.assertEqual(document.indexes, {})
entry = document.catalog_entry()
snake = lambda name: re.sub(r"(?<!^)(?=[A-Z])", "_", name).lower() # noqa: E731
twin = define_database_entry_type(
SIMULATION_FACT_DOCUMENT_TYPE,
[(snake(m.member_name), m.slot) for m in entry.members],
name="fact_id",
indexes=aliases,
schema_name=entry.schema_name,
schema_version=entry.schema_version,
)
self.assertEqual(entry.canonical_schema_json, twin.catalog_entry().canonical_schema_json)
self.assertEqual(entry.content_hash, twin.catalog_entry().content_hash)
self.assertEqual(entry.members, twin.catalog_entry().members)
self.assertEqual({m.index_alias for m in entry.members if m.index_alias}, set(aliases))
def test_facts_sharing_every_role_value_are_all_admitted(self) -> None:
from cultnet_py.cultmesh_contracts import CultMeshSimulationFact, simulation_fact_document as document
cache = cache_of(document)
for number in range(3):
fact = CultMeshSimulationFact.from_candidate(
{"shardId": "s", "shardEpoch": 1, "frame": 4, "subjectId": "x", "claimKind": "k", "claimHash": f"h{number}",
"witnessCount": 1, "supportWeight": 1.0, "totalWeight": 1.0, "confidence": 1.0}
)
cache.put(document, f"simulation:{number}", fact)
self.assertEqual(len(cache.get_all(document)), 3)
self.assertIsNone(cache.get_key_by_index(document, "shard", "s"))
class VectorTests(unittest.TestCase):
def test_values_rows(self) -> None:
document = item_doc(indexes={"sku": "sku"}, roles={})
for label, kind, value, text in msgpack.unpackb((VECTORS / "values.msgpack").read_bytes(), raw=False):
with self.subTest(label=label):
self.assertEqual(_index_text(value), text)
cache = cache_of(document)
cache.put(document, "item:1", Item(sku=value))
if text is None:
cache.put(document, "item:2", Item(sku=value)) # blank: two records commit
self.assertEqual(len(cache.get_all(document)), 2)
else:
self.assertEqual(cache.get_key_by_index(document, "sku", text), "item:1")
with self.assertRaises(CultCacheError):
cache.put(document, "item:2", Item(sku=value))
def test_store_verdicts(self) -> None:
unit = define_database_entry_type
outcomes = msgpack.unpackb((VECTORS / "outcomes.msgpack").read_bytes(), raw=False)
self.assertEqual(len(outcomes), 3)
for name, alias, verdict, key_a, key_b in outcomes:
for roles_first in (False, True): # the verdict does not depend on declaration order
with self.subTest(file=name, alias=alias, roles_first=roles_first), tempfile.TemporaryDirectory() as tmp:
path = Path(tmp) / name
shutil.copyfile(VECTORS / name, path)
before = path.read_bytes()
fields = [database_entry_field("sku", 0, codec=t.string), database_entry_field("category", 1, codec=t.string)]
declared = {"indexes": {alias: "sku"} if alias else {}, "roles": {} if alias else {"Sku": "sku"}}
if roles_first:
declared = dict(reversed(list(declared.items())))
document = unit(
"tests.unique_item",
fields,
schema_id="sha256:93bb890de83069061af696b23f94fd32fb3c2dad3c91ee645e8add56085f1882",
schema_name="tests.unique_item",
**declared,
)
cache = cache_of(document, store=SingleFileMessagePackBackingStore(path))
if verdict == "open":
cache.pull_all_backing_stores()
self.assertGreaterEqual(len(cache.get_all(document)), 2)
else:
with self.assertRaises(CultCacheError) as caught:
cache.pull_all_backing_stores()
self.assertEqual(str(caught.exception), refusal(key_a, key_b, "tests.unique_item", alias))
self.assertNotIn("S-1", str(caught.exception))
self.assertEqual(cache.snapshot(), {})
self.assertEqual(path.read_bytes(), before)
if __name__ == "__main__":
unittest.main()

View file

@ -692,7 +692,7 @@ simulation_fact_document: DocumentDefinition[CultMeshSimulationFact] = define_da
],
cls=CultMeshSimulationFact,
name="fact_id",
indexes={"shard": "shard_id", "frame": "frame", "subject": "subject_id", "claim_kind": "claim_kind"},
roles={"shard": "shard_id", "frame": "frame", "subject": "subject_id", "claim_kind": "claim_kind"},
schema_name=SIMULATION_FACT_DOCUMENT_TYPE,
schema_version=SIMULATION_FACT_SCHEMA_VERSION,
)