cultcache-py: update tests, first-call snippet, docs, changelog (update-with-retry-py)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Libby 2026-11-13 20:09:50 +00:00
commit 1b4f8ffd77
6 changed files with 731 additions and 6 deletions

View file

@ -19,7 +19,7 @@ this page.
| **C#** | [first call, tested](#c-first-call); [update, tested](#c-update-with-retry) | [package](../../packages/cultmath/README.md); snippet not yet | [package](../../src/GameCult.Geometry/README.md); snippet not yet | [first call, tested](#c-cultnet-first-call); [package](../../src/GameCult.Networking) | [quickstart](../../src/GameCult.Mesh/docs/durable-node-quickstart.md); run by `scripts/verify-eve-getting-started.ps1` |
| **TypeScript** | [first call, tested](#typescript-first-call); [update, tested](#typescript-update-with-retry) | not yet | not yet | [first call, tested](#typescript-cultnet-first-call); [package](../../packages/cultnet-ts/README.md) | [package](../../packages/cultmesh-ts/README.md); snippet not yet; [recipe: two-client .io game](io-game.md), tested with two live clients, browser page not yet run |
| **Rust** | [first call, tested](#rust-first-call); [update, tested](#rust-update-with-retry) | [native core](../../packages/cultmath/native/cultmath-core); snippet not yet | not yet | [first call, tested](#rust-cultnet-first-call); [package](../../packages/cultnet-rs/README.md), [examples](../../packages/cultnet-rs/examples) | [package](../../packages/cultmesh-rs); snippet not yet |
| **Python** | [first call, tested](#python-first-call) | not yet | not yet | [first call, tested](#python-cultnet-first-call); [package](../../packages/cultnet-py/README.md) | [package](../../packages/cultmesh-py/README.md); snippet not yet |
| **Python** | [first call, tested](#python-first-call); [update, tested](#python-update-with-retry) | not yet | not yet | [first call, tested](#python-cultnet-first-call); [package](../../packages/cultnet-py/README.md) | [package](../../packages/cultmesh-py/README.md); snippet not yet |
For the whole
picture of what each piece is for, see the
@ -305,6 +305,29 @@ def first_call() -> Item:
return reopened.get_required(item_doc, "item:potion")
```
## Python update with retry
`update` changes a record from what the store holds now and commits only if it is still that record. The source is the
`update` region of
[`packages/cultcache-py/tests/test_first_call.py`](../../packages/cultcache-py/tests/test_first_call.py), which uses the
definitions of the first call above; a test in that file fails if the block stops matching the file.
```python
def restock(path: Path):
cache = (
CultCache.builder()
.register_document_type(item_doc)
.add_generic_store(SingleFileMessagePackBackingStore(path))
.build()
)
cache.pull_all_backing_stores()
return cache.update(item_doc, "item:potion", lambda item: Item("Potion", "Consumable", (item.value if item else 0) + 10))
```
The write is durable when `update` returns. A lost race re-runs the function, so keep it free of side effects; when the
attempts run out the result's `outcome` is `UpdateOutcome.LOST_RACE` instead of a raise. `put` is the blind opt-out: last
writer wins.
## CultNet first calls
CultNet is schema-v0 messages in length-prefixed frames. The first call in each runtime is the same

View file

@ -4,10 +4,12 @@
### Added
- `CultCache.update` and `update_global` return `UpdateResult(outcome, document)`; `UpdateOutcome` is `UPDATED`, `DECLINED` or `LOST_RACE`. The single-file store gains `read(key)` (the record and its content version, the SHA-256 C# computes) and `commit_if(envelope, expected)`, both looked up by the cache and not `BackingStore` protocol members (variants:cut_spec:cut-update-with-retry-py).
- `read_payload_tree(type_name, payload, decoder=None)`: the exported owner of the one-value, depth-first payload read that `decode_payload` uses, for payloads read outside a definition (`cultnet-py`'s witness bundle decode needs it, so `cultnet-py` now requires `cultcache-py>=0.4.0`). The manifest version is 0.4.0, the minor these Unreleased breaking changes imply, set in this change so the workspace install resolves the new bound; the `## Unreleased` heading becomes `## [0.4.0]` when it is tagged.
### Changed
- A write that replaces a record is stored at a later `stored_at` than the record it replaces, in the single-file store, the JSONL store and a store-less cache, with no exception for identical content (`variants:ruling:py-rs-update-follow-csharp-restamp`); the cache holds the stamp the store wrote.
- `store_write_lock` looks for a link at the store path again once the lock is held, so a link put at the path while a writer waited on the lock is refused with `StoreLinkedError` and the lock released, instead of being replaced. `JsonLinesBackingStore` takes no cross-process lock (only its thread lock, under which it already refuses a link), so it has no lock wait to re-check across. Pinned by a test that puts the link while a foreign process holds the lock.
- `CultCache.is_writable` does not look at store paths: a link at a store path is not predicted, and the write refuses it with `StoreLinkedError`. Pinned by a test through a cache attached to a linked store.
- `JsonLinesBackingStore` writes LF line endings on Windows. It wrote CRLF there, through `write_text` newline translation, before the commit_store_file change (#154, adbbb780).
@ -17,6 +19,8 @@
### Breaking
- `update(document, key, apply, *, max_attempts=5)` and `update_global(document, apply, *, max_attempts=5)` replace the blind pair. Each attempt reads the record from the home store, runs `apply` on a copy (None when absent) outside every lock, and commits only if the record is unchanged, running `apply` again on a lost race; it returns `UpdateResult`, not the value. `apply` returning None writes nothing; a lost race that outlasts `max_attempts` returns `LOST_RACE`; a store without `read` and `commit_if` (JSON lines among them) refuses before `apply`. `put` is the blind write.
- `push` returns the `stored_at` it wrote and `push_all` a dict of them by key (`{}` when a soft push skipped); a store returning None keeps the cache's own stamp.
- `push_all(envelopes, soft=False)` on the single-file and JSONL stores and on the `BackingStore` protocol takes the soft flag, as TypeScript, Rust and C# do: a soft push writes only where nothing is at the path; over a store the reader reads, empty included, it returns without writing, and over a file the reader cannot read it refuses (`StoreUnreadableError` for the single-file store). The `BackingStore` protocol has `pull_or_absent()`: the envelopes the reader reads, `None` when nothing is at the path, a raise when the path cannot be read; its default is `pull_all()`, which cannot say absent, so a store that inherits it is always read. The default `push_all` decides soft by it (writes where it says absent, skips over a store it read, refuses where it raises), and the single-file and JSONL stores expose their pull through it; an inheriting store that can tell nothing-at-the-path overrides `pull_or_absent`, not `push_all`. Before, a soft push through the default always skipped. A store kind that overrides `push_all(envelopes)` without the keyword fails when a caller passes `soft`.
- A write at a store path that is a link, live or dangling, is refused with the new `StoreLinkedError` (an `OSError`, exported from `cultcache_py`; `.path` names the path) before the lock and the replace, for the single-file store and for `JsonLinesBackingStore` (`push`, `push_all`, `delete`), and nothing is written. Before, a write at a live link replaced the link with a regular file, and at a dangling link both stores raised `FileNotFoundError`. A caller catching `FileNotFoundError` there now catches `StoreLinkedError` (`OSError`); configure the link's target as the store path. On Windows a junction or any reparse point counts as a link. Reads through a live link work.
- A record whose payload carries a slot past its reader's declared slots, at any depth (the document, a nested record, a list element, a union arm, a unit arm), is read, held read-only, and every write or removal of its key (`put`, `put_global`, `update`, `put_envelope`, `put_envelopes`, `delete`, `delete_global`) is refused with `SchemaConflictError` before any store is touched (`cultlib-gaps:ruling:extra-trailing-slots`). Reading dropped those slots silently and the next write shed them. `SchemaConflictError.unread_slots` is an `UnreadSlots(type_name, slot_count, declared_count, occurrences, depth0_only)` and the message names the type and both counts, never a slot value. A depth-0 mark is consented, and the record writable, only when the record is stored under an id the document lists in `compatible_schema_ids` and that id's catalog entry has slots past the document's, covering the mark's slot count (`cultlib-gaps:ruling:slot-mark-consent`); a nested mark, or a mark under the document's own id, keeps the record read-only whatever is declared. Missing trailing slots keep their declared defaults. An untouched marked record keeps its stored bytes through every other write. Added `CultCache.is_writable(document, key)` (advisory; the write still decides: it also answers False for a foreign-carried key, for a key held as a type the document does not own, and for a global document at a non-global key; `delete` asks the same judges, so removing a key held as another type is refused with `SchemaConflictError` unless the document owns that record's schema id (its own or one it lists compatible), in which case the removal removes that record, its file row and its lookups, as a write of the key would replace it; it returned silently and removed nothing), `DocumentDefinition.decode_payload_marked`, and `UnreadSlots`. A document with a custom `decode` (no slot codec) never marks. A store holding a variant record is still unreadable here. Implies a minor bump while the version is 0.x (0.3.0 to 0.4.0): code that wrote back a record from a newer writer now gets `SchemaConflictError`.

View file

@ -94,8 +94,7 @@ store.
- `put(...)`
- `put_envelopes(...)`
- `put_global(...)`
- `update(...)`
- `update_global(...)`
- `update(document, key, apply, *, max_attempts=5)` and `update_global(document, apply, *, max_attempts=5)`: the conditional golden path; reads the record as its home store holds it now, runs `apply` on a fresh copy, writes only if the record is unchanged and runs `apply` again on a lost race; returns `UpdateResult(outcome, document)` with `UpdateOutcome.UPDATED`, `DECLINED` (`apply` returned `None`) or `LOST_RACE` (see below)
- `delete(...)`
- `delete_global(...)`
- `is_writable(document, key)` (advisory: would a write or removal of the key be refused; see below)
@ -143,6 +142,8 @@ used by the TypeScript, Rust, and C# runtimes:
canonicalSchemaJson, compatibleSchemaIds, members]`
- records: `[key, schemaId, storedAt, payload]`
`update` needs a home store with the optional `read(key)` (the record at the key, of any type, and its content version, from one read) and `commit_if(envelope, expected)` (write only if the record at the key is still that version, under the store's write lock). The content version is the lowercase hex SHA-256 of the record as the store writes it, `[key, schemaId, storedAt, payload]`, the same value C# computes (`tests/vectors/content-version`). The single-file store has both; `JsonLinesBackingStore` and a store without them refuse `update` before `apply` runs. A cache with no store copies its held instance. Every write that replaces a record is stored at a later `stored_at` than the one it replaces, in each store kind and in a store-less cache; `push` and `push_all` return the stamp written, and the cache holds that stamp.
Large corpora should use a sharded store or a real database. A single snapshot file is a scalpel, not a forklift.
## Store format and rollback

View file

@ -663,6 +663,7 @@ class CultCache:
self._state.names[(type, keys.name)] = envelope.key
for index, index_value in keys.indexes.items():
self._state.indexes[(type, index, index_value)] = envelope.key
return "committed"
def _refuse_retype(self, document: DocumentDefinition[Any], written: list[CultCacheEnvelope]) -> list[tuple[str, str]]:
"""The one key-identity judge (variants:ruling:cache-refuses-retype): a key holds one record whatever its type, so a write

View file

@ -4,6 +4,8 @@
import unittest
from pathlib import Path
from cultcache_py import UpdateOutcome
# snippet:begin
import tempfile
from dataclasses import asdict, dataclass
@ -51,15 +53,44 @@ def first_call() -> Item:
# snippet:end
# snippet:update:begin
def restock(path: Path):
cache = (
CultCache.builder()
.register_document_type(item_doc)
.add_generic_store(SingleFileMessagePackBackingStore(path))
.build()
)
cache.pull_all_backing_stores()
return cache.update(item_doc, "item:potion", lambda item: Item("Potion", "Consumable", (item.value if item else 0) + 10))
# snippet:update:end
class FirstCallTests(unittest.TestCase):
def test_first_call_round_trips_through_a_second_cache(self) -> None:
self.assertEqual(first_call(), Item("Potion", "Consumable", 50))
def test_docs_show_the_tested_snippet(self) -> None:
def test_update_with_retry_through_a_second_cache(self) -> None:
path = Path(tempfile.mkdtemp()) / "restock.cc"
restock(path)
result = restock(path)
self.assertIs(result.outcome, UpdateOutcome.UPDATED)
reopened = (
CultCache.builder()
.register_document_type(item_doc)
.add_generic_store(SingleFileMessagePackBackingStore(path))
.build()
)
reopened.pull_all_backing_stores()
self.assertEqual(reopened.get_required(item_doc, "item:potion"), Item("Potion", "Consumable", 20))
def test_docs_show_the_tested_snippets(self) -> None:
source = Path(__file__).read_text(encoding="utf-8")
snippet = source.split("# snippet:begin\n", 1)[1].split("# snippet:end\n", 1)[0]
docs = (Path(__file__).parents[3] / "docs" / "start" / "README.md").read_text(encoding="utf-8")
self.assertIn("```python\n" + snippet + "```", docs)
for region in ("snippet", "snippet:update"):
with self.subTest(region=region):
snippet = source.split(f"# {region}:begin\n", 1)[1].split(f"# {region}:end\n", 1)[0]
self.assertIn("```python\n" + snippet + "```", docs)
if __name__ == "__main__":

View file

@ -0,0 +1,665 @@
"""CultCache.update (variants:cut_spec:cut-update-with-retry-py): read the record as its home store holds it, apply a function,
commit only if the record is unchanged, run the function again on a lost race. Mirrors C# UpdateTests and the TS update tests."""
from __future__ import annotations
import re
import shutil
import tempfile
import threading
import time
import unittest
from dataclasses import asdict, dataclass
from datetime import datetime, timezone
from pathlib import Path
from unittest import mock
from cultcache_py import (
CultCache,
CultCacheEnvelope,
JsonLinesBackingStore,
SchemaConflictError,
SingleFileMessagePackBackingStore,
UpdateOutcome,
UpdateResult,
database_entry_field,
define_database_entry_type,
define_document_type,
leaves as t,
)
from cultcache_py.cache import GLOBAL_KEY, CultCacheError
from cultcache_py.stores import _parse_stored_at, mint_stored_at, record_version, stored_at_over
@dataclass
class Item:
name: str = "n"
value: int = 0
sku: str | None = None
def _doc(type: str, **extra):
return define_document_type(type, encode=asdict, decode=lambda raw: Item(**raw), **extra)
item_doc = _doc("item")
other_doc = _doc("other")
sku_doc = _doc("skuitem", indexes={"sku": "sku"})
global_doc = _doc("gitem", global_document=True)
@dataclass
class Wide:
name: str = ""
extra: str = ""
@dataclass
class Narrow:
name: str = ""
ALPHA = "2099-01-01T00:00:00.0000000Z"
class Fixture(unittest.TestCase):
def setUp(self) -> None:
self.dir = Path(tempfile.mkdtemp())
self.addCleanup(shutil.rmtree, self.dir, ignore_errors=True)
self.path = self.dir / "state.cc"
def cache(self, *documents, path: Path | None = None, store=True) -> CultCache:
builder = CultCache.builder()
for document in documents or (item_doc,):
builder.register_document_type(document)
if store:
builder.add_generic_store(SingleFileMessagePackBackingStore(path or self.path))
cache = builder.build()
if store:
cache.pull_all_backing_stores()
return cache
def stored(self, key: str = "k", document=item_doc):
"""What a fresh cache reads at the key."""
return self.cache(document).get(document, key)
def bytes(self) -> bytes | None:
return self.path.read_bytes() if self.path.exists() else None
class Counting:
"""An apply that records what it was given."""
def __init__(self, fn=None) -> None:
self.seen: list = []
self.fn = fn or (lambda item, calls: Item("n", (item.value if item else 0) + 1))
def __call__(self, item):
self.seen.append(None if item is None else item.value)
# An always-losing apply must fail by assertion when the attempt bound is gone, not hang.
assert len(self.seen) <= 10, "apply ran past the attempt bound"
return self.fn(item, len(self.seen))
@property
def calls(self) -> int:
return len(self.seen)
class UpdateTests(Fixture):
def test_two_writers_updating_lose_no_increment(self) -> None:
results: list[UpdateResult] = []
calls = [0]
lock = threading.Lock()
a, b = self.cache(), self.cache()
a.put(item_doc, "k", Item("n", 0))
def bump(item):
with lock:
calls[0] += 1
time.sleep(0)
return Item("n", item.value + 1)
def run(cache) -> None:
for _ in range(50):
results.append(cache.update(item_doc, "k", bump, max_attempts=200))
threads = [threading.Thread(target=run, args=(cache,)) for cache in (a, b)]
for thread in threads:
thread.start()
for thread in threads:
thread.join()
self.assertTrue(all(result.outcome is UpdateOutcome.UPDATED for result in results))
self.assertEqual(self.stored().value, 100)
self.assertGreater(calls[0], 100)
def test_a_lost_attempt_reruns_apply_on_the_fresh_value(self) -> None:
cache, rival = self.cache(), self.cache()
cache.put(item_doc, "k", Item("n", 1))
rival.pull_all_backing_stores()
apply = Counting(lambda item, calls: (rival.put(item_doc, "k", Item("n", 100)) if calls == 1 else None) or Item("n", item.value + 1))
result = cache.update(item_doc, "k", apply)
self.assertEqual(apply.seen, [1, 100])
self.assertIs(result.outcome, UpdateOutcome.UPDATED)
self.assertEqual(result.document.value, 101)
self.assertEqual(self.stored().value, 101)
def test_exhausted_attempts_return_lost_race(self) -> None:
cache, rival = self.cache(), self.cache()
cache.put(item_doc, "k", Item("n", 1))
rival.pull_all_backing_stores()
apply = Counting(lambda item, calls: rival.put(item_doc, "k", Item("n", 100 * calls)) or Item("n", -1))
result = cache.update(item_doc, "k", apply, max_attempts=3)
self.assertEqual(apply.calls, 3)
self.assertIs(result.outcome, UpdateOutcome.LOST_RACE)
self.assertEqual(result.document.value, 300) # the pull before LOST_RACE shows the winner
self.assertEqual(self.stored().value, 300)
def test_max_attempts_bounds(self) -> None:
cache, rival = self.cache(), self.cache()
cache.put(item_doc, "k", Item("n", 1))
rival.pull_all_backing_stores()
def always_losing(item, calls):
rival.put(item_doc, "k", Item("n", 100 + calls))
return Item("n", -1)
apply = Counting(always_losing)
result = cache.update(item_doc, "k", apply, max_attempts=1)
self.assertEqual((apply.calls, result.outcome), (1, UpdateOutcome.LOST_RACE))
for bad, error in ((0, ValueError), (-1, ValueError), (True, TypeError), (1.5, TypeError), (None, TypeError)):
with self.subTest(max_attempts=bad):
untouched = Counting()
with self.assertRaises(error):
cache.update(item_doc, "k", untouched, max_attempts=bad)
self.assertEqual(untouched.calls, 0)
with self.assertRaises(TypeError):
cache.update(item_doc, "k", "not callable")
with self.assertRaises(TypeError):
cache.update(item_doc, "k", None)
def test_apply_gets_a_fresh_copy(self) -> None:
cache, rival = self.cache(), self.cache()
cache.put(item_doc, "k", Item("n", 1))
rival.pull_all_backing_stores()
arguments: list[Item] = []
def apply(item):
arguments.append(item)
self.assertIsNot(item, cache.get(item_doc, "k"))
if len(arguments) == 1:
item.value = 777 # changing the copy in place changes nothing the cache or the next attempt sees
rival.put(item_doc, "k", Item("n", 5))
self.assertEqual(cache.get(item_doc, "k").value, 1)
return Item("n", item.value + 1)
result = cache.update(item_doc, "k", apply)
self.assertEqual([item.value for item in arguments], [777, 5])
self.assertIsNot(arguments[0], arguments[1])
self.assertEqual(result.document.value, 6)
def test_an_in_place_change_to_the_held_value_is_neither_applied_nor_persisted(self) -> None:
cache = self.cache()
cache.put(item_doc, "k", Item("n", 1))
cache.get(item_doc, "k").value = 999
apply = Counting(lambda item, calls: Item("n", item.value + 1))
result = cache.update(item_doc, "k", apply)
self.assertEqual(apply.seen, [1])
self.assertEqual(self.stored().value, 2)
result.document.value = 999
cache.update(item_doc, "k", apply)
self.assertEqual(apply.seen, [1, 2])
self.assertEqual(self.stored().value, 3)
bare = self.cache(store=False) # a cache with no store copies its held instance
bare.put(item_doc, "k", Item("n", 1))
bare.get(item_doc, "k").value = 999
seen = Counting(lambda item, calls: Item("n", item.value + 1))
self.assertEqual(bare.update(item_doc, "k", seen).document.value, 2)
self.assertEqual(seen.seen, [1])
def test_none_declines(self) -> None:
cache = self.cache()
cache.put(item_doc, "k", Item("n", 1))
before = self.bytes()
result = cache.update(item_doc, "k", lambda item: None)
self.assertIs(result.outcome, UpdateOutcome.DECLINED)
self.assertEqual(result.document, cache.get(item_doc, "k"))
self.assertEqual(self.bytes(), before)
absent = cache.update(item_doc, "missing", lambda item: None)
self.assertEqual((absent.outcome, absent.document), (UpdateOutcome.DECLINED, None))
self.assertEqual(self.bytes(), before)
empty = Fixture.cache(self, path=self.dir / "empty.cc")
self.assertEqual(empty.update(item_doc, "k", lambda item: None).outcome, UpdateOutcome.DECLINED)
self.assertFalse((self.dir / "empty.cc").exists())
def test_a_declined_or_lost_attempt_installs_nothing(self) -> None:
cache, rival = self.cache(), self.cache()
cache.put(item_doc, "k", Item("n", 1))
rival.pull_all_backing_stores()
rival.put(item_doc, "k", Item("n", 5))
self.assertIs(cache.update(item_doc, "k", lambda item: None).outcome, UpdateOutcome.DECLINED)
self.assertEqual(cache.get(item_doc, "k").value, 1)
cache.pull_all_backing_stores()
self.assertEqual(cache.get(item_doc, "k").value, 5)
during: list[int] = []
def apply(item, calls):
during.append(cache.get(item_doc, "k").value)
if calls == 1:
rival.put(item_doc, "k", Item("n", 9))
return Item("n", item.value + 1)
cache.update(item_doc, "k", Counting(apply))
self.assertEqual(during, [5, 5]) # the lost attempt published nothing mid-update
def test_absent_key_rival_removal_and_a_store_that_holds_nothing(self) -> None:
a, b = self.cache(), self.cache()
bump = lambda item: Item("n", (item.value if item else 0) + 1) # noqa: E731
self.assertIs(a.update(item_doc, "k", bump).outcome, UpdateOutcome.UPDATED)
self.assertIs(b.update(item_doc, "k", bump).outcome, UpdateOutcome.UPDATED)
self.assertEqual(self.stored().value, 2)
cache, rival = self.cache(), self.cache()
cache.put(item_doc, "d", Item("n", 1))
rival.pull_all_backing_stores()
apply = Counting(lambda item, calls: (rival.delete(item_doc, "d") if calls == 1 else None) or Item("n", (item.value if item else 0) + 1))
cache.update(item_doc, "d", apply)
self.assertEqual(apply.seen, [1, None])
self.assertEqual(self.stored("d").value, 1)
holder = self.cache()
holder.put(item_doc, "h", Item("n", 3))
rival.pull_all_backing_stores()
rival.delete(item_doc, "h")
nothing = Counting(lambda item, calls: Item("n", 1))
holder.update(item_doc, "h", nothing)
self.assertEqual(nothing.seen, [None])
def test_a_rival_writing_and_restoring_the_value_still_reruns_apply(self) -> None:
cache, rival = self.cache(), self.cache()
cache.put(item_doc, "k", Item("n", 1))
rival.pull_all_backing_stores()
def apply(item, calls):
if calls == 1:
rival.put(item_doc, "k", Item("n", 5))
rival.put(item_doc, "k", Item("n", 1))
return Item("n", item.value + 1)
counting = Counting(apply)
cache.update(item_doc, "k", counting)
self.assertEqual(counting.seen, [1, 1])
self.assertEqual(self.stored().value, 2)
def test_a_stale_caches_first_attempt_reads_the_store(self) -> None:
a, b = self.cache(), self.cache()
a.put(item_doc, "k", Item("n", 1))
b.pull_all_backing_stores()
a.put(item_doc, "k", Item("n", 7))
apply = Counting(lambda item, calls: Item("n", item.value + 1))
b.update(item_doc, "k", apply)
self.assertEqual(apply.seen, [7])
self.assertEqual(self.stored().value, 8)
class UpdateRefusalTests(Fixture):
def refused(self, cache: CultCache, key: str = "k", document=item_doc, *errors) -> None:
before = self.bytes()
apply = Counting()
with self.assertRaises(errors or (CultCacheError, SchemaConflictError)):
cache.update(document, key, apply)
self.assertEqual(apply.calls, 0)
self.assertEqual(self.bytes(), before)
def test_refusals_before_apply(self) -> None:
writer = self.cache(item_doc, other_doc)
writer.put(other_doc, "k", Item("n", 1))
self.refused(writer) # the record at the key is another document's, held
self.refused(self.fresh(item_doc, other_doc)) # unpulled: the store's record is another document's
self.refused(self.cache(item_doc)) # foreign: the store holds a type this cache never registered
self.refused(self.fresh(item_doc))
storeless = self.cache(item_doc, other_doc, store=False)
storeless.put(other_doc, "k", Item("n", 1))
apply = Counting()
with self.assertRaises((CultCacheError, SchemaConflictError)):
storeless.update(item_doc, "k", apply)
self.assertEqual(apply.calls, 0)
def fresh(self, *documents) -> CultCache:
"""A cache that has not pulled: whatever it reads, it reads from the store at update."""
builder = CultCache.builder()
for document in documents:
builder.register_document_type(document)
return builder.add_generic_store(SingleFileMessagePackBackingStore(self.path)).build()
def test_a_home_without_read_and_commit_if_refuses(self) -> None:
jsonl = JsonLinesBackingStore(self.dir / "state.jsonl")
cache = CultCache.builder().register_document_type(item_doc).add_generic_store(jsonl).build()
cache.pull_all_backing_stores()
cache.put(item_doc, "k", Item("n", 1))
before = (self.dir / "state.jsonl").read_bytes()
apply = Counting()
with self.assertRaisesRegex(CultCacheError, "cannot write conditionally"):
cache.update(item_doc, "k", apply)
self.assertEqual(apply.calls, 0)
self.assertEqual((self.dir / "state.jsonl").read_bytes(), before)
self.assertFalse(hasattr(JsonLinesBackingStore, "read") or hasattr(JsonLinesBackingStore, "commit_if"))
class Bare:
def pull_all(self):
return []
def push(self, envelope):
raise AssertionError("update wrote through a store that cannot write conditionally")
def push_all(self, envelopes, soft=False):
raise AssertionError("push_all")
def delete(self, type, key):
raise AssertionError("delete")
bare = CultCache.builder().register_document_type(item_doc).add_generic_store(Bare()).build()
with self.assertRaisesRegex(CultCacheError, "Bare"):
bare.update(item_doc, "k", apply)
self.assertEqual(apply.calls, 0)
def test_read_only_records_refuse_before_apply(self) -> None:
wide = define_database_entry_type(
"tests.rw",
[database_entry_field("name", 0, codec=t.string), database_entry_field("extra", 1, codec=t.string)],
cls=Wide,
schema_id="tests.rw.v1",
)
narrow = define_database_entry_type("tests.rw", [database_entry_field("name", 0, codec=t.string)], cls=Narrow, schema_id="tests.rw.v1")
self.cache(wide).put(wide, "k", Wide("n", "x"))
before = self.bytes()
for pulled in (True, False): # held read-only, then a store record the cache never loaded
with self.subTest(pulled=pulled):
cache = self.cache(narrow) if pulled else self.fresh(narrow)
apply = Counting()
with self.assertRaises(SchemaConflictError) as caught:
cache.update(narrow, "k", apply)
self.assertEqual(caught.exception.record_key, "k")
self.assertEqual(apply.calls, 0)
self.assertEqual(self.bytes(), before)
def test_a_non_global_key_for_a_global_document_refuses(self) -> None:
cache = self.cache(global_doc)
apply = Counting()
with self.assertRaises(CultCacheError):
cache.update(global_doc, "not-global", apply)
self.assertEqual(apply.calls, 0)
def test_refusals_after_apply_are_not_retried(self) -> None:
cache = self.cache()
cache.put(item_doc, "k", Item("n", 1))
before = self.bytes()
boom = Counting(lambda item, calls: (_ for _ in ()).throw(RuntimeError("boom")))
with self.assertRaisesRegex(RuntimeError, "boom"):
cache.update(item_doc, "k", boom)
self.assertEqual(boom.calls, 1)
self.assertEqual(self.bytes(), before)
indexed = self.cache(sku_doc, path=self.dir / "sku.cc")
indexed.put(sku_doc, "a", Item("n", 1, "X"))
indexed.put(sku_doc, "b", Item("n", 1, "Y"))
before = (self.dir / "sku.cc").read_bytes()
clash = Counting(lambda item, calls: Item("n", 2, "X"))
with self.assertRaisesRegex(CultCacheError, "unique index"):
indexed.update(sku_doc, "b", clash)
self.assertEqual(clash.calls, 1)
self.assertEqual((self.dir / "sku.cc").read_bytes(), before)
class UpdateOwnWritesTests(Fixture):
def test_own_writes_and_the_held_instance(self) -> None:
cache = self.cache()
cache.put(item_doc, "k", Item("n", 1))
apply = Counting(lambda item, calls: Item("n", item.value + 1))
cache.update(item_doc, "k", apply)
second = cache.update(item_doc, "k", apply)
self.assertEqual(apply.calls, 2)
self.assertIs(second.document, cache.get(item_doc, "k"))
self.assertEqual(second.document.value, 3)
def sideways(item, calls):
cache.put(item_doc, "side", Item("n", 5)) # apply runs outside the cache's hold
return Item("n", item.value + 1)
sideways_apply = Counting(sideways)
result = cache.update(item_doc, "k", sideways_apply)
self.assertEqual((result.outcome, sideways_apply.calls), (UpdateOutcome.UPDATED, 1))
self.assertEqual(self.stored("side").value, 5)
def test_store_less_cache(self) -> None:
cache = self.cache(store=False)
cache.put(item_doc, "k", Item("n", 0))
results: list[UpdateResult] = []
def bump(item):
time.sleep(0)
return Item("n", item.value + 1)
def run() -> None:
for _ in range(50):
results.append(cache.update(item_doc, "k", bump, max_attempts=500))
threads = [threading.Thread(target=run) for _ in range(2)]
for thread in threads:
thread.start()
for thread in threads:
thread.join()
self.assertTrue(all(result.outcome is UpdateOutcome.UPDATED for result in results))
self.assertEqual(cache.get(item_doc, "k").value, 100)
# A store that becomes the type's home while apply runs holds a record the cache never read: the commit loses.
cache = self.cache(store=False) # holds no record of the type, so a store may become its home
other = self.cache(path=self.dir / "late.cc")
other.put(item_doc, "fresh", Item("n", 40))
late = SingleFileMessagePackBackingStore(self.dir / "late.cc")
def attach(item, calls):
if calls == 1:
cache.add_generic_store(late)
return Item("n", (item.value if item else 0) + 1)
apply = Counting(attach)
result = cache.update(item_doc, "fresh", apply)
self.assertEqual(apply.seen, [None, 40])
self.assertIs(result.outcome, UpdateOutcome.UPDATED)
self.assertEqual(self.cache(path=self.dir / "late.cc").get(item_doc, "fresh").value, 41)
def test_ledger_under_contention(self) -> None:
caches = [self.cache() for _ in range(3)]
outcomes: list[UpdateOutcome] = []
def run(cache) -> None:
for _ in range(25):
outcomes.append(cache.update(item_doc, "k", lambda item: Item("n", (item.value if item else 0) + 1)).outcome)
threads = [threading.Thread(target=run, args=(cache,)) for cache in caches]
for thread in threads:
thread.start()
for thread in threads:
thread.join()
self.assertEqual(self.stored().value, outcomes.count(UpdateOutcome.UPDATED))
self.assertEqual(len(outcomes), 75)
class UpdateLegacyGlobalTests(Fixture):
def plant(self, key: str = "legacy") -> None:
entry = global_doc.catalog_entry()
SingleFileMessagePackBackingStore(self.path).push(
CultCacheEnvelope.create(key=key, type=global_doc.type, payload=global_doc.encode_payload(Item("n", 1)), schema_id=entry.schema_id, catalog_entry=entry)
)
def test_update_global_updates_the_global_record(self) -> None:
cache = self.cache(global_doc)
result = cache.update_global(global_doc, lambda item: Item("n", (item.value if item else 0) + 1))
self.assertIs(result.outcome, UpdateOutcome.UPDATED)
self.assertEqual(self.cache(global_doc).get(global_doc, GLOBAL_KEY).value, 1)
def test_a_legacy_global_is_refused_until_put_global_retires_it(self) -> None:
self.plant()
cache = self.cache(global_doc) # loaded from the legacy key
before = self.bytes()
for call in (lambda a: cache.update_global(global_doc, a), lambda a: cache.update(global_doc, GLOBAL_KEY, a)):
apply = Counting()
with self.assertRaisesRegex(CultCacheError, "put_global"):
call(apply)
self.assertEqual(apply.calls, 0)
self.assertEqual(self.bytes(), before)
# Judged from the cache's load, not only from the store: the legacy record is gone from the store, the cache still holds it.
SingleFileMessagePackBackingStore(self.path).delete(global_doc.type, "legacy")
apply = Counting()
with self.assertRaisesRegex(CultCacheError, "put_global"):
cache.update_global(global_doc, apply)
self.assertEqual(apply.calls, 0)
cache.put_global(global_doc, Item("n", 5))
self.assertEqual(cache.update_global(global_doc, lambda item: Item("n", item.value + 1)).document.value, 6)
def test_a_pull_inside_apply_that_loads_a_legacy_global_fails_the_commit(self) -> None:
cache = self.cache(global_doc)
def apply(item):
self.plant()
cache.pull_all_backing_stores()
return Item("n", 9)
with self.assertRaisesRegex(CultCacheError, "put_global"):
cache.update_global(global_doc, apply)
self.assertEqual([envelope.key for envelope in SingleFileMessagePackBackingStore(self.path).pull_all()], ["legacy"])
def test_an_unpulled_cache_judges_the_legacy_global_from_the_store(self) -> None:
self.plant()
before = self.bytes()
cache = SingleFileMessagePackBackingStore(self.path)
unpulled = CultCache.builder().register_document_type(global_doc).add_generic_store(cache).build()
apply = Counting()
with self.assertRaisesRegex(CultCacheError, "put_global"):
unpulled.update_global(global_doc, apply)
self.assertEqual(apply.calls, 0)
self.assertEqual(self.bytes(), before)
class StoreRestampTests(Fixture):
def envelope(self, value: int, stored_at: str = ALPHA, key: str = "k") -> CultCacheEnvelope:
entry = item_doc.catalog_entry()
return CultCacheEnvelope(key, "item", item_doc.encode_payload(Item("n", value)), stored_at, entry.schema_id, entry)
def stamp_on_disk(self, store, key: str = "k") -> str:
return next(envelope.stored_at for envelope in store.pull_all() if envelope.key == key)
def later_than_alpha(self, text: str) -> bool:
return _parse_stored_at(text) > _parse_stored_at(ALPHA)
def test_commit_if_and_the_restamp_at_the_store(self) -> None:
store = SingleFileMessagePackBackingStore(self.path)
self.assertIsNone(store.read("k"))
stamp = store.commit_if(self.envelope(1), None) # None against an absent key commits
self.assertEqual(stamp, ALPHA)
envelope, version = store.read("k")
self.assertEqual(version, record_version(envelope))
before = self.bytes()
self.assertIsNone(store.commit_if(self.envelope(2, "2100-01-01T00:00:00Z"), None)) # a record is there: not absent
self.assertIsNone(store.commit_if(self.envelope(2, "2100-01-01T00:00:00Z"), "0" * 64))
self.assertEqual(self.bytes(), before)
written = store.commit_if(self.envelope(2), version) # a staged stamp that is no later is minted over
self.assertTrue(self.later_than_alpha(written))
self.assertEqual(self.stamp_on_disk(store), written)
self.assertEqual(stamp_of(store, "k"), written)
def test_every_replacing_write_is_stored_strictly_later(self) -> None:
for make in (SingleFileMessagePackBackingStore, JsonLinesBackingStore):
for name in ("push", "push_all", "commit_if"):
if name == "commit_if" and make is JsonLinesBackingStore:
continue
with self.subTest(store=make.__name__, write=name):
path = self.dir / f"{make.__name__}-{name}"
store = make(path)
store.push(self.envelope(1))
if name == "push":
reported = store.push(self.envelope(2))
elif name == "push_all":
reported = store.push_all([self.envelope(2)])["k"]
else:
reported = store.commit_if(self.envelope(2), store.read("k")[1])
self.assertTrue(self.later_than_alpha(reported))
self.assertEqual(self.stamp_on_disk(store), reported)
def test_the_cache_holds_the_stamp_the_store_wrote(self) -> None:
for name in ("put", "put_envelope", "put_envelopes", "update"):
with self.subTest(write=name):
path = self.dir / name
SingleFileMessagePackBackingStore(path).push(self.envelope(1))
cache = self.cache(path=path)
self.assertEqual(cache.get_envelope(item_doc, "k").stored_at, ALPHA)
if name == "put":
cache.put(item_doc, "k", Item("n", 2))
elif name == "put_envelope":
cache.put_envelope(item_doc, self.envelope(2))
elif name == "put_envelopes":
cache.put_envelopes(item_doc, [self.envelope(2)])
else:
cache.update(item_doc, "k", lambda item: Item("n", 2))
held = cache.get_envelope(item_doc, "k").stored_at
self.assertTrue(self.later_than_alpha(held))
self.assertEqual(held, self.stamp_on_disk(SingleFileMessagePackBackingStore(path)))
def test_a_store_less_cache_stamps_a_replacing_write_later(self) -> None:
cache = self.cache(store=False)
cache.put_envelope(item_doc, self.envelope(1))
cache.put_envelope(item_doc, self.envelope(2, "2000-01-01T00:00:00Z"))
self.assertTrue(self.later_than_alpha(cache.get_envelope(item_doc, "k").stored_at))
def test_stored_at_parses_on_a_python_that_refuses_z_and_seven_digits(self) -> None:
"""Python 3.10's fromisoformat takes neither (requires-python is >=3.10): the parse owns both, so it is pinned against a strict
reader on any interpreter."""
class Strict(datetime):
@classmethod
def fromisoformat(cls, text):
if text.endswith("Z") or re.search(r"\.\d{4,5}\b|\.\d{7,}", text):
raise ValueError(text)
return datetime.fromisoformat(text)
with mock.patch("cultcache_py.stores.datetime", Strict):
for text in ("2026-01-01T00:00:00.1234567+00:00", "2026-01-01T00:00:00Z", "2026-01-01T00:00:00.1234567Z"):
with self.subTest(text=text):
parsed = _parse_stored_at(text)
self.assertIsNotNone(parsed)
self.assertEqual(parsed.microsecond, 0 if text.endswith("00Z") else 123456)
def test_stamp_parsing_and_install(self) -> None:
for text in ("2026-01-01T00:00:00.1234567+00:00", "2026-01-01T00:00:00Z", "2026-01-01T00:00:00.123456+00:00", "2026-01-01T00:00:00"):
with self.subTest(text=text):
self.assertIsNotNone(_parse_stored_at(text))
self.assertEqual(_parse_stored_at(text).tzinfo, timezone.utc)
self.assertIsNone(_parse_stored_at("garbage"))
self.assertIsNone(_parse_stored_at(None))
now = datetime(2030, 1, 1, tzinfo=timezone.utc)
self.assertEqual(mint_stored_at("garbage", now), "2030-01-01T00:00:00.000000+00:00")
self.assertEqual(stored_at_over("2000-01-01T00:00:00Z", "2000-01-01T00:00:00Z")[:4], str(datetime.now(timezone.utc).year)) # no later: minted at now
self.assertEqual(mint_stored_at("2030-01-01T00:00:00Z", now), "2030-01-01T00:00:00.000001+00:00") # equal is not later
self.assertEqual(mint_stored_at("2099-01-01T00:00:00.1234567Z", now), "2099-01-01T00:00:00.123457+00:00")
self.assertEqual(stored_at_over("2000-01-01T00:00:00Z", "2099-01-01T00:00:00.1234567Z"), "2099-01-01T00:00:00.123457+00:00")
for reported in (True, 1, "garbage"):
with self.subTest(reported=reported):
class Lying(SingleFileMessagePackBackingStore):
def push(self, envelope):
super().push(envelope)
return reported
cache = CultCache.builder().register_document_type(item_doc).add_generic_store(Lying(self.dir / f"lying-{reported}")).build()
cache.pull_all_backing_stores()
cache.put_envelope(item_doc, self.envelope(1, "2026-02-02T00:00:00.000000+00:00"))
self.assertEqual(cache.get_envelope(item_doc, "k").stored_at, "2026-02-02T00:00:00.000000+00:00")
def stamp_of(store, key: str) -> str:
return store.read(key)[0].stored_at
if __name__ == "__main__":
unittest.main()