CultLib/packages/cultcache-py/README.md
2026-06-14 20:08:25 +01:00

468 lines
18 KiB
Markdown

# cultcache-py
`cultcache-py` is the Python runtime package for CultCache, CultNet, and
CultMesh. Callers work with registered domain documents while the cache owns
schema identity, routing, globals, name lookups, indexes, and backing-store
envelopes.
It is intentionally small. It is not an ORM, a database, or distributed consensus wearing rented authority.
## Current Shape
- document types are registered explicitly
- a registered cache can be used in memory without attaching a backing store
- `SingleFileMessagePackBackingStore` writes `cultcache.store.v1` snapshots:
`[format, schemaCatalog, records]`
- records store raw MessagePack payload bytes under schema-catalog identity
- payloads decode only through the registered document definition for that type
- unknown persisted types fail closed
- global documents are singleton-style per type
- type-specific backing stores beat generic backing stores
- the JSONL store uses only the Python standard library for bootstrap consumers
- `define_database_entry_type(...)` emits Rust/C#-style slot-indexed
MessagePack array payloads for cross-runtime `DatabaseEntry` contracts
- `cultnet_py` exposes schema-v0 MessagePack helpers and 4-byte big-endian
frame helpers for raw document state, subscriptions, shard catch-up,
simulation observations, and witness artifact bundles
- `cultmesh_py` exposes a cache-backed local node surface for Python tools
## Example
```python
from dataclasses import dataclass, asdict
from cultcache_py import CultCache, define_document_type
@dataclass
class Settings:
theme: str
retries: int
settings_doc = define_document_type(
"settings",
encode=lambda value: asdict(value),
decode=lambda payload: Settings(**payload),
global_document=True,
)
cache = (
CultCache.builder()
.register_document_type(settings_doc)
.build()
)
cache.put_global(settings_doc, Settings(theme="ash", retries=3))
settings = cache.get_required_global(settings_doc)
```
Add `JsonLinesBackingStore` or `SingleFileMessagePackBackingStore` when the
cache should persist state; local in-memory cache reads and writes do not need a
store.
## Public Surface
- `define_document_type(...)`
- `define_database_entry_type(...)`
- `database_entry_field(...)`
- `define_document_registry(...)`
- `CultCache.builder()`
- `register_document_type(...)`
- `register_registry(...)`
- `register_name_lookup(...)`
- `register_index(...)`
- `add_backing_store(...)`
- `add_generic_store(...)`
- `pull_all_backing_stores()`
- `get(...)`
- `get_required(...)`
- `get_all(...)`
- `get_key_by_name(...)`
- `get_by_name(...)`
- `get_key_by_index(...)`
- `get_by_index(...)`
- `get_global(...)`
- `get_required_global(...)`
- `put(...)`
- `put_envelopes(...)`
- `put_global(...)`
- `update(...)`
- `update_global(...)`
- `delete(...)`
- `delete_global(...)`
- `snapshot()`
## Persistence Model
Backing stores persist envelopes, not caller domain objects. The cache decodes payloads through registered document definitions and rejects unknown type discriminators. This keeps Python's dynamic runtime from turning persistence into an open polymorphic sewer with a cheerful docstring.
`JsonLinesBackingStore` is the dependency-free control-plane store. It rewrites an atomic JSONL snapshot and base64-encodes payload bytes. It is designed for compact state spines, settings, ledgers, and bootstrap surfaces.
`SingleFileMessagePackBackingStore` follows the shared CultCache v1 store shape
used by the TypeScript, Rust, and C# runtimes:
- top-level MessagePack value: `[format, schemaCatalog, records]`
- `format`: `cultcache.store.v1`
- catalog entries: `[schemaId, schemaName, schemaVersion, contentHash,
canonicalSchemaJson, compatibleSchemaIds, members]`
- records: `[key, schemaId, storedAt, payload]`
Large corpora should use a sharded store or a real database. A single snapshot file is a scalpel, not a forklift.
## DatabaseEntry Slot Contract
For cross-runtime cache entries, use `define_database_entry_type(...)` instead
of the generic JSON formatter:
```python
from dataclasses import dataclass
from cultcache_py import define_database_entry_type
@dataclass
class Settings:
theme: str
retries: int = 0
settings_doc = define_database_entry_type(
"settings",
[
("theme", 0),
("retries", 1, 0),
],
cls=Settings,
)
```
The payload is a MessagePack array. Field keys are durable slot indexes:
- key `0` writes array slot 0
- key `4` writes array slot 4
- unused slots are written as nil
- deleted fields should leave their slots reserved
- newly added fields should use new keys
- fields with defaults tolerate missing or nil slots when older payloads are read
That matches the Rust `#[derive(DatabaseEntry)]` formatter shape and the C#
`[Key(n)]` intent: schema evolution comes from stable field slots, not source
member order.
## CultNet And CultMesh
`cultnet_py` provides the Python schema-v0 wire helpers and an interop peer:
```python
from datetime import UTC, datetime
from cultnet_py import (
CultNetSecret,
CultNetServerSecurityOptions,
CultNetRawClient,
CultNetSimulationObservation,
CultNetWitnessArtifactBundle,
apply_raw_snapshot,
apply_shard_log_response,
compute_simulation_claim_hash,
database_subscribe,
decode_frame,
encode_frame,
hello,
parse_message,
shard_catalog_request,
simulation_observation,
witness_artifact_bundle,
)
payload = hello(runtime_id="python-runtime").to_bytes()
message = parse_message(decode_frame(encode_frame(payload)))
client = CultNetRawClient("127.0.0.1", 3075)
catalog = client.fetch_schema_catalog(kinds=["wire_message"])
snapshot = client.fetch_snapshot_response(schema_ids=["cultnet.interop-note"])
shard_catalog = client.fetch_shard_catalog(schema_ids=["cultnet.interop-note"])
# With a local CultCache and the matching registered document definitions:
# applied = apply_raw_snapshot(cache, [note_doc], snapshot)
subscription = database_subscribe(subscription_id="ui", schema_ids=["cultnet.interop-note"])
shard_request = shard_catalog_request(message_id="shards", schema_ids=["cultnet.interop-note"])
shard_log = client.fetch_shard_log_response(shard_id="interop", shard_epoch=1, after_sequence=0)
# log_changes = apply_shard_log_response(cache, [note_doc], shard_log)
with client.subscribe_database(subscription_id="ui", schema_ids=["cultnet.interop-note"]) as live:
initial_snapshot = live.read_next_snapshot_response()
claim_hash = compute_simulation_claim_hash("frame:42", "subject:player-1", "hit")
observation = simulation_observation(
message_id="obs-1",
witness_runtime_id="python-runtime",
shard_id="interop",
shard_epoch=1,
frame=42,
subject_id="player-1",
claim_kind="hit",
claim_hash=claim_hash,
)
typed_observation = CultNetSimulationObservation.from_wire(observation.to_wire())
witness = witness_artifact_bundle(
bundle_id="bundle-1",
witness_kind="interop-proof",
captured_at="2026-06-13T00:00:00Z",
subject={"documentType": "cultnet.interop-note", "subjectId": "note:python"},
contracts=[{"role": "payload", "schemaId": "cultnet.interop-note"}],
artifacts=[{"role": "log", "uri": "cultcache://bundle-1/log", "mediaType": "text/plain"}],
provenance={"pipelineId": "interop", "runId": "run-1", "runtimeId": "python-runtime"},
)
typed_witness = CultNetWitnessArtifactBundle.from_wire(witness)
witness_payload = typed_witness.to_payload()
security = CultNetServerSecurityOptions.development()
token = CultNetSecret.create_session_token(
"318fb4b6-ff5e-4c4f-b911-d81807de53a8",
datetime(2035, 1, 1, tzinfo=UTC),
security,
session_version=1,
)
session = CultNetSecret.try_validate_session_token(token, security)
```
AES-GCM string/byte encryption helpers are available through the same
`CultNetSecret` surface when the optional `crypto` extra is installed.
The peer can serve, dial, and probe the same raw-state interop lane used by the
TypeScript, Rust, and C# test peers:
```powershell
python -m cultnet_py.interop_peer serve --runtime-id python-peer --runtime-kind python --display-name "Python Peer" --agent-id python-agent --advertise-host 127.0.0.1 --tcp-port 3075 --discovery-port 4075 --discovery-group 239.77.44.11 --schema-path ..\cultnet-ts\integration\contracts\cultnet.interop-note.schema.json
python -m cultnet_py.interop_peer dial --runtime-id python-client --runtime-kind python --display-name "Python Client" --agent-id python-client --target-host 127.0.0.1 --target-port 3075 --schema-path ..\cultnet-ts\integration\contracts\cultnet.interop-note.schema.json
python -m cultnet_py.interop_peer probe --runtime-id python-prober --discovery-port 4075 --discovery-group 239.77.44.11
```
`cultmesh_py` includes a local cache-backed node, schema-v0 helpers for the
CultMesh Verse catalog and peer exchange wire messages, local and verified
authority lease checks, stream transport negotiation, committed simulation fact
documents, and local prediction reconciliation:
```python
from datetime import UTC, datetime, timedelta
from cultcache_py import define_database_entry_type
from cultnet_py import (
CultNetClientAuthorityScope,
CultNetFileShardReplicaCursorStore,
CultNetRawClient,
CultNetSchemaShardLogFetcher,
CultNetSchemaShardSnapshotFetcher,
CultNetShardDescriptor,
CultNetShardReplicator,
CultNetShardReplicatorOptions,
)
from cultmesh_py import (
CultMesh,
CultMeshAuthorityLease,
CultMeshGameSessionOptions,
CultMeshHmacAuthorityLeaseVerifier,
CultMeshNodeOptions,
peer_exchange_request,
)
note_doc = define_database_entry_type(
"mesh.note",
[("body", 0), ("name", 1), ("kind", 2)],
name="name",
indexes={"kind": "kind"},
)
node = CultMesh.start_node(
"mesh.cc",
runtime_id="python-runtime",
options=CultMeshNodeOptions(enable_durable_shard_logs=True),
)
node.database.register_document(note_doc)
unsubscribe = node.database.watch_record(note_doc, "note:1", lambda change: print(change.change_kind))
unsubscribe_named = node.database.watch_by_name(note_doc, "intro", lambda change: print(change.record_key))
unsubscribe_kind = node.database.watch_by_index(note_doc, "kind", "guide", lambda change: print(change.record_key))
node.database.put(note_doc, "note:1", {"body": "hello", "name": "intro", "kind": "guide"})
live_note = node.database.get_required(note_doc, "note:1")
put_message = node.database.put_raw_message(
note_doc,
"note:2",
{"body": "wire me", "name": "wire", "kind": "interop"},
shard_id="interop",
shard_epoch=1,
)
snapshot_response = node.database.build_snapshot_response(schema_ids=[note_doc.catalog_entry().schema_id])
snapshot_wire = node.database.create_snapshot_response(schema_ids=[note_doc.catalog_entry().schema_id])
delete_message = node.database.delete_raw_message(note_doc, "note:2", shard_id="interop", shard_epoch=1)
shard_log_response = node.database.build_shard_log_response(shard_id="interop", shard_epoch=1)
shard_log_wire = node.database.create_shard_log_response(shard_id="interop", shard_epoch=1)
peers = CultMesh.create_peer_catalog()
response = peers.create_response(peer_exchange_request("pex-1", verse_id="local"))
lease_verifier = CultMeshHmacAuthorityLeaseVerifier({"odin": b"shared-lease-key"})
lease_catalog = CultMesh.create_authority_lease_catalog(
signature_verifier=lease_verifier.verify,
require_verified_signatures=True,
)
lease_valid_from = datetime.now(UTC)
lease = lease_verifier.issue(CultMeshAuthorityLease(
lease_id="lease:python-runtime",
verse_id="local",
peer_id="python-runtime",
roles=("shard-primary",),
valid_from=lease_valid_from,
expires_at=lease_valid_from + timedelta(minutes=5),
shard_ids=("interop",),
issuer_runtime_id="odin",
))
lease_catalog.upsert(lease)
session = CultMesh.create_game_session(
node,
CultMeshGameSessionOptions(
client_authority_scopes=(
CultNetClientAuthorityScope("python-runtime", schema_ids=(note_doc.catalog_entry().schema_id,), key_prefix="input:python"),
),
),
)
server = CultMesh.serve_node(
node,
peer_catalog=peers,
observation_hub=session.observation_hub,
port=3075,
max_snapshot_documents=1000,
max_snapshot_bytes=4 * 1024 * 1024,
)
```
The same local server can be launched as a package daemon when another runtime
or operator process needs a CultNet endpoint without embedding Python glue:
```powershell
python -m cultmesh_py.daemon --runtime-id python-runtime --display-name "Python Runtime" --host 127.0.0.1 --port 3075 --cache-file mesh.cc --enable-durable-shard-logs --seed-interop-note --seed-shard-id interop --verse-id local --role shard-primary --ready-file mesh.ready.json
```
The ready file and stdout line are process readiness hints. The served state and
capability truth still live on the framed CultNet/CultMesh endpoint. Use
`--register-interop-note` to expose the package interop-note schema without
seeding a note, or `--seed-interop-note` to register it and publish one local
record for wire probes. Seeded records are committed through the raw mutation
path, so peers can read them through both snapshot and shard-log catch-up. Add
`--verse-id` and one or more `--role` values to publish the launched endpoint in
its own Verse and peer catalogs for discovery clients. When the daemon is
launched with `--cache-file`, `--enable-durable-shard-logs`, and
`--shard-log-file`, a later daemon process can rehydrate and serve the persisted
snapshot and shard log without reseeding.
`CultMeshPeerHealthMonitor` probes peer-card endpoints with `cultnet.hello.v0`
and preserves the runtime id, display name, document types, message versions,
and mutation contracts reported by the peer.
```python
client = CultMesh.create_verse_discovery_client("127.0.0.1", 3075)
verses = client.fetch_verses(transport_version="cultmesh.v0")
mesh_peers = client.fetch_peers(verse_id="python-interop", roles=["read-replica"])
client.sync_peer_catalog(peers, verse_id="python-interop", roles=["read-replica"])
client.fanout_peer_catalog(peers, verse_id="python-interop", roles=["read-replica"])
raw_client = CultNetRawClient("127.0.0.1", 3075)
node.database.sync_snapshot(raw_client, schema_ids=[note_doc.catalog_entry().schema_id])
node.database.sync_shard_log(raw_client, shard_id="interop", shard_epoch=1)
replicator = CultNetShardReplicator(
node.database,
CultNetShardReplicatorOptions(
fetcher=CultNetSchemaShardLogFetcher(),
snapshot_fetcher=CultNetSchemaShardSnapshotFetcher(),
cursor_store=CultNetFileShardReplicaCursorStore("mesh.replica-cursors.msgpack"),
poll_interval_seconds=1.0,
on_error=lambda error: print(error),
),
)
replica_shard = CultNetShardDescriptor(
shard_id="interop",
owner_runtime_id="primary-runtime",
epoch=1,
schema_ids=(note_doc.catalog_entry().schema_id,),
primary_endpoints=("cultnet://127.0.0.1:3075",),
)
replicator.pull_once(replica_shard)
replicator.start([replica_shard])
replicator.stop()
server.stop()
streams = CultMesh.create_stream_catalog()
claim_hash = "accepted-claim-hash"
facts = CultMesh.create_simulation_fact_committer(node)
committed = facts.commit({
"shardId": "arena",
"shardEpoch": 4,
"frame": 100,
"subjectId": "bob",
"claimKind": "hit",
"claimHash": claim_hash,
"witnessCount": 2,
"supportWeight": 2.0,
"totalWeight": 2.0,
"confidence": 1.0,
"hasQuorum": True,
})
session_commits = session.submit_and_commit(typed_observation)
prediction = session.predict(note_doc, "input:python:move", {"body": "predicted input"})
unsubscribe()
```
## Wire Parity
See [PARITY.md](PARITY.md) for the current evidence map, gates, and things this
package still does not claim.
The Python interop peer is `cultcache_py.interop`:
```powershell
python -m cultcache_py.interop write --file cache.cc --runtime-id python
python -m cultcache_py.interop read --file cache.cc
```
Current receipts:
- `packages/cultcache-ts/test/cult-cache.test.ts` includes Python in the shared
CultCache v1 parity matrix with TypeScript, Rust, and C#.
- `packages/cultnet-ts/test/interop/cultnet-interop.test.ts` includes Python in
the live TS/Rust/C#/Python schema-v0 peer ring: discovery, hello, schema
catalog, wire-message catalog discovery, raw snapshot, raw document
put/delete, mutation receipt, fire-command receipt, database subscription
changes, shard catalog, shard log catch-up, simulation consensus candidates,
committed simulation fact snapshots, Python session prediction reconciliation,
and witness artifact bundle round-trips. The same test asks the Python peer
for CultMesh Verse catalog and peer exchange responses over the CultNet pipe,
then verifies the public `CultMeshDiscoveryClient` can fetch typed Python
Verse and peer descriptors from that live endpoint. It also verifies
`CultNetRawClient` can fetch the live Python peer's schema catalog, raw
snapshot, and shard catalog.
- `packages/cultcache-py/tests/test_cultcache.py` covers Python CultMesh
Verse catalog, peer exchange, authority lease, stream negotiation, CultNet
helper shapes, raw snapshot/shard-log application, simulation claim hashing,
consensus aggregation, game-session fact commits, local prediction
reconciliation, committed simulation fact payload slots, and witness artifact
bundle payload slots.
- `cultcache_py`, `cultnet_py`, and `cultmesh_py` ship `py.typed` markers so
downstream type checkers can inspect the package surface instead of treating
the runtime as an untyped xenos swamp.
## Performance Baseline
The package ships a lightweight benchmark for Python-owned hot paths. In a
CultLib source checkout, it can also run the C# reference benchmark beside it:
```powershell
python -m cultcache_py.benchmark --records 1000 --json
python -m cultcache_py.compare_csharp --records 1000 --json
python -m cultcache_py.verify --json
```
It reports slot-indexed `DatabaseEntry` encode/decode throughput, framed
CultNet MessagePack parse throughput, raw snapshot application into a
registered cache, and public `CultCache` upsert/get throughput. The C# compare
command also runs `packages/cultcache-py/tools/GameCult.Caching.Benchmark` and
reports median Python-to-C# ratios for the shared public cache operations. It
uses three samples by default; pass `--samples` when you need a different
evidence shape. Both runtimes precompute record keys before measured cache
upsert/get loops so the comparison is about public cache access, not string
construction noise. The public in-memory cache owner stores records in
type-partitioned dictionaries so hot `get(...)` calls avoid per-call
composite-key allocation. Treat the benchmark numbers as a local baseline, not
a claim that Python matches the C# reference in every workload.