CultLib/packages/cultcache-ts/README.md
Libby 7002c252ed
Some checks failed
CultMesh QUIC native / Build win32-x64 (push) Has been skipped
CultMesh QUIC native / Build linux-x64 (push) Has been skipped
CultMesh QUIC native / Release native binaries (push) Has been skipped
CultMesh QUIC native / Verify committed binaries match SHA256SUMS (push) Has been cancelled
CultMesh portability and performance / ubuntu-latest reactive semantics and leases (push) Has been cancelled
Element ids: the existing-member route and one-way v3/v5 rollback, documented (#98)
2026-11-11 08:53:13 +00:00

427 lines
21 KiB
Markdown

# CultCacheTS
`CultCacheTS` is the TypeScript port of the useful part of GameCult's `CultCache` idea: consumer code talks to typed domain documents, while a closed-world persistence layer deals with schemas, routing, name lookups, indexes, globals, and raw MessagePack bytes.
This is not an ORM in a fake mustache. It is for cases where you want:
- one cache API for heterogeneous domain documents
- typed reads and writes instead of every script massaging anonymous blobs
- lookup by type, logical key, name, and indexed property
- singleton-style global documents
- backing-store routing instead of every caller pawing raw files directly
- a persistence boundary that refuses unregistered polymorphic junk
## Current Shape
- document types are registered explicitly
- a document schema only needs a `parse(...)` function, so generated JSON-schema
contracts can plug in directly without growing a second hand-written Zod body
- each persisted record stores:
- `key`
- `schemaId`
- raw MessagePack `payload` bytes
- `storedAt`
- the single-file store writes the CultCache v1 snapshot shape:
- `formatVersion`
- embedded schema catalog
- record array
- payload bytes only decode through the registered document definition for that schema
- name and index maps live in the cache, not in the backing store
- global documents are treated as a singleton per type
- backing stores are adapters, not the public data model
That keeps the polymorphic boundary tight without making callers write boilerplate by candlelight.
## Reading And Writing Payloads
A document's bytes travel as a `DocumentPayload`: in a store envelope, a CultNet record and a
realtime frame. At run time it is the same `Uint8Array`, so nothing on the wire changes. At compile
time it is opaque: it is assignable to and from neither `Uint8Array` nor `ArrayLike<number>`, so no
MessagePack decoder accepts it.
What the type guarantees is that a payload cannot be read or assigned directly: a decoder call, an
assignment to `Uint8Array` and `new Uint8Array(payload)` fail to compile. It does not stop what the
runtime value allows, because the value is a `Uint8Array`. These routes read it without the owner:
`instanceof Uint8Array` or `ArrayBuffer.isView` narrowing, `Object.values`, `Object.assign` onto a
`Uint8Array`, a JSON round trip, `Reflect.get`, and a cast. A read made that way sees the raw bytes,
which for a codec or formatter document are not the document. No lint or guard closes them. Read
through `decodeDocumentPayload`, and when you need the bytes themselves (hashing, signing, a store
write) take them with `documentPayloadBytes`, the sanctioned escape.
- `decodeDocumentPayload(definition, payload)` reads a payload through the definition's formatter,
codec or plain map and then its schema. The definition is required.
- `encodeDocumentPayload(definition, value)` is the writer: the schema first, then the formatter,
codec or plain map.
- `documentPayloadFromBytes(bytes)` is where bytes enter (a record you validated, a store read, a
writer of pre-encoded or signed bytes) and `documentPayloadBytes(payload)` is where they leave (a
store write, a frame encode, hashing or signing). Reach for them only at those edges.
```ts
const note = decodeDocumentPayload(noteDocument, envelope.payload); // parsed Note
const payload = encodeDocumentPayload(noteDocument, note); // DocumentPayload
```
Browsers import the same names from `@gamecult/cultcache-ts/document`, which holds no Node built-ins. The whole cache without the single-file store is `@gamecult/cultcache-ts/browser`, which is also what bundlers resolve for the package root (`browser` condition); bring your own `CacheBackingStore`.
## Store format and rollback
A store whose elements hold an id is marked `cultcache.store.v3` by the C# runtime. This runtime reads a v3 store, keeps the header it read and carries the ids through a rewrite; it never drops the mark, so it has no carry-back. A cultcache-ts from before element ids refuses a v3 store by name, so rolling back to it after a store has been written as v3 is one-way: restore from a copy taken before upgrading, and upgrade every reader before any writer. (The C# line is different: CultLib 1.x does not refuse; see the GameCult.Caching README.)
## `.cc` Inspection
CultCache files may use the `.cc` extension. The bytes are still the canonical
`cultcache.store.v1` MessagePack snapshot; the extension is the human handle, not
a second format.
`inspectCultCacheBytes(...)` reads `.cc`, `.msgpack`, or `.mpack` bytes and
returns the snapshot header, schema catalog, records, and decoded MessagePack
payload previews without registering application schemas. Presentation runtimes
such as Huginn consume that inspection result; they do not own the storage
format.
## Example
```ts
import { z } from "zod";
import {
CultCache,
SingleFileMessagePackBackingStore,
defineDocumentRegistry,
defineDocumentType,
} from "@gamecult/cultcache-ts";
const itemDocument = defineDocumentType({
type: "item",
schemaId: "gamecult.item.v1",
schemaName: "item",
schemaVersion: "item.v1",
schema: z.object({
name: z.string(),
sku: z.string(),
category: z.string(),
value: z.number().int(),
}),
name: "name",
indexes: {
sku: "sku", // unique: a sku names one item
},
roles: {
category: "category", // a selection alias: many items share a category
},
});
const settingsDocument = defineDocumentType({
type: "settings",
schema: z.object({
theme: z.string(),
retries: z.number().int().nonnegative(),
}),
global: true,
});
const registry = defineDocumentRegistry(itemDocument, settingsDocument);
const cache = CultCache.builder()
.withRegistry(registry)
.withGenericStore(new SingleFileMessagePackBackingStore("cache.msgpack"))
.build();
await cache.pullAllBackingStores();
await cache.put(itemDocument, "item:potion", {
name: "Potion",
sku: "POT-001",
category: "Consumable",
value: 50,
});
await cache.putGlobal(settingsDocument, {
theme: "ash",
retries: 3,
});
const potion = cache.getByName(itemDocument, "Potion");
const bySku = cache.getByIndex(itemDocument, "sku", "POT-001");
const settings = cache.getRequiredGlobal(settingsDocument);
```
## Public Surface
- `defineDocumentType(...)`
- `defineDocumentRegistry(...)`
- `CultCache.builder()`
- `registerDocumentType(...)`
- `registerRegistry(...)`
- `registerNameLookup(...)`
- `registerIndex(...)`
- `addBackingStore(...)`
- `addGenericBackingStore(...)`
- `pullAllBackingStores()`
- `get(...)`
- `getRequired(...)`
- `getAll(...)`
- `getKeyByName(...)`
- `getByName(...)`
- `getKeyByIndex(...)`
- `getByIndex(...)`
- `getGlobal(...)`
- `getRequiredGlobal(...)`
- `put(...)`
- `putGlobal(...)`
- `update(...)`
- `updateGlobal(...)`
- `delete(...)`
- `deleteGlobal(...)`
- `snapshot()`
- `getEnvelope(...)`
- `getRequiredEnvelope(...)`
- `getGlobalEnvelope(...)`
- `putEnvelope(...)`
Document definitions may also carry persistence metadata for cross-language
stores: `schemaId`, `schemaName`, `schemaVersion`, `contentHash`,
`canonicalSchemaJson`, `compatibleSchemaIds`, and slot `members`.
## Backing Store Routing
```ts
const cache = CultCache.builder()
.withRegistry(registry)
.withBackingStore(new SingleFileMessagePackBackingStore("settings.cc"), settingsDocument)
.withGenericStore(new SingleFileMessagePackBackingStore("cache.cc"))
.build();
```
A document type has exactly one home store
(`src/GameCult.Caching/Contracts/cultcache-store-composition.md` in CultLib):
- a store added with types owns those types; otherwise the generic store is the home
- a second generic store throws, and so does a type claimed by a second store;
a refused registration attaches nothing
- `put`, `putEnvelope` and `delete` touch exactly one store; nothing is mirrored
- a write whose type has no home store throws
## Name, Index, and Global Semantics
`name`, `indexes` and `roles` can be declared on the document definition:
```ts
const playerDocument = defineDocumentType({
type: "player",
schema: playerSchema,
name: "displayName",
indexes: {
email: "email", // unique lookup
},
roles: {
faction: "faction", // selection alias, many players share one
},
});
```
`indexes` are unique (variants:ruling:unique-indexes; C# `[CultIndex]`, Rust `#[cultcache(index)]`). Within one type
and one alias no two records hold one non-blank value, judged on the result of every write and of every load; a
refusal reads `Records <k1> and <k2> (<schema name>) both hold unique index '<alias>'; an index value names one record.`
(keys in ordinal order, never the value), and writes nothing. The contract is the verdict, which stores and writes are
refused; the text is informational. On a load TS judges record by record in file order and names the first collision
it meets, where C# judges the whole store, so the two runtimes can name different pairs of one store. A blank value (absent, empty, or only whitespace) is
never indexed and never conflicts. Two types may share an alias and a value. Index values are compared as the C#
reference renders them: strings as is (never trimmed), `True`/`False`, integers as digits, a non-integral number as
its exact decimal expansion; NaN and infinities have no value. `roles` name selection and reference aliases (the
member's catalog `indexAlias`, so the catalog bytes are the same either way); they imply nothing about uniqueness and
are never indexed for lookup, so `getByIndex` on a role answers `undefined`. An alias is declared in one of the two.
### Migrating an existing `indexes` key
Apply per key, and do it BEFORE the CultLib checkout your tools load is rebuilt or updated. A consumer that loads
CultLib at run time (Bifrost reads `$VOIDBOT_CULTLIB_ROOT`, else the sibling `../CultLib` dist) has no pin to hold: its
next start after the checkout moves is the bump. A consumer that pins a version applies this before it bumps the pin.
Moving an `indexes` key to `roles` is safe on the old CultLib too when the definition lists `members` explicitly (an old
codec definition without `members` derives `indexAlias` from `indexes` only, so a moved key would lose its catalog alias),
so the consumer edit lands first:
1. Nothing looks records up by the key (`getByIndex`, `getKeyByIndex`, `getIdByIndex`; Python `get_by_index`,
`get_key_by_index`, `node.get_by_index`, `watch_by_index`; Rust `get_by_index`): it is a role. Move it to `roles`;
where a definition lists `members` explicitly, delete it from `indexes` and keep the member's `indexAlias` (the
catalog is the same).
2. Something looks it up: it stays unique only when two live records can never hold one non-blank value, because the
record key is derived from that value alone, or the value is an outside identity (email, sku, external id).
Enums, statuses, kinds, categories, parent or foreign ids, shared hashes and timestamps are roles; a lookup on
one already answered an arbitrary record, so move it to a selection (`any_of`) or `getAll` plus a filter.
3. Before the checkout moves, open each real `.cc` once with the new definitions. A refusal names both keys and the alias and
writes nothing.
Misclassifying costs differently each way. A role declared unique: the second record holding a shared value is
refused at commit, and a store already holding one refuses to open, so that store is down until the key is
reclassified (loud, names both keys and the alias). A unique key declared a role: lookups by it answer nothing and
duplicates stop being refused.
The Bifrost TS stores (at 781e6ea) call no lookup anywhere, so by rule 1 every key is a role and no key becomes unique.
All six definitions list `members` explicitly, each with its `indexAlias`, so the edit is deleting each `indexes` block:
the catalog is unchanged and the edit is safe on the CultLib it loads today. It must land before that checkout is
updated:
| Definition | Keys | Verdict |
|---|---|---|
| `update-request` (`tools/agent-transport.mjs`) | `targetRepoName`, `targetAgentIdentity`, `status` | role: many requests per repo and agent; status is an enum |
| `crossing_receipt` (`tools/bifrost-crossing-documents.mjs`) | `commandId`, `crossingKind`, `status` | role: receipts are `crossing_<commandId>_<activityKind>`, several per command; the rest are enums |
| release authority (`tools/bifrost-repository-release-authority.mjs`) | `repositoryFullName`, `upstreamRef`, `commitSha`, `status` | role: the authority id is derived from repo, ref and sha, so one sha under two refs is two records |
| `topic` (`tools/governance-threads.mjs`) | `jurisdictionRepoName`, `jurisdictionAgentIdentity`, `status` | role: many topics per jurisdiction |
| `comment` (`tools/governance-threads.mjs`) | `topicId`, `authorId`, `stance` | role: many comments per topic; stance is an enum |
| `advertisement` (`tools/provider-advertisement.mjs`) | `serviceName` | role: one record holds it today, but nothing looks it up |
You can also register them later if you feel like changing the machine after boot:
```ts
await cache.registerNameLookup(playerDocument, "displayName");
await cache.registerIndex(playerDocument, "email", "email");
```
Registration, `addBackingStore`, `pullAllBackingStores` and every write and
delete run one at a time through a per-cache queue and return promises; reads
are synchronous. A write validates the whole record (home store, global key,
name and index accessors) before any store is touched, so a refused write
changes neither store nor cache. Registering an accessor that throws on a held
record installs nothing.
Global documents are singleton-style per type, always stored under
`CultCache.GLOBAL_KEY` (`__global__`); writing one under any other key throws:
```ts
await cache.putGlobal(settingsDocument, { theme: "ash", retries: 3 });
const settings = cache.getRequiredGlobal(settingsDocument);
```
If persisted state contains two globals for the same type, `pullAllBackingStores()` fails closed instead of pretending the contradiction is charming.
## Typed Formatters
Each document definition can provide a custom formatter:
```ts
const combatLogDocument = defineDocumentType({
type: "combat-log",
schema: combatLogSchema,
formatter: {
encode(value) {
return myFastBinaryEncoder(value);
},
decode(payload) {
return myFastBinaryDecoder(payload);
},
},
});
```
If you do not supply one, `CultCacheTS` uses MessagePack plus Zod validation by default.
The 256-container nesting bound (`MAX_PAYLOAD_DEPTH`, refused with `CultCachePayloadDepthError`) covers the MessagePack the
library itself writes: the default format and a codec. A payload your formatter writes is opaque bytes in any format, so the
library does not walk it as MessagePack; its format and its bounds are your formatter's.
Zod is no longer mandatory. Any schema object with a `parse(...)` function will
do, which means generated Ajv-backed JSON-schema contracts can feed the cache
directly as long as they expose the same parse surface.
The important boundary is this:
- payload bytes are persisted as bytes
- decoding goes through the registered formatter for that document type
- the decoded value is still validated against the declared schema
So callers get typed values, and the persistence layer does not become an open polymorphic sewer.
## Slot Records and Unions
`codec` writes the reference runtimes' record shape: a MessagePack array indexed by slot key, nil in a gap, and a
union as `[armKey, [armSlots...]]`. A union value is `{ kind: armName, ...armFields }`.
```ts
import { codecs, defineDocumentType } from "@gamecult/cultcache-ts";
const { slots, union, list, lazy, string } = codecs;
const NS = "MyApp.Scene+"; // the C# names of the types, as the reference writes them into a store's schema catalog
type Node =
| { kind: "Box"; id: string; children: (Node | null)[] }
| { kind: "Text"; id: string; content: string }
| { kind: "Image"; id: string; uri: string; alt: string };
const node: codecs.Codec<Node | null> = union<Node>("Node", {
0: ["Box", slots<{ id: string; children: (Node | null)[] }>({ 0: ["id", string], 1: ["children", list(lazy(() => node))] })],
1: ["Text", slots<{ id: string; content: string }>({ 0: ["id", string], 1: ["content", string] })],
5: ["Image", slots<{ id: string; uri: string; alt: string }>({ 0: ["id", string], 1: ["uri", string], 3: ["alt", string] })], // slot 2 retired
}, `${NS}Node`);
const sceneDocument = defineDocumentType({
type: "scene",
schema: sceneSchema,
codec: slots<Scene>({ 0: ["id", string], 1: ["root", node], 2: ["title", string, ""] }),
name: "id",
});
```
- A slot is `[name, codec, default?]`. A gap is written nil and skipped on read; an extra trailing slot is skipped; a
slot missing from the payload (or nil, when a default is declared) takes its default, and is refused without one.
- A union refuses an undeclared arm key on read, naming the union, the key and the declared keys. Nothing falls back
to a default arm. An arm with no slots is `codecs.unit` and is written `[key, []]`.
- A refusal names the union, the field and the slot; it never echoes the value it refused.
- The leaves are `string`, `boolean`, `bytes`, `int32`, `int64`, `float32` and `float64`, each writing the MessagePack the
C# type writes and refusing what C# refuses (a float in an int slot, an `int` outside Int32). `int64` reads a number
when the value is a safe integer and a bigint beyond 2^53, and refuses a number beyond 2^53 on write. `list`, `array`
and `nullable` wrap a leaf, a union or a record. A `float32` slot rounds an integer once, as the reference's cast does.
- Every codec carries the reference's type name, and a store's schema catalog is derived from those names, because the
reference refuses a store whose slot names another type. The leaves know theirs; give a union or a record its C# name
as the last argument of `union(name, arms, typeName)` or `slots(spec, typeName)`. A codec with a slot it cannot name
refuses to derive members: pass `members` instead.
- An arm key is an integer within Int32 (negative too), in any MessagePack integer encoding. A float key (even `1.0`) is
refused on read, and a union refuses a key outside Int32 when it is defined. A slot key is within 0 to Int32 max.
- `codec` supplies the document's formatter and, unless `members` is given, its catalog members. A definition may not
carry both `codec` and `formatter`. `schema.parse` still runs on every decoded value.
## Raw Envelope Fast Path
For bit-compatible neighbors such as CultNet peers that already share the same
formatter contract, `CultCacheTS` can now move the persisted envelope directly:
- `getEnvelope(...)` / `getRequiredEnvelope(...)` export the canonical
MessagePack payload bytes for a typed document
- `putEnvelope(...)` ingests that envelope into another cache instance without
re-encoding the payload first
Raw envelopes may carry `schemaId` and catalog metadata. Domain callers should
not need that sludge. Replication, migration tooling, and wire-compatibility
tests do.
That still is not magic shared memory. The cache decodes once so lookups and
typed reads stay honest. But it stops doing the stupid part where identical
MessagePack payload bytes get decoded into a generic value, then re-encoded
into the same bytes again just to cross the room.
## Security Model
`CultCacheTS` should never deserialize arbitrary persisted payloads into arbitrary runtime shapes.
It only accepts persisted records whose `schemaId` or legacy `type` discriminator resolves to a registered document definition on the cache instance. Unknown schemas fail immediately. Known schemas decode through the registered formatter and schema only.
That is the TypeScript version of the original CultCache trick:
- closed discriminator set
- explicit registration
- typed formatter path
- no runtime type-name roulette
## Persistence Model
`SingleFileMessagePackBackingStore` is the first concrete store.
- it writes atomic CultCache v1 MessagePack snapshots
- the top-level MessagePack value is `[formatVersion, schemaCatalog, records]`
- each record is `[key, schemaId, storedAt, payload]`
- each catalog entry is `[schemaId, schemaName, schemaVersion, contentHash, canonicalSchemaJson, compatibleSchemaIds, members]`
- it still reads the older TS/Rust-style envelope array (object payloads are normalized in memory; opening never rewrites the file) and replaces it with a v1 snapshot on the next push, delete or flush
- it queues local writes in order, then takes the shared `<file>.lock` (`flock` on Unix, `LockFileEx` on Windows, via koffi, loaded on the first write) for each one, so C#, Rust and TypeScript writers exclude each other; the read a write merges from is taken under that lock
- it fsyncs the staged file and, on Unix, the directory after the replace; on Windows it replaces with a POSIX-semantics rename and falls back to a write-through `MoveFileExW`, retrying briefly on sharing violations
- it persists raw MessagePack payload bytes inside schema-addressed records
It is a control-plane store, not a tiny god: every write rewrites the whole file.