Some checks failed
CultMesh QUIC native / Build win32-x64 (push) Has been skipped
CultMesh QUIC native / Build linux-x64 (push) Has been skipped
CultMesh QUIC native / Release native binaries (push) Has been skipped
CultMesh QUIC native / Verify committed binaries match SHA256SUMS (push) Has been cancelled
CultMesh portability and performance / ubuntu-latest reactive semantics and leases (push) Has been cancelled
427 lines
21 KiB
Markdown
427 lines
21 KiB
Markdown
# CultCacheTS
|
|
|
|
`CultCacheTS` is the TypeScript port of the useful part of GameCult's `CultCache` idea: consumer code talks to typed domain documents, while a closed-world persistence layer deals with schemas, routing, name lookups, indexes, globals, and raw MessagePack bytes.
|
|
|
|
This is not an ORM in a fake mustache. It is for cases where you want:
|
|
|
|
- one cache API for heterogeneous domain documents
|
|
- typed reads and writes instead of every script massaging anonymous blobs
|
|
- lookup by type, logical key, name, and indexed property
|
|
- singleton-style global documents
|
|
- backing-store routing instead of every caller pawing raw files directly
|
|
- a persistence boundary that refuses unregistered polymorphic junk
|
|
|
|
## Current Shape
|
|
|
|
- document types are registered explicitly
|
|
- a document schema only needs a `parse(...)` function, so generated JSON-schema
|
|
contracts can plug in directly without growing a second hand-written Zod body
|
|
- each persisted record stores:
|
|
- `key`
|
|
- `schemaId`
|
|
- raw MessagePack `payload` bytes
|
|
- `storedAt`
|
|
- the single-file store writes the CultCache v1 snapshot shape:
|
|
- `formatVersion`
|
|
- embedded schema catalog
|
|
- record array
|
|
- payload bytes only decode through the registered document definition for that schema
|
|
- name and index maps live in the cache, not in the backing store
|
|
- global documents are treated as a singleton per type
|
|
- backing stores are adapters, not the public data model
|
|
|
|
That keeps the polymorphic boundary tight without making callers write boilerplate by candlelight.
|
|
|
|
## Reading And Writing Payloads
|
|
|
|
A document's bytes travel as a `DocumentPayload`: in a store envelope, a CultNet record and a
|
|
realtime frame. At run time it is the same `Uint8Array`, so nothing on the wire changes. At compile
|
|
time it is opaque: it is assignable to and from neither `Uint8Array` nor `ArrayLike<number>`, so no
|
|
MessagePack decoder accepts it.
|
|
|
|
What the type guarantees is that a payload cannot be read or assigned directly: a decoder call, an
|
|
assignment to `Uint8Array` and `new Uint8Array(payload)` fail to compile. It does not stop what the
|
|
runtime value allows, because the value is a `Uint8Array`. These routes read it without the owner:
|
|
`instanceof Uint8Array` or `ArrayBuffer.isView` narrowing, `Object.values`, `Object.assign` onto a
|
|
`Uint8Array`, a JSON round trip, `Reflect.get`, and a cast. A read made that way sees the raw bytes,
|
|
which for a codec or formatter document are not the document. No lint or guard closes them. Read
|
|
through `decodeDocumentPayload`, and when you need the bytes themselves (hashing, signing, a store
|
|
write) take them with `documentPayloadBytes`, the sanctioned escape.
|
|
|
|
- `decodeDocumentPayload(definition, payload)` reads a payload through the definition's formatter,
|
|
codec or plain map and then its schema. The definition is required.
|
|
- `encodeDocumentPayload(definition, value)` is the writer: the schema first, then the formatter,
|
|
codec or plain map.
|
|
- `documentPayloadFromBytes(bytes)` is where bytes enter (a record you validated, a store read, a
|
|
writer of pre-encoded or signed bytes) and `documentPayloadBytes(payload)` is where they leave (a
|
|
store write, a frame encode, hashing or signing). Reach for them only at those edges.
|
|
|
|
```ts
|
|
const note = decodeDocumentPayload(noteDocument, envelope.payload); // parsed Note
|
|
const payload = encodeDocumentPayload(noteDocument, note); // DocumentPayload
|
|
```
|
|
|
|
Browsers import the same names from `@gamecult/cultcache-ts/document`, which holds no Node built-ins. The whole cache without the single-file store is `@gamecult/cultcache-ts/browser`, which is also what bundlers resolve for the package root (`browser` condition); bring your own `CacheBackingStore`.
|
|
|
|
## Store format and rollback
|
|
|
|
A store whose elements hold an id is marked `cultcache.store.v3` by the C# runtime. This runtime reads a v3 store, keeps the header it read and carries the ids through a rewrite; it never drops the mark, so it has no carry-back. A cultcache-ts from before element ids refuses a v3 store by name, so rolling back to it after a store has been written as v3 is one-way: restore from a copy taken before upgrading, and upgrade every reader before any writer. (The C# line is different: CultLib 1.x does not refuse; see the GameCult.Caching README.)
|
|
|
|
## `.cc` Inspection
|
|
|
|
CultCache files may use the `.cc` extension. The bytes are still the canonical
|
|
`cultcache.store.v1` MessagePack snapshot; the extension is the human handle, not
|
|
a second format.
|
|
|
|
`inspectCultCacheBytes(...)` reads `.cc`, `.msgpack`, or `.mpack` bytes and
|
|
returns the snapshot header, schema catalog, records, and decoded MessagePack
|
|
payload previews without registering application schemas. Presentation runtimes
|
|
such as Huginn consume that inspection result; they do not own the storage
|
|
format.
|
|
|
|
## Example
|
|
|
|
```ts
|
|
import { z } from "zod";
|
|
import {
|
|
CultCache,
|
|
SingleFileMessagePackBackingStore,
|
|
defineDocumentRegistry,
|
|
defineDocumentType,
|
|
} from "@gamecult/cultcache-ts";
|
|
|
|
const itemDocument = defineDocumentType({
|
|
type: "item",
|
|
schemaId: "gamecult.item.v1",
|
|
schemaName: "item",
|
|
schemaVersion: "item.v1",
|
|
schema: z.object({
|
|
name: z.string(),
|
|
sku: z.string(),
|
|
category: z.string(),
|
|
value: z.number().int(),
|
|
}),
|
|
name: "name",
|
|
indexes: {
|
|
sku: "sku", // unique: a sku names one item
|
|
},
|
|
roles: {
|
|
category: "category", // a selection alias: many items share a category
|
|
},
|
|
});
|
|
|
|
const settingsDocument = defineDocumentType({
|
|
type: "settings",
|
|
schema: z.object({
|
|
theme: z.string(),
|
|
retries: z.number().int().nonnegative(),
|
|
}),
|
|
global: true,
|
|
});
|
|
|
|
const registry = defineDocumentRegistry(itemDocument, settingsDocument);
|
|
|
|
const cache = CultCache.builder()
|
|
.withRegistry(registry)
|
|
.withGenericStore(new SingleFileMessagePackBackingStore("cache.msgpack"))
|
|
.build();
|
|
|
|
await cache.pullAllBackingStores();
|
|
|
|
await cache.put(itemDocument, "item:potion", {
|
|
name: "Potion",
|
|
sku: "POT-001",
|
|
category: "Consumable",
|
|
value: 50,
|
|
});
|
|
|
|
await cache.putGlobal(settingsDocument, {
|
|
theme: "ash",
|
|
retries: 3,
|
|
});
|
|
|
|
const potion = cache.getByName(itemDocument, "Potion");
|
|
const bySku = cache.getByIndex(itemDocument, "sku", "POT-001");
|
|
const settings = cache.getRequiredGlobal(settingsDocument);
|
|
```
|
|
|
|
## Public Surface
|
|
|
|
- `defineDocumentType(...)`
|
|
- `defineDocumentRegistry(...)`
|
|
- `CultCache.builder()`
|
|
- `registerDocumentType(...)`
|
|
- `registerRegistry(...)`
|
|
- `registerNameLookup(...)`
|
|
- `registerIndex(...)`
|
|
- `addBackingStore(...)`
|
|
- `addGenericBackingStore(...)`
|
|
- `pullAllBackingStores()`
|
|
- `get(...)`
|
|
- `getRequired(...)`
|
|
- `getAll(...)`
|
|
- `getKeyByName(...)`
|
|
- `getByName(...)`
|
|
- `getKeyByIndex(...)`
|
|
- `getByIndex(...)`
|
|
- `getGlobal(...)`
|
|
- `getRequiredGlobal(...)`
|
|
- `put(...)`
|
|
- `putGlobal(...)`
|
|
- `update(...)`
|
|
- `updateGlobal(...)`
|
|
- `delete(...)`
|
|
- `deleteGlobal(...)`
|
|
- `snapshot()`
|
|
- `getEnvelope(...)`
|
|
- `getRequiredEnvelope(...)`
|
|
- `getGlobalEnvelope(...)`
|
|
- `putEnvelope(...)`
|
|
|
|
Document definitions may also carry persistence metadata for cross-language
|
|
stores: `schemaId`, `schemaName`, `schemaVersion`, `contentHash`,
|
|
`canonicalSchemaJson`, `compatibleSchemaIds`, and slot `members`.
|
|
|
|
## Backing Store Routing
|
|
|
|
```ts
|
|
const cache = CultCache.builder()
|
|
.withRegistry(registry)
|
|
.withBackingStore(new SingleFileMessagePackBackingStore("settings.cc"), settingsDocument)
|
|
.withGenericStore(new SingleFileMessagePackBackingStore("cache.cc"))
|
|
.build();
|
|
```
|
|
|
|
A document type has exactly one home store
|
|
(`src/GameCult.Caching/Contracts/cultcache-store-composition.md` in CultLib):
|
|
|
|
- a store added with types owns those types; otherwise the generic store is the home
|
|
- a second generic store throws, and so does a type claimed by a second store;
|
|
a refused registration attaches nothing
|
|
- `put`, `putEnvelope` and `delete` touch exactly one store; nothing is mirrored
|
|
- a write whose type has no home store throws
|
|
|
|
## Name, Index, and Global Semantics
|
|
|
|
`name`, `indexes` and `roles` can be declared on the document definition:
|
|
|
|
```ts
|
|
const playerDocument = defineDocumentType({
|
|
type: "player",
|
|
schema: playerSchema,
|
|
name: "displayName",
|
|
indexes: {
|
|
email: "email", // unique lookup
|
|
},
|
|
roles: {
|
|
faction: "faction", // selection alias, many players share one
|
|
},
|
|
});
|
|
```
|
|
|
|
`indexes` are unique (variants:ruling:unique-indexes; C# `[CultIndex]`, Rust `#[cultcache(index)]`). Within one type
|
|
and one alias no two records hold one non-blank value, judged on the result of every write and of every load; a
|
|
refusal reads `Records <k1> and <k2> (<schema name>) both hold unique index '<alias>'; an index value names one record.`
|
|
(keys in ordinal order, never the value), and writes nothing. The contract is the verdict, which stores and writes are
|
|
refused; the text is informational. On a load TS judges record by record in file order and names the first collision
|
|
it meets, where C# judges the whole store, so the two runtimes can name different pairs of one store. A blank value (absent, empty, or only whitespace) is
|
|
never indexed and never conflicts. Two types may share an alias and a value. Index values are compared as the C#
|
|
reference renders them: strings as is (never trimmed), `True`/`False`, integers as digits, a non-integral number as
|
|
its exact decimal expansion; NaN and infinities have no value. `roles` name selection and reference aliases (the
|
|
member's catalog `indexAlias`, so the catalog bytes are the same either way); they imply nothing about uniqueness and
|
|
are never indexed for lookup, so `getByIndex` on a role answers `undefined`. An alias is declared in one of the two.
|
|
|
|
### Migrating an existing `indexes` key
|
|
|
|
Apply per key, and do it BEFORE the CultLib checkout your tools load is rebuilt or updated. A consumer that loads
|
|
CultLib at run time (Bifrost reads `$VOIDBOT_CULTLIB_ROOT`, else the sibling `../CultLib` dist) has no pin to hold: its
|
|
next start after the checkout moves is the bump. A consumer that pins a version applies this before it bumps the pin.
|
|
Moving an `indexes` key to `roles` is safe on the old CultLib too when the definition lists `members` explicitly (an old
|
|
codec definition without `members` derives `indexAlias` from `indexes` only, so a moved key would lose its catalog alias),
|
|
so the consumer edit lands first:
|
|
|
|
1. Nothing looks records up by the key (`getByIndex`, `getKeyByIndex`, `getIdByIndex`; Python `get_by_index`,
|
|
`get_key_by_index`, `node.get_by_index`, `watch_by_index`; Rust `get_by_index`): it is a role. Move it to `roles`;
|
|
where a definition lists `members` explicitly, delete it from `indexes` and keep the member's `indexAlias` (the
|
|
catalog is the same).
|
|
2. Something looks it up: it stays unique only when two live records can never hold one non-blank value, because the
|
|
record key is derived from that value alone, or the value is an outside identity (email, sku, external id).
|
|
Enums, statuses, kinds, categories, parent or foreign ids, shared hashes and timestamps are roles; a lookup on
|
|
one already answered an arbitrary record, so move it to a selection (`any_of`) or `getAll` plus a filter.
|
|
3. Before the checkout moves, open each real `.cc` once with the new definitions. A refusal names both keys and the alias and
|
|
writes nothing.
|
|
|
|
Misclassifying costs differently each way. A role declared unique: the second record holding a shared value is
|
|
refused at commit, and a store already holding one refuses to open, so that store is down until the key is
|
|
reclassified (loud, names both keys and the alias). A unique key declared a role: lookups by it answer nothing and
|
|
duplicates stop being refused.
|
|
|
|
The Bifrost TS stores (at 781e6ea) call no lookup anywhere, so by rule 1 every key is a role and no key becomes unique.
|
|
All six definitions list `members` explicitly, each with its `indexAlias`, so the edit is deleting each `indexes` block:
|
|
the catalog is unchanged and the edit is safe on the CultLib it loads today. It must land before that checkout is
|
|
updated:
|
|
|
|
| Definition | Keys | Verdict |
|
|
|---|---|---|
|
|
| `update-request` (`tools/agent-transport.mjs`) | `targetRepoName`, `targetAgentIdentity`, `status` | role: many requests per repo and agent; status is an enum |
|
|
| `crossing_receipt` (`tools/bifrost-crossing-documents.mjs`) | `commandId`, `crossingKind`, `status` | role: receipts are `crossing_<commandId>_<activityKind>`, several per command; the rest are enums |
|
|
| release authority (`tools/bifrost-repository-release-authority.mjs`) | `repositoryFullName`, `upstreamRef`, `commitSha`, `status` | role: the authority id is derived from repo, ref and sha, so one sha under two refs is two records |
|
|
| `topic` (`tools/governance-threads.mjs`) | `jurisdictionRepoName`, `jurisdictionAgentIdentity`, `status` | role: many topics per jurisdiction |
|
|
| `comment` (`tools/governance-threads.mjs`) | `topicId`, `authorId`, `stance` | role: many comments per topic; stance is an enum |
|
|
| `advertisement` (`tools/provider-advertisement.mjs`) | `serviceName` | role: one record holds it today, but nothing looks it up |
|
|
|
|
You can also register them later if you feel like changing the machine after boot:
|
|
|
|
```ts
|
|
await cache.registerNameLookup(playerDocument, "displayName");
|
|
await cache.registerIndex(playerDocument, "email", "email");
|
|
```
|
|
|
|
Registration, `addBackingStore`, `pullAllBackingStores` and every write and
|
|
delete run one at a time through a per-cache queue and return promises; reads
|
|
are synchronous. A write validates the whole record (home store, global key,
|
|
name and index accessors) before any store is touched, so a refused write
|
|
changes neither store nor cache. Registering an accessor that throws on a held
|
|
record installs nothing.
|
|
|
|
Global documents are singleton-style per type, always stored under
|
|
`CultCache.GLOBAL_KEY` (`__global__`); writing one under any other key throws:
|
|
|
|
```ts
|
|
await cache.putGlobal(settingsDocument, { theme: "ash", retries: 3 });
|
|
const settings = cache.getRequiredGlobal(settingsDocument);
|
|
```
|
|
|
|
If persisted state contains two globals for the same type, `pullAllBackingStores()` fails closed instead of pretending the contradiction is charming.
|
|
|
|
## Typed Formatters
|
|
|
|
Each document definition can provide a custom formatter:
|
|
|
|
```ts
|
|
const combatLogDocument = defineDocumentType({
|
|
type: "combat-log",
|
|
schema: combatLogSchema,
|
|
formatter: {
|
|
encode(value) {
|
|
return myFastBinaryEncoder(value);
|
|
},
|
|
decode(payload) {
|
|
return myFastBinaryDecoder(payload);
|
|
},
|
|
},
|
|
});
|
|
```
|
|
|
|
If you do not supply one, `CultCacheTS` uses MessagePack plus Zod validation by default.
|
|
|
|
The 256-container nesting bound (`MAX_PAYLOAD_DEPTH`, refused with `CultCachePayloadDepthError`) covers the MessagePack the
|
|
library itself writes: the default format and a codec. A payload your formatter writes is opaque bytes in any format, so the
|
|
library does not walk it as MessagePack; its format and its bounds are your formatter's.
|
|
|
|
Zod is no longer mandatory. Any schema object with a `parse(...)` function will
|
|
do, which means generated Ajv-backed JSON-schema contracts can feed the cache
|
|
directly as long as they expose the same parse surface.
|
|
|
|
The important boundary is this:
|
|
|
|
- payload bytes are persisted as bytes
|
|
- decoding goes through the registered formatter for that document type
|
|
- the decoded value is still validated against the declared schema
|
|
|
|
So callers get typed values, and the persistence layer does not become an open polymorphic sewer.
|
|
|
|
## Slot Records and Unions
|
|
|
|
`codec` writes the reference runtimes' record shape: a MessagePack array indexed by slot key, nil in a gap, and a
|
|
union as `[armKey, [armSlots...]]`. A union value is `{ kind: armName, ...armFields }`.
|
|
|
|
```ts
|
|
import { codecs, defineDocumentType } from "@gamecult/cultcache-ts";
|
|
const { slots, union, list, lazy, string } = codecs;
|
|
|
|
const NS = "MyApp.Scene+"; // the C# names of the types, as the reference writes them into a store's schema catalog
|
|
|
|
type Node =
|
|
| { kind: "Box"; id: string; children: (Node | null)[] }
|
|
| { kind: "Text"; id: string; content: string }
|
|
| { kind: "Image"; id: string; uri: string; alt: string };
|
|
|
|
const node: codecs.Codec<Node | null> = union<Node>("Node", {
|
|
0: ["Box", slots<{ id: string; children: (Node | null)[] }>({ 0: ["id", string], 1: ["children", list(lazy(() => node))] })],
|
|
1: ["Text", slots<{ id: string; content: string }>({ 0: ["id", string], 1: ["content", string] })],
|
|
5: ["Image", slots<{ id: string; uri: string; alt: string }>({ 0: ["id", string], 1: ["uri", string], 3: ["alt", string] })], // slot 2 retired
|
|
}, `${NS}Node`);
|
|
|
|
const sceneDocument = defineDocumentType({
|
|
type: "scene",
|
|
schema: sceneSchema,
|
|
codec: slots<Scene>({ 0: ["id", string], 1: ["root", node], 2: ["title", string, ""] }),
|
|
name: "id",
|
|
});
|
|
```
|
|
|
|
- A slot is `[name, codec, default?]`. A gap is written nil and skipped on read; an extra trailing slot is skipped; a
|
|
slot missing from the payload (or nil, when a default is declared) takes its default, and is refused without one.
|
|
- A union refuses an undeclared arm key on read, naming the union, the key and the declared keys. Nothing falls back
|
|
to a default arm. An arm with no slots is `codecs.unit` and is written `[key, []]`.
|
|
- A refusal names the union, the field and the slot; it never echoes the value it refused.
|
|
- The leaves are `string`, `boolean`, `bytes`, `int32`, `int64`, `float32` and `float64`, each writing the MessagePack the
|
|
C# type writes and refusing what C# refuses (a float in an int slot, an `int` outside Int32). `int64` reads a number
|
|
when the value is a safe integer and a bigint beyond 2^53, and refuses a number beyond 2^53 on write. `list`, `array`
|
|
and `nullable` wrap a leaf, a union or a record. A `float32` slot rounds an integer once, as the reference's cast does.
|
|
- Every codec carries the reference's type name, and a store's schema catalog is derived from those names, because the
|
|
reference refuses a store whose slot names another type. The leaves know theirs; give a union or a record its C# name
|
|
as the last argument of `union(name, arms, typeName)` or `slots(spec, typeName)`. A codec with a slot it cannot name
|
|
refuses to derive members: pass `members` instead.
|
|
- An arm key is an integer within Int32 (negative too), in any MessagePack integer encoding. A float key (even `1.0`) is
|
|
refused on read, and a union refuses a key outside Int32 when it is defined. A slot key is within 0 to Int32 max.
|
|
- `codec` supplies the document's formatter and, unless `members` is given, its catalog members. A definition may not
|
|
carry both `codec` and `formatter`. `schema.parse` still runs on every decoded value.
|
|
|
|
## Raw Envelope Fast Path
|
|
|
|
For bit-compatible neighbors such as CultNet peers that already share the same
|
|
formatter contract, `CultCacheTS` can now move the persisted envelope directly:
|
|
|
|
- `getEnvelope(...)` / `getRequiredEnvelope(...)` export the canonical
|
|
MessagePack payload bytes for a typed document
|
|
- `putEnvelope(...)` ingests that envelope into another cache instance without
|
|
re-encoding the payload first
|
|
|
|
Raw envelopes may carry `schemaId` and catalog metadata. Domain callers should
|
|
not need that sludge. Replication, migration tooling, and wire-compatibility
|
|
tests do.
|
|
|
|
That still is not magic shared memory. The cache decodes once so lookups and
|
|
typed reads stay honest. But it stops doing the stupid part where identical
|
|
MessagePack payload bytes get decoded into a generic value, then re-encoded
|
|
into the same bytes again just to cross the room.
|
|
|
|
## Security Model
|
|
|
|
`CultCacheTS` should never deserialize arbitrary persisted payloads into arbitrary runtime shapes.
|
|
|
|
It only accepts persisted records whose `schemaId` or legacy `type` discriminator resolves to a registered document definition on the cache instance. Unknown schemas fail immediately. Known schemas decode through the registered formatter and schema only.
|
|
|
|
That is the TypeScript version of the original CultCache trick:
|
|
|
|
- closed discriminator set
|
|
- explicit registration
|
|
- typed formatter path
|
|
- no runtime type-name roulette
|
|
|
|
## Persistence Model
|
|
|
|
`SingleFileMessagePackBackingStore` is the first concrete store.
|
|
|
|
- it writes atomic CultCache v1 MessagePack snapshots
|
|
- the top-level MessagePack value is `[formatVersion, schemaCatalog, records]`
|
|
- each record is `[key, schemaId, storedAt, payload]`
|
|
- each catalog entry is `[schemaId, schemaName, schemaVersion, contentHash, canonicalSchemaJson, compatibleSchemaIds, members]`
|
|
- it still reads the older TS/Rust-style envelope array (object payloads are normalized in memory; opening never rewrites the file) and replaces it with a v1 snapshot on the next push, delete or flush
|
|
- it queues local writes in order, then takes the shared `<file>.lock` (`flock` on Unix, `LockFileEx` on Windows, via koffi, loaded on the first write) for each one, so C#, Rust and TypeScript writers exclude each other; the read a write merges from is taken under that lock
|
|
- it fsyncs the staged file and, on Unix, the directory after the replace; on Windows it replaces with a POSIX-semantics rename and falls back to a write-through `MoveFileExW`, retrying briefly on sharing violations
|
|
- it persists raw MessagePack payload bytes inside schema-addressed records
|
|
|
|
It is a control-plane store, not a tiny god: every write rewrites the whole file.
|