diff --git a/contracts/cultnet/cultnet.database-change-raw.schema.json b/contracts/cultnet/cultnet.database-change-raw.schema.json index d214ed6b..f1cd97eb 100644 --- a/contracts/cultnet/cultnet.database-change-raw.schema.json +++ b/contracts/cultnet/cultnet.database-change-raw.schema.json @@ -13,8 +13,8 @@ "document": { "$ref": "./cultnet.raw-document-record.schema.json" }, "recordKey": { "type": "string", "minLength": 1 }, "schemaId": { "type": "string", "minLength": 1 }, - "commitId": { "type": "string", "minLength": 1 }, - "commitCount": { "type": "integer", "minimum": 2 } + "commitId": { "type": ["string", "null"], "minLength": 1 }, + "commitCount": { "type": ["integer", "null"], "minimum": 2 } }, "allOf": [ { diff --git a/docs/start/README.md b/docs/start/README.md index c388f8b6..e52f7d73 100644 --- a/docs/start/README.md +++ b/docs/start/README.md @@ -94,7 +94,6 @@ const itemDocument = defineDocumentType({ schemaVersion: "gamecult.first_call_item.v1", schema: z.object({ name: z.string(), category: z.string(), value: z.number().int() }), name: "name", - indexes: { category: "category" }, }); const registry = defineDocumentRegistry(itemDocument); diff --git a/packages/cultcache-ts/CHANGELOG.md b/packages/cultcache-ts/CHANGELOG.md index eefea3b5..9850dfe4 100644 --- a/packages/cultcache-ts/CHANGELOG.md +++ b/packages/cultcache-ts/CHANGELOG.md @@ -5,6 +5,15 @@ All notable changes to `@gamecult/cultcache-ts` are recorded here, following ## [Unreleased] +### Breaking + +- `indexes` are unique, as C# `[CultIndex]`: a write or a load that leaves two records of a type holding one non-blank + value under one alias is refused naming both keys and the alias, and writes nothing. Index text is the C# canonical + text (booleans `True`/`False`, integers as digits, doubles as the exact expansion, blank values never indexed), so + an empty string is no longer a value. `registerIndex` over held duplicates installs nothing. A new `roles` map names + the many-row selection aliases (never indexed for lookup, same catalog `indexAlias`). The per-key migration rule is + in the README; classify every key before the CultLib your tools load is updated. The refusal verdict is the contract, its text is informational. + ### Changed - Opening a legacy (pre-v1) store is a pure read: payloads that were not bytes are normalized in memory and the diff --git a/packages/cultcache-ts/README.md b/packages/cultcache-ts/README.md index dc53c42d..97fb3cdd 100644 --- a/packages/cultcache-ts/README.md +++ b/packages/cultcache-ts/README.md @@ -93,12 +93,16 @@ const itemDocument = defineDocumentType({ schemaVersion: "item.v1", schema: z.object({ name: z.string(), + sku: z.string(), category: z.string(), value: z.number().int(), }), name: "name", indexes: { - category: "category", + sku: "sku", // unique: a sku names one item + }, + roles: { + category: "category", // a selection alias: many items share a category }, }); @@ -122,6 +126,7 @@ await cache.pullAllBackingStores(); await cache.put(itemDocument, "item:potion", { name: "Potion", + sku: "POT-001", category: "Consumable", value: 50, }); @@ -132,7 +137,7 @@ await cache.putGlobal(settingsDocument, { }); const potion = cache.getByName(itemDocument, "Potion"); -const consumable = cache.getByIndex(itemDocument, "category", "Consumable"); +const bySku = cache.getByIndex(itemDocument, "sku", "POT-001"); const settings = cache.getRequiredGlobal(settingsDocument); ``` @@ -194,7 +199,7 @@ A document type has exactly one home store ## Name, Index, and Global Semantics -`name` and `indexes` can be declared on the document definition: +`name`, `indexes` and `roles` can be declared on the document definition: ```ts const playerDocument = defineDocumentType({ @@ -202,17 +207,70 @@ const playerDocument = defineDocumentType({ schema: playerSchema, name: "displayName", indexes: { - faction: "faction", - email: "email", + email: "email", // unique lookup + }, + roles: { + faction: "faction", // selection alias, many players share one }, }); ``` +`indexes` are unique (variants:ruling:unique-indexes; C# `[CultIndex]`, Rust `#[cultcache(index)]`). Within one type +and one alias no two records hold one non-blank value, judged on the result of every write and of every load; a +refusal reads `Records and () both hold unique index ''; an index value names one record.` +(keys in ordinal order, never the value), and writes nothing. The contract is the verdict, which stores and writes are +refused; the text is informational. On a load TS judges record by record in file order and names the first collision +it meets, where C# judges the whole store, so the two runtimes can name different pairs of one store. A blank value (absent, empty, or only whitespace) is +never indexed and never conflicts. Two types may share an alias and a value. Index values are compared as the C# +reference renders them: strings as is (never trimmed), `True`/`False`, integers as digits, a non-integral number as +its exact decimal expansion; NaN and infinities have no value. `roles` name selection and reference aliases (the +member's catalog `indexAlias`, so the catalog bytes are the same either way); they imply nothing about uniqueness and +are never indexed for lookup, so `getByIndex` on a role answers `undefined`. An alias is declared in one of the two. + +### Migrating an existing `indexes` key + +Apply per key, and do it BEFORE the CultLib checkout your tools load is rebuilt or updated. A consumer that loads +CultLib at run time (Bifrost reads `$VOIDBOT_CULTLIB_ROOT`, else the sibling `../CultLib` dist) has no pin to hold: its +next start after the checkout moves is the bump. A consumer that pins a version applies this before it bumps the pin. +Moving an `indexes` key to `roles` is safe on the old CultLib too when the definition lists `members` explicitly (an old +codec definition without `members` derives `indexAlias` from `indexes` only, so a moved key would lose its catalog alias), +so the consumer edit lands first: + +1. Nothing looks records up by the key (`getByIndex`, `getKeyByIndex`, `getIdByIndex`; Python `get_by_index`, + `get_key_by_index`, `node.get_by_index`, `watch_by_index`; Rust `get_by_index`): it is a role. Move it to `roles`; + where a definition lists `members` explicitly, delete it from `indexes` and keep the member's `indexAlias` (the + catalog is the same). +2. Something looks it up: it stays unique only when two live records can never hold one non-blank value, because the + record key is derived from that value alone, or the value is an outside identity (email, sku, external id). + Enums, statuses, kinds, categories, parent or foreign ids, shared hashes and timestamps are roles; a lookup on + one already answered an arbitrary record, so move it to a selection (`any_of`) or `getAll` plus a filter. +3. Before the checkout moves, open each real `.cc` once with the new definitions. A refusal names both keys and the alias and + writes nothing. + +Misclassifying costs differently each way. A role declared unique: the second record holding a shared value is +refused at commit, and a store already holding one refuses to open, so that store is down until the key is +reclassified (loud, names both keys and the alias). A unique key declared a role: lookups by it answer nothing and +duplicates stop being refused. + +The Bifrost TS stores (at 781e6ea) call no lookup anywhere, so by rule 1 every key is a role and no key becomes unique. +All six definitions list `members` explicitly, each with its `indexAlias`, so the edit is deleting each `indexes` block: +the catalog is unchanged and the edit is safe on the CultLib it loads today. It must land before that checkout is +updated: + +| Definition | Keys | Verdict | +|---|---|---| +| `update-request` (`tools/agent-transport.mjs`) | `targetRepoName`, `targetAgentIdentity`, `status` | role: many requests per repo and agent; status is an enum | +| `crossing_receipt` (`tools/bifrost-crossing-documents.mjs`) | `commandId`, `crossingKind`, `status` | role: receipts are `crossing__`, several per command; the rest are enums | +| release authority (`tools/bifrost-repository-release-authority.mjs`) | `repositoryFullName`, `upstreamRef`, `commitSha`, `status` | role: the authority id is derived from repo, ref and sha, so one sha under two refs is two records | +| `topic` (`tools/governance-threads.mjs`) | `jurisdictionRepoName`, `jurisdictionAgentIdentity`, `status` | role: many topics per jurisdiction | +| `comment` (`tools/governance-threads.mjs`) | `topicId`, `authorId`, `stance` | role: many comments per topic; stance is an enum | +| `advertisement` (`tools/provider-advertisement.mjs`) | `serviceName` | role: one record holds it today, but nothing looks it up | + You can also register them later if you feel like changing the machine after boot: ```ts await cache.registerNameLookup(playerDocument, "displayName"); -await cache.registerIndex(playerDocument, "faction", "faction"); +await cache.registerIndex(playerDocument, "email", "email"); ``` Registration, `addBackingStore`, `pullAllBackingStores` and every write and diff --git a/packages/cultcache-ts/src/cult-cache.ts b/packages/cultcache-ts/src/cult-cache.ts index 88ab94e9..775118d4 100644 --- a/packages/cultcache-ts/src/cult-cache.ts +++ b/packages/cultcache-ts/src/cult-cache.ts @@ -224,7 +224,7 @@ export class CultCache { return this.#serial(() => { const registered = this.#requireDefinition(definition); const nameAccessor = this.#compileAccessor(definition.type, "name", accessor); - const derived = this.#deriveTypeLookups(definition.type, nameAccessor, registered.indexAccessors); + const derived = this.#deriveTypeLookups(definition.type, registered.catalogEntry.schemaName, nameAccessor, registered.indexAccessors); this.#installAccessors(registered, nameAccessor, registered.indexAccessors, derived); return definition; }); @@ -243,7 +243,7 @@ export class CultCache { const registered = this.#requireDefinition(definition); const indexAccessors = new Map(registered.indexAccessors); indexAccessors.set(indexName, this.#compileAccessor(definition.type, `index "${indexName}"`, accessor)); - const derived = this.#deriveTypeLookups(definition.type, registered.nameAccessor, indexAccessors); + const derived = this.#deriveTypeLookups(definition.type, registered.catalogEntry.schemaName, registered.nameAccessor, indexAccessors); this.#installAccessors(registered, registered.nameAccessor, indexAccessors, derived); return definition; }); @@ -418,6 +418,7 @@ export class CultCache { return key ? this.get(definition, key) : undefined; } + // A unique index: one key per non-blank value. A roles alias is never indexed and answers undefined. getKeyByIndex( definition: TDefinition, indexName: string, @@ -579,7 +580,7 @@ export class CultCache { if (existingBySchemaName && existingBySchemaName !== registered) { throw refuse(existingBySchemaName, registered.catalogEntry, `schema name "${registered.catalogEntry.schemaName}"`); } - const derived = this.#deriveTypeLookups(definition.type, nameAccessor, indexAccessors); + const derived = this.#deriveTypeLookups(definition.type, registered.catalogEntry.schemaName, nameAccessor, indexAccessors); // Every check and accessor has run; install. this.#definitions.set(definition.type, registered); @@ -795,11 +796,13 @@ export class CultCache { switch (typeof value) { case "string": - return value; - case "number": + return isBlankIndexText(value) ? undefined : value; case "boolean": + return value ? "True" : "False"; case "bigint": - return String(value); + return value.toString(); + case "number": + return canonicalNumberText(value); default: throw new Error(`CultCache ${label} accessor for type "${type}" produced a non-scalar value.`); } @@ -894,6 +897,8 @@ export class CultCache { } } + const lookups = this.#lookupsFor(registered.nameAccessor, registered.indexAccessors, value); + this.#refuseIndexSharing(state, entry.type, registered.catalogEntry.schemaName, entry.key, lookups); return { ...entry, payload: this.#cloneBytes(entry.payload), @@ -902,10 +907,22 @@ export class CultCache { schemaId: registered.catalogEntry.schemaId, catalogEntry: registered.catalogEntry, value, - lookups: this.#lookupsFor(registered.nameAccessor, registered.indexAccessors, value), + lookups, }; } + // The one index rule (variants:ruling:unique-indexes): within one type and one unique alias, no two records hold one + // non-blank value. The held key at the landing record's own key is its previous self and keeps its value. The message + // names both keys, ordinal, and the alias, never the value. + #refuseIndexSharing(state: HydratedState, type: string, schemaName: string, key: string, lookups: EntryLookups): void { + for (const alias of [...lookups.indexes.keys()].sort(compareOrdinal)) { + const held = state.indexToKeyMaps.get(this.#indexId(type, alias))?.get(lookups.indexes.get(alias)!); + if (held !== undefined && held !== key) { + throw indexSharingRefusal(held, key, schemaName, alias); + } + } + } + // Infallible half of admission: replaces any held record at the same key and installs the admitted one. #installEntry(state: HydratedState, registered: RegisteredDefinition, hydrated: HydratedEntry): void { const entryId = this.#entryId(hydrated.type, hydrated.key); @@ -966,12 +983,14 @@ export class CultCache { // Runs the candidate accessors over every held record of the type into local maps. #deriveTypeLookups( type: string, + schemaName: string, nameAccessor: Accessor | undefined, indexAccessors: ReadonlyMap, ): DerivedTypeLookups { const entryLookups: Array<[HydratedEntry, EntryLookups]> = []; const nameMap = nameAccessor ? new Map() : undefined; const indexMaps = new Map>(); + const sharing = new Map(); for (const indexName of indexAccessors.keys()) { indexMaps.set(indexName, new Map()); } @@ -983,10 +1002,23 @@ export class CultCache { nameMap?.set(lookups.name, entry.key); } for (const [indexName, indexValue] of lookups.indexes) { - indexMaps.get(indexName)?.set(indexValue, entry.key); + const map = indexMaps.get(indexName)!; + const holder = map.get(indexValue); + if (holder === undefined) { + map.set(indexValue, entry.key); + } else { + sharing.set(`${indexName}\u0000${indexValue}`, [...(sharing.get(`${indexName}\u0000${indexValue}`) ?? [holder]), entry.key]); + } } } + // Judged whole: any value held by two records refuses, naming the two smallest of its keys. Held records already + // passed the judge for every alias installed before, so only the alias being registered can be shared here. + for (const [id, keys] of sharing) { + keys.sort(compareOrdinal); + throw indexSharingRefusal(keys[0], keys[1], schemaName, id.slice(0, id.indexOf("\u0000"))); + } + return { entryLookups, nameMap, indexMaps }; } @@ -1057,3 +1089,42 @@ export class CultCache { return documentPayloadFromBytes(new Uint8Array(documentPayloadBytes(payload))); } } + +function compareOrdinal(a: string, b: string): number { + return a < b ? -1 : a > b ? 1 : 0; +} + +function indexSharingRefusal(keyA: string, keyB: string, schemaName: string, alias: string): Error { + const [first, second] = compareOrdinal(keyA, keyB) <= 0 ? [keyA, keyB] : [keyB, keyA]; + return new Error(`Records ${first} and ${second} (${schemaName}) both hold unique index '${alias}'; an index value names one record.`); +} + +// Blank is no value: empty or only C# White_Space characters. Not JavaScript's \s, which counts U+FEFF and omits U+0085. +const BLANK_INDEX_TEXT = /^[\t-\r \u0085\u00A0\u1680\u2000-\u200A\u2028\u2029\u202F\u205F\u3000]*$/; + +function isBlankIndexText(value: string): boolean { + return BLANK_INDEX_TEXT.test(value); +} + +// The canonical text of a number, as C# RenderCanonicalNumber: integers as their digits (-0 is 0), a non-integral finite +// double as its exact decimal expansion, NaN and infinities as no value. +function canonicalNumberText(value: number): string | undefined { + if (!Number.isFinite(value)) { + return undefined; + } + if (Number.isInteger(value)) { + return BigInt(value).toString(); + } + + // value = mantissa * 2^exponent with exponent < 0, so value = mantissa * 5^k / 10^k for k = -exponent. + const view = new DataView(new ArrayBuffer(8)); + view.setFloat64(0, value); + const bits = view.getBigUint64(0); + const biasedExponent = Number((bits >> 52n) & 0x7ffn); + const fraction = bits & 0xfffffffffffffn; + const mantissa = biasedExponent === 0 ? fraction : fraction | (1n << 52n); + const k = -((biasedExponent === 0 ? 1 : biasedExponent) - 1075); + const digits = (mantissa * 5n ** BigInt(k)).toString().padStart(k + 1, "0"); + const text = `${digits.slice(0, digits.length - k)}.${digits.slice(digits.length - k)}`.replace(/0+$/, "").replace(/\.$/, ""); + return value < 0 ? `-${text}` : text; +} diff --git a/packages/cultcache-ts/src/document.ts b/packages/cultcache-ts/src/document.ts index 99ec170c..bc4ace88 100644 --- a/packages/cultcache-ts/src/document.ts +++ b/packages/cultcache-ts/src/document.ts @@ -32,23 +32,22 @@ export function defineDocumentType( throw new Error(`CultCache document type "${definition.type}" declares an invalid name accessor.`); } - if (Array.isArray(definition.indexes)) { - const seen = new Set(); - for (const index of definition.indexes) { - if (!index.name || index.name.trim().length === 0) { - throw new Error(`CultCache document type "${definition.type}" declares an index with an empty name.`); - } - if (seen.has(index.name)) { - throw new Error(`CultCache document type "${definition.type}" declares duplicate index "${index.name}".`); - } - seen.add(index.name); - } - } else if (definition.indexes) { - for (const name of Object.keys(definition.indexes)) { + const declared = new Set(); + for (const [kind, set] of [["index", definition.indexes], ["role", definition.roles]] as const) { + const names = new Set(); + for (const name of Array.isArray(set) ? set.map((entry) => entry.name) : set ? Object.keys(set) : []) { if (!name || name.trim().length === 0) { - throw new Error(`CultCache document type "${definition.type}" declares an index with an empty name.`); + throw new Error(`CultCache document type "${definition.type}" declares ${kind === "index" ? "an index" : "a role"} with an empty name.`); } + if (names.has(name)) { + throw new Error(`CultCache document type "${definition.type}" declares duplicate ${kind} "${name}".`); + } + if (kind === "role" && declared.has(name)) { + throw new Error(`CultCache document type "${definition.type}" declares "${name}" as both an index and a role; an index already names its role.`); + } + names.add(name); } + for (const name of names) declared.add(name); } if (definition.members) { @@ -138,10 +137,13 @@ export function resolveDocumentMembers(definition: AnyCultCacheDocumentDefinitio return definition.members ?? []; } const codec = definition.codec; - const indexAlias = (slotName: string) => - !Array.isArray(definition.indexes) && definition.indexes - ? Object.entries(definition.indexes).find(([, accessor]) => accessor === slotName)?.[0] ?? null - : null; + const indexAlias = (slotName: string) => { + for (const set of [definition.indexes, definition.roles]) { + const alias = !Array.isArray(set) && set ? Object.entries(set).find(([, accessor]) => accessor === slotName)?.[0] : undefined; + if (alias !== undefined) return alias; + } + return null; + }; return codec.members.map((member, index) => ({ ...member, isName: (definition.name as unknown) === codec.names[index], diff --git a/packages/cultcache-ts/src/types.ts b/packages/cultcache-ts/src/types.ts index 88c20f2e..6f6c0923 100644 --- a/packages/cultcache-ts/src/types.ts +++ b/packages/cultcache-ts/src/types.ts @@ -59,9 +59,21 @@ export interface CultCacheDocumentDefinition>; + /** + * Unique lookup aliases. A non-blank value names one record of the type: a write that would leave two records holding + * one value, and a store that already holds two, is refused naming both keys and the alias. Blank values (absent, + * empty, or only whitespace) are never indexed. `getByIndex` and `getKeyByIndex` answer from these. + */ indexes?: | Readonly>>> | readonly CultCacheDocumentIndexDefinition>[]; + /** + * Selection and reference aliases: the member's catalog `indexAlias`. A role implies nothing about uniqueness, many + * records may share a value, and it is never indexed for lookup. An alias is declared in `indexes` or `roles`, not both. + */ + roles?: + | Readonly>>> + | readonly CultCacheDocumentIndexDefinition>[]; formatter?: CultCacheDocumentFormatter>; codec?: SlotsCodec & object>; } diff --git a/packages/cultcache-ts/test/first-call.test.ts b/packages/cultcache-ts/test/first-call.test.ts index 3c0e22e5..45f6edd2 100644 --- a/packages/cultcache-ts/test/first-call.test.ts +++ b/packages/cultcache-ts/test/first-call.test.ts @@ -24,7 +24,6 @@ const itemDocument = defineDocumentType({ schemaVersion: "gamecult.first_call_item.v1", schema: z.object({ name: z.string(), category: z.string(), value: z.number().int() }), name: "name", - indexes: { category: "category" }, }); const registry = defineDocumentRegistry(itemDocument); diff --git a/packages/cultcache-ts/test/unique-index.test.ts b/packages/cultcache-ts/test/unique-index.test.ts new file mode 100644 index 00000000..67af97fb --- /dev/null +++ b/packages/cultcache-ts/test/unique-index.test.ts @@ -0,0 +1,504 @@ +import assert from "node:assert/strict"; +import { existsSync } from "node:fs"; +import { copyFile, mkdtemp, readFile, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { test } from "node:test"; + +import { decode } from "@msgpack/msgpack"; +import { z } from "zod"; + +import { codecs } from "../src"; +import { CultCache } from "../src/cult-cache"; +import { defineDocumentType, resolveDocumentMembers } from "../src/document"; +import { SingleFileMessagePackBackingStore } from "../src/single-file-messagepack-backing-store"; +import type { AnyCultCacheDocumentDefinition } from "../src/types"; + +// variants:ruling:unique-indexes. `indexes` are unique within one type, `roles` are selection aliases. The vectors in +// tests/vectors/unique-index are written by the C# reference; the rest of the behaviour is pinned here. + +function findVectors(from: string): string { + for (let dir = from; ; dir = resolve(dir, "..")) { + if (existsSync(join(dir, "tests", "vectors", "unique-index"))) return join(dir, "tests", "vectors", "unique-index"); + if (resolve(dir, "..") === dir) throw new Error("tests/vectors/unique-index not found above the test."); + } +} +const vectors = findVectors(__dirname); + +const CANARY = "CANARY-uidx-5e1b"; + +const skuSchema = z.object({ sku: z.any().optional(), group: z.string().optional() }); + +function item(type: string, where: "indexes" | "roles" | "none" = "indexes", schemaName = `tests.${type}`) { + return defineDocumentType({ + type, + schemaId: `${schemaName}.v1`, + schemaName, + schemaVersion: `${schemaName}.v1`, + schema: skuSchema, + ...(where === "none" ? {} : { [where]: { sku: "sku" } }), + }); +} + +// A definition whose payload is never read back, so a value of any scalar kind (a bigint included) can be held and indexed. +function scalar(type: string) { + const held: unknown[] = []; + return defineDocumentType({ + type, + schemaId: `tests.${type}.v1`, + schemaName: `tests.${type}`, + schemaVersion: `tests.${type}.v1`, + schema: skuSchema, + formatter: { + encode: (value: unknown) => new Uint8Array([held.push(value) - 1]), + decode: (payload: Uint8Array) => held[payload[0]] as never, + }, + indexes: { sku: "sku" }, + }); +} + +function memory(...definitions: AnyCultCacheDocumentDefinition[]) { + let builder = CultCache.builder(); + for (const definition of definitions) builder = builder.withDocumentType(definition); + return builder.build(); +} + +function refusal(keyA: string, keyB: string, schemaName: string, alias: string) { + return `Records ${keyA} and ${keyB} (${schemaName}) both hold unique index '${alias}'; an index value names one record.`; +} + +test("a put of a held value at another key is refused naming both keys and the alias, and nothing moves", async () => { + const dir = await mkdtemp(join(tmpdir(), "uidx-")); + try { + const file = join(dir, "items.cc"); + const document = item("item"); + const cache = CultCache.builder().withDocumentType(document).withGenericStore(new SingleFileMessagePackBackingStore(file)).build(); + await cache.pullAllBackingStores(); + await cache.put(document, "zeta", { sku: CANARY }); + const before = await readFile(file); + const snapshot = cache.snapshot(); + + await assert.rejects(cache.put(document, "alpha", { sku: CANARY }), (error: Error) => { + assert.equal(error.message, refusal("alpha", "zeta", "tests.item", "sku")); + assert.ok(!error.message.includes(CANARY), "the value is never echoed"); + return true; + }); + assert.deepEqual(await readFile(file), before, "the store is byte-identical"); + assert.deepEqual(cache.snapshot(), snapshot); + assert.equal(cache.get(document, "alpha"), undefined); + assert.equal(cache.getKeyByIndex(document, "sku", CANARY), "zeta"); + } finally { + await rm(dir, { recursive: true, force: true }); + } +}); + +test("rewriting a record's own key with its own value lands", async () => { + const document = item("item"); + const cache = memory(document); + await cache.put(document, "a", { sku: "S", group: "one" }); + await cache.put(document, "a", { sku: "S", group: "two" }); + assert.equal(cache.getKeyByIndex(document, "sku", "S"), "a"); + assert.equal(cache.get(document, "a")?.group, "two"); +}); + +test("a removed or changed holder releases its value", async () => { + const document = item("item"); + const cache = memory(document); + await cache.put(document, "b", { sku: "S" }); + await assert.rejects(cache.put(document, "a", { sku: "S" }), /Records a and b/u); + assert.equal(cache.getKeyByIndex(document, "sku", "S"), "b", "the refused write did not take the value"); + + await cache.delete(document, "b"); + assert.equal(cache.getKeyByIndex(document, "sku", "S"), undefined); + await cache.put(document, "a", { sku: "S" }); + assert.equal(cache.getKeyByIndex(document, "sku", "S"), "a"); + + await cache.put(document, "a", { sku: "T" }); + await cache.put(document, "b", { sku: "S" }); + assert.equal(cache.getKeyByIndex(document, "sku", "S"), "b"); + assert.equal(cache.getKeyByIndex(document, "sku", "T"), "a"); +}); + +test("the keys in a refusal are in ordinal order whichever one lands second", async () => { + const document = item("item"); + const cache = memory(document); + await cache.put(document, "m", { sku: "S" }); + await assert.rejects(cache.put(document, "z", { sku: "S" }), { message: refusal("m", "z", "tests.item", "sku") }); + await assert.rejects(cache.put(document, "B", { sku: "S" }), { message: refusal("B", "m", "tests.item", "sku") }); +}); + +// The C# White_Space set, and nothing else, is blank. Every UTF-16 code unit is judged, so a rule written for a few +// characters (or for JavaScript's \s, which counts U+FEFF and omits U+0085) cannot pass. +const CSHARP_WHITE_SPACE = (code: number) => + (code >= 0x09 && code <= 0x0d) || code === 0x20 || code === 0x85 || code === 0xa0 || code === 0x1680 || + (code >= 0x2000 && code <= 0x200a) || code === 0x2028 || code === 0x2029 || code === 0x202f || code === 0x205f || code === 0x3000; + +test("blank is exactly the C# White_Space set: every code unit, alone, repeated and padded", async () => { + const document = item("item"); + const cache = memory(document); + const disagreements: string[] = []; + for (let code = 0; code <= 0xffff; code++) { + if (code >= 0xd800 && code <= 0xdfff) continue; // lone surrogates do not survive MessagePack text + const character = String.fromCharCode(code); + await cache.put(document, "x", { sku: character }); + let blank = true; + try { + await cache.put(document, "y", { sku: character }); + } catch { + blank = false; + } + if (blank !== CSHARP_WHITE_SPACE(code)) disagreements.push(code.toString(16)); + await cache.delete(document, "x"); + await cache.delete(document, "y"); + } + assert.deepEqual(disagreements, []); + + // Not trimmed: a padded value is the value, distinct from its trimmed form. + await cache.put(document, "x", { sku: " a " }); + await cache.put(document, "y", { sku: "a" }); + assert.equal(cache.getKeyByIndex(document, "sku", " a "), "x"); + assert.equal(cache.getKeyByIndex(document, "sku", "a"), "y"); + await assert.rejects(cache.put(document, "z", { sku: " a " }), /Records x and z/u); +}); + +test("blank values are never indexed and never conflict: empty, spaces, absent", async () => { + const document = item("item"); + const cache = memory(document); + for (const [key, sku] of [["a", ""], ["b", " "], ["c", "\u0085"], ["d", undefined], ["e", "　"]] as const) { + await cache.put(document, key, { sku }); + } + assert.equal(cache.getKeyByIndex(document, "sku", ""), undefined); + assert.equal(cache.getKeyByIndex(document, "sku", " "), undefined); + // U+FEFF and U+001F are values. + await cache.put(document, "f", { sku: "" }); + await assert.rejects(cache.put(document, "g", { sku: "" }), /Records f and g/u); + await cache.put(document, "h", { sku: "\u001F" }); + await assert.rejects(cache.put(document, "i", { sku: "\u001F" }), /Records h and i/u); +}); + +test("two types may share an alias and a value", async () => { + const first = item("first"); + const second = item("second"); + const cache = memory(first, second); + await cache.put(first, "a", { sku: "S" }); + await cache.put(second, "b", { sku: "S" }); + assert.equal(cache.getKeyByIndex(first, "sku", "S"), "a"); + assert.equal(cache.getKeyByIndex(second, "sku", "S"), "b"); +}); + +test("roles may share values and are not looked up", async () => { + const document = item("item", "roles"); + const cache = memory(document); + await cache.put(document, "a", { sku: "S" }); + await cache.put(document, "b", { sku: "S" }); + await cache.put(document, "c", { sku: "S" }); + assert.equal(cache.getKeyByIndex(document, "sku", "S"), undefined); + assert.equal(cache.getByIndex(document, "sku", "S"), undefined); +}); + +test("an alias declared in indexes and roles is refused at definition; each map is checked like indexes", () => { + const base = { type: "bad", schema: skuSchema, schemaId: "tests.bad.v1", schemaName: "tests.bad", schemaVersion: "tests.bad.v1" }; + assert.throws( + () => defineDocumentType({ ...base, indexes: { sku: "sku" }, roles: { sku: "sku" } }), + { message: 'CultCache document type "bad" declares "sku" as both an index and a role; an index already names its role.' }, + ); + assert.throws(() => defineDocumentType({ ...base, roles: { " ": "sku" } }), /declares a role with an empty name/u); + assert.throws( + () => defineDocumentType({ ...base, roles: [{ name: "r", accessor: "sku" }, { name: "r", accessor: "group" }] }), + /declares duplicate role "r"/u, + ); + assert.doesNotThrow(() => defineDocumentType({ ...base, indexes: { sku: "sku" }, roles: { group: "group" } })); +}); + +test("a codec definition's catalog entry is identical with an alias under indexes or roles", async () => { + const codec = codecs.slots<{ sku: string; group: string }>({ 0: ["sku", codecs.string], 1: ["group", codecs.string] }, "T.Item"); + const make = (where: "indexes" | "roles") => + defineDocumentType({ + type: "codec_item", + schemaId: "tests.codec_item.v1", + schemaName: "tests.codec_item", + schemaVersion: "tests.codec_item.v1", + schema: z.object({ sku: z.string(), group: z.string() }), + codec, + [where]: { sku: "sku" }, + }); + const catalog = async (definition: AnyCultCacheDocumentDefinition) => { + const cache = memory(definition); + await cache.put(definition, "a", { sku: "S", group: "g" } as never); + return cache.snapshot()[0].catalogEntry!; + }; + const members = (definition: AnyCultCacheDocumentDefinition) => resolveDocumentMembers(definition); + assert.deepEqual(members(make("indexes")), members(make("roles"))); + assert.equal(members(make("roles")).find((member) => member.memberName === "Sku")?.indexAlias, "sku"); + const [unique, role] = [await catalog(make("indexes")), await catalog(make("roles"))]; + assert.equal(unique.canonicalSchemaJson, role.canonicalSchemaJson); + assert.equal(unique.contentHash, role.contentHash); + assert.equal(unique.schemaId, role.schemaId); +}); + +test("vectors values.msgpack: every row's canonical text, blank rows no value", async () => { + const rows = decode(await readFile(join(vectors, "values.msgpack")), { useBigInt64: true }) as [string, string, unknown, string | null][]; + assert.ok(rows.length >= 20); + for (const [label, kind, value, text] of rows) { + const document = scalar(`value_${label.replaceAll(/[^a-z0-9]/gu, "_")}`); + const cache = memory(document); + void kind; + const normalized = value; + await cache.put(document, "one", { sku: normalized }); + if (text === null) { + await cache.put(document, "two", { sku: normalized }); + assert.equal(cache.get(document, "two") !== undefined, true, `${label}: a blank value commits twice`); + for (const probe of ["", " ", "NaN", "0", "Infinity", "-Infinity", "null", "undefined"]) { + assert.equal(cache.getKeyByIndex(document, "sku", probe), undefined, `${label}: found by '${probe}'`); + } + } else { + assert.equal(cache.getKeyByIndex(document, "sku", text), "one", `${label}: found by its text`); + await assert.rejects(cache.put(document, "two", { sku: normalized }), /Records one and two/u, label); + } + } +}); + +// An independent oracle for the double rule: the text must denote exactly the double, so N / 10^d == mantissa * 2^exponent +// in exact integer arithmetic, with no trailing zero and no bare point. A rule written to a few spellings cannot pass. +function exactlyDenotes(text: string, value: number): boolean { + const match = /^(-?)(\d+)(?:\.(\d*[1-9]))?$/u.exec(text); + if (!match) return false; + const decimals = match[3]?.length ?? 0; + const numerator = BigInt(match[2] + (match[3] ?? "")); + const view = new DataView(new ArrayBuffer(8)); + view.setFloat64(0, Math.abs(value)); + const bits = view.getBigUint64(0); + const biased = Number((bits >> 52n) & 0x7ffn); + const mantissa = biased === 0 ? bits & 0xfffffffffffffn : (bits & 0xfffffffffffffn) | (1n << 52n); + const exponent = (biased === 0 ? 1 : biased) - 1075; + const lhs = numerator * (exponent >= 0 ? 1n : 2n ** BigInt(-exponent)); + const rhs = mantissa * (exponent >= 0 ? 2n ** BigInt(exponent) : 1n) * 10n ** BigInt(decimals); + return (match[1] === "-") === (value < 0) && lhs === rhs; +} + +test("a number's index text is its exact value: integers as digits, doubles as the exact expansion, NaN and infinities none", async () => { + const document = scalar("number"); + const cache = memory(document); + const probes = [0.1, -0.1, 2.5e-7, 1.5, -123.456, 5e-324, 2.2250738585072014e-308, 1e-10, 123456789.123456789, Number.MIN_VALUE * 3, + ...Array.from({ length: 200 }, (_, i) => Math.sin(i + 1) * 10 ** ((i % 40) - 20)), 1e21 + 2 ** 20 + 0.5 * 0]; + // Round trip through lookups: the exact expansion of each probe is found, and it denotes the probe exactly. + for (const value of probes) { + if (Number.isInteger(value)) continue; + await cache.put(document, "k", { sku: value }); + const candidates = candidateTexts(value); + const found = candidates.filter((candidate) => cache.getKeyByIndex(document, "sku", candidate) === "k"); + assert.equal(found.length, 1, `${value}: exactly one spelling is its index text`); + assert.ok(exactlyDenotes(found[0], value), `${value}: ${found[0]} denotes it exactly`); + } + + await cache.put(document, "k", { sku: 1e21 }); + assert.equal(cache.getKeyByIndex(document, "sku", "1000000000000000000000"), "k"); + await cache.put(document, "k", { sku: -0 }); + assert.equal(cache.getKeyByIndex(document, "sku", "0"), "k"); + await cache.put(document, "k", { sku: 12n }); + assert.equal(cache.getKeyByIndex(document, "sku", "12"), "k"); + await cache.put(document, "k", { sku: true }); + assert.equal(cache.getKeyByIndex(document, "sku", "True"), "k"); + await cache.put(document, "k", { sku: false }); + assert.equal(cache.getKeyByIndex(document, "sku", "False"), "k"); + for (const value of [Number.NaN, Number.POSITIVE_INFINITY, Number.NEGATIVE_INFINITY]) { + await cache.put(document, "p", { sku: value }); + await cache.put(document, "q", { sku: value }); + } +}); + +// Spellings a lookup could be probed with: the exact decimal expansion computed by an arbitrary-precision route. +function candidateTexts(value: number): string[] { + const view = new DataView(new ArrayBuffer(8)); + view.setFloat64(0, Math.abs(value)); + const bits = view.getBigUint64(0); + const biased = Number((bits >> 52n) & 0x7ffn); + const mantissa = biased === 0 ? bits & 0xfffffffffffffn : (bits & 0xfffffffffffffn) | (1n << 52n); + const k = 1075 - (biased === 0 ? 1 : biased); + const scaled = (mantissa * 5n ** BigInt(k)).toString().padStart(k + 1, "0"); + const point = scaled.length - k; + const text = `${scaled.slice(0, point)}.${scaled.slice(point)}`.replace(/0+$/u, "").replace(/\.$/u, ""); + return [(value < 0 ? "-" : "") + text, String(value), value.toString()].filter((spelling, i, all) => all.indexOf(spelling) === i); +} + +function twoDefinitions(schemaId: string, where: "indexes" | "roles") { + return defineDocumentType({ + type: "unique_item", + schemaId, + schemaName: "tests.unique_item", + schemaVersion: "tests.unique_item.v1", + schema: z.object({ sku: z.string(), category: z.string() }), + codec: codecs.slots<{ sku: string; category: string }>({ 0: ["sku", codecs.string], 1: ["category", codecs.string] }, "UqVecItem"), + [where]: { Sku: "sku" }, + }); +} + +test("vectors duplicate-store.cc and blank-store.cc hold their verdicts", async () => { + const outcomes = decode(await readFile(join(vectors, "outcomes.msgpack"))) as [string, string | null, string, string | null, string | null][]; + assert.equal(outcomes.length, 3); + const dir = await mkdtemp(join(tmpdir(), "uidx-vec-")); + try { + for (const [name, alias, outcome, key1, key2] of outcomes) { + const file = join(dir, `${outcomes.indexOf(outcomes.find((row) => row[0] === name && row[1] === alias)!)}-${name}`); + await copyFile(join(vectors, name), file); + const before = await readFile(file); + const [envelope] = await new SingleFileMessagePackBackingStore(file).pullAll(); + const document = twoDefinitions(envelope.schemaId!, alias === null ? "roles" : "indexes"); + const cache = CultCache.builder().withDocumentType(document).withGenericStore(new SingleFileMessagePackBackingStore(file)).build(); + const stateBefore = cache.snapshot(); + if (outcome === "open") { + await cache.pullAllBackingStores(); + assert.equal(cache.snapshot().length, name === "blank-store.cc" ? 2 : 4); + if (alias === null) assert.equal(cache.getKeyByIndex(document, "Sku", "S-1"), undefined); + } else { + await assert.rejects(cache.pullAllBackingStores(), (error: Error) => { + assert.equal(error.message, refusal(key1!, key2!, "tests.unique_item", alias!)); + assert.ok(!error.message.includes("S-1")); + return true; + }); + assert.deepEqual(cache.snapshot(), stateBefore, "a refused load admits nothing"); + } + assert.deepEqual(await readFile(file), before, `${name} is untouched`); + } + } finally { + await rm(dir, { recursive: true, force: true }); + } +}); + +test("a refused load leaves the cache's prior state as it was, and a later good load still works", async () => { + const dir = await mkdtemp(join(tmpdir(), "uidx-load-")); + try { + const bad = join(dir, "bad.cc"); + await copyFile(join(vectors, "duplicate-store.cc"), bad); + const [envelope] = await new SingleFileMessagePackBackingStore(bad).pullAll(); + const document = twoDefinitions(envelope.schemaId!, "indexes"); + const cache = CultCache.builder().withDocumentType(document).withGenericStore(new SingleFileMessagePackBackingStore(bad)).build(); + await assert.rejects(cache.pullAllBackingStores(), /Records item:a and item:b/u); + assert.equal(cache.get(document, "item:a"), undefined); + assert.equal(cache.getKeyByIndex(document, "Sku", "S-1"), undefined); + } finally { + await rm(dir, { recursive: true, force: true }); + } +}); + +test("registerIndex over held duplicates installs nothing", async () => { + const document = item("item", "roles"); + const cache = memory(document); + await cache.put(document, "a", { sku: "S", group: "g" }); + await cache.put(document, "b", { sku: "S", group: "h" }); + await assert.rejects(cache.registerIndex(document, "sku", "sku"), { message: refusal("a", "b", "tests.item", "sku") }); + assert.equal(cache.getKeyByIndex(document, "sku", "S"), undefined, "no map was installed"); + await cache.put(document, "c", { sku: "S", group: "i" }); + await cache.registerIndex(document, "group", "group"); + assert.equal(cache.getKeyByIndex(document, "group", "h"), "b"); + await assert.rejects(cache.put(document, "d", { sku: "x", group: "h" }), /Records b and d/u); +}); + +test("registerIndex over distinct held values installs the lookup", async () => { + const document = item("item", "none"); + const cache = memory(document); + await cache.put(document, "a", { sku: "S" }); + await cache.registerIndex(document, "sku", "sku"); + assert.equal(cache.getKeyByIndex(document, "sku", "S"), "a"); +}); + +// ---- a type with several unique indexes: the second alias is judged, in alias order, in both judges ---- + +const pairSchema = z.object({ a: z.string().optional(), b: z.string().optional() }); + +function pair(where: { indexes?: Record; roles?: Record }) { + return defineDocumentType({ + type: "pair", + schemaId: "tests.pair.v1", + schemaName: "tests.pair", + schemaVersion: "tests.pair.v1", + schema: pairSchema, + ...where, + }); +} + +// Declared b first, so insertion order is the opposite of alias order. +const BOTH_UNIQUE = { indexes: { b: "b", a: "a" } as Record }; + +test("a collision on the second unique alias is refused, with a value on the first alias that does not collide", async () => { + const document = pair(BOTH_UNIQUE); + const cache = memory(document); + await cache.put(document, "k1", { a: "x1", b: "shared" }); + await assert.rejects(cache.put(document, "k2", { a: "x2", b: "shared" }), { message: refusal("k1", "k2", "tests.pair", "b") }); + await assert.rejects(cache.put(document, "k2", { a: "x1", b: "y" }), { message: refusal("k1", "k2", "tests.pair", "a") }); + assert.equal(cache.get(document, "k2"), undefined); +}); + +test("when two aliases collide the first alias in ordinal order is named, and its keys are in ordinal order", async () => { + const document = pair(BOTH_UNIQUE); + const cache = memory(document); + await cache.put(document, "m", { a: "same-a", b: "same-b" }); + await assert.rejects(cache.put(document, "c", { a: "same-a", b: "same-b" }), { message: refusal("c", "m", "tests.pair", "a") }); + // The colliding holders differ per alias: a is held by m, b by n; alias a is still the one named. + await cache.put(document, "n", { a: "other-a", b: "other-b" }); + await assert.rejects(cache.put(document, "z", { a: "same-a", b: "other-b" }), { message: refusal("m", "z", "tests.pair", "a") }); +}); + +test("a load judges every unique alias: a store collision on the second alias refuses the open", async () => { + const dir = await mkdtemp(join(tmpdir(), "uidx-pair-")); + try { + for (const [name, records, expected] of [ + ["second-only", [["k1", { a: "1", b: "s" }], ["k2", { a: "2", b: "s" }]], refusal("k1", "k2", "tests.pair", "b")], + ["both-aliases", [["m", { a: "s", b: "s" }], ["c", { a: "s", b: "s" }]], refusal("c", "m", "tests.pair", "a")], + ] as const) { + const file = join(dir, `${name}.cc`); + const roles = pair({ roles: { b: "b", a: "a" } }); + const writer = CultCache.builder().withDocumentType(roles).withGenericStore(new SingleFileMessagePackBackingStore(file)).build(); + for (const [key, value] of records) await writer.put(roles, key, value); + const before = await readFile(file); + + const unique = pair(BOTH_UNIQUE); + const reader = CultCache.builder().withDocumentType(unique).withGenericStore(new SingleFileMessagePackBackingStore(file)).build(); + await assert.rejects(reader.pullAllBackingStores(), { message: expected }, name); + assert.deepEqual(reader.snapshot(), [], "a refused load admits nothing"); + assert.deepEqual(await readFile(file), before); + + const rolesReader = CultCache.builder().withDocumentType(roles).withGenericStore(new SingleFileMessagePackBackingStore(file)).build(); + await rolesReader.pullAllBackingStores(); + assert.equal(rolesReader.snapshot().length, 2, `${name}: the same store opens under roles`); + } + } finally { + await rm(dir, { recursive: true, force: true }); + } +}); + +test("registerIndex of a second unique alias judges it, naming the two smallest keys whatever order they were written in", async () => { + const document = pair({ indexes: { a: "a" }, roles: { b: "b" } }); + const cache = memory(document); + await cache.put(document, "z", { a: "1", b: "s" }); + await cache.put(document, "m", { a: "2", b: "s" }); + await cache.put(document, "q", { a: "3", b: "s" }); + await assert.rejects(cache.registerIndex(document, "b", "b"), { message: refusal("m", "q", "tests.pair", "b") }); + assert.equal(cache.getKeyByIndex(document, "a", "1"), "z", "the alias already installed is intact"); + assert.equal(cache.getKeyByIndex(document, "b", "s"), undefined); +}); + +test("verdict parity: the same stores are refused whatever the load order, and the first collision refuses (text is informational)", async () => { + const dir = await mkdtemp(join(tmpdir(), "uidx-verdict-")); + try { + // Three records, one pair colliding; whichever position the pair has in the file, the store is refused and a clean store opens. + const roles = pair({ roles: { a: "a" } }); + for (const [name, values, refused] of [ + ["pair-first", ["s", "s", "t"], true], + ["pair-last", ["t", "s", "s"], true], + ["pair-split", ["s", "t", "s"], true], + ["distinct", ["r", "s", "t"], false], + ] as const) { + const file = join(dir, `${name}.cc`); + const writer = CultCache.builder().withDocumentType(roles).withGenericStore(new SingleFileMessagePackBackingStore(file)).build(); + for (const [i, a] of values.entries()) await writer.put(roles, `r${i}`, { a }); + const unique = pair({ indexes: { a: "a" } }); + const reader = CultCache.builder().withDocumentType(unique).withGenericStore(new SingleFileMessagePackBackingStore(file)).build(); + if (refused) await assert.rejects(reader.pullAllBackingStores(), /both hold unique index 'a'/u, name); + else await reader.pullAllBackingStores(); + assert.equal(reader.snapshot().length, refused ? 0 : 3, name); + } + } finally { + await rm(dir, { recursive: true, force: true }); + } +}); diff --git a/packages/cultnet-ts/README.md b/packages/cultnet-ts/README.md index 07996956..7d1b9a6a 100644 --- a/packages/cultnet-ts/README.md +++ b/packages/cultnet-ts/README.md @@ -66,6 +66,7 @@ Current message families: - `cultnet.snapshot_response.v0` - `cultnet.snapshot_response_raw.v0` - `cultnet.snapshot_request.v1`, `cultnet.database_subscribe.v1`, `cultnet.snapshot_response_raw.v1` (typed-selection carriers; no selection engine yet) +- `cultnet.database_change_raw.v0` (optional `commitId` and `commitCount >= 2` mark the changes of one commit; carried and validated here, no TS code groups or applies by them) - `cultnet.sample.change_name.v0` - `cultnet.sample.chat.v0` diff --git a/packages/cultnet-ts/src/contracts.ts b/packages/cultnet-ts/src/contracts.ts index 8dd37ced..b733c959 100644 --- a/packages/cultnet-ts/src/contracts.ts +++ b/packages/cultnet-ts/src/contracts.ts @@ -424,6 +424,10 @@ export interface CultNetDatabaseChangeRawMessage { document?: CultNetRawDocumentRecord; recordKey?: string; schemaId?: string; + /** Opaque, one subscription: changes sharing a commitId are the changes of one commit. */ + commitId?: string | null; + /** How many changes the commit carries, at least two. The C# server sends null on a change outside a grouped commit. */ + commitCount?: number | null; } export interface CultNetSchemaDescriptor { @@ -1397,6 +1401,15 @@ function validateDatabaseChangeRawMessage( ); } + if (candidate.commitId !== undefined && candidate.commitId !== null && + (typeof candidate.commitId !== "string" || candidate.commitId.length === 0)) { + throw new Error("Validation failed for cultnet.database_change_raw.v0: /commitId: must be a non-empty string"); + } + if (candidate.commitCount !== undefined && candidate.commitCount !== null && + (typeof candidate.commitCount !== "number" || !Number.isInteger(candidate.commitCount) || candidate.commitCount < 2)) { + throw new Error("Validation failed for cultnet.database_change_raw.v0: /commitCount: must be an integer of at least 2"); + } + if (candidate.changeKind === "removed") { requireNonEmptyString(candidate.recordKey, "/recordKey"); requireNonEmptyString(candidate.schemaId, "/schemaId"); diff --git a/packages/cultnet-ts/test/cultnet.test.ts b/packages/cultnet-ts/test/cultnet.test.ts index 37d963a8..ac4f539d 100644 --- a/packages/cultnet-ts/test/cultnet.test.ts +++ b/packages/cultnet-ts/test/cultnet.test.ts @@ -54,6 +54,7 @@ import { validateGhostlightAgentStateGenerated, validateGhostlightAgentState, shardServes, + type CultNetDatabaseChangeRawMessage, type CultNetLoginMessage, type GhostlightAgentStateShape, type GhostlightAgentStateDocument, @@ -1688,6 +1689,55 @@ test("database subscription contracts match the C# live document lane", () => { assert.equal(removed.schemaVersion, "cultnet.database_change_raw.v0"); assert.equal(removed.recordKey, "counter:main"); + const grouped = parseCultNetMessage({ + schemaVersion: "cultnet.database_change_raw.v0", + messageId: "change-counter-3", + subscriptionId: "counter:browser-1", + changeKind: "removed", + recordKey: "counter:main", + schemaId: "sample.counter_state.v1", + commitId: "c1", + commitCount: 2, + }); + assert.equal((grouped as CultNetDatabaseChangeRawMessage).commitId, "c1"); + assert.equal((grouped as CultNetDatabaseChangeRawMessage).commitCount, 2); + assert.deepEqual(encodeCultNetMessageForWire(grouped), grouped, "both commit fields travel"); + assert.equal((removed as CultNetDatabaseChangeRawMessage).commitId, undefined, "a change without them parses as before"); + const changeWith = (extra: Record) => () => parseCultNetMessage({ + schemaVersion: "cultnet.database_change_raw.v0", + messageId: "change-counter-4", + subscriptionId: "counter:browser-1", + changeKind: "removed", + recordKey: "counter:main", + schemaId: "sample.counter_state.v1", + ...extra, + }); + // The C# server writes commitId and commitCount as nil on every change outside a grouped commit, so null is absent. + const ungrouped = changeWith({ commitId: null, commitCount: null })() as CultNetDatabaseChangeRawMessage; + assert.equal(ungrouped.commitId, null); + assert.equal(ungrouped.commitCount, null); + assert.doesNotThrow(changeWith({ commitId: null })); + assert.doesNotThrow(changeWith({ commitCount: null })); + assert.doesNotThrow(() => parseCultNetMessage({ + schemaVersion: "cultnet.database_change_raw.v0", + messageId: "change-counter-5", + subscriptionId: "counter:browser-1", + changeKind: "added", + document: { + schemaId: "sample.counter_state.v1", + recordKey: "counter:main", + storedAt: "2026-08-17T00:00:00Z", + payloadEncoding: "messagepack", + payload: new Uint8Array([0x81, 0xa5, 0x63, 0x6f, 0x75, 0x6e, 0x74, 0x01]), + }, + recordKey: null, + schemaId: null, + commitId: null, + commitCount: null, + })); + for (const commitId of ["", 5]) assert.throws(changeWith({ commitId }), /\/commitId/, `commitId ${JSON.stringify(commitId)}`); + for (const commitCount of [1, 0, 2.5, "two", -3]) assert.throws(changeWith({ commitCount }), /\/commitCount/, `commitCount ${commitCount}`); + const unsubscribe = parseCultNetMessage({ schemaVersion: "cultnet.database_unsubscribe.v0", messageId: "unsubscribe-counter", diff --git a/tests/GameCult.Caching.Tests/UniqueIndexVectorTests.cs b/tests/GameCult.Caching.Tests/UniqueIndexVectorTests.cs new file mode 100644 index 00000000..d7c68459 --- /dev/null +++ b/tests/GameCult.Caching.Tests/UniqueIndexVectorTests.cs @@ -0,0 +1,294 @@ +#nullable enable +using System; +using System.Buffers; +using System.Collections.Generic; +using System.Globalization; +using System.IO; +using System.Linq; +using System.Reflection; +using GameCult.Caching; +using GameCult.Caching.MessagePack; +using MessagePack; +using NUnit.Framework; + +namespace GameCult.Caching.Tests +{ + // variants:ruling:unique-indexes. The canonical index text and the unique-index load verdicts in tests/vectors/unique-index. + // The C# reference is the only writer (CULTCACHE_WRITE_VECTORS=1); every other run reads the committed bytes and + // compares them byte for byte. Other runtimes are held to the same text and the same verdicts. + public class UniqueIndexVectorTests + { + private const string WriteVariable = "CULTCACHE_WRITE_VECTORS"; + private static readonly DateTimeOffset StoredAt = new(2026, 1, 1, 0, 0, 0, TimeSpan.Zero); + private static bool Writing => Environment.GetEnvironmentVariable(WriteVariable) == "1"; + + private static string VectorDirectory() + { + for (var dir = new DirectoryInfo(TestContext.CurrentContext.TestDirectory); dir != null; dir = dir.Parent) + { + var vectors = Path.Combine(dir.FullName, "tests", "vectors"); + if (Directory.Exists(vectors)) return Path.Combine(vectors, "unique-index"); + } + + throw new DirectoryNotFoundException($"tests/vectors not found above {TestContext.CurrentContext.TestDirectory}."); + } + + private static string VectorPath(string name) => Path.Combine(VectorDirectory(), name); + + private static void Check(string name, byte[] reference) + { + var path = VectorPath(name); + if (Writing) + { + Directory.CreateDirectory(Path.GetDirectoryName(path)!); + File.WriteAllBytes(path, reference); + return; + } + + Assert.That(File.Exists(path), Is.True, $"vector {name} is committed"); + Assert.That(Convert.ToHexString(reference), Is.EqualTo(Convert.ToHexString(File.ReadAllBytes(path))), $"vector {name}"); + } + + private static byte[] Committed(string name) + { + var path = VectorPath(name); + Assert.That(File.Exists(path), Is.True, $"vector {name} is committed"); + return File.ReadAllBytes(path); + } + + // ---- values.msgpack: [label, kind, value, text-or-nil] ---- + + private sealed record Row(string Label, string Kind, object Value, string? Text); + + private static readonly MethodInfo RenderCanonicalNumber = + typeof(CultDocumentRegistry).GetMethod("RenderCanonicalNumber", BindingFlags.NonPublic | BindingFlags.Static)!; + + private static object? ByIndex(CultCache cache, Type type, string alias, string value) => + typeof(CultCache).GetMethod(nameof(CultCache.GetByIndex))!.MakeGenericMethod(type).Invoke(cache, new object[] { alias, value }); + + private static string? Number(object raw) => (string?)RenderCanonicalNumber.Invoke(null, new[] { raw }); + + // Strings and booleans carry the text the rule states; numbers carry whatever the reference renders, never hand-typed. + private static List Rows() => new() + { + new("plain", "string", "S-1", "S-1"), + new("empty", "string", "", null), + new("space", "string", " ", null), + new("nel", "string", "\u0085", null), + new("nbsp", "string", "\u00A0", null), + new("ideographic-space", "string", "\u3000", null), + new("vertical-tab", "string", "\u000B", null), + new("bom", "string", "\uFEFF", "\uFEFF"), + new("unit-separator", "string", "\u001F", "\u001F"), + new("padded", "string", " a ", " a "), + new("int64-max", "int64", long.MaxValue, Number(long.MaxValue)), + new("int64-min", "int64", long.MinValue, Number(long.MinValue)), + new("uint64-max", "uint64", ulong.MaxValue, Number(ulong.MaxValue)), + new("double-zero", "double", 0.0, Number(0.0)), + new("double-negative-zero", "double", -0.0, Number(-0.0)), + new("double-tenth", "double", 0.1, Number(0.1)), + new("double-1e21", "double", 1e21, Number(1e21)), + new("double-2.5e-7", "double", 2.5e-7, Number(2.5e-7)), + new("double-nan", "double", double.NaN, Number(double.NaN)), + new("double-infinity", "double", double.PositiveInfinity, Number(double.PositiveInfinity)), + new("bool-true", "bool", true, "True"), + new("bool-false", "bool", false, "False"), + }; + + private static byte[] ValuesFile() + { + var buffer = new ArrayBufferWriter(); + var writer = new MessagePackWriter(buffer); + var rows = Rows(); + writer.WriteArrayHeader(rows.Count); + foreach (var row in rows) + { + writer.WriteArrayHeader(4); + writer.Write(row.Label); + writer.Write(row.Kind); + switch (row.Value) + { + case string s: writer.Write(s); break; + case long l: writer.Write(l); break; + case ulong u: writer.Write(u); break; + case double d: writer.Write(d); break; + case bool b: writer.Write(b); break; + default: throw new InvalidOperationException(row.Kind); + } + + if (row.Text == null) writer.WriteNil(); else writer.Write(row.Text); + } + + writer.Flush(); + return buffer.WrittenSpan.ToArray(); + } + + private static readonly Dictionary KindTypes = new[] + { + ("string", typeof(string)), ("int64", typeof(long)), ("uint64", typeof(ulong)), ("double", typeof(double)), ("bool", typeof(bool)) + }.ToDictionary(pair => pair.Item1, pair => EmittedDocumentTypes.Emit( + "UqKind" + pair.Item1, "tests.uq_kind_" + pair.Item1, "tests.uq_kind_" + pair.Item1 + ".v1", + new[] { new EmittedDocumentTypes.Field("Value", pair.Item2, 0, Index: "v") })); + + [Test] + public void ValuesFileIsTheReferencesCanonicalText() + { + Check("values.msgpack", ValuesFile()); + + var reader = new MessagePackReader(Committed("values.msgpack")); + var count = reader.ReadArrayHeader(); + var rows = Rows(); + Assert.That(count, Is.EqualTo(rows.Count)); + foreach (var row in rows) + { + Assert.That(reader.ReadArrayHeader(), Is.EqualTo(4)); + Assert.That(reader.ReadString(), Is.EqualTo(row.Label)); + Assert.That(reader.ReadString(), Is.EqualTo(row.Kind)); + reader.Skip(); + var text = reader.TryReadNil() ? null : reader.ReadString(); + Assert.That(text, Is.EqualTo(row.Text), row.Label); + + var type = KindTypes[row.Kind]; + var registry = CultDocumentRegistry.ForTypes(new[] { type }); + using var cache = CultCacheMessagePack.Create(Path.Combine(Path.GetTempPath(), $"cultlib-uidxv-{Guid.NewGuid():N}.cc"), new CultCacheOpenOptions { Registry = registry }); + cache.Commit(batch => + { + batch.Upsert(type, EmittedDocumentTypes.New(type, ("Value", row.Value)), new CultRecordKey("one")); + batch.Upsert(type, EmittedDocumentTypes.New(type, ("Value", row.Value)), new CultRecordKey(text == null ? "two" : "one")); + }); + if (text == null) + { + Assert.That(cache.Get(new CultRecordKey("two")), Is.Not.Null, $"{row.Label}: a blank value commits twice"); + foreach (var probe in new[] { "", " ", "NaN", "0", "Infinity", "∞" }) + Assert.That(ByIndex(cache, type, "v", probe), Is.Null, $"{row.Label}: found by '{probe}'"); + } + else + { + Assert.That(ByIndex(cache, type, "v", text), Is.Not.Null, $"{row.Label}: found by its text"); + } + } + } + + // ---- the stores: written through a role, judged by an index ---- + + private static readonly EmittedDocumentTypes.Field[] RoleFields = + { + new("Sku", typeof(string), 0, Role: "Sku"), new("Category", typeof(string), 1) + }; + + private static readonly EmittedDocumentTypes.Field[] IndexFields = + { + new("Sku", typeof(string), 0, Index: "Sku"), new("Category", typeof(string), 1) + }; + + private static readonly Type ItemRole = EmittedDocumentTypes.Emit("UqVecItemRole", "tests.unique_item", "tests.unique_item.v1", RoleFields); + private static readonly Type ItemIndex = EmittedDocumentTypes.Emit("UqVecItemIndex", "tests.unique_item", "tests.unique_item.v1", IndexFields); + + private static readonly (string Key, string Sku, string Category)[] Items = + { + ("item:a", "S-1", "tools"), ("item:b", "S-1", "tools"), ("item:c", " ", "parts"), ("item:d", "\u0085", "parts") + }; + + private static byte[] StoreBytes(string[] keys) + { + var directory = Path.Combine(Path.GetTempPath(), $"cultlib-uidxs-{Guid.NewGuid():N}"); + Directory.CreateDirectory(directory); + try + { + var path = Path.Combine(directory, "store.cc"); + using (var cache = CultCacheMessagePack.Create(path, new CultCacheOpenOptions + { + Registry = CultDocumentRegistry.ForTypes(new[] { ItemRole }), StoreFlushOnDispose = true + })) + { + cache.Clock = () => StoredAt; + foreach (var item in Items.Where(item => keys.Contains(item.Key))) + cache.UpsertAsync(ItemRole, EmittedDocumentTypes.New(ItemRole, ("Sku", item.Sku), ("Category", item.Category)), new CultRecordKey(item.Key)).GetAwaiter().GetResult(); + cache.FlushAsync().GetAwaiter().GetResult(); + } + + return File.ReadAllBytes(path); + } + finally + { + Directory.Delete(directory, recursive: true); + } + } + + private static readonly (string File, string? Alias, string Outcome, string? K1, string? K2)[] Outcomes = + { + ("duplicate-store.cc", "Sku", "refused", "item:a", "item:b"), + ("duplicate-store.cc", null, "open", null, null), + ("blank-store.cc", "Sku", "open", null, null), + }; + + private static byte[] OutcomesFile() + { + var buffer = new ArrayBufferWriter(); + var writer = new MessagePackWriter(buffer); + writer.WriteArrayHeader(Outcomes.Length); + foreach (var row in Outcomes) + { + writer.WriteArrayHeader(5); + writer.Write(row.File); + foreach (var text in new[] { row.Alias, row.Outcome, row.K1, row.K2 }) + if (text == null) writer.WriteNil(); else writer.Write(text); + } + + writer.Flush(); + return buffer.WrittenSpan.ToArray(); + } + + private static string? NextText(ref MessagePackReader reader) => reader.TryReadNil() ? null : reader.ReadString(); + + [Test] + public void StoreVectorsHoldTheirVerdicts() + { + Check("duplicate-store.cc", StoreBytes(new[] { "item:a", "item:b", "item:c", "item:d" })); + Check("blank-store.cc", StoreBytes(new[] { "item:c", "item:d" })); + Check("outcomes.msgpack", OutcomesFile()); + + var reader = new MessagePackReader(Committed("outcomes.msgpack")); + var count = reader.ReadArrayHeader(); + Assert.That(count, Is.EqualTo(Outcomes.Length)); + for (var i = 0; i < count; i++) + { + Assert.That(reader.ReadArrayHeader(), Is.EqualTo(5)); + var file = reader.ReadString()!; + var alias = NextText(ref reader); + var outcome = NextText(ref reader)!; + var k1 = NextText(ref reader); + var k2 = NextText(ref reader); + + var directory = Path.Combine(Path.GetTempPath(), $"cultlib-uidxo-{Guid.NewGuid():N}"); + Directory.CreateDirectory(directory); + try + { + var path = Path.Combine(directory, "store.cc"); + File.WriteAllBytes(path, Committed(file)); + var before = File.ReadAllBytes(path); + var type = alias == null ? ItemRole : ItemIndex; + var registry = CultDocumentRegistry.ForTypes(new[] { type }); + if (outcome == "open") + { + using var cache = CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = registry }); + Assert.That(cache.Get(new CultRecordKey("item:c")), Is.Not.Null, file); + } + else + { + var refused = Assert.Throws(() => CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = registry }).Dispose())!; + Assert.That(refused.Message, Is.EqualTo( + $"Records {k1} and {k2} (tests.unique_item) both hold unique index '{alias}'; an index value names one record.")); + Assert.That(refused.Message, Does.Not.Contain("S-1")); + } + + Assert.That(File.ReadAllBytes(path), Is.EqualTo(before), $"{file} is untouched"); + } + finally + { + Directory.Delete(directory, recursive: true); + } + } + } + } +} diff --git a/tests/vectors/unique-index/README.md b/tests/vectors/unique-index/README.md new file mode 100644 index 00000000..548c8845 --- /dev/null +++ b/tests/vectors/unique-index/README.md @@ -0,0 +1,24 @@ +# unique-index vectors + +Written only by the C# reference, `tests/GameCult.Caching.Tests/UniqueIndexVectorTests.cs`, with +`CULTCACHE_WRITE_VECTORS=1`. Every other run reads these files and compares them byte for byte; a changed vector is a +reviewed commit. Other runtimes are held to the same canonical text and the same verdicts (`variants:ruling:unique-indexes`). + +## Files + +- `values.msgpack`: an array of `[label, kind, value, text-or-nil]`, kind one of `string`, `int64`, `uint64`, `double`, + `bool` (decode integers 64-bit wide). `text` is the canonical index text of a member holding `value`: a record holding + it is found by `GetByIndex(alias, text)`. A nil text is a blank value: no value, never indexed, never judged, found by + no text, and two records holding it commit. Strings are as is (never trimmed), blank being empty or only C# + `White_Space` characters (U+0009-000D, U+0020, U+0085, U+00A0, U+1680, U+2000-200A, U+2028, U+2029, U+202F, U+205F, + U+3000); U+FEFF and U+001F are values. Booleans are `True` and `False`. Integers are their decimal digits. A + non-integral finite double is its exact decimal expansion (written by C# `RenderExactDouble`, never hand-typed), + `-0.0` is `0`, NaN and infinities are nil. +- `duplicate-store.cc`: a single-file store, schema `tests.unique_item` v1, slots `[Sku: string, Category: string]`, + stored-at pinned to `2026-01-01T00:00:00Z`. Records `item:a` and `item:b` both hold Sku `S-1`; `item:c` holds Sku + `" "` and `item:d` holds Sku `U+0085`. Written through a role Sku. +- `blank-store.cc`: the same, only `item:c` and `item:d`. +- `outcomes.msgpack`: rows `[file, unique alias or nil, open|refused, k1 or nil, k2 or nil]`. Opening `file` with that + alias a unique index (nil: a role, no unique alias) yields the verdict. A refusal reads, byte for byte, + `Records and (tests.unique_item) both hold unique index ''; an index value names one record.` + with `k1 < k2` ordinal, never contains the value, and leaves the file byte-identical. diff --git a/tests/vectors/unique-index/blank-store.cc b/tests/vectors/unique-index/blank-store.cc new file mode 100644 index 00000000..2b69facc Binary files /dev/null and b/tests/vectors/unique-index/blank-store.cc differ diff --git a/tests/vectors/unique-index/duplicate-store.cc b/tests/vectors/unique-index/duplicate-store.cc new file mode 100644 index 00000000..4caec451 Binary files /dev/null and b/tests/vectors/unique-index/duplicate-store.cc differ diff --git a/tests/vectors/unique-index/outcomes.msgpack b/tests/vectors/unique-index/outcomes.msgpack new file mode 100644 index 00000000..2ff46011 --- /dev/null +++ b/tests/vectors/unique-index/outcomes.msgpack @@ -0,0 +1 @@ +“•²duplicate-store.cc£Sku§refused¦item:a¦item:b•²duplicate-store.ccÀ¤openÀÀ•®blank-store.cc£Sku¤openÀÀ \ No newline at end of file diff --git a/tests/vectors/unique-index/values.msgpack b/tests/vectors/unique-index/values.msgpack new file mode 100644 index 00000000..bbf93b0f Binary files /dev/null and b/tests/vectors/unique-index/values.msgpack differ