From 1cac14574a0acb922c37d4feedba0f03ebdea814 Mon Sep 17 00:00:00 2001
From: Meta Vens <rwvens@hotmail.com>
Date: Fri, 8 May 2026 00:16:55 +0100
Subject: [PATCH] Specify canonical CultCache persistence format

---
 notes/operator-interventions.md               |  11 +
 .../Contracts/cultcache-persistence-format.md | 352 ++++++++++++++++++
 2 files changed, 363 insertions(+)
 create mode 100644 notes/operator-interventions.md
 create mode 100644 src/GameCult.Caching/Contracts/cultcache-persistence-format.md

diff --git a/notes/operator-interventions.md b/notes/operator-interventions.md
new file mode 100644
index 00000000..aa97c6f2
--- /dev/null
+++ b/notes/operator-interventions.md
@@ -0,0 +1,11 @@
+# Operator Interventions
+
+## 2026-05-08
+
+- External Codex session from `E:\Projects\EpiphanyAgent` added
+  `src/GameCult.Caching/Contracts/cultcache-persistence-format.md`.
+- Intent: write down the next canonical CultCache persistence-format design in
+  the C# source-of-truth repo before Rust/TS implementations harden the wrong
+  shape.
+- Existing unrelated worktree changes in `src/GameCult.Networking/PlayerData.cs`
+  and `src/GameCult.Unity/Assets/UI/Default Resolver.asset` were left alone.
diff --git a/src/GameCult.Caching/Contracts/cultcache-persistence-format.md b/src/GameCult.Caching/Contracts/cultcache-persistence-format.md
new file mode 100644
index 00000000..9cb79fa8
--- /dev/null
+++ b/src/GameCult.Caching/Contracts/cultcache-persistence-format.md
@@ -0,0 +1,352 @@
+# CultCache Persistence Format
+
+This document is the canonical design receipt for the next CultCache storage
+format. `GameCult.Caching` in C# is the source of truth for this contract. Rust,
+TypeScript, and other siblings should follow once the shape stabilizes here.
+
+The goal is to preserve CultCache's best property: application code should work
+with domain data, not with a little parade of storage metadata stapled onto
+every object until the object forgets what it was for.
+
+## Why This Exists
+
+The current stack has two conflicting virtues:
+
+- domain ergonomics: users want plain typed objects
+- interop durability: caches in different languages need enough persisted
+  information to read each other's data, compare schemas, and perform soft
+  migration when shapes drift
+
+The old C# shape leaned hard toward ergonomics by exposing `DatabaseEntry.ID`
+directly and letting backing stores carry a lot of implicit responsibility. The
+newer Rust draft leaned hard toward persistence machinery by wrapping payloads
+in a heavier envelope. Both capture part of the truth. Neither is the final
+machine.
+
+The intended format keeps storage metadata out of domain objects while making
+the persisted store self-describing enough for cross-language readers and
+training-data archaeology.
+
+## Core Rule
+
+Domain payloads contain domain truth only.
+
+These are **not** domain fields and should not be exposed on app-facing types:
+
+- CultCache record key / GUID
+- persistence schema id
+- storage timestamp
+
+Those belong to the CultCache persistence layer.
+
+## Canonical Store Shape
+
+A persisted CultCache file should contain:
+
+1. store header
+2. embedded schema catalog
+3. record set
+
+High-level shape:
+
+```json
+{
+  "formatVersion": "cultcache.store.v1",
+  "catalog": {
+    "catalogVersion": "cultcache.schema_catalog.v1",
+    "schemas": []
+  },
+  "records": []
+}
+```
+
+The on-disk encoding does not have to be JSON. MessagePack is the expected
+default. The shape above is explanatory.
+
+## Store Header
+
+Required header fields:
+
+- `formatVersion`
+- `catalog`
+- `records`
+
+Optional header fields:
+
+- `createdAt`
+- `updatedAt`
+- `storeId`
+- `producer`
+- `notes`
+
+These are store-level bookkeeping fields, not per-record domain metadata.
+
+## Embedded Schema Catalog
+
+The store must embed a catalog entry for every schema referenced by the records
+it contains.
+
+This is not decorative paperwork. It is what allows another CultCache
+implementation to inspect an old store, resolve the schema for each record, and
+decide whether it can:
+
+- read directly
+- soft-migrate with warnings
+- or refuse the data honestly
+
+### Schema Catalog Entry
+
+Each schema descriptor should contain at least:
+
+- `schemaId`
+- `schemaName`
+- `schemaVersion`
+- `contentHash`
+- `canonicalSchema`
+
+Recommended optional fields:
+
+- `migrationFamily`
+- `supersedesSchemaIds`
+- `compatibleReaderFamilies`
+- `persistenceEncoding`
+- `notes`
+
+Example:
+
+```json
+{
+  "schemaId": "sha256:8b52c6...",
+  "schemaName": "gamecult.player-data",
+  "schemaVersion": "1.2.0",
+  "contentHash": "sha256:52ff9c...",
+  "migrationFamily": "player-data",
+  "supersedesSchemaIds": ["sha256:0ce41a..."],
+  "persistenceEncoding": "messagepack-array-v1",
+  "canonicalSchema": {
+    "kind": "messagepack-array-object",
+    "fields": [
+      { "slot": 0, "name": "Name", "type": "string", "required": true },
+      { "slot": 1, "name": "Email", "type": "string", "required": true },
+      { "slot": 2, "name": "Level", "type": "int32", "required": false, "default": 0 }
+    ]
+  }
+}
+```
+
+### `schemaId`
+
+`schemaId` is the stable machine key. It should be derived from the canonical
+semantic persistence schema, not from:
+
+- repo path
+- C# type name
+- Rust type name
+- TypeScript interface name
+- generated file name
+
+If two implementations describe the same persisted shape, they should derive
+the same `schemaId` regardless of origin language.
+
+### `contentHash`
+
+`contentHash` is separate from `schemaId`.
+
+- `schemaId` answers: "is this the same semantic persistence shape?"
+- `contentHash` answers: "is this the exact same canonical schema document?"
+
+This distinction matters because descriptions, examples, and other non-semantic
+surface details may change without changing the actual persisted shape.
+
+## Record Shape
+
+Each persisted record should carry only the storage metadata needed to route and
+reconstruct the domain object:
+
+- `key`
+- `schemaId`
+- `storedAt`
+- `payload`
+
+Example:
+
+```json
+{
+  "key": "e1439d7e-8da8-4b63-a0b1-238c815a8a17",
+  "schemaId": "sha256:8b52c6...",
+  "storedAt": "2026-05-08T12:00:00Z",
+  "payload": "<messagepack bytes>"
+}
+```
+
+The payload is the domain object only. It does not repeat the key, schema id,
+or stored timestamp.
+
+## Canonical Semantic Schema Hash
+
+`schemaId` should be derived from a canonical semantic schema hash.
+
+The canonicalization step must ignore noise that should not change storage
+identity, such as:
+
+- whitespace
+- key ordering
+- documentation prose
+- examples
+- source-generator quirks
+
+It must preserve the parts that *do* define persistence compatibility:
+
+- field slot numbers
+- field types
+- required versus defaulted presence
+- nullability
+- collection/container semantics
+- enum/value constraints when those affect stored meaning
+- persistence encoding shape, such as MessagePack array layout
+
+The output should be a stable string such as:
+
+- `sha256:<hex>`
+
+This is the value persisted in each record's `schemaId`.
+
+## Soft Migration
+
+Soft migration is a first-class requirement. The format must support a reader
+encountering a record whose embedded schema is not byte-identical to its local
+native schema while still consuming the record safely when the slot contract is
+compatible enough.
+
+Reader flow:
+
+1. Read record header.
+2. Resolve `schemaId` against the embedded catalog.
+3. Resolve the local registered schema for the requested native type.
+4. Compare embedded versus local canonical persistence schemas.
+5. Choose one:
+   - `exact_match`
+   - `soft_migrate`
+   - `reject`
+
+### Exact Match
+
+Read directly when:
+
+- schema ids match exactly
+- or equivalent schema comparison says the persisted and local canonical shapes
+  are identical
+
+### Soft Migration
+
+Allowed when:
+
+- persisted schema has missing slots that the local schema can default
+- persisted schema has extra slots the local reader can safely ignore or preserve
+- slot meanings remain compatible
+
+Soft migration should emit warnings, not silent shrugs.
+
+Typical warnings:
+
+- missing slot defaulted
+- unknown extra slot ignored
+- nullable/default behavior widened
+- field preserved but not surfaced natively
+
+### Reject
+
+Reject when:
+
+- slot meanings conflict
+- required local data cannot be reconstructed
+- stored type and local expected type are incompatible
+- persistence encoding families differ in a non-compatible way
+
+Do not bluff here. A false clean read is worse than a refusal.
+
+## API Ergonomics
+
+The public CultCache API should hide persistence metadata by default.
+
+Desired feel:
+
+- callers work with `PlayerData`, not `PlayerData` plus storage sludge
+- cache methods own key generation or key routing
+- global and named-entry flows stay ergonomic
+- advanced callers can still inspect raw records and catalogs when needed
+
+This implies a split between:
+
+- domain-facing API
+- raw persistence/interop API
+
+### Domain-Facing Surface
+
+Domain callers should interact with:
+
+- typed adds/puts
+- typed gets
+- named lookups
+- index lookups
+- global entry helpers
+
+They should not have to care about `schemaId`, `storedAt`, or raw payload bytes
+unless they are explicitly doing migration, tooling, or replication work.
+
+### Raw Persistence Surface
+
+The lower-level API should still expose:
+
+- record inspection
+- schema catalog inspection
+- raw record import/export
+- migration diagnostics
+- exact payload preservation for bit-compatible neighbors
+
+That is the cache's responsibility, not the domain model's.
+
+## CultNet Alignment
+
+CultCache and CultNet should not grow rival schema religions.
+
+The schema descriptor shape used by the CultCache embedded catalog should align
+with the CultNet schema catalog as closely as practical. In particular:
+
+- `schemaId`
+- `schemaVersion`
+- `contentHash`
+- canonical schema body
+
+should mean the same thing across persistence and wire discovery.
+
+Where CultNet needs additional wire-oriented metadata, extend around the shared
+core instead of redefining the core.
+
+## Migration From Current C# Shape
+
+The current C# `DatabaseEntry` base type exposes `ID` directly. That is useful
+today but not the desired long-term boundary.
+
+Planned direction:
+
+1. define the v1 persistence format here
+2. implement store header, catalog, and record shape in C#
+3. move storage identity out of the domain-facing `DatabaseEntry` contract
+4. keep compatibility shims long enough to read older stores
+5. let Rust, TypeScript, and other siblings follow the stabilized C# contract
+
+The transition should preserve the existing soft-migration advantage rather than
+trading it for fake purity.
+
+## Near-Term Rules
+
+Until this format is implemented:
+
+- treat this document as the canonical target
+- do not add new storage metadata to domain payloads
+- prefer schema catalogs over repeated per-document metadata
+- preserve the distinction between semantic schema identity and exact schema
+  document hash
+- keep cross-language interop as a first-class design constraint, not a later
+  apology