From 40a30e714b3ec489842ab796b107e54f3bc8697c Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sat, 10 Oct 2026 21:04:23 +0200 Subject: [PATCH] cultlib-gaps: union parity map (body facts, step 0b, cut order) Imagination pass imagination-cl-union-parity: MessagePack-CSharp's union miss branch read and run, a CultCache-layer arm guard proved without a fork, Aetheria's and StreamPixels' stores walked for arms and slots, and the Rust, TS and Python slot and union behaviour measured. Co-Authored-By: Claude Opus 5.5 --- docs/cultcache-union-parity-map.md | 207 +++++++++++++++++++++++++++++ 1 file changed, 207 insertions(+) create mode 100644 docs/cultcache-union-parity-map.md diff --git a/docs/cultcache-union-parity-map.md b/docs/cultcache-union-parity-map.md new file mode 100644 index 00000000..245e7a1a --- /dev/null +++ b/docs/cultcache-union-parity-map.md @@ -0,0 +1,207 @@ +# CultCache Union Parity: Map + +Campaign `cultlib-gaps`, follow-up `cultlib-gaps:follow_up:union-parity-and-unknown-arms` (from +`thing2:follow_up:cultcache-union-parity`), under operator ruling `thing2:ruling:contract-decode-tolerance` +("CultCache refuses unknown union arms in every runtime, the C# reference included"). This file holds body facts, +the model page and rationale; the cut specs and the question are documents in the mind and cite this file by +section and fact number. + +Pinned base: `origin/main` at `aaf0d7f6` (2026-10-10). Imagination pass `imagination-cl-union-parity`, session +`self-2026-10-10c-eureka`. Probes live under the session scratchpad, `imagination-cl-union-parity/` (`cs` payload +probe, `cs2` store probe, `rs` payload probe, `py` store walkers, `il` decompiled MessagePack-CSharp). None of it is +committed. Earlier evidence is not repeated here: Eve `docs/thing2-map.md` P12-P13 (the cross-runtime probe) and +`docs/research/cultcache-union-decode-tolerance-prior-art.md` (prior art). + +## Cut order + +1. `union-refusal-cs`: the C# reference refuses an unknown union arm on read and a subtype the union does not name + on write. Ruled; nothing blocks it. +2. `union-golden`: the recursive-union golden and the arm refusal goldens in `tests/vectors/union-parity/`, + written by the C# reference and held by its tests. +3. `union-derive-rs` and `union-codec-ts`, in either order or together: cultcache-rs and cultcache-ts gain a + union (and, in TS, a slot record) codec that writes the reference shape and decodes the golden byte-identically. + `union-codec-py` follows the same golden for cultcache-py. +4. `slot-tolerance`: extra trailing slots, per `cultlib-gaps:question:extra-trailing-slots`. Blocked on the + operator; it changes every runtime's slot rule at once, so it waits for 1-3 rather than reopening them. +5. `union-parity-release`: changelogs and tags, Breaking where `docs/semver-policy.md` says so. + +Thing2's `cut-contract-rs` waits on step 3 (Rust) only: its first step pins cultcache-rs at the commit that lands +`union-derive-rs` and re-runs P12 or the `union-golden` vector both ways. + +Why this order: the C# reference is the wire authority, so its refusal and its golden come first and the other +runtimes are held to bytes C# wrote, as `document-variants-c1` was (`docs/runtime-parity-scope.md`). Arms are +ruled and slots are not, so the slot rule is its own cut rather than a blocked step inside every runtime's cut. + +## Model page (step 0b) + +| Kind | What names it | Over time | Who decides | +|---|---|---|---| +| Union arm | An integer key, `[Union(k, typeof(Arm))]` on the abstract base (C#), assigned by the union's author. The wire carries `[k, armSlots]`; the arm's type name never travels. `nil` is a null reference, not an arm. | **Added:** a key never used before. **Retired:** the attribute is removed or commented out and the key is never reused (Aetheria `Behaviors.cs:160,173`; Aetheria-legacy `Behaviors.cs:37-38`). **Renamed type:** free, the key is the identity. **Unknown to a reader** (a newer writer's arm, or a retired arm in an old store): refused, naming the union and the key (ruling). | The union's author assigns keys. The C# reference defines the shape; every runtime keeps parity with it (doctrine, `runtime-parity-scope.md`). | +| Arm subtype on write | The value's runtime type must be one the union names. | A subtype the union does not name is today written as `nil` (U4). After cut 1 it is refused. | The union's author, by naming the arm. | +| Arm slots, record slots | `[Key(n)]` integers, base-class keys included (an arm is a flat array). Gaps are written `nil`. | **Added:** appended at the next free key. **Retired:** kept as a gap, never reused (Aetheria `ItemInstance.cs:35`, `EntitySerializer.cs` key 5). **Missing in an older store:** the reader's default (C# field initializer; Rust and Python need a declared default). **Extra past the reader's last key:** skipped silently today in every runtime (U3, U6); the question decides. | The type's author. The slot rule is CultCache's (`src/GameCult.Caching/Contracts/cultcache-schema-compatibility.md`). | +| Serde-enum member (Rust) | serde's external tag: the variant **name** (`{"Name": fields}`, a unit variant the bare string). | Persisted already by Rust consumers (section C). They keep their bytes: the union derive is opt-in and serde enums are untouched. | The Rust consumer. Not a cross-runtime shape. | + +## U. Mechanism facts (C# reference, Rust, TS) + +- **U1. The miss branch is `Skip` then `null`, in both union formatters MessagePack-CSharp 3.1.7 can hand out.** + `DynamicUnionResolver.BuildDeserialize` (decompiled from `MessagePack.dll` 3.1.7 net9.0 with ilspycmd): after + `ReadArrayHeader() == 2` (else `InvalidOperationException "Invalid Union data was detected"`) and `ReadInt32`, a + non-zero-start key set is mapped through `keyToJumpMap.TryGetValue`, a miss sets the index to `-1`, the `switch` + default calls `reader.Skip()`, and the result local stays `null`. The source generator's `UnionTemplate` + (`MessagePackAnalyzer` 3.1.7, `MessagePack.SourceGenerator.dll`) emits the same: `default: reader.Skip(); break;` + with `result = null`. Its serializer writes `nil` when `typeToKeyAndJumpMap` has no entry for `value.GetType()`. +- **U2. The C# reference decodes an unknown arm to null and does not re-encode to its bytes.** Probe `cs` + (payload built by hand with `MessagePackWriter`, decoded with `CultDocumentMessagePackSerialization.OptionsFor`): + unknown key 9 in a union field gives `Root=NULL`, in a list `Items=[TextNode,NULL]`; re-encode equal: False in + both. The formatter the reference composite hands out for the union is `MessagePack.Formatters.NodeFormatter1` + (DynamicUnionResolver's emitted type; the probe assembly carries no generated resolver). +- **U3. Missing and extra trailing slots, C# payload level.** Same probe: a payload one slot short decodes with + the field initializer (`Added=42`); a payload one slot long decodes with the extra slot skipped. Neither + re-encodes to its bytes. +- **U4. The reference writes an unnamed subtype as nil.** Same probe: `Root = new RogueNode()` (a subclass the + union does not list) serializes to `94 a2 64 32 c0 90 2a`: the root slot is `nil`, and it reads back `Root=NULL`. + Silent loss on the write side, the mirror of U2. +- **U5. Refusal can live at the CultCache layer; no MessagePack-CSharp fork.** Same probe, `strict` options: a + resolver placed ahead of `StandardResolver` returns, for any abstract or interface type with `[Union]`s, a + formatter that peeks (`CreatePeekReader`) the `[key, ...]` header, throws + `MessagePackSerializationException("union Node has no arm 9")` for a key the union does not list, refuses to + serialize an unnamed subtype, and otherwise delegates to the formatter `StandardResolver` would have handed out. + Baseline round-trips byte-identical; both unknown-arm cases and U4 are refused; slot behaviour is unchanged. + Every C# decode path reaches the composite through `CultDocumentMessagePackSerialization.OptionsFor` + (`CultCacheMessagePack.CreateCodec`, `DirectoryMessagePackBackingStore.cs:345`, `CultMesh.cs:2779,3493,3499`, + `CultMeshPrimitives.cs:1158,1230,2357`), so one resolver covers the cache, the directory store and CultMesh. + Consumer resolvers named by `[CultCacheFormatterResolver]` sit ahead of `CultDocumentResolver` today + (`CultDocumentMessagePackSerialization.cs:58-62`); the guard goes first and delegates to the rest of the + composite, so a consumer formatter still encodes the arm while the key check still runs. +- **U6. A top-level extra slot is dropped on write-back.** Probe `cs2`: a C#-written store, mutated as a newer + writer would (catalog member slot 2 added, payload slot 2 `"newer writer data"`), is pulled, the record edited + and flushed. The read succeeds, and the written payload is `['n1', 'edited by the older build']`: the newer + writer's slot is gone. (The schema id was left unchanged, so the migration report said `Kind=Exact`; with a + changed id the documented drift path reports `IgnoredExtraSlots`, `CultCache.cs:840-924`, but writes back the + same way.) +- **U7. Rust today.** Probe `rs` (`cultcache-rs` at `aaf0d7f6`, payloads built with `rmpv`, decoded with + `rmp_serde::from_slice`): a serde enum's unknown variant is refused (`unknown variant 'Exploded'`); the C# arm + shape `[0, [...]]` is refused (`expected variant identifier`); a non-nil value in a gap slot is skipped; a missing + trailing slot is refused (`missing field 'added'`) unless the field is `#[cultcache(default)]`; an extra trailing + slot is skipped (`cultcache-rs-derive/src/lib.rs`, the `IgnoredAny` loop after the slot loop). The derive + accepts named structs only (`lib.rs:51-67`) and writes an empty member list into the catalog + (`src/lib.rs:2706`). +- **U8. TypeScript today.** No typed codec: with no `formatter` a document encodes as a MessagePack **map** of the + zod-parsed object and decodes raw (`cult-cache.ts:679-692`; `@msgpack/msgpack` `encode({id:'d',version:7})` is + `82 a2 69 64 ...`, a fixmap, where the reference writes an array); positional slots exist only in hand-written + formatters (`test/cult-cache.test.ts:1263-1310`). P13 measured that a hand codec matches whichever encoding its + author chose. +- **U9. Python today.** `define_database_entry_type` (`packages/cultcache-py/src/cultcache_py/documents.py:185-276`) + is a slot codec: gaps written `nil`, extra slots ignored, a missing slot defaulted or refused. No union codec. +- **U10. The written contracts.** `cultcache-schema-compatibility.md:75-91` classifies "persisted slot is missing + locally and can be ignored" as soft-migratable drift, reported as `IgnoredExtraSlots`; + `cultcache-persistence-format.md:215-218` says a store reader "never skips a slot it does not understand" for + record envelope slots, and every runtime refuses `extra-slot-full-payload.msgpack` + (`tests/vectors/document-variants-c0`). `docs/persisted-record-encoding.md` says consumers verify canonicality + by re-encoding and comparing bytes. The two contracts disagree about payload slots; the question below is + about that disagreement. +- **U11. Fixture home.** `tests/vectors//`: committed bytes every runtime's tests read + (`document-variants-c0` by Python `generate.py`, `document-variants-c1` written by C#), consumed by + `tests/GameCult.Caching.Tests/PreCut2StoreFormatTests.cs`, `packages/cultcache-rs/src/lib.rs`, + `packages/cultcache-ts/test/cult-cache.test.ts` and `packages/cultcache-py/tests/test_cultcache.py`. The CultNet + precedent for a reference-written vector is `CULTNET_WRITE_VECTORS=1` (`docs/cultnet-error-contract-cut.md:170`). + +## A. Aetheria (C#, the only consumer persisting `[Union]`s) + +Aetheria consumes the reference through Unity packages pinned by tag (`Packages/manifest.json:55-57`: +`caching-unity-v1.4.0`, `cultlib-unity-v1.0.60`), so a CultLib change reaches it only when it bumps the pin. +Its only consumer resolver is `MathResolver` (`AssemblyInfo.cs:4`), which supplies no union formatter. + +- **A1. Unions inside persisted documents** (Aetheria `aac2cf91`, `Assets/Scripts/ServerShared`): + `BehaviorData` (`Behaviors/Behaviors.cs:147-191`, keys 0-11, 15, 16, 18, 20-24, 28, 31-40; 12 and 26 retired, + 14, 17, 19, 25, 27, 29, 30 commented out) in `List Behaviors` of the item data documents + (`ItemData.cs:346,368`); `WeaponData` (`Behaviors/Weapon.cs:15-18`); `EntityPack` (`EntitySerializer.cs:172-173`) + in `aetheria.savedzone` contents and its own `Children`; `ItemInstance` and `CraftedItemInstance` + (`ItemInstance.cs:18-31`, a union whose arms are also a union) in entity cargo and docking bays + (`EntitySerializer.cs:186-187`); `Provenance` (`Provenance.cs:96-98`) in `aetheria.provenanceledger` + (`Provenance.cs:11-15`, `Lot.Origin`); `SavedActionBarBinding` (`SavedGame.cs:221-223`) in `aetheria.savedgame`; + the input layout rows and keys (`InputLayout.cs:29-50`) in `aetheria.inputlayout`. `Brush` + (`Environment.cs:73`) was not traced to a persisted document. +- **A2. `GameData/Aetheria.cc` carries no unknown arm.** Walker `py/store_unions.py` over the git-tracked game + database (12.7 MB, `cultcache.store.v1`, 238 records, 13 catalog entries): `Behaviors` arm keys + {0:3, 1:2, 2:10, 3:5, 4:2, 5:2, 6:3, 8:7, 9:23, 11:2, 16:4, 23:7, 24:5, 31:1, 32:1, 34:1, 35:2, 37:4, 38:10, 39:2, + 40:1}; none retired or commented out; no `nil` element. +- **A3. The run save carries no unknown arm, and does carry missing trailing slots.** Walker `py/run_unions.py` + over `GameData/run.cc` (2026-09-30): `EntityPack` arms 0 and 1 (3 each), `ItemInstance` arm 2 (12), + `Provenance` arm 0 (64), `SavedActionBarBinding` 9 `nil` (empty bar slots). The `EntityPack` arms carry 21 slots + (orbital) and 20 (ship) against today's 22: the `Boss` slot (key 21, added 2026-10-07 in `db4617d8`) is missing, + so today's Aetheria reads this save only because a missing trailing slot defaults. `py/gaps.py`: no non-nil + retired slot (ItemInstance 2, 9, 10; EntityPack 5). `GameData/stale-2026-09-22/run.cc` is the same shape; + `player.cc` holds only `aetheria.playersettings`. `py/slots.py`: every top-level record matches its own store's + catalog exactly. +- **A4. What changes for Aetheria.** After it adopts a CultLib with cut 1, its current stores load unchanged + (A2, A3). A store written later by a build that has added an arm is refused by a build that lacks it, naming the + union and key, where today the arm silently becomes `null` (a gear item quietly loses a behaviour, a lot loses + its provenance) and the next save writes the loss back. Builds already pinned to 1.0.60 keep the null path; a + shipped binary cannot change. Aetheria-legacy does not use CultCache and has no store. +- **A5. Other C# stores.** `%LOCALAPPDATA%Low/GameCult/Aetheria/aetheria-unity.cc` (2026-08-08, 1.2 KB) is a + `cultcache.store.v2.directory-indexed` header from an older layout, not a union carrier. + +## C. Rust, TypeScript and other consumers + +Eyes audit for this pass (session `self-2026-10-10c-eureka`), `git grep` at each repo's `origin/HEAD`, Rust reach +by a lexical script (it can over-count; it does not under-count named fields). Aetheria and CultLib's tests excluded. + +- **C1. Rust: no `DatabaseEntry` record crosses runtimes as a union, and none uses the reference arm shape.** + Records that reach serde enums, each keeping its bytes because the derive is opt-in: + Huginn `epiphany.pipeline.resolution.v2` (`ResolutionOutcome`, external tag, `eureka-pipeline/src/lib.rs:746`, + live on Yggdrasil); Idunn `idunn.deployment_transaction` and `idunn.admitted_generation` (v4 and legacy forms; + internally tagged `#[serde(tag = ..., deny_unknown_fields)]` enums in `control_plane.rs`, `deployment_plan.rs`, + and two `#[serde(untagged)]` enums kept so older transactions decode, `drivers.rs:259,387`; live at + `/var/lib/gamecult/idunn/control.cc`); CodexConnector `gamecult.codex.replay_record.v1` (`CodexReplayState`, + `lib.rs:1017`); Epiphany's Atlas, model-request, persona and receipt records (internal, adjacent, external and + untagged representations; `.cc` files under `F:\Projects\.epiphany-runtime\` and on Yggdrasil). Unit-only serde + enums (a bare string) are widespread in Idunn, Huginn (`Faculty`, the pipeline's unit enums), CodexConnector and + Epiphany. No enum reaches any record in Odin, Muninn, Ghostlight, Huginn `mind-body`, or cultnet-rs and + cultmesh-rs production records. None of these is read by another runtime as a union today, so none is affected + by the arm refusal; the derive changes nothing for them. +- **C2. TypeScript: no consumer encodes a tagged union.** The only hand formatters are StreamPixels + `apps/service/src/realtime/overlay-state-documents.ts:190,232,277` (positional arrays, no union). Bifrost, + Heimdall, Hermodr, Odin scripts, Stonks, Vili, VoidBot (vendored `cultcache-ts`) and weksa have none. +- **C3. C# outside Aetheria: StreamPixels' overlay declares unions; its tracked store holds none of them.** + `apps/overlay-unity/Assets/Scripts/Assembly-CSharp/` (`8d40541`): `ActivationCondition` (10 arms), + `BattleEffect` (2), `StatusEffect` (4), `ItemInstance` (3), `EquippableItemInstance` (2), reachable from + `streampixels.status`, `streampixels.action_item`, `streampixels.skill` and gear documents. Walker + `py/sp_catalog.py` over `apps/overlay-unity/Data/catalog.cc` (`cultcache.store.v1`, 20 records): only + `outfit`, `skin_color`, `hair_style`, `emote` and `hair_color` records, and no `[int, array]` pair at any depth. + Its only consumer resolver is `MathResolver` (`Catalog/CatalogAssembly.cs:6`). CultLib's own + `GameCult.Networking` `Message` union (`Messages.cs:10`) is a CultNet wire message, decoded with + `MessageSerialization.Options` (`MessageSerialization.cs:7`, `DefaultOptions`), outside `OptionsFor`: the arm + guard does not reach it (follow-up). +- **C4. Python and Kotlin: no union-shaped record.** repixelizer uses `define_database_entry_type` with flat + slots; `cultmesh-kotlin` documents are `data class`es with no sealed type. + +So the stores the arm refusal can affect today are Aetheria's (A2-A4, none carrying an unknown arm) and +StreamPixels' overlay catalog (C3, carrying no arm). Every other runtime's persisted enums are serde shapes that +the opt-in derive leaves alone. + +## Rationale + +- **Why a guard formatter, not a fork or a re-implementation.** U5 shows a guard that only checks the key and the + subtype and delegates the encoding is enough. It keeps MessagePack-CSharp's union encoding as the one encoder, + costs one small file, and covers every path that already goes through `OptionsFor`. Forking the generated + template or vendoring `DynamicUnionResolver` would put a second encoder in CultLib. +- **Why the Rust derive is opt-in.** Serde enums already persisted by Rust consumers keep their bytes + (`docs/persisted-record-encoding.md`); a changed default would make every such store unreadable or + non-canonical. A new derive with explicit arm keys is how a Rust type says "this crosses runtimes". +- **Why arms use newtype variants over slot structs.** A C# arm is a subclass whose slots include the base class's + keys. A Rust newtype variant over a struct with keyed slots is the same thing, and it needs the slot code the + `DatabaseEntry` derive already has, factored so both derives share one slot generator. +- **Why the slot rule is a separate, asked cut.** The ruling settles arms and asks for extra trailing slots to be + "settled against the same re-encode rule". Two written contracts disagree (U10), and the current behaviour drops + data silently (U6), so the choice is the operator's and it should change every runtime at once. +- **The recommendation on extra slots is refusal.** An extra trailing slot is the slot-level twin of an unknown + arm: data from a newer writer that the reader's types do not describe. The operator ruled refusal for arms + knowing it costs an older reader the newer record; the same reasoning gives refusal for slots, and record + envelope slots are already refused in every runtime (U10). The alternative with precedent is + `variants:ruling:undeclared-read-only` (a record a type can read but does not declare loads read-only): it would + let an older reader see a newer record without overwriting it, but only where a catalog shows the extra slot + (top level), while most evolution here is nested (A3's `Boss` slot sits inside a union arm). +- **Missing trailing slots stay tolerated.** The reader knows a missing slot and its default, so it carries no + bytes it does not understand, and Aetheria's live run save depends on it (A3). A store whose guard compares + re-encoded bytes (Idunn's) refuses such a record by its own stricter rule, which is that guard's purpose.