cultlib-gaps: union parity map (body facts, step 0b, cut order)
Imagination pass imagination-cl-union-parity: MessagePack-CSharp's union miss branch read and run, a CultCache-layer arm guard proved without a fork, Aetheria's and StreamPixels' stores walked for arms and slots, and the Rust, TS and Python slot and union behaviour measured. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Meta Vens
parent
aaf0d7f6a7
commit
40a30e714b
1 changed files with 207 additions and 0 deletions
207
docs/cultcache-union-parity-map.md
Normal file
207
docs/cultcache-union-parity-map.md
Normal file
|
|
@ -0,0 +1,207 @@
|
|||
# CultCache Union Parity: Map
|
||||
|
||||
Campaign `cultlib-gaps`, follow-up `cultlib-gaps:follow_up:union-parity-and-unknown-arms` (from
|
||||
`thing2:follow_up:cultcache-union-parity`), under operator ruling `thing2:ruling:contract-decode-tolerance`
|
||||
("CultCache refuses unknown union arms in every runtime, the C# reference included"). This file holds body facts,
|
||||
the model page and rationale; the cut specs and the question are documents in the mind and cite this file by
|
||||
section and fact number.
|
||||
|
||||
Pinned base: `origin/main` at `aaf0d7f6` (2026-10-10). Imagination pass `imagination-cl-union-parity`, session
|
||||
`self-2026-10-10c-eureka`. Probes live under the session scratchpad, `imagination-cl-union-parity/` (`cs` payload
|
||||
probe, `cs2` store probe, `rs` payload probe, `py` store walkers, `il` decompiled MessagePack-CSharp). None of it is
|
||||
committed. Earlier evidence is not repeated here: Eve `docs/thing2-map.md` P12-P13 (the cross-runtime probe) and
|
||||
`docs/research/cultcache-union-decode-tolerance-prior-art.md` (prior art).
|
||||
|
||||
## Cut order
|
||||
|
||||
1. `union-refusal-cs`: the C# reference refuses an unknown union arm on read and a subtype the union does not name
|
||||
on write. Ruled; nothing blocks it.
|
||||
2. `union-golden`: the recursive-union golden and the arm refusal goldens in `tests/vectors/union-parity/`,
|
||||
written by the C# reference and held by its tests.
|
||||
3. `union-derive-rs` and `union-codec-ts`, in either order or together: cultcache-rs and cultcache-ts gain a
|
||||
union (and, in TS, a slot record) codec that writes the reference shape and decodes the golden byte-identically.
|
||||
`union-codec-py` follows the same golden for cultcache-py.
|
||||
4. `slot-tolerance`: extra trailing slots, per `cultlib-gaps:question:extra-trailing-slots`. Blocked on the
|
||||
operator; it changes every runtime's slot rule at once, so it waits for 1-3 rather than reopening them.
|
||||
5. `union-parity-release`: changelogs and tags, Breaking where `docs/semver-policy.md` says so.
|
||||
|
||||
Thing2's `cut-contract-rs` waits on step 3 (Rust) only: its first step pins cultcache-rs at the commit that lands
|
||||
`union-derive-rs` and re-runs P12 or the `union-golden` vector both ways.
|
||||
|
||||
Why this order: the C# reference is the wire authority, so its refusal and its golden come first and the other
|
||||
runtimes are held to bytes C# wrote, as `document-variants-c1` was (`docs/runtime-parity-scope.md`). Arms are
|
||||
ruled and slots are not, so the slot rule is its own cut rather than a blocked step inside every runtime's cut.
|
||||
|
||||
## Model page (step 0b)
|
||||
|
||||
| Kind | What names it | Over time | Who decides |
|
||||
|---|---|---|---|
|
||||
| Union arm | An integer key, `[Union(k, typeof(Arm))]` on the abstract base (C#), assigned by the union's author. The wire carries `[k, armSlots]`; the arm's type name never travels. `nil` is a null reference, not an arm. | **Added:** a key never used before. **Retired:** the attribute is removed or commented out and the key is never reused (Aetheria `Behaviors.cs:160,173`; Aetheria-legacy `Behaviors.cs:37-38`). **Renamed type:** free, the key is the identity. **Unknown to a reader** (a newer writer's arm, or a retired arm in an old store): refused, naming the union and the key (ruling). | The union's author assigns keys. The C# reference defines the shape; every runtime keeps parity with it (doctrine, `runtime-parity-scope.md`). |
|
||||
| Arm subtype on write | The value's runtime type must be one the union names. | A subtype the union does not name is today written as `nil` (U4). After cut 1 it is refused. | The union's author, by naming the arm. |
|
||||
| Arm slots, record slots | `[Key(n)]` integers, base-class keys included (an arm is a flat array). Gaps are written `nil`. | **Added:** appended at the next free key. **Retired:** kept as a gap, never reused (Aetheria `ItemInstance.cs:35`, `EntitySerializer.cs` key 5). **Missing in an older store:** the reader's default (C# field initializer; Rust and Python need a declared default). **Extra past the reader's last key:** skipped silently today in every runtime (U3, U6); the question decides. | The type's author. The slot rule is CultCache's (`src/GameCult.Caching/Contracts/cultcache-schema-compatibility.md`). |
|
||||
| Serde-enum member (Rust) | serde's external tag: the variant **name** (`{"Name": fields}`, a unit variant the bare string). | Persisted already by Rust consumers (section C). They keep their bytes: the union derive is opt-in and serde enums are untouched. | The Rust consumer. Not a cross-runtime shape. |
|
||||
|
||||
## U. Mechanism facts (C# reference, Rust, TS)
|
||||
|
||||
- **U1. The miss branch is `Skip` then `null`, in both union formatters MessagePack-CSharp 3.1.7 can hand out.**
|
||||
`DynamicUnionResolver.BuildDeserialize` (decompiled from `MessagePack.dll` 3.1.7 net9.0 with ilspycmd): after
|
||||
`ReadArrayHeader() == 2` (else `InvalidOperationException "Invalid Union data was detected"`) and `ReadInt32`, a
|
||||
non-zero-start key set is mapped through `keyToJumpMap.TryGetValue`, a miss sets the index to `-1`, the `switch`
|
||||
default calls `reader.Skip()`, and the result local stays `null`. The source generator's `UnionTemplate`
|
||||
(`MessagePackAnalyzer` 3.1.7, `MessagePack.SourceGenerator.dll`) emits the same: `default: reader.Skip(); break;`
|
||||
with `result = null`. Its serializer writes `nil` when `typeToKeyAndJumpMap` has no entry for `value.GetType()`.
|
||||
- **U2. The C# reference decodes an unknown arm to null and does not re-encode to its bytes.** Probe `cs`
|
||||
(payload built by hand with `MessagePackWriter`, decoded with `CultDocumentMessagePackSerialization.OptionsFor`):
|
||||
unknown key 9 in a union field gives `Root=NULL`, in a list `Items=[TextNode,NULL]`; re-encode equal: False in
|
||||
both. The formatter the reference composite hands out for the union is `MessagePack.Formatters.NodeFormatter1`
|
||||
(DynamicUnionResolver's emitted type; the probe assembly carries no generated resolver).
|
||||
- **U3. Missing and extra trailing slots, C# payload level.** Same probe: a payload one slot short decodes with
|
||||
the field initializer (`Added=42`); a payload one slot long decodes with the extra slot skipped. Neither
|
||||
re-encodes to its bytes.
|
||||
- **U4. The reference writes an unnamed subtype as nil.** Same probe: `Root = new RogueNode()` (a subclass the
|
||||
union does not list) serializes to `94 a2 64 32 c0 90 2a`: the root slot is `nil`, and it reads back `Root=NULL`.
|
||||
Silent loss on the write side, the mirror of U2.
|
||||
- **U5. Refusal can live at the CultCache layer; no MessagePack-CSharp fork.** Same probe, `strict` options: a
|
||||
resolver placed ahead of `StandardResolver` returns, for any abstract or interface type with `[Union]`s, a
|
||||
formatter that peeks (`CreatePeekReader`) the `[key, ...]` header, throws
|
||||
`MessagePackSerializationException("union Node has no arm 9")` for a key the union does not list, refuses to
|
||||
serialize an unnamed subtype, and otherwise delegates to the formatter `StandardResolver` would have handed out.
|
||||
Baseline round-trips byte-identical; both unknown-arm cases and U4 are refused; slot behaviour is unchanged.
|
||||
Every C# decode path reaches the composite through `CultDocumentMessagePackSerialization.OptionsFor`
|
||||
(`CultCacheMessagePack.CreateCodec`, `DirectoryMessagePackBackingStore.cs:345`, `CultMesh.cs:2779,3493,3499`,
|
||||
`CultMeshPrimitives.cs:1158,1230,2357`), so one resolver covers the cache, the directory store and CultMesh.
|
||||
Consumer resolvers named by `[CultCacheFormatterResolver]` sit ahead of `CultDocumentResolver` today
|
||||
(`CultDocumentMessagePackSerialization.cs:58-62`); the guard goes first and delegates to the rest of the
|
||||
composite, so a consumer formatter still encodes the arm while the key check still runs.
|
||||
- **U6. A top-level extra slot is dropped on write-back.** Probe `cs2`: a C#-written store, mutated as a newer
|
||||
writer would (catalog member slot 2 added, payload slot 2 `"newer writer data"`), is pulled, the record edited
|
||||
and flushed. The read succeeds, and the written payload is `['n1', 'edited by the older build']`: the newer
|
||||
writer's slot is gone. (The schema id was left unchanged, so the migration report said `Kind=Exact`; with a
|
||||
changed id the documented drift path reports `IgnoredExtraSlots`, `CultCache.cs:840-924`, but writes back the
|
||||
same way.)
|
||||
- **U7. Rust today.** Probe `rs` (`cultcache-rs` at `aaf0d7f6`, payloads built with `rmpv`, decoded with
|
||||
`rmp_serde::from_slice`): a serde enum's unknown variant is refused (`unknown variant 'Exploded'`); the C# arm
|
||||
shape `[0, [...]]` is refused (`expected variant identifier`); a non-nil value in a gap slot is skipped; a missing
|
||||
trailing slot is refused (`missing field 'added'`) unless the field is `#[cultcache(default)]`; an extra trailing
|
||||
slot is skipped (`cultcache-rs-derive/src/lib.rs`, the `IgnoredAny` loop after the slot loop). The derive
|
||||
accepts named structs only (`lib.rs:51-67`) and writes an empty member list into the catalog
|
||||
(`src/lib.rs:2706`).
|
||||
- **U8. TypeScript today.** No typed codec: with no `formatter` a document encodes as a MessagePack **map** of the
|
||||
zod-parsed object and decodes raw (`cult-cache.ts:679-692`; `@msgpack/msgpack` `encode({id:'d',version:7})` is
|
||||
`82 a2 69 64 ...`, a fixmap, where the reference writes an array); positional slots exist only in hand-written
|
||||
formatters (`test/cult-cache.test.ts:1263-1310`). P13 measured that a hand codec matches whichever encoding its
|
||||
author chose.
|
||||
- **U9. Python today.** `define_database_entry_type` (`packages/cultcache-py/src/cultcache_py/documents.py:185-276`)
|
||||
is a slot codec: gaps written `nil`, extra slots ignored, a missing slot defaulted or refused. No union codec.
|
||||
- **U10. The written contracts.** `cultcache-schema-compatibility.md:75-91` classifies "persisted slot is missing
|
||||
locally and can be ignored" as soft-migratable drift, reported as `IgnoredExtraSlots`;
|
||||
`cultcache-persistence-format.md:215-218` says a store reader "never skips a slot it does not understand" for
|
||||
record envelope slots, and every runtime refuses `extra-slot-full-payload.msgpack`
|
||||
(`tests/vectors/document-variants-c0`). `docs/persisted-record-encoding.md` says consumers verify canonicality
|
||||
by re-encoding and comparing bytes. The two contracts disagree about payload slots; the question below is
|
||||
about that disagreement.
|
||||
- **U11. Fixture home.** `tests/vectors/<name>/`: committed bytes every runtime's tests read
|
||||
(`document-variants-c0` by Python `generate.py`, `document-variants-c1` written by C#), consumed by
|
||||
`tests/GameCult.Caching.Tests/PreCut2StoreFormatTests.cs`, `packages/cultcache-rs/src/lib.rs`,
|
||||
`packages/cultcache-ts/test/cult-cache.test.ts` and `packages/cultcache-py/tests/test_cultcache.py`. The CultNet
|
||||
precedent for a reference-written vector is `CULTNET_WRITE_VECTORS=1` (`docs/cultnet-error-contract-cut.md:170`).
|
||||
|
||||
## A. Aetheria (C#, the only consumer persisting `[Union]`s)
|
||||
|
||||
Aetheria consumes the reference through Unity packages pinned by tag (`Packages/manifest.json:55-57`:
|
||||
`caching-unity-v1.4.0`, `cultlib-unity-v1.0.60`), so a CultLib change reaches it only when it bumps the pin.
|
||||
Its only consumer resolver is `MathResolver` (`AssemblyInfo.cs:4`), which supplies no union formatter.
|
||||
|
||||
- **A1. Unions inside persisted documents** (Aetheria `aac2cf91`, `Assets/Scripts/ServerShared`):
|
||||
`BehaviorData` (`Behaviors/Behaviors.cs:147-191`, keys 0-11, 15, 16, 18, 20-24, 28, 31-40; 12 and 26 retired,
|
||||
14, 17, 19, 25, 27, 29, 30 commented out) in `List<BehaviorData> Behaviors` of the item data documents
|
||||
(`ItemData.cs:346,368`); `WeaponData` (`Behaviors/Weapon.cs:15-18`); `EntityPack` (`EntitySerializer.cs:172-173`)
|
||||
in `aetheria.savedzone` contents and its own `Children`; `ItemInstance` and `CraftedItemInstance`
|
||||
(`ItemInstance.cs:18-31`, a union whose arms are also a union) in entity cargo and docking bays
|
||||
(`EntitySerializer.cs:186-187`); `Provenance` (`Provenance.cs:96-98`) in `aetheria.provenanceledger`
|
||||
(`Provenance.cs:11-15`, `Lot.Origin`); `SavedActionBarBinding` (`SavedGame.cs:221-223`) in `aetheria.savedgame`;
|
||||
the input layout rows and keys (`InputLayout.cs:29-50`) in `aetheria.inputlayout`. `Brush`
|
||||
(`Environment.cs:73`) was not traced to a persisted document.
|
||||
- **A2. `GameData/Aetheria.cc` carries no unknown arm.** Walker `py/store_unions.py` over the git-tracked game
|
||||
database (12.7 MB, `cultcache.store.v1`, 238 records, 13 catalog entries): `Behaviors` arm keys
|
||||
{0:3, 1:2, 2:10, 3:5, 4:2, 5:2, 6:3, 8:7, 9:23, 11:2, 16:4, 23:7, 24:5, 31:1, 32:1, 34:1, 35:2, 37:4, 38:10, 39:2,
|
||||
40:1}; none retired or commented out; no `nil` element.
|
||||
- **A3. The run save carries no unknown arm, and does carry missing trailing slots.** Walker `py/run_unions.py`
|
||||
over `GameData/run.cc` (2026-09-30): `EntityPack` arms 0 and 1 (3 each), `ItemInstance` arm 2 (12),
|
||||
`Provenance` arm 0 (64), `SavedActionBarBinding` 9 `nil` (empty bar slots). The `EntityPack` arms carry 21 slots
|
||||
(orbital) and 20 (ship) against today's 22: the `Boss` slot (key 21, added 2026-10-07 in `db4617d8`) is missing,
|
||||
so today's Aetheria reads this save only because a missing trailing slot defaults. `py/gaps.py`: no non-nil
|
||||
retired slot (ItemInstance 2, 9, 10; EntityPack 5). `GameData/stale-2026-09-22/run.cc` is the same shape;
|
||||
`player.cc` holds only `aetheria.playersettings`. `py/slots.py`: every top-level record matches its own store's
|
||||
catalog exactly.
|
||||
- **A4. What changes for Aetheria.** After it adopts a CultLib with cut 1, its current stores load unchanged
|
||||
(A2, A3). A store written later by a build that has added an arm is refused by a build that lacks it, naming the
|
||||
union and key, where today the arm silently becomes `null` (a gear item quietly loses a behaviour, a lot loses
|
||||
its provenance) and the next save writes the loss back. Builds already pinned to 1.0.60 keep the null path; a
|
||||
shipped binary cannot change. Aetheria-legacy does not use CultCache and has no store.
|
||||
- **A5. Other C# stores.** `%LOCALAPPDATA%Low/GameCult/Aetheria/aetheria-unity.cc` (2026-08-08, 1.2 KB) is a
|
||||
`cultcache.store.v2.directory-indexed` header from an older layout, not a union carrier.
|
||||
|
||||
## C. Rust, TypeScript and other consumers
|
||||
|
||||
Eyes audit for this pass (session `self-2026-10-10c-eureka`), `git grep` at each repo's `origin/HEAD`, Rust reach
|
||||
by a lexical script (it can over-count; it does not under-count named fields). Aetheria and CultLib's tests excluded.
|
||||
|
||||
- **C1. Rust: no `DatabaseEntry` record crosses runtimes as a union, and none uses the reference arm shape.**
|
||||
Records that reach serde enums, each keeping its bytes because the derive is opt-in:
|
||||
Huginn `epiphany.pipeline.resolution.v2` (`ResolutionOutcome`, external tag, `eureka-pipeline/src/lib.rs:746`,
|
||||
live on Yggdrasil); Idunn `idunn.deployment_transaction` and `idunn.admitted_generation` (v4 and legacy forms;
|
||||
internally tagged `#[serde(tag = ..., deny_unknown_fields)]` enums in `control_plane.rs`, `deployment_plan.rs`,
|
||||
and two `#[serde(untagged)]` enums kept so older transactions decode, `drivers.rs:259,387`; live at
|
||||
`/var/lib/gamecult/idunn/control.cc`); CodexConnector `gamecult.codex.replay_record.v1` (`CodexReplayState`,
|
||||
`lib.rs:1017`); Epiphany's Atlas, model-request, persona and receipt records (internal, adjacent, external and
|
||||
untagged representations; `.cc` files under `F:\Projects\.epiphany-runtime\` and on Yggdrasil). Unit-only serde
|
||||
enums (a bare string) are widespread in Idunn, Huginn (`Faculty`, the pipeline's unit enums), CodexConnector and
|
||||
Epiphany. No enum reaches any record in Odin, Muninn, Ghostlight, Huginn `mind-body`, or cultnet-rs and
|
||||
cultmesh-rs production records. None of these is read by another runtime as a union today, so none is affected
|
||||
by the arm refusal; the derive changes nothing for them.
|
||||
- **C2. TypeScript: no consumer encodes a tagged union.** The only hand formatters are StreamPixels
|
||||
`apps/service/src/realtime/overlay-state-documents.ts:190,232,277` (positional arrays, no union). Bifrost,
|
||||
Heimdall, Hermodr, Odin scripts, Stonks, Vili, VoidBot (vendored `cultcache-ts`) and weksa have none.
|
||||
- **C3. C# outside Aetheria: StreamPixels' overlay declares unions; its tracked store holds none of them.**
|
||||
`apps/overlay-unity/Assets/Scripts/Assembly-CSharp/` (`8d40541`): `ActivationCondition` (10 arms),
|
||||
`BattleEffect` (2), `StatusEffect` (4), `ItemInstance` (3), `EquippableItemInstance` (2), reachable from
|
||||
`streampixels.status`, `streampixels.action_item`, `streampixels.skill` and gear documents. Walker
|
||||
`py/sp_catalog.py` over `apps/overlay-unity/Data/catalog.cc` (`cultcache.store.v1`, 20 records): only
|
||||
`outfit`, `skin_color`, `hair_style`, `emote` and `hair_color` records, and no `[int, array]` pair at any depth.
|
||||
Its only consumer resolver is `MathResolver` (`Catalog/CatalogAssembly.cs:6`). CultLib's own
|
||||
`GameCult.Networking` `Message` union (`Messages.cs:10`) is a CultNet wire message, decoded with
|
||||
`MessageSerialization.Options` (`MessageSerialization.cs:7`, `DefaultOptions`), outside `OptionsFor`: the arm
|
||||
guard does not reach it (follow-up).
|
||||
- **C4. Python and Kotlin: no union-shaped record.** repixelizer uses `define_database_entry_type` with flat
|
||||
slots; `cultmesh-kotlin` documents are `data class`es with no sealed type.
|
||||
|
||||
So the stores the arm refusal can affect today are Aetheria's (A2-A4, none carrying an unknown arm) and
|
||||
StreamPixels' overlay catalog (C3, carrying no arm). Every other runtime's persisted enums are serde shapes that
|
||||
the opt-in derive leaves alone.
|
||||
|
||||
## Rationale
|
||||
|
||||
- **Why a guard formatter, not a fork or a re-implementation.** U5 shows a guard that only checks the key and the
|
||||
subtype and delegates the encoding is enough. It keeps MessagePack-CSharp's union encoding as the one encoder,
|
||||
costs one small file, and covers every path that already goes through `OptionsFor`. Forking the generated
|
||||
template or vendoring `DynamicUnionResolver` would put a second encoder in CultLib.
|
||||
- **Why the Rust derive is opt-in.** Serde enums already persisted by Rust consumers keep their bytes
|
||||
(`docs/persisted-record-encoding.md`); a changed default would make every such store unreadable or
|
||||
non-canonical. A new derive with explicit arm keys is how a Rust type says "this crosses runtimes".
|
||||
- **Why arms use newtype variants over slot structs.** A C# arm is a subclass whose slots include the base class's
|
||||
keys. A Rust newtype variant over a struct with keyed slots is the same thing, and it needs the slot code the
|
||||
`DatabaseEntry` derive already has, factored so both derives share one slot generator.
|
||||
- **Why the slot rule is a separate, asked cut.** The ruling settles arms and asks for extra trailing slots to be
|
||||
"settled against the same re-encode rule". Two written contracts disagree (U10), and the current behaviour drops
|
||||
data silently (U6), so the choice is the operator's and it should change every runtime at once.
|
||||
- **The recommendation on extra slots is refusal.** An extra trailing slot is the slot-level twin of an unknown
|
||||
arm: data from a newer writer that the reader's types do not describe. The operator ruled refusal for arms
|
||||
knowing it costs an older reader the newer record; the same reasoning gives refusal for slots, and record
|
||||
envelope slots are already refused in every runtime (U10). The alternative with precedent is
|
||||
`variants:ruling:undeclared-read-only` (a record a type can read but does not declare loads read-only): it would
|
||||
let an older reader see a newer record without overwriting it, but only where a catalog shows the extra slot
|
||||
(top level), while most evolution here is nested (A3's `Boss` slot sits inside a union arm).
|
||||
- **Missing trailing slots stay tolerated.** The reader knows a missing slot and its default, so it carries no
|
||||
bytes it does not understand, and Aetheria's live run save depends on it (A3). A store whose guard compares
|
||||
re-encoded bytes (Idunn's) refuses such a record by its own stricter rule, which is that guard's purpose.
|
||||
Loading…
Reference in a new issue