From bda2629d593d9d91a269d8d7d1377ae0492a1490 Mon Sep 17 00:00:00 2001 From: Libby Date: Wed, 11 Nov 2026 08:35:50 +0000 Subject: [PATCH 1/4] Element ids: document the trailing derived-member route and name it in the hex refusal Co-Authored-By: Claude Sonnet 5.5 --- docs/document-variants-id-sweep.md | 2 + src/GameCult.Caching/CultElementIds.cs | 2 +- src/GameCult.Caching/README.md | 48 +++++++++++++++++++ .../ElementIdEngineTests.cs | 2 +- 4 files changed, 52 insertions(+), 2 deletions(-) diff --git a/docs/document-variants-id-sweep.md b/docs/document-variants-id-sweep.md index 03fad9cb..74ecd023 100644 --- a/docs/document-variants-id-sweep.md +++ b/docs/document-variants-id-sweep.md @@ -4,6 +4,8 @@ Data for the per-consumer sweep in docs/document-variants-cut.md, C2. Every obje Method: a static scan of `F:\Projects\` C# files for types marked `[CultDocument(`, walked through keyed members, base classes, dictionary values and `[Union]` subtypes. Read-only; no consumer was edited. Worktree copies (`Aetheria-catalog`, `CultLib-*`, and so on) are excluded. Struct elements are values and need no id. Non-C# consumers carry the id as an ordinary member and are not listed. The scan is text-based: a document registered without a literal `[CultDocument(` attribute, or a type whose members lack an integer `[Key]`, is missed, so the sweep pass should confirm each repo by registering its types once. +A type that already has a readable string id keeps it and adds a trailing keyed string member marked `[CultElementId(nameof(ExistingId))]`; see the Element ids section of `src/GameCult.Caching/README.md`. + Lines: `Type file:line (via Owner.Member)`, paths relative to the consumer root. diff --git a/src/GameCult.Caching/CultElementIds.cs b/src/GameCult.Caching/CultElementIds.cs index 38dfaaae..97a183e9 100644 --- a/src/GameCult.Caching/CultElementIds.cs +++ b/src/GameCult.Caching/CultElementIds.cs @@ -399,7 +399,7 @@ namespace GameCult.Caching continue; if (shape.Derive == null && !IsRandomId(id!)) throw new CultElementIdException( - $"Record {rootPath}: element id '{id}' in the list at {path} is not 12 lowercase hex characters. An id you write by hand must be 12 lowercase hex; for readable ids, derive them from a member with [CultElementId(nameof(Member))].", + $"Record {rootPath}: element id '{id}' in the list at {path} is not 12 lowercase hex characters. An id you write by hand must be 12 lowercase hex. To keep an existing readable id, leave that member as it is and add a trailing keyed string member marked [CultElementId(nameof(Member))], where Member is the existing one; the element id is derived from it once and then persisted.", rootPath, path, id); if (!taken.Add(id!)) throw Duplicate(id!, path); diff --git a/src/GameCult.Caching/README.md b/src/GameCult.Caching/README.md index 471361dd..32c53940 100644 --- a/src/GameCult.Caching/README.md +++ b/src/GameCult.Caching/README.md @@ -228,6 +228,54 @@ the text of that member (a string or number on the same type, unique within its list), and a derived id may be any text. A readable name set by hand on a non-derived list id is refused; the same member outside a list is unconstrained. +### A type that already has a readable id + +If the element type already has a readable string id (`Name`, `Slug`, a catalog +key), leave it as it is. Add a trailing keyed string member marked +`[CultElementId(nameof(ExistingId))]`; the element id is derived from the +existing member the first time the element is written, and then persisted: + +```csharp +[MessagePackObject] +public sealed class Recipe +{ + [Key(0)] public string Name = string.Empty; // existing readable id + [Key(1)] public int Yield; + [Key(2)] [CultElementId(nameof(Name))] public string ElementId = string.Empty; // new, last +} +``` + +Marking `Name` itself `[CultElementId]` is refused (`Name` is not 12 lowercase +hex), and that is deliberate: + +- Minted and hand-written ids share one grammar, 12 lowercase hex, so they can + never collide; a derived id is the one readable form and is checked for + uniqueness within its list. +- An id the consumer can edit is not a stable override address. Variants address + an element by its id, so a field that is renamed in place would silently move + the overrides to a different element, or to none. +- A derived id is persisted once. Renaming `Name` later does not change + `ElementId`, so the element and every override that addresses it stay put. + `ElementId` is the address; `Name` goes back to being ordinary data. + +Take the next free `[Key]` slot, after the highest key on the type (and above +the subclasses' keys on a union base). + +## Store format and rollback + +A store that holds an element id is written as `cultcache.store.v3` (a directory +store as `cultcache.store.v5.directory-content-addressed-pages`, its `v4` form +otherwise). A single-file store that holds a variant but no id is `v2`, and a store with +neither stays `v1` (`v4` for a directory). Any CultLib from before element ids refuses a v3 or v5 store by name. +That refusal is on purpose, since an older reader would skip the id slots and +rewrite the elements without them. + +Rolling a consumer back to an older CultLib pin after it has written a v3 or v5 +store is therefore one-way. There is no carry-back: a whole-store write drops the +mark only when nothing in the store holds an id, and a type with a +`[CultElementId]` member always writes one. Restore from a copy taken before +upgrading. Upgrade every reader before any writer. + ## Persistence discipline Persistence stays manual by default. Mutations mark the cache and attached diff --git a/tests/GameCult.Caching.Tests/ElementIdEngineTests.cs b/tests/GameCult.Caching.Tests/ElementIdEngineTests.cs index 2bc1d9df..a54251ed 100644 --- a/tests/GameCult.Caching.Tests/ElementIdEngineTests.cs +++ b/tests/GameCult.Caching.Tests/ElementIdEngineTests.cs @@ -142,7 +142,7 @@ namespace GameCult.Caching.Tests public void TheHexRefusalNamesTheDerivedFormForReadableIds() { var refusal = Assert.Throws(() => Assign(new HHasIds { Items = { new EIded { Id = "root" } } }, "r", deterministic: false))!; - Assert.That(refusal.Message, Does.Contain("12 lowercase hex").And.Contain("[CultElementId(nameof(Member))]")); + Assert.That(refusal.Message, Does.Contain("12 lowercase hex").And.Contain("[CultElementId(nameof(Member))]").And.Contain("trailing keyed string member").And.Contain("existing readable id")); } [Test] -- 2.47.3 From 14fb0856e882a57deb94afed193e5b83045b1701 Mon Sep 17 00:00:00 2001 From: Libby Date: Wed, 11 Nov 2026 08:35:50 +0000 Subject: [PATCH 2/4] Docs: a store holding an element id is v3 (v5 directory); rolling back after writing is one-way, no carry-back Co-Authored-By: Claude Sonnet 5.5 --- docs/nuget-packaging.md | 9 +++++++++ packages/cultcache-py/CHANGELOG.md | 2 +- packages/cultcache-py/README.md | 4 ++++ packages/cultcache-rs/CHANGELOG.md | 2 +- packages/cultcache-rs/README.md | 4 ++++ packages/cultcache-ts/CHANGELOG.md | 1 + packages/cultcache-ts/README.md | 4 ++++ src/GameCult.Caching.MessagePack/README.md | 4 ++++ unity/org.gamecult.cultlib/CHANGELOG.md | 2 +- 9 files changed, 29 insertions(+), 3 deletions(-) diff --git a/docs/nuget-packaging.md b/docs/nuget-packaging.md index 301e54eb..a307f35b 100644 --- a/docs/nuget-packaging.md +++ b/docs/nuget-packaging.md @@ -27,3 +27,12 @@ Install the Unity package from the immutable release tag: ```text https://github.com/GameCult/CultLib.git?path=/unity/org.gamecult.cultlib#cultlib-unity-v2.0.0 ``` + +## Rolling back across the store format + +A consumer that upgrades and writes a store holding an element id leaves a +`cultcache.store.v3` single-file store (`v5` directory store). A `GameCult.Caching` +from before element ids refuses it by name, so pinning an older package after +writing is one-way. There is no carry-back; restore from a copy taken before +upgrading. Upgrade readers before writers. Detail: the Element ids section of +`src/GameCult.Caching/README.md`. diff --git a/packages/cultcache-py/CHANGELOG.md b/packages/cultcache-py/CHANGELOG.md index 641cc0ed..cf5898c7 100644 --- a/packages/cultcache-py/CHANGELOG.md +++ b/packages/cultcache-py/CHANGELOG.md @@ -11,7 +11,7 @@ - `indexes` are unique, as C# `[CultIndex]`: a write, load or `register_index` that leaves two records of a type holding one non-blank value under one alias is refused (`CultCacheError`) naming both keys and the alias, and writes nothing. Index text is the C# canonical text (`True`/`False`, integers as digits, floats as the exact expansion, blank values never indexed), so an empty or whitespace string is no longer a value. A new `roles` map names the many-row selection aliases (never indexed for lookup, same catalog `index_alias`). Classify every `indexes` key per the README before updating. The verdict is the contract, the text is informational. -- A single-file store written as `cultcache.store.v3` is read and written: a record that holds an element id (an owned list element with an `[CultElementId]` member, carried through every write) marks the store v3, and a v3 store is carried through a rewrite with its ids. A runtime from before this change refuses a v3 store by name. +- A single-file store written as `cultcache.store.v3` is read and written: a record that holds an element id (an owned list element with an `[CultElementId]` member, carried through every write) marks the store v3, and a v3 store is carried through a rewrite with its ids. A runtime from before this change refuses a v3 store by name. Rolling back to an older release after a store has been written as v3 is one-way: there is no carry-back, so restore from a copy taken before upgrading. - A write over a file the reader refuses (`StoreUnreadableError`, from `push`, `push_all` and `delete`) refuses and leaves the file as it is. A flush used to replace such a file. A write set cannot merge with a disk it cannot read, so the file is repaired or removed by its owner before the store writes again. - Added `StoreUnreadableError` (a `ValueError`) for a store file this runtime cannot read, and `SchemaConflictError` (a `ValueError`) for a write the catalog cannot describe. A store is exactly three slots; a schema is never recovered from a record's payload. diff --git a/packages/cultcache-py/README.md b/packages/cultcache-py/README.md index ab586ff6..2ba13875 100644 --- a/packages/cultcache-py/README.md +++ b/packages/cultcache-py/README.md @@ -139,6 +139,10 @@ used by the TypeScript, Rust, and C# runtimes: Large corpora should use a sharded store or a real database. A single snapshot file is a scalpel, not a forklift. +## Store format and rollback + +A store whose records hold an element id is marked `cultcache.store.v3` (`v5` for a C# directory store). This runtime reads and writes a v3 single-file store and carries ids through a rewrite. A CultLib or cultcache-py from before element ids refuses a v3 or v5 store by name, so rolling back to an older pin after a store has been written that way is one-way. There is no carry-back: a whole-store write drops the mark only when no record holds an id. Restore from a copy taken before upgrading, and upgrade every reader before any writer. + ## DatabaseEntry Slot Contract For cross-runtime cache entries, use `define_database_entry_type(...)` instead diff --git a/packages/cultcache-rs/CHANGELOG.md b/packages/cultcache-rs/CHANGELOG.md index f136bccf..61f6e4dc 100644 --- a/packages/cultcache-rs/CHANGELOG.md +++ b/packages/cultcache-rs/CHANGELOG.md @@ -8,7 +8,7 @@ only in git history. ### Breaking -- A single-file store written as `cultcache.store.v3` is read and written: a record that holds an element id (an owned list element with an `[CultElementId]` member, carried through every write) marks the store v3, and a v3 store is carried through a rewrite with its ids. A runtime from before this change refuses a v3 store by name. +- A single-file store written as `cultcache.store.v3` is read and written: a record that holds an element id (an owned list element with an `[CultElementId]` member, carried through every write) marks the store v3, and a v3 store is carried through a rewrite with its ids. A runtime from before this change refuses a v3 store by name. Rolling back to an older release after a store has been written as v3 is one-way: there is no carry-back, so restore from a copy taken before upgrading. - A write over a file the reader refuses (a zero-byte file, a truncated store, a stamp this runtime does not read) refuses with `CultCacheStoreUnreadable` and leaves the file as it is. A whole-store write used to replace such a file. This reverses 0.3.1, where `push_all` replaced a file with no readable stamp (healing it); the stamp rule now has one owner, the reader every write asks under the lock, and it accepts v1 and v3 (0.3.1 refused v3). - `#[cultcache(index)]` means unique, as in C# `RefuseIndexSharing`: within one type and one alias no two records hold one non-blank value. A put, batch, push or load that would make two records hold a value is refused, naming both keys and the alias, and a store that already holds a duplicate does not open. A grouping index is no longer expressible; a consumer that relied on shared values must drop the `index` and look up by another way. - `bool` index and lookup text renders `True` / `False`, as C# does, where it rendered `true` / `false`. diff --git a/packages/cultcache-rs/README.md b/packages/cultcache-rs/README.md index 1dab4fc7..2d815c1e 100644 --- a/packages/cultcache-rs/README.md +++ b/packages/cultcache-rs/README.md @@ -394,6 +394,10 @@ fn outcome(error: &anyhow::Error) { } ``` +## Store format and rollback + +A store whose records hold an element id is marked `cultcache.store.v3`. This runtime reads and writes a v3 single-file store and carries ids through a rewrite. A CultLib or cultcache-rs from before element ids refuses it by name, so rolling back to an older pin after a store has been written that way is one-way. There is no carry-back: a whole-store write drops the mark only when no record holds an id. Restore from a copy taken before upgrading, and upgrade every reader before any writer. + ## Near-Term Ergonomic Improvements 1. **Derive macro** diff --git a/packages/cultcache-ts/CHANGELOG.md b/packages/cultcache-ts/CHANGELOG.md index 9850dfe4..84d6592c 100644 --- a/packages/cultcache-ts/CHANGELOG.md +++ b/packages/cultcache-ts/CHANGELOG.md @@ -7,6 +7,7 @@ All notable changes to `@gamecult/cultcache-ts` are recorded here, following ### Breaking +- A single-file store marked `cultcache.store.v3` (it holds an element id, written by the C# runtime) is read, and a rewrite keeps the mark and the ids; this runtime does not decide when a document carries ids. A runtime from before this change refuses a v3 store by name, so rolling back to an older release after a store has been written as v3 is one-way: there is no carry-back, so restore from a copy taken before upgrading. - `indexes` are unique, as C# `[CultIndex]`: a write or a load that leaves two records of a type holding one non-blank value under one alias is refused naming both keys and the alias, and writes nothing. Index text is the C# canonical text (booleans `True`/`False`, integers as digits, doubles as the exact expansion, blank values never indexed), so diff --git a/packages/cultcache-ts/README.md b/packages/cultcache-ts/README.md index 97fb3cdd..f78d66ac 100644 --- a/packages/cultcache-ts/README.md +++ b/packages/cultcache-ts/README.md @@ -63,6 +63,10 @@ const payload = encodeDocumentPayload(noteDocument, note); // DocumentPayload Browsers import the same names from `@gamecult/cultcache-ts/document`, which holds no Node built-ins. The whole cache without the single-file store is `@gamecult/cultcache-ts/browser`, which is also what bundlers resolve for the package root (`browser` condition); bring your own `CacheBackingStore`. +## Store format and rollback + +A store whose records hold an element id is marked `cultcache.store.v3` (`v5` for a C# directory store). This runtime reads a v3 store and keeps the mark and the ids through a rewrite; it does not decide when a document carries ids. A CultLib or cultcache-ts from before element ids refuses a v3 or v5 store by name, so rolling back to an older pin after a store has been written that way is one-way. There is no carry-back: a whole-store write drops the mark only when no record holds an id. Restore from a copy taken before upgrading, and upgrade every reader before any writer. + ## `.cc` Inspection CultCache files may use the `.cc` extension. The bytes are still the canonical diff --git a/src/GameCult.Caching.MessagePack/README.md b/src/GameCult.Caching.MessagePack/README.md index d34490f7..e5c8071c 100644 --- a/src/GameCult.Caching.MessagePack/README.md +++ b/src/GameCult.Caching.MessagePack/README.md @@ -59,6 +59,10 @@ await cache.PullAllBackingStoresAsync(); - The raw CultNet document lane should treat these payload bytes as already blessed, not decode and re-encode them for sport. +## Store format and rollback + +A store whose records hold an element id is marked `cultcache.store.v3` (a directory store `v5`). The single-file and directory stores here write the mark from content; see the Element ids section of the GameCult.Caching README. A CultLib or GameCult.Caching from before element ids refuses it by name, so rolling back to an older pin after a store has been written that way is one-way. There is no carry-back: a whole-store write drops the mark only when no record holds an id. Restore from a copy taken before upgrading, and upgrade every reader before any writer. + ## MessagePack source generator NuGet consumers get `[assembly: MessagePackAssumedFormattable(typeof(CultRecordRef<>))]` from the package, so a diff --git a/unity/org.gamecult.cultlib/CHANGELOG.md b/unity/org.gamecult.cultlib/CHANGELOG.md index ef98338b..f9304755 100644 --- a/unity/org.gamecult.cultlib/CHANGELOG.md +++ b/unity/org.gamecult.cultlib/CHANGELOG.md @@ -10,7 +10,7 @@ Releases as 3.0.0 (major). - A single-file store flush refuses a file it cannot read (empty, truncated, an unreadable stamp or body) with `CultStoreUnreadableException` and leaves the file as it is. A flush used to replace such a file. A write set cannot merge with a disk it cannot read, so the file is repaired or removed by its owner before the store writes again. - A single-file store reads v1, v2 and v3 stamps and refuses v4 and later by name; a directory store reads v4 and v5 and refuses v6 and later by name. Each judges the stamp before the body, so an unreadable body never makes a newer store look like damage. -- A store is written in the format its content needs, and CultLib 2.0 refuses the newer ones by name. A single-file store holding a document variant is written as `cultcache.store.v2`, one holding an element id as `cultcache.store.v3`; a directory store holding an element id is written as `cultcache.store.v5` (`v4` otherwise). A store is marked only when a record actually holds an id or a variant. Upgrade readers before writers: a 2.0 reader that opens such a store is refused, not misread. +- A store is written in the format its content needs, and CultLib 2.0 refuses the newer ones by name. A single-file store holding a document variant is written as `cultcache.store.v2`, one holding an element id as `cultcache.store.v3`; a directory store holding an element id is written as `cultcache.store.v5` (`v4` otherwise). A store is marked only when a record actually holds an id or a variant. Upgrade readers before writers: a 2.0 reader that opens such a store is refused, not misread. Rolling back to a 2.0 pin after a store has been written as v3 or v5 is one-way: there is no carry-back (a whole-store write drops the mark only when no record holds an id, and a type with a `[CultElementId]` member always writes one), so restore from a copy taken before upgrading. - Writes are write sets. A flush or commit replaces the records it staged and copies every other record forward as stored, under the store's lock, instead of rewriting the whole store from the cache's memory. A record the cache does not know is kept, and an untouched record keeps a dropped member's slot through every write (never reuse a dropped slot; see the schema-compatibility contract). - A write or removal at a key whose stored record the writer does not own is refused with `CultSchemaConflictException`, and the file is left as it was. A writer owns a record whose schema id it declares (owns or lists compatible) and a record its type reads whole under compatible drift: that record is writable and is restamped under the type's own id on write, as in 2.0.1. A record a type reads only by dropping persisted slots stays read-only until its id is declared compatible, which accepts the loss (this differs from 2.0.1, which shed the slots silently). A cache owns through its registry; a cacheless CultMesh write owns through the catalog entry it writes. - A write replaces the record at a key only when the type it writes there owns that record, decided per key and the same for a cache and for a cacheless `WriteDocument`, `WriteRecord` or unattached-store `PushAll`, in a single-file store and a directory store. The written type owns the stored record when its entry declares the stored id (its own id, or one it lists compatible), or when it is the type that reads the record's published entry and reads it whole (no persisted slot unread): an older id of its own schema name that it reads whole is its own and is restamped, as in 2.0.1. Anything else is refused with `CultSchemaConflictException` and the file left as it was: a record of another registered type, under its current id or an old id it lists, and a record the written type would read only by dropping persisted slots. No registry enters the decision, `CultDocumentRegistry.Shared` included, so a lent registry never changes the answer. This changes a cache: where 2.x let a cache holding two types replace one's record at a key with the other's, it now refuses, with or without a backing store. Retype a key by removing it and adding the new record, in one batch (`batch.Remove(key)` then `batch.Upsert(...)`) or staged apart (`Remove` then `UpsertAsync`); the removal is judged as a removal, so a record no registered type owns is still refused. A removal or check goes to the registry's removal rule, and a cacheless call, which has no registry, removes a record only when an entry it writes owns it (a `Delete` then `PushAll` on an unattached store writes no entry and is refused). -- 2.47.3 From 567eac90565bb788b002dd7d1468557fdb026684 Mon Sep 17 00:00:00 2001 From: Libby Date: Wed, 11 Nov 2026 08:43:06 +0000 Subject: [PATCH 3/4] Docs and tests: 1.x pins do not refuse v3/v5, the mark follows content, non-C# runtimes never drop it; pin the recipe and the narrow drop to v1 Co-Authored-By: Claude Sonnet 5.5 --- docs/nuget-packaging.md | 7 ++- packages/cultcache-py/README.md | 2 +- packages/cultcache-rs/README.md | 2 +- packages/cultcache-ts/README.md | 2 +- src/GameCult.Caching.MessagePack/README.md | 2 +- src/GameCult.Caching/README.md | 28 ++++++---- .../GameCult.Caching.Tests/ElementIdTests.cs | 52 +++++++++++++++++++ unity/org.gamecult.cultlib/CHANGELOG.md | 2 +- 8 files changed, 77 insertions(+), 20 deletions(-) diff --git a/docs/nuget-packaging.md b/docs/nuget-packaging.md index a307f35b..c83929e0 100644 --- a/docs/nuget-packaging.md +++ b/docs/nuget-packaging.md @@ -31,8 +31,7 @@ https://github.com/GameCult/CultLib.git?path=/unity/org.gamecult.cultlib#cultlib ## Rolling back across the store format A consumer that upgrades and writes a store holding an element id leaves a -`cultcache.store.v3` single-file store (`v5` directory store). A `GameCult.Caching` -from before element ids refuses it by name, so pinning an older package after -writing is one-way. There is no carry-back; restore from a copy taken before -upgrading. Upgrade readers before writers. Detail: the Element ids section of +`cultcache.store.v3` single-file store (`v5` directory store). The first CultLib release that refuses such a store by name is `cultlib-unity-v2.0.0` (`caching-unity-v2.0.0`). Earlier pins, the 1.x line (`cultlib-unity-v1.0.61`, `cultlib-v1.0.32`), read the header string and never compare it: rolling back to one of them reads a v3 or v5 store silently, drops the ids, and can rewrite the elements without them. That is the dangerous case. There is +no general carry-back; restore from a copy taken before upgrading, and upgrade +readers before writers. Detail: the Store format and rollback section of `src/GameCult.Caching/README.md`. diff --git a/packages/cultcache-py/README.md b/packages/cultcache-py/README.md index 2ba13875..236009ed 100644 --- a/packages/cultcache-py/README.md +++ b/packages/cultcache-py/README.md @@ -141,7 +141,7 @@ Large corpora should use a sharded store or a real database. A single snapshot f ## Store format and rollback -A store whose records hold an element id is marked `cultcache.store.v3` (`v5` for a C# directory store). This runtime reads and writes a v3 single-file store and carries ids through a rewrite. A CultLib or cultcache-py from before element ids refuses a v3 or v5 store by name, so rolling back to an older pin after a store has been written that way is one-way. There is no carry-back: a whole-store write drops the mark only when no record holds an id. Restore from a copy taken before upgrading, and upgrade every reader before any writer. +A store whose elements hold an id is marked `cultcache.store.v3` by the C# runtime. This runtime reads and writes a v3 single-file store, keeps the header it read and carries the ids through a rewrite; it never drops the mark, so it has no carry-back. A cultcache-py from before element ids refuses a v3 store by name, so rolling back to it after a store has been written as v3 is one-way: restore from a copy taken before upgrading, and upgrade every reader before any writer. (The C# line is different: CultLib 1.x does not refuse; see the GameCult.Caching README.) ## DatabaseEntry Slot Contract diff --git a/packages/cultcache-rs/README.md b/packages/cultcache-rs/README.md index 2d815c1e..45724843 100644 --- a/packages/cultcache-rs/README.md +++ b/packages/cultcache-rs/README.md @@ -396,7 +396,7 @@ fn outcome(error: &anyhow::Error) { ## Store format and rollback -A store whose records hold an element id is marked `cultcache.store.v3`. This runtime reads and writes a v3 single-file store and carries ids through a rewrite. A CultLib or cultcache-rs from before element ids refuses it by name, so rolling back to an older pin after a store has been written that way is one-way. There is no carry-back: a whole-store write drops the mark only when no record holds an id. Restore from a copy taken before upgrading, and upgrade every reader before any writer. +A store whose elements hold an id is marked `cultcache.store.v3` by the C# runtime. This runtime reads and writes a v3 single-file store, keeps the header it read and carries the ids through a rewrite; it never drops the mark, so it has no carry-back. A cultcache-rs from before element ids refuses a v3 store by name, so rolling back to it after a store has been written as v3 is one-way: restore from a copy taken before upgrading, and upgrade every reader before any writer. (The C# line is different: CultLib 1.x does not refuse; see the GameCult.Caching README.) ## Near-Term Ergonomic Improvements diff --git a/packages/cultcache-ts/README.md b/packages/cultcache-ts/README.md index f78d66ac..21e76701 100644 --- a/packages/cultcache-ts/README.md +++ b/packages/cultcache-ts/README.md @@ -65,7 +65,7 @@ Browsers import the same names from `@gamecult/cultcache-ts/document`, which hol ## Store format and rollback -A store whose records hold an element id is marked `cultcache.store.v3` (`v5` for a C# directory store). This runtime reads a v3 store and keeps the mark and the ids through a rewrite; it does not decide when a document carries ids. A CultLib or cultcache-ts from before element ids refuses a v3 or v5 store by name, so rolling back to an older pin after a store has been written that way is one-way. There is no carry-back: a whole-store write drops the mark only when no record holds an id. Restore from a copy taken before upgrading, and upgrade every reader before any writer. +A store whose elements hold an id is marked `cultcache.store.v3` by the C# runtime. This runtime reads a v3 store, keeps the header it read and carries the ids through a rewrite; it never drops the mark, so it has no carry-back. A cultcache-ts from before element ids refuses a v3 store by name, so rolling back to it after a store has been written as v3 is one-way: restore from a copy taken before upgrading, and upgrade every reader before any writer. (The C# line is different: CultLib 1.x does not refuse; see the GameCult.Caching README.) ## `.cc` Inspection diff --git a/src/GameCult.Caching.MessagePack/README.md b/src/GameCult.Caching.MessagePack/README.md index e5c8071c..9643372d 100644 --- a/src/GameCult.Caching.MessagePack/README.md +++ b/src/GameCult.Caching.MessagePack/README.md @@ -61,7 +61,7 @@ await cache.PullAllBackingStoresAsync(); ## Store format and rollback -A store whose records hold an element id is marked `cultcache.store.v3` (a directory store `v5`). The single-file and directory stores here write the mark from content; see the Element ids section of the GameCult.Caching README. A CultLib or GameCult.Caching from before element ids refuses it by name, so rolling back to an older pin after a store has been written that way is one-way. There is no carry-back: a whole-store write drops the mark only when no record holds an id. Restore from a copy taken before upgrading, and upgrade every reader before any writer. +A store whose elements hold an id is marked `cultcache.store.v3` (a directory store `v5`); the single-file and directory stores here write the mark from content. See the Store format and rollback section of the GameCult.Caching README. The first CultLib release that refuses such a store by name is `cultlib-unity-v2.0.0` (`caching-unity-v2.0.0`). Earlier pins, the 1.x line (`cultlib-unity-v1.0.61`, `cultlib-v1.0.32`), read the header string and never compare it: rolling back to one of them reads a v3 or v5 store silently, drops the ids, and can rewrite the elements without them. That is the dangerous case. The only carry-back is narrow and lossy (a whole-store write that leaves no id in the store drops the mark and the ids); otherwise restore from a copy taken before upgrading, and upgrade every reader before any writer. ## MessagePack source generator diff --git a/src/GameCult.Caching/README.md b/src/GameCult.Caching/README.md index 32c53940..63faa68f 100644 --- a/src/GameCult.Caching/README.md +++ b/src/GameCult.Caching/README.md @@ -233,7 +233,7 @@ non-derived list id is refused; the same member outside a list is unconstrained. If the element type already has a readable string id (`Name`, `Slug`, a catalog key), leave it as it is. Add a trailing keyed string member marked `[CultElementId(nameof(ExistingId))]`; the element id is derived from the -existing member the first time the element is written, and then persisted: +existing member on load or first write, and then persisted: ```csharp [MessagePackObject] @@ -258,23 +258,29 @@ hex), and that is deliberate: `ElementId`, so the element and every override that addresses it stay put. `ElementId` is the address; `Name` goes back to being ordinary data. +Adopting the recipe on data that already exists: the existing member becomes an +id source, so it must be non-empty and unique within its list. A record with a +duplicate value refuses to load (`CultElementIdException`, duplicate id) and an +empty one is refused at write, so clean such data first. + Take the next free `[Key]` slot, after the highest key on the type (and above the subclasses' keys on a union base). ## Store format and rollback A store that holds an element id is written as `cultcache.store.v3` (a directory -store as `cultcache.store.v5.directory-content-addressed-pages`, its `v4` form -otherwise). A single-file store that holds a variant but no id is `v2`, and a store with -neither stays `v1` (`v4` for a directory). Any CultLib from before element ids refuses a v3 or v5 store by name. -That refusal is on purpose, since an older reader would skip the id slots and -rewrite the elements without them. +store as `cultcache.store.v5.directory-content-addressed-pages`, `v4` otherwise). +A single-file store that holds a variant but no id is `v2`, and a store with +neither stays `v1`. The mark follows content: a whole-store write that leaves no +element holding an id drops it. -Rolling a consumer back to an older CultLib pin after it has written a v3 or v5 -store is therefore one-way. There is no carry-back: a whole-store write drops the -mark only when nothing in the store holds an id, and a type with a -`[CultElementId]` member always writes one. Restore from a copy taken before -upgrading. Upgrade every reader before any writer. +The first CultLib release that refuses such a store by name is `cultlib-unity-v2.0.0` (`caching-unity-v2.0.0`). Earlier pins, the 1.x line (`cultlib-unity-v1.0.61`, `cultlib-v1.0.32`), read the header string and never compare it: rolling back to one of them reads a v3 or v5 store silently, drops the ids, and can rewrite the elements without them. That is the dangerous case. + +Rolling a consumer back after it has written a v3 or v5 store is therefore +one-way. The only carry-back is narrow and lossy: a whole-store write that leaves +no id anywhere in the store (every record rewritten without list elements) writes +the store unmarked again, and the ids are gone. Otherwise restore from a copy +taken before upgrading. Upgrade every reader before any writer. ## Persistence discipline diff --git a/tests/GameCult.Caching.Tests/ElementIdTests.cs b/tests/GameCult.Caching.Tests/ElementIdTests.cs index 1b6b3fe2..a6c68c94 100644 --- a/tests/GameCult.Caching.Tests/ElementIdTests.cs +++ b/tests/GameCult.Caching.Tests/ElementIdTests.cs @@ -389,6 +389,44 @@ namespace GameCult.Caching.Tests Assert.That(refusal.Message, Does.Contain("'7'")); } + // The documented recipe for a type that already has a readable id (src/GameCult.Caching/README.md, Element ids). + [Test] + public void ATrailingMemberDerivedFromAnExistingReadableIdIsPersistedOnceAndTheStoreIsMarkedV3() + { + var registry = CultDocumentRegistry.ForTypes(new[] { typeof(RecipeBook) }); + var path = PathOf("recipes.cc"); + var key = new CultRecordKey("book"); + using (var cache = CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = registry })) + cache.Commit(batch => batch.Upsert(typeof(RecipeBook), new RecipeBook { Recipes = { new IdRecipe { Name = "bread" }, new IdRecipe { Name = "soup" } } }, key)); + Assert.That(HeaderOf(path), Is.EqualTo("cultcache.store.v3")); + + using (var cache = CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = registry })) + { + var book = cache.Get(key)!; + Assert.That(book.Recipes.Select(recipe => recipe.ElementId), Is.EqualTo(new[] { "bread", "soup" })); + book.Recipes[0].Name = "loaf"; + cache.Commit(batch => batch.Upsert(typeof(RecipeBook), book, key)); + } + + using var again = CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = registry }); + var reread = again.Get(key)!; + Assert.That(reread.Recipes.Select(recipe => recipe.Name), Is.EqualTo(new[] { "loaf", "soup" })); + Assert.That(reread.Recipes.Select(recipe => recipe.ElementId), Is.EqualTo(new[] { "bread", "soup" }), "renaming the source does not move the element"); + } + + [Test] + public void AWholeStoreWriteThatLeavesNoIdInTheStoreDropsTheMarkBackToV1() + { + var registry = CultDocumentRegistry.ForTypes(new[] { typeof(RecipeBook) }); + var path = PathOf("drop.cc"); + var key = new CultRecordKey("book"); + using var cache = CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = registry }); + cache.Commit(batch => batch.Upsert(typeof(RecipeBook), new RecipeBook { Recipes = { new IdRecipe { Name = "bread" } } }, key)); + Assert.That(HeaderOf(path), Is.EqualTo("cultcache.store.v3")); + cache.Commit(batch => batch.Upsert(typeof(RecipeBook), new RecipeBook(), key)); + Assert.That(HeaderOf(path), Is.EqualTo("cultcache.store.v1"), "the mark follows content: no element holds an id, so the store is unmarked"); + } + // ---- C2a fix batch: the marker, refusals at load, override values, atomic minting ---- private const string HexA = "aaaaaaaaaaaa"; @@ -1359,6 +1397,20 @@ namespace GameCult.Caching.Tests [Key(1)] public List Labels { get; set; } = new(); } + [CultDocument("tests.element_id_recipe_book", "tests.element_id_recipe_book.v1")] + [MessagePackObject] + public sealed class RecipeBook + { + [Key(0)] public List Recipes { get; set; } = new(); + } + + [MessagePackObject] + public sealed class IdRecipe + { + [Key(0)] public string Name { get; set; } = ""; + [Key(1)] [CultElementId(nameof(Name))] public string ElementId { get; set; } = ""; + } + [MessagePackObject] public sealed class IdLabel { diff --git a/unity/org.gamecult.cultlib/CHANGELOG.md b/unity/org.gamecult.cultlib/CHANGELOG.md index f9304755..83389b7b 100644 --- a/unity/org.gamecult.cultlib/CHANGELOG.md +++ b/unity/org.gamecult.cultlib/CHANGELOG.md @@ -10,7 +10,7 @@ Releases as 3.0.0 (major). - A single-file store flush refuses a file it cannot read (empty, truncated, an unreadable stamp or body) with `CultStoreUnreadableException` and leaves the file as it is. A flush used to replace such a file. A write set cannot merge with a disk it cannot read, so the file is repaired or removed by its owner before the store writes again. - A single-file store reads v1, v2 and v3 stamps and refuses v4 and later by name; a directory store reads v4 and v5 and refuses v6 and later by name. Each judges the stamp before the body, so an unreadable body never makes a newer store look like damage. -- A store is written in the format its content needs, and CultLib 2.0 refuses the newer ones by name. A single-file store holding a document variant is written as `cultcache.store.v2`, one holding an element id as `cultcache.store.v3`; a directory store holding an element id is written as `cultcache.store.v5` (`v4` otherwise). A store is marked only when a record actually holds an id or a variant. Upgrade readers before writers: a 2.0 reader that opens such a store is refused, not misread. Rolling back to a 2.0 pin after a store has been written as v3 or v5 is one-way: there is no carry-back (a whole-store write drops the mark only when no record holds an id, and a type with a `[CultElementId]` member always writes one), so restore from a copy taken before upgrading. +- A store is written in the format its content needs, and CultLib 2.0 refuses the newer ones by name. A single-file store holding a document variant is written as `cultcache.store.v2`, one holding an element id as `cultcache.store.v3`; a directory store holding an element id is written as `cultcache.store.v5` (`v4` otherwise). A store is marked only when a record actually holds an id or a variant. Upgrade readers before writers: a 2.0 reader that opens such a store is refused, not misread. Rolling back to a 2.0 pin after a store has been written as v3 or v5 is one-way: the only carry-back is narrow and lossy (a whole-store write that leaves no element holding an id writes the store unmarked, and the ids are gone), so restore from a copy taken before upgrading. 1.x pins do not refuse such a store at all; rolling back to one reads it silently and can rewrite the elements without ids. - Writes are write sets. A flush or commit replaces the records it staged and copies every other record forward as stored, under the store's lock, instead of rewriting the whole store from the cache's memory. A record the cache does not know is kept, and an untouched record keeps a dropped member's slot through every write (never reuse a dropped slot; see the schema-compatibility contract). - A write or removal at a key whose stored record the writer does not own is refused with `CultSchemaConflictException`, and the file is left as it was. A writer owns a record whose schema id it declares (owns or lists compatible) and a record its type reads whole under compatible drift: that record is writable and is restamped under the type's own id on write, as in 2.0.1. A record a type reads only by dropping persisted slots stays read-only until its id is declared compatible, which accepts the loss (this differs from 2.0.1, which shed the slots silently). A cache owns through its registry; a cacheless CultMesh write owns through the catalog entry it writes. - A write replaces the record at a key only when the type it writes there owns that record, decided per key and the same for a cache and for a cacheless `WriteDocument`, `WriteRecord` or unattached-store `PushAll`, in a single-file store and a directory store. The written type owns the stored record when its entry declares the stored id (its own id, or one it lists compatible), or when it is the type that reads the record's published entry and reads it whole (no persisted slot unread): an older id of its own schema name that it reads whole is its own and is restamped, as in 2.0.1. Anything else is refused with `CultSchemaConflictException` and the file left as it was: a record of another registered type, under its current id or an old id it lists, and a record the written type would read only by dropping persisted slots. No registry enters the decision, `CultDocumentRegistry.Shared` included, so a lent registry never changes the answer. This changes a cache: where 2.x let a cache holding two types replace one's record at a key with the other's, it now refuses, with or without a backing store. Retype a key by removing it and adding the new record, in one batch (`batch.Remove(key)` then `batch.Upsert(...)`) or staged apart (`Remove` then `UpsertAsync`); the removal is judged as a removal, so a record no registered type owns is still refused. A removal or check goes to the registry's removal rule, and a cacheless call, which has no registry, removes a record only when an entry it writes owns it (a `Delete` then `PushAll` on an unattached store writes no entry and is refused). -- 2.47.3 From 3e9fb543f28409787d2bb6f284d4b2b717b3e77e Mon Sep 17 00:00:00 2001 From: Libby Date: Wed, 11 Nov 2026 08:50:18 +0000 Subject: [PATCH 4/4] Docs: refusal by name differs by store kind and pin (tags read); the C# runtime decides the mark; empty and duplicate sources refuse to load Co-Authored-By: Claude Sonnet 5.5 --- docs/nuget-packaging.md | 2 +- docs/runtime-parity-scope.md | 2 +- packages/cultcache-py/CHANGELOG.md | 2 +- packages/cultcache-rs/CHANGELOG.md | 2 +- packages/cultcache-ts/CHANGELOG.md | 2 +- src/GameCult.Caching.MessagePack/README.md | 2 +- src/GameCult.Caching/README.md | 6 +++--- unity/org.gamecult.cultlib/CHANGELOG.md | 2 +- 8 files changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/nuget-packaging.md b/docs/nuget-packaging.md index c83929e0..841deedf 100644 --- a/docs/nuget-packaging.md +++ b/docs/nuget-packaging.md @@ -31,7 +31,7 @@ https://github.com/GameCult/CultLib.git?path=/unity/org.gamecult.cultlib#cultlib ## Rolling back across the store format A consumer that upgrades and writes a store holding an element id leaves a -`cultcache.store.v3` single-file store (`v5` directory store). The first CultLib release that refuses such a store by name is `cultlib-unity-v2.0.0` (`caching-unity-v2.0.0`). Earlier pins, the 1.x line (`cultlib-unity-v1.0.61`, `cultlib-v1.0.32`), read the header string and never compare it: rolling back to one of them reads a v3 or v5 store silently, drops the ids, and can rewrite the elements without them. That is the dangerous case. There is +`cultcache.store.v3` single-file store (`v5` directory store). Refusal by name depends on the store kind and the pin. A **directory** store (`v5`) is refused by every Unity pin from `cultlib-unity-v1.0.57` and `caching-unity-v1.2.0` on (the earliest tags of each line whose `DirectoryMessagePackBackingStore` reads only the `v4` stamp: "only ... is readable"). A **single-file** store (`v3`) is refused only from `cultlib-unity-v2.0.0` and `caching-unity-v2.0.0`: `cultlib-unity-v1.0.61` and `caching-unity-v1.4.0` read the header string and never compare it. The NuGet line (`cultlib-v1.0.32`, the last `cultlib-v` tag) refuses neither: its directory store writes `cultcache.store.v1.directory` and never checks the stamp on read. Rolling back a single-file store to any of those pins reads it silently, drops the ids, and can rewrite the elements without them. That is the dangerous case. There is no general carry-back; restore from a copy taken before upgrading, and upgrade readers before writers. Detail: the Store format and rollback section of `src/GameCult.Caching/README.md`. diff --git a/docs/runtime-parity-scope.md b/docs/runtime-parity-scope.md index e001bb1d..2f54e2fe 100644 --- a/docs/runtime-parity-scope.md +++ b/docs/runtime-parity-scope.md @@ -16,7 +16,7 @@ Cross-runtime parity has three layers: Every store reader refuses, by name, a header or record slot it does not understand (vectors: `tests/vectors/document-variants-c0`). Element ids: a store that can hold them declares `cultcache.store.v3`; TypeScript, - Python and Rust read it and keep the header, and readers older than ids refuse it + Python and Rust read it and keep the header, and the C# runtime decides the mark from content, and these runtimes never drop it. A C# reader from `cultlib-unity-v2.0.0` on refuses a v3 single-file store it cannot read (earlier pins do not) (vectors: `tests/vectors/document-variants-c2a`). Document variants: C# resolves them (`cultcache.store.v2`); every other runtime, CultMesh single-file reads, the directory store and CultNet refuse a variant by name diff --git a/packages/cultcache-py/CHANGELOG.md b/packages/cultcache-py/CHANGELOG.md index cf5898c7..d440ddca 100644 --- a/packages/cultcache-py/CHANGELOG.md +++ b/packages/cultcache-py/CHANGELOG.md @@ -11,7 +11,7 @@ - `indexes` are unique, as C# `[CultIndex]`: a write, load or `register_index` that leaves two records of a type holding one non-blank value under one alias is refused (`CultCacheError`) naming both keys and the alias, and writes nothing. Index text is the C# canonical text (`True`/`False`, integers as digits, floats as the exact expansion, blank values never indexed), so an empty or whitespace string is no longer a value. A new `roles` map names the many-row selection aliases (never indexed for lookup, same catalog `index_alias`). Classify every `indexes` key per the README before updating. The verdict is the contract, the text is informational. -- A single-file store written as `cultcache.store.v3` is read and written: a record that holds an element id (an owned list element with an `[CultElementId]` member, carried through every write) marks the store v3, and a v3 store is carried through a rewrite with its ids. A runtime from before this change refuses a v3 store by name. Rolling back to an older release after a store has been written as v3 is one-way: there is no carry-back, so restore from a copy taken before upgrading. +- A single-file store written as `cultcache.store.v3` is read and written: a store the C# runtime marked v3 (it holds a record with an element id: an owned list element with a `[CultElementId]` member) keeps its header here, and is carried through every rewrite with its ids; this runtime never drops the mark. A runtime from before this change refuses a v3 store by name. Rolling back to an older release after a store has been written as v3 is one-way: there is no carry-back, so restore from a copy taken before upgrading. - A write over a file the reader refuses (`StoreUnreadableError`, from `push`, `push_all` and `delete`) refuses and leaves the file as it is. A flush used to replace such a file. A write set cannot merge with a disk it cannot read, so the file is repaired or removed by its owner before the store writes again. - Added `StoreUnreadableError` (a `ValueError`) for a store file this runtime cannot read, and `SchemaConflictError` (a `ValueError`) for a write the catalog cannot describe. A store is exactly three slots; a schema is never recovered from a record's payload. diff --git a/packages/cultcache-rs/CHANGELOG.md b/packages/cultcache-rs/CHANGELOG.md index 61f6e4dc..15011491 100644 --- a/packages/cultcache-rs/CHANGELOG.md +++ b/packages/cultcache-rs/CHANGELOG.md @@ -8,7 +8,7 @@ only in git history. ### Breaking -- A single-file store written as `cultcache.store.v3` is read and written: a record that holds an element id (an owned list element with an `[CultElementId]` member, carried through every write) marks the store v3, and a v3 store is carried through a rewrite with its ids. A runtime from before this change refuses a v3 store by name. Rolling back to an older release after a store has been written as v3 is one-way: there is no carry-back, so restore from a copy taken before upgrading. +- A single-file store written as `cultcache.store.v3` is read and written: a store the C# runtime marked v3 (it holds a record with an element id: an owned list element with a `[CultElementId]` member) keeps its header here, and is carried through every rewrite with its ids; this runtime never drops the mark. A runtime from before this change refuses a v3 store by name. Rolling back to an older release after a store has been written as v3 is one-way: there is no carry-back, so restore from a copy taken before upgrading. - A write over a file the reader refuses (a zero-byte file, a truncated store, a stamp this runtime does not read) refuses with `CultCacheStoreUnreadable` and leaves the file as it is. A whole-store write used to replace such a file. This reverses 0.3.1, where `push_all` replaced a file with no readable stamp (healing it); the stamp rule now has one owner, the reader every write asks under the lock, and it accepts v1 and v3 (0.3.1 refused v3). - `#[cultcache(index)]` means unique, as in C# `RefuseIndexSharing`: within one type and one alias no two records hold one non-blank value. A put, batch, push or load that would make two records hold a value is refused, naming both keys and the alias, and a store that already holds a duplicate does not open. A grouping index is no longer expressible; a consumer that relied on shared values must drop the `index` and look up by another way. - `bool` index and lookup text renders `True` / `False`, as C# does, where it rendered `true` / `false`. diff --git a/packages/cultcache-ts/CHANGELOG.md b/packages/cultcache-ts/CHANGELOG.md index 84d6592c..1d0e81a2 100644 --- a/packages/cultcache-ts/CHANGELOG.md +++ b/packages/cultcache-ts/CHANGELOG.md @@ -28,7 +28,7 @@ All notable changes to `@gamecult/cultcache-ts` are recorded here, following ### Breaking -- A single-file store written as `cultcache.store.v3` is read and written: a record that holds an element id (an owned list element with an `[CultElementId]` member, carried through every write) marks the store v3, and a v3 store is carried through a rewrite with its ids. A runtime from before this change refuses a v3 store by name. +- A single-file store written as `cultcache.store.v3` is read and written: a store the C# runtime marked v3 (it holds a record with an element id: an owned list element with a `[CultElementId]` member) keeps its header here, and is carried through every rewrite with its ids; this runtime never drops the mark. A runtime from before this change refuses a v3 store by name. - A single-file store refuses a zero-byte file as `StoreUnreadableError`; only a path with nothing at it is an empty store. A header this runtime does not read is refused, and the refusal echoes the header only as `cultcache.store.v`. - A write over a file the reader refuses (`StoreUnreadableError`, from `push`, `pushAll` and `delete`) refuses and leaves the file as it is. A flush used to replace such a file. A write set cannot merge with a disk it cannot read, so the file is repaired or removed by its owner before the store writes again. A full `pushAll` over a file with no readable stamp does not replace it: the reader refuses it. The stamp rule has one owner, `refuseUnreadFormat`, asked only by the one reader that every write consults under the lock. The reader judges the stamp from the header before it decodes the body, so a newer store is refused by name whether or not its body is whole. - A dangling symbolic link at the store path is an I/O error with code `ENOENT`, not an empty store. diff --git a/src/GameCult.Caching.MessagePack/README.md b/src/GameCult.Caching.MessagePack/README.md index 9643372d..5e7156b0 100644 --- a/src/GameCult.Caching.MessagePack/README.md +++ b/src/GameCult.Caching.MessagePack/README.md @@ -61,7 +61,7 @@ await cache.PullAllBackingStoresAsync(); ## Store format and rollback -A store whose elements hold an id is marked `cultcache.store.v3` (a directory store `v5`); the single-file and directory stores here write the mark from content. See the Store format and rollback section of the GameCult.Caching README. The first CultLib release that refuses such a store by name is `cultlib-unity-v2.0.0` (`caching-unity-v2.0.0`). Earlier pins, the 1.x line (`cultlib-unity-v1.0.61`, `cultlib-v1.0.32`), read the header string and never compare it: rolling back to one of them reads a v3 or v5 store silently, drops the ids, and can rewrite the elements without them. That is the dangerous case. The only carry-back is narrow and lossy (a whole-store write that leaves no id in the store drops the mark and the ids); otherwise restore from a copy taken before upgrading, and upgrade every reader before any writer. +A store whose elements hold an id is marked `cultcache.store.v3` (a directory store `v5`); the single-file and directory stores here write the mark from content. See the Store format and rollback section of the GameCult.Caching README. Refusal by name depends on the store kind and the pin. A **directory** store (`v5`) is refused by every Unity pin from `cultlib-unity-v1.0.57` and `caching-unity-v1.2.0` on (the earliest tags of each line whose `DirectoryMessagePackBackingStore` reads only the `v4` stamp: "only ... is readable"). A **single-file** store (`v3`) is refused only from `cultlib-unity-v2.0.0` and `caching-unity-v2.0.0`: `cultlib-unity-v1.0.61` and `caching-unity-v1.4.0` read the header string and never compare it. The NuGet line (`cultlib-v1.0.32`, the last `cultlib-v` tag) refuses neither: its directory store writes `cultcache.store.v1.directory` and never checks the stamp on read. Rolling back a single-file store to any of those pins reads it silently, drops the ids, and can rewrite the elements without them. That is the dangerous case. The only carry-back is narrow and lossy (a whole-store write that leaves no id in the store drops the mark and the ids); otherwise restore from a copy taken before upgrading, and upgrade every reader before any writer. ## MessagePack source generator diff --git a/src/GameCult.Caching/README.md b/src/GameCult.Caching/README.md index 63faa68f..6defed61 100644 --- a/src/GameCult.Caching/README.md +++ b/src/GameCult.Caching/README.md @@ -260,8 +260,8 @@ hex), and that is deliberate: Adopting the recipe on data that already exists: the existing member becomes an id source, so it must be non-empty and unique within its list. A record with a -duplicate value refuses to load (`CultElementIdException`, duplicate id) and an -empty one is refused at write, so clean such data first. +duplicate value and an empty one both refuse to load +(`CultElementIdException`), so clean such data first. Take the next free `[Key]` slot, after the highest key on the type (and above the subclasses' keys on a union base). @@ -274,7 +274,7 @@ A single-file store that holds a variant but no id is `v2`, and a store with neither stays `v1`. The mark follows content: a whole-store write that leaves no element holding an id drops it. -The first CultLib release that refuses such a store by name is `cultlib-unity-v2.0.0` (`caching-unity-v2.0.0`). Earlier pins, the 1.x line (`cultlib-unity-v1.0.61`, `cultlib-v1.0.32`), read the header string and never compare it: rolling back to one of them reads a v3 or v5 store silently, drops the ids, and can rewrite the elements without them. That is the dangerous case. +Refusal by name depends on the store kind and the pin. A **directory** store (`v5`) is refused by every Unity pin from `cultlib-unity-v1.0.57` and `caching-unity-v1.2.0` on (the earliest tags of each line whose `DirectoryMessagePackBackingStore` reads only the `v4` stamp: "only ... is readable"). A **single-file** store (`v3`) is refused only from `cultlib-unity-v2.0.0` and `caching-unity-v2.0.0`: `cultlib-unity-v1.0.61` and `caching-unity-v1.4.0` read the header string and never compare it. The NuGet line (`cultlib-v1.0.32`, the last `cultlib-v` tag) refuses neither: its directory store writes `cultcache.store.v1.directory` and never checks the stamp on read. Rolling back a single-file store to any of those pins reads it silently, drops the ids, and can rewrite the elements without them. That is the dangerous case. Rolling a consumer back after it has written a v3 or v5 store is therefore one-way. The only carry-back is narrow and lossy: a whole-store write that leaves diff --git a/unity/org.gamecult.cultlib/CHANGELOG.md b/unity/org.gamecult.cultlib/CHANGELOG.md index 83389b7b..3d03439a 100644 --- a/unity/org.gamecult.cultlib/CHANGELOG.md +++ b/unity/org.gamecult.cultlib/CHANGELOG.md @@ -10,7 +10,7 @@ Releases as 3.0.0 (major). - A single-file store flush refuses a file it cannot read (empty, truncated, an unreadable stamp or body) with `CultStoreUnreadableException` and leaves the file as it is. A flush used to replace such a file. A write set cannot merge with a disk it cannot read, so the file is repaired or removed by its owner before the store writes again. - A single-file store reads v1, v2 and v3 stamps and refuses v4 and later by name; a directory store reads v4 and v5 and refuses v6 and later by name. Each judges the stamp before the body, so an unreadable body never makes a newer store look like damage. -- A store is written in the format its content needs, and CultLib 2.0 refuses the newer ones by name. A single-file store holding a document variant is written as `cultcache.store.v2`, one holding an element id as `cultcache.store.v3`; a directory store holding an element id is written as `cultcache.store.v5` (`v4` otherwise). A store is marked only when a record actually holds an id or a variant. Upgrade readers before writers: a 2.0 reader that opens such a store is refused, not misread. Rolling back to a 2.0 pin after a store has been written as v3 or v5 is one-way: the only carry-back is narrow and lossy (a whole-store write that leaves no element holding an id writes the store unmarked, and the ids are gone), so restore from a copy taken before upgrading. 1.x pins do not refuse such a store at all; rolling back to one reads it silently and can rewrite the elements without ids. +- A store is written in the format its content needs, and CultLib 2.0 refuses the newer single-file ones by name. A single-file store holding a document variant is written as `cultcache.store.v2`, one holding an element id as `cultcache.store.v3`; a directory store holding an element id is written as `cultcache.store.v5` (`v4` otherwise). A store is marked only when a record actually holds an id or a variant. Upgrade readers before writers: a 2.0 reader that opens such a store is refused, not misread. Rolling back to a 2.0 pin after a store has been written as v3 or v5 is one-way: the only carry-back is narrow and lossy (a whole-store write that leaves no element holding an id writes the store unmarked, and the ids are gone), so restore from a copy taken before upgrading. Refusal by name is not uniform in 1.x: a v5 directory store is refused from `cultlib-unity-v1.0.57` (`caching-unity-v1.2.0`), but a v3 single-file store is read silently by every pin before `cultlib-unity-v2.0.0` (including `cultlib-unity-v1.0.61`, `caching-unity-v1.4.0`, `cultlib-v1.0.32`), which drops the ids and can rewrite the elements without them. - Writes are write sets. A flush or commit replaces the records it staged and copies every other record forward as stored, under the store's lock, instead of rewriting the whole store from the cache's memory. A record the cache does not know is kept, and an untouched record keeps a dropped member's slot through every write (never reuse a dropped slot; see the schema-compatibility contract). - A write or removal at a key whose stored record the writer does not own is refused with `CultSchemaConflictException`, and the file is left as it was. A writer owns a record whose schema id it declares (owns or lists compatible) and a record its type reads whole under compatible drift: that record is writable and is restamped under the type's own id on write, as in 2.0.1. A record a type reads only by dropping persisted slots stays read-only until its id is declared compatible, which accepts the loss (this differs from 2.0.1, which shed the slots silently). A cache owns through its registry; a cacheless CultMesh write owns through the catalog entry it writes. - A write replaces the record at a key only when the type it writes there owns that record, decided per key and the same for a cache and for a cacheless `WriteDocument`, `WriteRecord` or unattached-store `PushAll`, in a single-file store and a directory store. The written type owns the stored record when its entry declares the stored id (its own id, or one it lists compatible), or when it is the type that reads the record's published entry and reads it whole (no persisted slot unread): an older id of its own schema name that it reads whole is its own and is restamped, as in 2.0.1. Anything else is refused with `CultSchemaConflictException` and the file left as it was: a record of another registered type, under its current id or an old id it lists, and a record the written type would read only by dropping persisted slots. No registry enters the decision, `CultDocumentRegistry.Shared` included, so a lent registry never changes the answer. This changes a cache: where 2.x let a cache holding two types replace one's record at a key with the other's, it now refuses, with or without a backing store. Retype a key by removing it and adding the new record, in one batch (`batch.Remove(key)` then `batch.Upsert(...)`) or staged apart (`Remove` then `UpsertAsync`); the removal is judged as a removal, so a record no registered type owns is still refused. A removal or check goes to the registry's removal rule, and a cacheless call, which has no registry, removes a record only when an entry it writes owns it (a `Delete` then `PushAll` on an unattached store writes no entry and is refused). -- 2.47.3