diff --git a/docs/nuget-packaging.md b/docs/nuget-packaging.md index a307f35b..c83929e0 100644 --- a/docs/nuget-packaging.md +++ b/docs/nuget-packaging.md @@ -31,8 +31,7 @@ https://github.com/GameCult/CultLib.git?path=/unity/org.gamecult.cultlib#cultlib ## Rolling back across the store format A consumer that upgrades and writes a store holding an element id leaves a -`cultcache.store.v3` single-file store (`v5` directory store). A `GameCult.Caching` -from before element ids refuses it by name, so pinning an older package after -writing is one-way. There is no carry-back; restore from a copy taken before -upgrading. Upgrade readers before writers. Detail: the Element ids section of +`cultcache.store.v3` single-file store (`v5` directory store). The first CultLib release that refuses such a store by name is `cultlib-unity-v2.0.0` (`caching-unity-v2.0.0`). Earlier pins, the 1.x line (`cultlib-unity-v1.0.61`, `cultlib-v1.0.32`), read the header string and never compare it: rolling back to one of them reads a v3 or v5 store silently, drops the ids, and can rewrite the elements without them. That is the dangerous case. There is +no general carry-back; restore from a copy taken before upgrading, and upgrade +readers before writers. Detail: the Store format and rollback section of `src/GameCult.Caching/README.md`. diff --git a/packages/cultcache-py/README.md b/packages/cultcache-py/README.md index 2ba13875..236009ed 100644 --- a/packages/cultcache-py/README.md +++ b/packages/cultcache-py/README.md @@ -141,7 +141,7 @@ Large corpora should use a sharded store or a real database. A single snapshot f ## Store format and rollback -A store whose records hold an element id is marked `cultcache.store.v3` (`v5` for a C# directory store). This runtime reads and writes a v3 single-file store and carries ids through a rewrite. A CultLib or cultcache-py from before element ids refuses a v3 or v5 store by name, so rolling back to an older pin after a store has been written that way is one-way. There is no carry-back: a whole-store write drops the mark only when no record holds an id. Restore from a copy taken before upgrading, and upgrade every reader before any writer. +A store whose elements hold an id is marked `cultcache.store.v3` by the C# runtime. This runtime reads and writes a v3 single-file store, keeps the header it read and carries the ids through a rewrite; it never drops the mark, so it has no carry-back. A cultcache-py from before element ids refuses a v3 store by name, so rolling back to it after a store has been written as v3 is one-way: restore from a copy taken before upgrading, and upgrade every reader before any writer. (The C# line is different: CultLib 1.x does not refuse; see the GameCult.Caching README.) ## DatabaseEntry Slot Contract diff --git a/packages/cultcache-rs/README.md b/packages/cultcache-rs/README.md index 2d815c1e..45724843 100644 --- a/packages/cultcache-rs/README.md +++ b/packages/cultcache-rs/README.md @@ -396,7 +396,7 @@ fn outcome(error: &anyhow::Error) { ## Store format and rollback -A store whose records hold an element id is marked `cultcache.store.v3`. This runtime reads and writes a v3 single-file store and carries ids through a rewrite. A CultLib or cultcache-rs from before element ids refuses it by name, so rolling back to an older pin after a store has been written that way is one-way. There is no carry-back: a whole-store write drops the mark only when no record holds an id. Restore from a copy taken before upgrading, and upgrade every reader before any writer. +A store whose elements hold an id is marked `cultcache.store.v3` by the C# runtime. This runtime reads and writes a v3 single-file store, keeps the header it read and carries the ids through a rewrite; it never drops the mark, so it has no carry-back. A cultcache-rs from before element ids refuses a v3 store by name, so rolling back to it after a store has been written as v3 is one-way: restore from a copy taken before upgrading, and upgrade every reader before any writer. (The C# line is different: CultLib 1.x does not refuse; see the GameCult.Caching README.) ## Near-Term Ergonomic Improvements diff --git a/packages/cultcache-ts/README.md b/packages/cultcache-ts/README.md index f78d66ac..21e76701 100644 --- a/packages/cultcache-ts/README.md +++ b/packages/cultcache-ts/README.md @@ -65,7 +65,7 @@ Browsers import the same names from `@gamecult/cultcache-ts/document`, which hol ## Store format and rollback -A store whose records hold an element id is marked `cultcache.store.v3` (`v5` for a C# directory store). This runtime reads a v3 store and keeps the mark and the ids through a rewrite; it does not decide when a document carries ids. A CultLib or cultcache-ts from before element ids refuses a v3 or v5 store by name, so rolling back to an older pin after a store has been written that way is one-way. There is no carry-back: a whole-store write drops the mark only when no record holds an id. Restore from a copy taken before upgrading, and upgrade every reader before any writer. +A store whose elements hold an id is marked `cultcache.store.v3` by the C# runtime. This runtime reads a v3 store, keeps the header it read and carries the ids through a rewrite; it never drops the mark, so it has no carry-back. A cultcache-ts from before element ids refuses a v3 store by name, so rolling back to it after a store has been written as v3 is one-way: restore from a copy taken before upgrading, and upgrade every reader before any writer. (The C# line is different: CultLib 1.x does not refuse; see the GameCult.Caching README.) ## `.cc` Inspection diff --git a/src/GameCult.Caching.MessagePack/README.md b/src/GameCult.Caching.MessagePack/README.md index e5c8071c..9643372d 100644 --- a/src/GameCult.Caching.MessagePack/README.md +++ b/src/GameCult.Caching.MessagePack/README.md @@ -61,7 +61,7 @@ await cache.PullAllBackingStoresAsync(); ## Store format and rollback -A store whose records hold an element id is marked `cultcache.store.v3` (a directory store `v5`). The single-file and directory stores here write the mark from content; see the Element ids section of the GameCult.Caching README. A CultLib or GameCult.Caching from before element ids refuses it by name, so rolling back to an older pin after a store has been written that way is one-way. There is no carry-back: a whole-store write drops the mark only when no record holds an id. Restore from a copy taken before upgrading, and upgrade every reader before any writer. +A store whose elements hold an id is marked `cultcache.store.v3` (a directory store `v5`); the single-file and directory stores here write the mark from content. See the Store format and rollback section of the GameCult.Caching README. The first CultLib release that refuses such a store by name is `cultlib-unity-v2.0.0` (`caching-unity-v2.0.0`). Earlier pins, the 1.x line (`cultlib-unity-v1.0.61`, `cultlib-v1.0.32`), read the header string and never compare it: rolling back to one of them reads a v3 or v5 store silently, drops the ids, and can rewrite the elements without them. That is the dangerous case. The only carry-back is narrow and lossy (a whole-store write that leaves no id in the store drops the mark and the ids); otherwise restore from a copy taken before upgrading, and upgrade every reader before any writer. ## MessagePack source generator diff --git a/src/GameCult.Caching/README.md b/src/GameCult.Caching/README.md index 32c53940..63faa68f 100644 --- a/src/GameCult.Caching/README.md +++ b/src/GameCult.Caching/README.md @@ -233,7 +233,7 @@ non-derived list id is refused; the same member outside a list is unconstrained. If the element type already has a readable string id (`Name`, `Slug`, a catalog key), leave it as it is. Add a trailing keyed string member marked `[CultElementId(nameof(ExistingId))]`; the element id is derived from the -existing member the first time the element is written, and then persisted: +existing member on load or first write, and then persisted: ```csharp [MessagePackObject] @@ -258,23 +258,29 @@ hex), and that is deliberate: `ElementId`, so the element and every override that addresses it stay put. `ElementId` is the address; `Name` goes back to being ordinary data. +Adopting the recipe on data that already exists: the existing member becomes an +id source, so it must be non-empty and unique within its list. A record with a +duplicate value refuses to load (`CultElementIdException`, duplicate id) and an +empty one is refused at write, so clean such data first. + Take the next free `[Key]` slot, after the highest key on the type (and above the subclasses' keys on a union base). ## Store format and rollback A store that holds an element id is written as `cultcache.store.v3` (a directory -store as `cultcache.store.v5.directory-content-addressed-pages`, its `v4` form -otherwise). A single-file store that holds a variant but no id is `v2`, and a store with -neither stays `v1` (`v4` for a directory). Any CultLib from before element ids refuses a v3 or v5 store by name. -That refusal is on purpose, since an older reader would skip the id slots and -rewrite the elements without them. +store as `cultcache.store.v5.directory-content-addressed-pages`, `v4` otherwise). +A single-file store that holds a variant but no id is `v2`, and a store with +neither stays `v1`. The mark follows content: a whole-store write that leaves no +element holding an id drops it. -Rolling a consumer back to an older CultLib pin after it has written a v3 or v5 -store is therefore one-way. There is no carry-back: a whole-store write drops the -mark only when nothing in the store holds an id, and a type with a -`[CultElementId]` member always writes one. Restore from a copy taken before -upgrading. Upgrade every reader before any writer. +The first CultLib release that refuses such a store by name is `cultlib-unity-v2.0.0` (`caching-unity-v2.0.0`). Earlier pins, the 1.x line (`cultlib-unity-v1.0.61`, `cultlib-v1.0.32`), read the header string and never compare it: rolling back to one of them reads a v3 or v5 store silently, drops the ids, and can rewrite the elements without them. That is the dangerous case. + +Rolling a consumer back after it has written a v3 or v5 store is therefore +one-way. The only carry-back is narrow and lossy: a whole-store write that leaves +no id anywhere in the store (every record rewritten without list elements) writes +the store unmarked again, and the ids are gone. Otherwise restore from a copy +taken before upgrading. Upgrade every reader before any writer. ## Persistence discipline diff --git a/tests/GameCult.Caching.Tests/ElementIdTests.cs b/tests/GameCult.Caching.Tests/ElementIdTests.cs index 1b6b3fe2..a6c68c94 100644 --- a/tests/GameCult.Caching.Tests/ElementIdTests.cs +++ b/tests/GameCult.Caching.Tests/ElementIdTests.cs @@ -389,6 +389,44 @@ namespace GameCult.Caching.Tests Assert.That(refusal.Message, Does.Contain("'7'")); } + // The documented recipe for a type that already has a readable id (src/GameCult.Caching/README.md, Element ids). + [Test] + public void ATrailingMemberDerivedFromAnExistingReadableIdIsPersistedOnceAndTheStoreIsMarkedV3() + { + var registry = CultDocumentRegistry.ForTypes(new[] { typeof(RecipeBook) }); + var path = PathOf("recipes.cc"); + var key = new CultRecordKey("book"); + using (var cache = CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = registry })) + cache.Commit(batch => batch.Upsert(typeof(RecipeBook), new RecipeBook { Recipes = { new IdRecipe { Name = "bread" }, new IdRecipe { Name = "soup" } } }, key)); + Assert.That(HeaderOf(path), Is.EqualTo("cultcache.store.v3")); + + using (var cache = CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = registry })) + { + var book = cache.Get(key)!; + Assert.That(book.Recipes.Select(recipe => recipe.ElementId), Is.EqualTo(new[] { "bread", "soup" })); + book.Recipes[0].Name = "loaf"; + cache.Commit(batch => batch.Upsert(typeof(RecipeBook), book, key)); + } + + using var again = CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = registry }); + var reread = again.Get(key)!; + Assert.That(reread.Recipes.Select(recipe => recipe.Name), Is.EqualTo(new[] { "loaf", "soup" })); + Assert.That(reread.Recipes.Select(recipe => recipe.ElementId), Is.EqualTo(new[] { "bread", "soup" }), "renaming the source does not move the element"); + } + + [Test] + public void AWholeStoreWriteThatLeavesNoIdInTheStoreDropsTheMarkBackToV1() + { + var registry = CultDocumentRegistry.ForTypes(new[] { typeof(RecipeBook) }); + var path = PathOf("drop.cc"); + var key = new CultRecordKey("book"); + using var cache = CultCacheMessagePack.Create(path, new CultCacheOpenOptions { Registry = registry }); + cache.Commit(batch => batch.Upsert(typeof(RecipeBook), new RecipeBook { Recipes = { new IdRecipe { Name = "bread" } } }, key)); + Assert.That(HeaderOf(path), Is.EqualTo("cultcache.store.v3")); + cache.Commit(batch => batch.Upsert(typeof(RecipeBook), new RecipeBook(), key)); + Assert.That(HeaderOf(path), Is.EqualTo("cultcache.store.v1"), "the mark follows content: no element holds an id, so the store is unmarked"); + } + // ---- C2a fix batch: the marker, refusals at load, override values, atomic minting ---- private const string HexA = "aaaaaaaaaaaa"; @@ -1359,6 +1397,20 @@ namespace GameCult.Caching.Tests [Key(1)] public List Labels { get; set; } = new(); } + [CultDocument("tests.element_id_recipe_book", "tests.element_id_recipe_book.v1")] + [MessagePackObject] + public sealed class RecipeBook + { + [Key(0)] public List Recipes { get; set; } = new(); + } + + [MessagePackObject] + public sealed class IdRecipe + { + [Key(0)] public string Name { get; set; } = ""; + [Key(1)] [CultElementId(nameof(Name))] public string ElementId { get; set; } = ""; + } + [MessagePackObject] public sealed class IdLabel { diff --git a/unity/org.gamecult.cultlib/CHANGELOG.md b/unity/org.gamecult.cultlib/CHANGELOG.md index f9304755..83389b7b 100644 --- a/unity/org.gamecult.cultlib/CHANGELOG.md +++ b/unity/org.gamecult.cultlib/CHANGELOG.md @@ -10,7 +10,7 @@ Releases as 3.0.0 (major). - A single-file store flush refuses a file it cannot read (empty, truncated, an unreadable stamp or body) with `CultStoreUnreadableException` and leaves the file as it is. A flush used to replace such a file. A write set cannot merge with a disk it cannot read, so the file is repaired or removed by its owner before the store writes again. - A single-file store reads v1, v2 and v3 stamps and refuses v4 and later by name; a directory store reads v4 and v5 and refuses v6 and later by name. Each judges the stamp before the body, so an unreadable body never makes a newer store look like damage. -- A store is written in the format its content needs, and CultLib 2.0 refuses the newer ones by name. A single-file store holding a document variant is written as `cultcache.store.v2`, one holding an element id as `cultcache.store.v3`; a directory store holding an element id is written as `cultcache.store.v5` (`v4` otherwise). A store is marked only when a record actually holds an id or a variant. Upgrade readers before writers: a 2.0 reader that opens such a store is refused, not misread. Rolling back to a 2.0 pin after a store has been written as v3 or v5 is one-way: there is no carry-back (a whole-store write drops the mark only when no record holds an id, and a type with a `[CultElementId]` member always writes one), so restore from a copy taken before upgrading. +- A store is written in the format its content needs, and CultLib 2.0 refuses the newer ones by name. A single-file store holding a document variant is written as `cultcache.store.v2`, one holding an element id as `cultcache.store.v3`; a directory store holding an element id is written as `cultcache.store.v5` (`v4` otherwise). A store is marked only when a record actually holds an id or a variant. Upgrade readers before writers: a 2.0 reader that opens such a store is refused, not misread. Rolling back to a 2.0 pin after a store has been written as v3 or v5 is one-way: the only carry-back is narrow and lossy (a whole-store write that leaves no element holding an id writes the store unmarked, and the ids are gone), so restore from a copy taken before upgrading. 1.x pins do not refuse such a store at all; rolling back to one reads it silently and can rewrite the elements without ids. - Writes are write sets. A flush or commit replaces the records it staged and copies every other record forward as stored, under the store's lock, instead of rewriting the whole store from the cache's memory. A record the cache does not know is kept, and an untouched record keeps a dropped member's slot through every write (never reuse a dropped slot; see the schema-compatibility contract). - A write or removal at a key whose stored record the writer does not own is refused with `CultSchemaConflictException`, and the file is left as it was. A writer owns a record whose schema id it declares (owns or lists compatible) and a record its type reads whole under compatible drift: that record is writable and is restamped under the type's own id on write, as in 2.0.1. A record a type reads only by dropping persisted slots stays read-only until its id is declared compatible, which accepts the loss (this differs from 2.0.1, which shed the slots silently). A cache owns through its registry; a cacheless CultMesh write owns through the catalog entry it writes. - A write replaces the record at a key only when the type it writes there owns that record, decided per key and the same for a cache and for a cacheless `WriteDocument`, `WriteRecord` or unattached-store `PushAll`, in a single-file store and a directory store. The written type owns the stored record when its entry declares the stored id (its own id, or one it lists compatible), or when it is the type that reads the record's published entry and reads it whole (no persisted slot unread): an older id of its own schema name that it reads whole is its own and is restamped, as in 2.0.1. Anything else is refused with `CultSchemaConflictException` and the file left as it was: a record of another registered type, under its current id or an old id it lists, and a record the written type would read only by dropping persisted slots. No registry enters the decision, `CultDocumentRegistry.Shared` included, so a lent registry never changes the answer. This changes a cache: where 2.x let a cache holding two types replace one's record at a key with the other's, it now refuses, with or without a backing store. Retype a key by removing it and adding the new record, in one batch (`batch.Remove(key)` then `batch.Upsert(...)`) or staged apart (`Remove` then `UpsertAsync`); the removal is judged as a removal, so a record no registered type owns is still refused. A removal or check goes to the registry's removal rule, and a cacheless call, which has no registry, removes a record only when an entry it writes owns it (a `Delete` then `PushAll` on an unattached store writes no entry and is refused).