From 365e8cd7813ee505682cf6aa2436a59c3ffb1d0d Mon Sep 17 00:00:00 2001 From: Meta Vens Date: Sun, 13 Sep 2026 15:37:25 +0200 Subject: [PATCH] Fold compare-exchange into the atomic commit cut (in progress) Saved mid-pass after a workstation crash: per-entry Expect and ExpectUnchanged on the one-store batch, TryCommit, a lock sidecar for the single-file store, the last-writer-wins flush contract, the race tests and probe P7, and full CultLib\packages paths for sibling runtimes. The Imagination pass is finishing the rest. Co-Authored-By: Claude Opus 5 --- docs/cultcache-migration-cut.md | 192 ++++++++++++++++++++++++++------ 1 file changed, 157 insertions(+), 35 deletions(-) diff --git a/docs/cultcache-migration-cut.md b/docs/cultcache-migration-cut.md index 86c1ec65c..8caf4c779 100644 --- a/docs/cultcache-migration-cut.md +++ b/docs/cultcache-migration-cut.md @@ -29,18 +29,18 @@ and its worktrees are taxidermy, not a consumer. Replication and mirrors are deleted everywhere. No bytes change.** - TypeScript routes by type and mirrors the rest: `addBackingStore(store, - ...types)` (`packages\cultcache-ts\src\cult-cache.ts:180-188`), + ...types)` (`CultLib\packages\cultcache-ts\src\cult-cache.ts:180-188`), `#resolveRoute` returns `{primary, mirrors}` (`:663-677`), and `put`, `putEnvelope`, `delete` push to `primary` then every mirror (`:401-402, 447-448, 502-503`; `StoreRoute.mirrors` at `:18`). - Rust: `add_backing_store`/`add_generic_backing_store` - (`packages\cultcache-rs\src\lib.rs:1950-1963`), `resolve_route_indices` + (`CultLib\packages\cultcache-rs\src\lib.rs:1950-1963`), `resolve_route_indices` (`:2363-2380`), mirror loops at `:2092-2094, 2213-2215, 2252-2254, 2323-2325`; `put_prepared_batch` already demands one route and one store (`:2174-2189`). Its README claims this "mirrors the C# behavior" - (`packages\cultcache-rs\README.md:216-241`), which is false. + (`CultLib\packages\cultcache-rs\README.md:216-241`), which is false. - Python: `stores_by_type: dict[str, list[BackingStore]]`, `generic_stores: - list` (`packages\cultcache-py\src\cultcache_py\cache.py:23-24`); write loops + list` (`CultLib\packages\cultcache-py\src\cultcache_py\cache.py:23-24`); write loops at `:199-201, 218-219, 239-241, 269-270`; resolver at `:300-302`. - C# replicates every record to every store (`CC:2097-2101, 2135-2139`) and pushes all entries into a newly attached store (`CC:1436-1439`). `[P2-A]`: @@ -49,13 +49,14 @@ Replication and mirrors are deleted everywhere. No bytes change.** - Nobody uses a mirror: every cache in `CultLib\src`, `CultLib\tests` and every consumer attaches one untyped store; the only multi-store cache anywhere is the Rust test `type_specific_store_routes_before_generic_store` - (`lib.rs:3714-3748`), whose types never share a store. No test asserts + (`CultLib\packages\cultcache-rs\src\lib.rs:3714-3748`), whose types never share a store. No test asserts mirroring. - No runtime writes store identity into the file: all write `[formatVersion, catalog[], records[]]`, records `[key, schemaId, storedAt, payload]` (C# `CultDocumentMessagePackSerialization.cs:180-251`; TS - `single-file-messagepack-backing-store.ts:203-232`; Rust `lib.rs:244-280, - 2401-2455`; Python `stores.py:137-152`). `storeId` + `CultLib\packages\cultcache-ts\src\single-file-messagepack-backing-store.ts:203-232`; + Rust `CultLib\packages\cultcache-rs\src\lib.rs:244-280, 2401-2455`; Python + `CultLib\packages\cultcache-py\src\cultcache_py\stores.py:137-152`). `storeId` (`cultcache-persistence-format.md:93`) is implemented nowhere. Decision: a document type has exactly one home store, chosen by its most @@ -337,7 +338,7 @@ the two Caching projects. | `soft` flag on `FlushAsync`/`FlushAllBackingStores`/`PushAll`/`CommitBatch` | no store reads it (`CC:2638`, `DMS:360`); Mimir (`EveDashboard:900`, `CultMeshMedia:422`) and Brokkr (`BrokkrCultMeshMirror.cs:68-124`) pass `soft: true` through `CultMesh.FlushAsync`/`CultNetLocal.FlushAsync` | delete from Caching; `CultMesh.FlushAsync(bool)` and `CultNetLocal.FlushAsync(bool)` keep their parameter and stop forwarding it. Removing it there and at the four external call sites is a named follow-up | | SoA: `Soa`, `CultSoaTable`, `CultSoaColumn`, `CultCacheSoaStore`, `CultCacheSoaTypeTable`, `CultCacheSoaMember` (`CultManagedDocument.cs:136-414`), `_soa` hooks in `CC` | tests only | **park**, not delete: built for a stated reason (ECS-style structure-of-arrays columns over cached documents, performance without compromising document ergonomics). Cut 2 step 1 tags the last commit holding it as `parked/cultcache-soa` before removal, adds `docs\parked-features.md`, and amends `docs\runtime-parity-scope.md:23-27, 33` to say the SoA table is parked at that tag | | `Document`, `CultManagedDocument`, `CultNetDatabase.Document` | tests only | delete | -| Transactions: `ExecuteTransactionAsync` (both), `CultCacheTransaction`, `CommitTransaction`, `VisibleStoredDocuments`, `_ambientTransaction`, `_transactionGate`, `CacheBackingStore.CommitBatch` (+ `DMS:308-351`), `CultNetDatabase.ExecuteTransactionAsync` (`CND:519-555`), `RequireTransactionsForAuthoritativeWrites` (`CND:305`), `EnsureAuthoritativeTransaction` (`CND:1791-1796`), `AfterCommit` (`CND:898-901`) | tests only in C#. Siblings: TS has no batch; Python has `put_envelopes` (`cache.py:227-251`, one document type per call, `push_all`, no rollback) used by `cultnet-py\...\replication.py:98`; Rust has a cache-level `put_prepared_batch` (`lib.rs:2162-2201`: one store per batch, all-or-nothing `push_all`) with no caller, and a **store-level** batch family (`compare_and_swap_batch`, `compare_exchange`, `compare_exchange_snapshot`, `delete_batch_if_unchanged`, `lib.rs:397-800`, `8f29ee5`) consumed by Odin (12 sites), Idunn (31), CodexConnector (1) and Ghostlight (`ghostlight-dungeon\src\app_session.rs:119, 371`, plus tests): every one passes an explicit batch value; none relies on ambient in-flight visibility. Operator: atomic multi-record commit is a desired capability | **replace, not delete**: the ambient `AsyncLocal` overlay, the `SemaphoreSlim` gate, `CultCacheTransaction`, `CommitTransaction` and `VisibleStoredDocuments` (~230 lines) go in Cut 3 in the same commit that lands `Commit(Action)` (2.2); `CacheBackingStore.CommitBatch` and `DMS:308-351` stay as the store side. `CultNetDatabase.ExecuteTransactionAsync`, `RequireTransactionsForAuthoritativeWrites`, `EnsureAuthoritativeTransaction`, `AfterCommit` are deleted (tests only; CultNet callers use `PutAsync`). `cultcache-persistence-format.md:42-55` is amended to the explicit-batch shape, not retracted | +| Transactions: `ExecuteTransactionAsync` (both), `CultCacheTransaction`, `CommitTransaction`, `VisibleStoredDocuments`, `_ambientTransaction`, `_transactionGate`, `CacheBackingStore.CommitBatch` (+ `DMS:308-351`), `CultNetDatabase.ExecuteTransactionAsync` (`CND:519-555`), `RequireTransactionsForAuthoritativeWrites` (`CND:305`), `EnsureAuthoritativeTransaction` (`CND:1791-1796`), `AfterCommit` (`CND:898-901`) | tests only in C#. Siblings: TS has no batch; Python has `put_envelopes` (`CultLib\packages\cultcache-py\src\cultcache_py\cache.py:227-251`, one document type per call, `push_all`, no rollback) used by `CultLib\packages\cultnet-py\src\cultnet_py\replication.py:98`; Rust has a cache-level `put_prepared_batch` (`CultLib\packages\cultcache-rs\src\lib.rs:2162-2201`: one store per batch, all-or-nothing `push_all`) with no caller, and a **store-level** batch family (`compare_and_swap_batch`, `compare_exchange`, `compare_exchange_snapshot`, `delete_batch_if_unchanged`, `CultLib\packages\cultcache-rs\src\lib.rs:397-800`, `8f29ee5`) consumed by Odin (12 sites), Idunn (31), CodexConnector (1) and Ghostlight (`ghostlight-dungeon\src\app_session.rs:119, 371`, plus tests): every one passes an explicit batch value; none relies on ambient in-flight visibility. Operator: atomic multi-record commit is a desired capability | **replace, not delete**: the ambient `AsyncLocal` overlay, the `SemaphoreSlim` gate, `CultCacheTransaction`, `CommitTransaction` and `VisibleStoredDocuments` (~230 lines) go in Cut 3 in the same commit that lands `Commit(Action)` (2.2); `CacheBackingStore.CommitBatch` and `DMS:308-351` stay as the store side. `CultNetDatabase.ExecuteTransactionAsync`, `RequireTransactionsForAuthoritativeWrites`, `EnsureAuthoritativeTransaction`, `AfterCommit` are deleted (tests only; CultNet callers use `PutAsync`). `cultcache-persistence-format.md:42-55` is amended to the explicit-batch shape, not retracted | | Directory store legacy formats `cultcache.store.v2.directory-indexed`, `v3.directory-immutable-pages`, v1 inline records: `LoadLegacyRecords`, `_legacyInlineRecords`, `LegacyRecordPath`, `MetadataRecordPath`, `_needsIndexUpgrade`, `_manifestUsesMetadataPages` | no store on disk anywhere scanned uses them; only v4 is written | delete; a v2/v3 manifest is refused with its format string | | Directory store `ReadStageProbe`, `FlushStageProbe`, stage constants, `ReadPersistedGeneration` | tests only | delete, with the tests that inject through them | | Directory store itself, `AcquireCommitLease`, `UseDirectoryStore`, `DefaultRecordDirectoryPath` | Ymir (`YmirWorldStateDocument.cs:248, 267`, `YmirServicePublicationDocument.cs:161`), AquaSynth (`IpaTrialResults.cs:190`) | keep (v4 read/write and the lease) | @@ -405,7 +406,13 @@ public sealed class CultCacheBatch // an explicit value; nothing public CultRecordHandle Upsert(T document, CultRecordHandle? handle = null) public CultRecordKey Upsert(Type type, object document, CultRecordKey? key = null) public void Remove(CultRecordKey key) + public void Expect(CultRecordKey key, object? current) // current = the instance this cache holds at key, or null = must be absent + public void ExpectUnchanged() // the home store's persisted content equals what this cache last loaded } +public enum CultCommitOutcome { Committed, Mismatch, Contended } +// on CultCache: +public bool Commit(Action stage) // false = a condition failed: no write, no memory change, no notification +public CultCommitOutcome TryCommit(Action stage) // one non-blocking lock attempt; otherwise identical public abstract class CacheBackingStore : IDisposable { @@ -414,6 +421,13 @@ public abstract class CacheBackingStore : IDisposable public IReadOnlyList LastSchemaMigrationReports internal Action Loaded, Unloaded; // set by the cache at attach public abstract void PullAll(); public abstract void Push(CultStoredDocument e); public abstract void Delete(CultStoredDocument e); public abstract void PushAll(); + public abstract CultCommitOutcome CommitBatch(CultCommitRequest request, bool wait); // one durable step under the store's lock +} +public sealed class CultCommitRequest // built by Commit from the batch; identities are (schemaId, storedAt) captured at stage time +{ + IReadOnlyList Upserts; IReadOnlyList Deletes; + IReadOnlyList<(CultRecordKey Key, string? SchemaId, string? StoredAt)> Expected; // null pair = must be absent + bool ExpectUnchanged; } ``` @@ -455,8 +469,9 @@ reads a global back (Q4). type per cache, `GetGlobal()` returns it or `null`, and a write without a handle keys it `global:{SchemaId}` (`ResolveKey`, `CC:2195-2198`, kept). The cache never creates one. Siblings already do this: TS enforces a single -`__global__` record on `put` and pull (`cult-cache.ts:394-399, 691-713`) and -invents nothing; Python likewise (`cache.py:125-128, 186-187`); Rust has no +`__global__` record on `put` and pull (`CultLib\packages\cultcache-ts\src\cult-cache.ts:394-399, +691-713`) and invents nothing; Python likewise +(`CultLib\packages\cultcache-py\src\cultcache_py\cache.py:125-128, 186-187`); Rust has no global concept. The C# key `global:{schemaId}` versus the siblings' `__global__` is an existing divergence, recorded, not changed (it would change bytes). @@ -571,9 +586,68 @@ Commit(stage): lock _gate: Admit(each upsert, null, durable: true); Evict(each delete, null, durable: true) publish one Change per record, after the store accepted ``` +**Conditional commit (compare-exchange).** The seven Rust variants +(`compare_and_swap_entry` :397, `insert_entry_if_absent` :422, +`compare_and_swap_batch` :441, `append_if_snapshot_unchanged` :495, +`compare_exchange` :713, `compare_exchange_snapshot` :759, +`try_compare_exchange_snapshot` :770, all in +`CultLib\packages\cultcache-rs\src\lib.rs`, expectations as +`CultCacheExpectedEnvelope` :291) collapse to two conditions on the batch: +- `Expect(key, current)`: `current` must be the instance the cache holds at + `key` (looked up through `_handles`; anything else throws) or `null` for + "must be absent". The batch captures the record's identity, `(SchemaId, + StoredAt)` from `_entries[key]`, at stage time. Only named keys are + constrained, so unrelated concurrent writes do not lose the race (Rust's + per-entry family). +- `ExpectUnchanged()`: the home store's persisted record set must equal what + the cache last loaded from it, compared as the ordered list of `(key, + SchemaId, StoredAt)` (Rust's snapshot family; fences unknown inserts). + +Identity is `(SchemaId, StoredAt)` rather than payload bytes because the +cache does not retain bytes and the in-memory object may have been mutated +since it was observed; it is sound only if every write to a key mints a +`StoredAt` strictly later than the record it replaces, so `Push`, +`CommitBatch` and the batch builder bump a minted `StoredAt` by one tick when +it is not later than the existing record's. `[P7]` measures why: minting with +`"O"` in a loop produces duplicates within one tick. `Commit` returns `false` +when a condition fails: no store write, no in-memory change, no notification. +`TryCommit` makes one non-blocking lock attempt and reports `Contended` +instead of waiting (Idunn's `try_` use); it is the same path with `wait: +false`, about ten lines. A conditional commit on a store with staged +single-record writes (`IsDirty`) throws: that view is neither what was +observed nor what is being committed, so flush first. + +Store side. Single-file store: a lock file beside the store +(`.lock`, `FileMode.OpenOrCreate`, `FileShare.None`, retry every 10 ms +for 30 s when `wait`, one attempt when not), mirroring +`DirectoryMessagePackBackingStore.AcquireCommitLease` (`DMS:928-949`); +under it: re-read the file, evaluate `Expected` against the re-read records +and `ExpectUnchanged` against the fingerprint taken at the last `PullAll` +or successful commit, apply upserts and deletes, write temp then +`File.Replace` (`CC:2659-2694` unchanged), refresh `Entries` and the +fingerprint, release. Plain `PushAll` takes the same lock so two processes +never interleave a write, but compares nothing: **a plain flush is +last-writer-wins**, and processes sharing a store must all use conditional +commit; the contract says so. Directory store: conditions are evaluated +under its existing `AcquireCommitLease` against the manifest re-read from +disk (`_durableIndex` identities are the same `(SchemaId, StoredAt)`), +`ExpectUnchanged` against the manifest's record set, then its existing +content-addressed pages and manifest write. `[P7]` proves the identity and +the lock: stale `Expect` detected after a concurrent write, unrelated writes +ignored, `Expect(null)` as create-once, `ExpectUnchanged` failing on an +unrelated insert, and two processes racing on one file with the lock +finishing at exactly `2N`. + +Parity: Rust already has both families and a `try_` (`with_exclusive_lock` +:700 uses `fs2` on a lock file); TS and Python have neither and are recorded +in the contract as not implementing conditional commit. Whether Rust's +`fs2` lock and C#'s `FileShare.None` open on the same lock file exclude each +other across runtimes on one file is not established here and is a named +follow-up; no store is shared across runtimes today. + The batch is an explicit value: reads during `stage` see committed state only, matching every consumer with a batch primitive (Rust `put_prepared_batch` -requires one store per batch, `lib.rs:2174-2186`; Ghostlight and Odin pass +requires one store per batch, `CultLib\packages\cultcache-rs\src\lib.rs:2174-2186`; Ghostlight and Odin pass explicit `compare_and_swap_batch`/`compare_exchange` values, section 2.0). No `AsyncLocal`, no ambient overlay. The store side is the existing `CacheBackingStore.CommitBatch` (`CC:2425-2452`: stage, `PushAll`, restore on @@ -628,20 +702,21 @@ ordering rule. | Change | Bytes on disk or wire | Runtimes that change | Verified by | |---|---|---|---| -| One home store per type; mirrors deleted | none; every routed file is a complete `cultcache.store.v1` snapshot | C# (Cut 3), TS, Rust, Python (Cut 5) | C# `RoutedStoresWriteTheSameBytesAsSingleStores`; the C# interop peer's `write-routed` mode writes `catalog.cc` and `run.cc` and the writer-by-reader loop in `packages\cultcache-ts\test\cult-cache.test.ts:536-617` reads both with TS, Rust and Python readers | +| One home store per type; mirrors deleted | none; every routed file is a complete `cultcache.store.v1` snapshot | C# (Cut 3), TS, Rust, Python (Cut 5) | C# `RoutedStoresWriteTheSameBytesAsSingleStores`; the C# interop peer's `write-routed` mode writes `catalog.cc` and `run.cc` and the writer-by-reader loop in `CultLib\packages\cultcache-ts\test\cult-cache.test.ts:536-617` reads both with TS, Rust and Python readers | | Attach hydrates; loading never writes; read-only stores | none | C# only (siblings already hydrate without writing) | Cut 3 tests | | Globals never invented; singleton enforced | none (key divergence recorded) | C# only | Cut 3 tests | | Runtime type decides schema | none for existing records | C# only | fixture ids in `cultcache-schema-compatibility.md:20-24` unchanged; `UpsertWeaponThroughGearHandleReloadsAsWeapon` | | Assignable lookups and watches | none | C# only | Cut 3 tests; Mesh and Networking suites | | Subtraction (SoA parked, managed documents, legacy directory formats, wrappers) | none; v4 directory manifests and v1 single files unchanged | C# only | full suites at their prior pass counts minus the deleted tests; external consumers build | | Atomic commit becomes an explicit batch on one home store | none (the store's commit is unchanged: single-file one replace, directory pages then manifest) | C# only; Rust already matches; TS and Python recorded as not implementing it | Cut 3 tests `BatchIsAllOrNothingOnStoreFailure`, `BatchAcrossTwoHomesThrows`, `BatchObserversSeeOnlyCommittedRecords` | +| Conditional commit (`Expect`, `ExpectUnchanged`) and the single-file lock file | none in the store file; a new `.lock` sidecar beside single-file stores (the directory store already has `.commit.lock`) | C# gains what Rust has (`compare_exchange` :713, `compare_exchange_snapshot` :759, `try_` :770 in `CultLib\packages\cultcache-rs\src\lib.rs`); TS and Python recorded as not implementing it; cross-runtime lock exclusion on one file is a follow-up probe | `ConditionalCommitTests.cs` (Cut 3), `[P7]` | | Per-assembly options, security comparer | payload bytes of consumer-declared schemas only | C# only | `InteropNoteBytesUnchanged`; `RefKeyedDictionaryRoundTrips` | | Contract text | none | docs in all four | review | The hosted workflow (`.github\workflows\cultnet-interop.yml`) tests CultNet frames only; `.cc` parity is the cultcache-ts test, run by `npm test` and on `cultcache-ts-v*` tags. Rust writes a stub catalog and second-precision -`storedAt` (`lib.rs:2397-2433`), a pre-existing byte-parity gap this cut +`storedAt` (`CultLib\packages\cultcache-rs\src\lib.rs:2397-2433`), a pre-existing byte-parity gap this cut neither widens nor closes. ## 4. Cut sequence @@ -672,13 +747,19 @@ the behavior change. amends `cultcache-persistence-format.md:42-55` ("Transaction Visibility" becomes: a commit is an explicit batch of records that resolve to one home store; the store commits it as one durable step; nothing is visible, to the - committing flow or to observers, until the store has accepted it; C#, Rust - implement it; TS, Python do not yet) and + committing flow or to observers, until the store has accepted it; a batch + may carry conditions, per-record `(schemaId, storedAt)` identity or + whole-store unchanged, evaluated under the store's exclusive lock, and a + failed condition is a lost race, not an error; **conditional commit is the + only safe multi-process write: a plain flush writes the whole snapshot and + is last-writer-wins, so processes sharing a store must all use conditional + commit**; C# and Rust implement batch and conditional commit; TS and Python + implement neither yet) and `cultcache-schema-compatibility.md` (runtime type decides schema; lookups assignable); amends `docs\runtime-parity-scope.md:23-27, 33` (SoA claim retracted); rewrites `CultLib\README.md:240-264`, `src\GameCult.Caching\GameCult.Caching.txt:157`, the generator README, and - `packages\cultcache-rs\README.md:216-241`. + `CultLib\packages\cultcache-rs\README.md:216-241`. - Adds `tests\GameCult.Caching.Tests\StoreRoutingTests.cs`, compiling against today's API, red for the stated reason: 1. `LoadingNeverWritesASecondStore` (seed A; attach A, pull, attach B, flush; @@ -841,6 +922,36 @@ the behavior change. key upserted earlier in the same batch returns the pre-commit value); `SingleUpsertAndBatchShareOneAdmissionPath` (a global singleton violation is refused identically through `UpsertAsync` and through a batch). +- Conditional commit, same commit as the batch: files `CC` (`CultCacheBatch.Expect`, + `ExpectUnchanged`, `CultCommitRequest`, `CultCommitOutcome`, `Commit` returning + `bool`, `TryCommit`, the `StoredAt` bump in the batch builder and `Push`), + `CultDocumentMessagePackSerialization.cs` (`SingleFileMessagePackBackingStore`: + lock file, fingerprint after `PullAll`, `CommitBatch(request, wait)`, + `PushAll` under the lock), `DMS` (`CommitBatch(request, wait)` evaluating + conditions under `AcquireCommitLease` against the re-read manifest). + Tests (behavioral, `tests\GameCult.Caching.Tests\ConditionalCommitTests.cs`): + `StaleExpectFailsWithNothingChanged` (two caches on one file; A commits a + new value; B's `Expect(key, itsInstance)` commit returns false; B's file + bytes hash, `Get`, `IsDirty` and `Watch` count unchanged); + `ExpectNullIsCreateOnce` (two caches both `Expect(key, null)` and upsert; the + second returns false); `PerEntryCommitSurvivesUnrelatedWrite` (A writes key + `b`; B's commit expecting only `a` returns true and the file holds both); + `ExpectUnchangedFailsOnUnrelatedInsert` (A inserts `c`; B's + `ExpectUnchanged` commit returns false); `TwoProcessesRacingExactlyOneWins` + (the test starts two copies of a small console project under `tests\` on + one file, each performing N conditional increments of one record with + observe-outside, re-check-under-lock; the final value is `2N` and each + process reports at least one `Mismatch`); `TryCommitReportsContended` (the + test holds `.lock` with `FileShare.None`; `TryCommit` returns + `Contended` at once with bytes unchanged; `Commit` on another thread waits + and succeeds after release); `PlainFlushNeverInterleaves` (one process + flushes in a loop while another commits conditionally; every read of the + file deserializes); `StoredAtIsStrictlyIncreasingPerKey` (a thousand + upserts of one key in a loop produce a thousand distinct `StoredAt`s); + `DirtyStoreRefusesConditionalCommit`; `CrossStoreConditionalBatchThrowsBeforeAnyStore` + (both stores' files unchanged, neither lock file created); + `DirectoryStoreHonorsConditions` (the four condition tests repeated against + a `DirectoryMessagePackBackingStore`). `BackingStoreTests.cs` transaction tests (`:1362, 1412, 1422, 1457, 1514, 1544, 1585, 1593`) are rewritten against `Commit` where they prove durability ordering, deleted where they proved the ambient overlay. @@ -883,26 +994,26 @@ the behavior change. - Repo/branch: `codex/cultcache-one-home-store`, independent of Cuts 2-4; three language-scoped tasks, parallel. -- TypeScript (`packages\cultcache-ts`): delete `mirrors` (`src\cult-cache.ts:18`), +- TypeScript (`CultLib\packages\cultcache-ts`, all TS lines below in its `src\cult-cache.ts`): delete `mirrors` (`src\cult-cache.ts:18`), the three mirror pushes (`:402, 448, 503`), the `slice(1)` lists (`:663-677`); `#resolveRoute(type): CacheBackingStore | undefined`; `addBackingStore` (`:180-188`) throws on a second untyped store or a claimed type. Tests: `rejects a second generic backing store`, `rejects a type registered to two stores`, `routes each type to its home store`; the writer-by-reader loop (`:536-617`) adds the two files from the C# `write-routed` mode. Command: - `npm test` in `packages\cultcache-ts`. Version `0.13.5` -> `0.14.0`. -- Rust (`packages\cultcache-rs`): delete the four mirror loops (`src\lib.rs:2092-2094, + `npm test` in `CultLib\packages\cultcache-ts`. Version `0.13.5` -> `0.14.0`. +- Rust (`CultLib\packages\cultcache-rs`, all Rust lines below in its `src\lib.rs`): delete the four mirror loops (`src\lib.rs:2092-2094, 2213-2215, 2252-2254, 2323-2325`) and the `route.len() != 1` branch (`:2174-2181`); `fn resolve_route_index(&self, type_id: &str) -> Option` replaces `resolve_route_indices` (`:2363-2380`); `add_backing_store` (`:1950-1959`) returns `Result<()>` and errors on a second generic store or a - claimed type; callers (`lib.rs` tests, `examples\cultcache_interop.rs:77`) + claimed type; callers (`src\lib.rs` tests, `examples\cultcache_interop.rs:77`, both under `CultLib\packages\cultcache-rs`) take the `Result`. Tests: `second_generic_store_is_rejected`, `type_claimed_twice_is_rejected`; `type_specific_store_routes_before_generic_store` - stays. Command: `cargo test` in `packages\cultcache-rs`. Version `0.1.0` -> + stays. Command: `cargo test` in `CultLib\packages\cultcache-rs`. Version `0.1.0` -> `0.2.0`. -- Python (`packages\cultcache-py`): `stores_by_type: dict[str, BackingStore]`, - `generic_store: BackingStore | None` (`cache.py:23-24`); `add_backing_store` +- Python (`CultLib\packages\cultcache-py`): `stores_by_type: dict[str, BackingStore]`, + `generic_store: BackingStore | None` (`CultLib\packages\cultcache-py\src\cultcache_py\cache.py:23-24`, all Python lines below in that file); `add_backing_store` (`:100-102`) raises on a claimed type, `add_generic_store` (`:104-105`) on a second store; `_store_for_type(type) -> BackingStore` raises when none (`:300-302`); the four write loops (`:199-201, 218-219, 239-241, 269-270`) @@ -1147,7 +1258,7 @@ estimates otherwise. |---|---|---|---| | 1 | 0 | ~150 doc, ~160 test | 0 | | 2 | `CultManagedDocument.cs` −355 (SoA parked at a tag, managed doc); `CC` −~670 (`///` 461, wrappers/unused/`soft`/subjects ~210); `CultCacheMessagePack.cs` −~60; `DMS` −~380 (legacy formats, filter, probes, `///`); `CultDocumentMessagePackSerialization.cs` −~100 (catalog helpers, `///`); `CND` −~60; deleted tests ~−450 | ~40 (`Loaded/Unloaded`, `NoWarn`), ~30 (`parked-features.md`) | public surface −20 members; 1 tag; 0 targets | -| 3 | ~95 (2.1/2.2 deletion lines) + ~230 (ambient transaction machinery) | ~150 impl (routing, globals, read-only, lookups) + ~70 (`CultCacheBatch`, `Commit`), ~520 test, ~30 interop peer | +`AddBackingStore(store, Type[])`, +`IsReadOnly`, +`ReadOnly`, +`Commit`, +`CultCacheBatch`; −ctor overload, −`MaterializeMissingGlobals`, −`ContainsDurableRecord`, −`PullOnOpen`, −`ExecuteTransactionAsync` ×2; −`AsyncLocal`, −`SemaphoreSlim` | +| 3 | ~95 (2.1/2.2 deletion lines) + ~230 (ambient transaction machinery) | ~150 impl (routing, globals, read-only, lookups) + ~70 (`CultCacheBatch`, `Commit`) + ~150 (conditions, request, outcome, `TryCommit`, single-file lock and conditional commit ~60, directory conditional commit ~40, `StoredAt` bump), ~720 test, ~30 interop peer, 1 small console project for the two-process race | +`AddBackingStore(store, Type[])`, +`IsReadOnly`, +`ReadOnly`, +`Commit`, +`TryCommit`, +`CultCacheBatch`, +`CultCommitRequest`, +`CultCommitOutcome`; −ctor overload, −`MaterializeMissingGlobals`, −`ContainsDurableRecord`, −`PullOnOpen`, −`ExecuteTransactionAsync` ×2; −`AsyncLocal`, −`SemaphoreSlim`; +1 lock-file sidecar per single-file store | | 4 | 1 | ~80 impl, ~110 test, 2 csproj lines | +1 assembly attribute; MessagePack's generator no longer flows to consumers | | 5 | TS ~18, Rust ~30, Python ~20 | TS ~12 + ~40 test, Rust ~15 + ~40 test, Python ~10 + ~30 test | three version bumps | | 6 | ~120 (reflection bridge, fallback) | ~300 editor, 1 attribute | Studio `1.0.0` -> `1.1.0`, +1 package dependency | @@ -1157,10 +1268,11 @@ estimates otherwise. | 10 | ~120 + ~280 (importer) + 8 | ~110, ~120 tests | −4 private file formats | Expected net: CultLib **−1,650** lines of source and −450 of tests in Cut 2, -then −325/+220 source and +520 tests in Cut 3 and +80/+110 in Cut 4: the two -Caching projects end near 3,000 lines with `CultCache.cs` near 950, against -5,259 and 2,704 today; the siblings shrink; Aetheria about **−33,000** lines, -two vendored dependencies and four private formats gone. +then −325/+370 source and +720 tests in Cut 3 and +80/+110 in Cut 4: the two +Caching projects end near 3,150 lines with `CultCache.cs` near 1,000, against +5,259 and 2,704 today, and C# gains the conditional commit Rust already has; +the siblings shrink; Aetheria about **−33,000** lines, two vendored +dependencies and four private formats gone. Proposed follow-ups, outside this migration: remove `soft` from `CultMesh.FlushAsync`/`CultNetLocal.FlushAsync` and its four external call @@ -1168,12 +1280,11 @@ sites; collapse `FlushOnDispose`/`StoreFlushOnDispose` into one flag; package the generator without the empty `GameCult.Caching.MessagePack.Analyzers` project; strip `///` from `CultDocumentContracts.cs` and `CultGeneratedDocumentMetadata.cs` (180 lines untouched by these cuts); unify -the global key across runtimes (changes bytes); a C# store-level -compare-exchange matching cultcache-rs's (`lib.rs:397-800`), which Odin, Idunn, -CodexConnector and Ghostlight consume in Rust and which the C# reference has -never had (optimistic concurrency between processes, distinct from the -in-process batch commit); batch commit in cultcache-ts and a cross-type -all-or-nothing batch in cultcache-py. +the global key across runtimes (changes bytes); batch commit and conditional +commit in `CultLib\packages\cultcache-ts` and a cross-type all-or-nothing and +conditional batch in `CultLib\packages\cultcache-py`; a probe that a Rust +`fs2` lock and a C# `FileShare.None` open on one `.lock` file exclude each +other, before any store is shared across runtimes. ## 6. Risks and rejected paths @@ -1198,6 +1309,15 @@ Risks: none exists (the only caller was a tests-only CultNet wrapper), and every sibling consumer passes explicit batches. Cut 2 parks SoA rather than retracting it. +- Conditional commit identity is `(schemaId, storedAt)`, not payload bytes; + it is sound only with the strictly-increasing `StoredAt` rule, which a + writer outside CultLib (another runtime writing the same file) need not + obey. Rust compares whole envelopes. Until the cross-runtime lock and + identity are probed together, a store is written by one runtime. +- The single-file lock file is a sidecar next to every single-file store; + a store on a read-only directory cannot commit conditionally (nor could it + flush), and a stale lock file after a crash is harmless (`FileShare.None` + is released with the process). - Deleting the v2/v3 directory formats refuses any store nobody found; if one exists it fails loudly with its format string and is rebuilt from source. - `ReactiveProperty` fields reachable from a document break without the @@ -1249,7 +1369,9 @@ reference CultLib `c2a9a6e` by project, and were run with | P5 | `cc-import-probe`: structural read of `GameData\AetherDB.msgpack` and one name file; depth; one `Faction` slot rewrite; `.cc` write and read-back; depth limit | `167 records; per tag: 0:13, 1:51, 2:25, 3:3, 13:12, 17:3, 29:4, 30:1, 31:18, 32:37`; `max nesting depth = 7 (tag 2); UntrustedData limit = 500`; `Australia.msgpack: outer array 2, tag 9, payload slots 3`; `Faction payload slots=16: 0:Binary(len16) 1-4:String 5:Map(n0) 6:Nil 7:Array(n3) 8:Array(n3) 9:Binary(len16) 10:Binary(len16) 11:Integer 12:Map(n12) 13-15:Integer`; stock options: `TypeAccessException: No hash-resistant equality comparer available for type: GameCult.Caching.CultRecordRef`; with the security subclass: `key=bbd619ed-… Name=Adrasteia Geoname=95c298ae-… Allegiance=12`; `reopened from .cc: … schema=aetheria.faction … Allegiance=12 payloadSlots=16`; `depth 100: ok; 499: ok; 600: MessagePackSerializationException`. Build: with MessagePack's generator active, `CS0426: The type name 'GameCult' does not exist in the type 'GeneratedMessagePackResolver'` (emitted `case 0: return new global::MessagePack.GeneratedMessagePackResolver.GameCult.Caching.CultRecordRefFormatter<…>()`); `ExcludeAssets="analyzers"` on direct references to `MessagePack`, `MessagePack.Annotations`, `MessagePackAnalyzer` does not remove it; an `Analyzer Remove` target `BeforeTargets="CoreCompile"` does. | | P6 | `cc-analyzer-probe`: scratch library `Lib` (netstandard2.1, `MessagePack` 3.1.7, a generic `Ref` struct with a custom formatter and resolver, standing in for `GameCult.Caching.MessagePack`) and a consumer `App` declaring `Dictionary, float>`; four library shapes built with `dotnet build App\App.csproj -c Debug -p:` | default: `CS0426: The type name 'Lib' does not exist in the type 'GeneratedMessagePackResolver'`; `ExcludeAssets="analyzers" PrivateAssets="analyzers"` on `Lib`'s `MessagePack` reference: same error; `[assembly: MessagePackKnownFormatter(typeof(RefFormatter<>))]` in `Lib`: same error (emitted `case 0: return new global::MessagePack.GeneratedMessagePackResolver.Lib.RefFormatter();`); `` in `Lib`: `Build succeeded.` | -| E1 | grep evidence (no code run) for the batch-commit shape | TS `cultcache-ts\src\*.ts`: no batch, transaction or atomic member; Python `cache.py:227-251` `put_envelopes` only, called from `cultnet-py\src\cultnet_py\replication.py:98`; Rust cache-level `put_prepared_batch` (`lib.rs:2162-2201`): no caller in `CultLib\packages` or in Odin, Idunn, Epiphany, CodexConnector, Ghostlight, Muninn, Ratatoskr; Rust store-level `compare_exchange*`/`compare_and_swap*`/`delete_batch_if_unchanged` (`lib.rs:397-800`): Odin 12 sites, Idunn 31, CodexConnector 1, Ghostlight `ghostlight-dungeon\src\app_session.rs:119, 371` (+3 tests) and `world\consumer.rs:788` ("a malformed or stale batch commits nothing"); Epiphany's crates: none. Every call passes an explicit expected/replacement batch; none reads staged state mid-commit. | +| E1 | grep evidence (no code run) for the batch-commit shape | TS `cultcache-ts\src\*.ts`: no batch, transaction or atomic member; Python `CultLib\packages\cultcache-py\src\cultcache_py\cache.py:227-251` `put_envelopes` only, called from `CultLib\packages\cultnet-py\src\cultnet_py\replication.py:98`; Rust cache-level `put_prepared_batch` (`CultLib\packages\cultcache-rs\src\lib.rs:2162-2201`): no caller in `CultLib\packages` or in Odin, Idunn, Epiphany, CodexConnector, Ghostlight, Muninn, Ratatoskr; Rust store-level `compare_exchange*`/`compare_and_swap*`/`delete_batch_if_unchanged` (`CultLib\packages\cultcache-rs\src\lib.rs:397-800`): Odin 12 sites, Idunn 31, CodexConnector 1, Ghostlight `ghostlight-dungeon\src\app_session.rs:119, 371` (+3 tests) and `Ghostlight\crates\ghostlight-dungeon\src\world\consumer.rs:788` ("a malformed or stale batch commits nothing"); Epiphany's crates: none. Every call passes an explicit expected/replacement batch; none reads staged state mid-commit. | + +| P7 | `cc-cas-probe`: conditional commit over the existing single-file snapshot format, implemented in the probe (identity `(schemaId, storedAt)`, `.lock` with `FileShare.None`, temp then `File.Replace`), two child processes racing on one file with the lock and without it | pending: see the reply that accompanies this revision; results are copied here when the run completes. | Not established by running code, marked as design: the routed C# cache itself (Cut 3's tests are its proof); the Studio's reflective struct path over